Hacker News new | past | comments | ask | show | jobs | submit login
The Eye – Open Directory Data Archive (the-eye.eu)
63 points by lelf 6 months ago | hide | past | favorite | 21 comments



:D

Very cool to see one of my internet homes here. I'm one of the mods/devs here and would love to highlight some of our recent datasets as well try my best to answer any questions! We're very active in the r/DataHoarders community on Reddit and regularly help out in the efforts to save large sites from disappearing offline as well as helping ease the distribution of hot content.

- https://old.reddit.com/r/DataHoarder/comments/exdka0// The Coronavirus Papers

- https://old.reddit.com/r/HofellerDocuments/comments/enj8f0// The Hofeller Files

- https://old.reddit.com/r/opendirectories/comments/cy8jry// eXoDOS v4.0

- https://the-eye.eu/public/Games/Flashpoint_7.1_Ultimate.7z Mirror of https://bluemaxima.org/flashpoint/

- https://searchin.the-eye.eu/ Full-text search of a few of datasets

- https://discord.gg/the-eye Our wonderful community on Discord


Related article discussing the coronavirus papers: https://www.vice.com/en_us/article/z3b3v5/archivists-are-byp...


Sounds awesome! But,

> community driven platform dedicated to the archiving and long-term preservation of any and all data

> We pay $650/month to cover all of our services ([...]) and are entirely community funded.

This sounds optimistic. What reason is there to think the community will continue the financial support in the long-term?


I know a good amount a professionals using it to dig olders builds of software. The Eye is actually better than Microsoft (at least for people without a msdn account) at archiving VS installers, Windows isos, etc.


As you brought this up I should mention we're currently moving those collections to faster storage and they will be available again later in the week. MSDN is still our most popular content by volume, crazy how much attention it's seen over the years.


Hi, There is no reason to think the community will continue the financial support in the long-term, even if we are running for years now, this is the reason why we offer free and unlimited download to people, we believe that sharing is the best way to preserve things. :)


The Eye has been around for several years, at least, so there’s some historical data. But “grab what you think you might want someday” has always been the rule for this sort of place.



what is this?

EDIT: nvm after scrolling a bit I see now


This site is a treasure box :3


After a brief poke, this seems like some highfalutin language around a random collection of pirated files, like an old FTP server from 1990... with the same "bandwidth exceeded" errors.


Which files did you experience an error? While we do rate limit the amount of parallel incoming connections, many of our directories have no bandwith limits and the site is hosted colo on a 10gbps line. You can also view our current traffic at https://the-eye.eu/traffic/.


What makes this different from the Internet Archive?


Our work works in parallel the Internet Archive. Preservation is all about storing content in multiple locations and we regularly hand off content to IA, as well as host it ourselves, when saving and sharing content.


This is going to sound like I am trying to throw shade but I really am not. I joined this community a few months back hoping to contribute, but despite numerous offers for help I was turned down nearly every time. There isn’t really any leveraging of the community within the discord or reddit or really anything that I can see. I rarely saw the administrators ask for any help for archiving purposes, and most of the data that is added to the community has no rhyme or reason but largely seems to be at the whim of the administrators.

The-Eye was fulfilling a niche for me and that was comic book archival. It seems the comic book industry at large has no regard for preserving comic books, and I found that The-Eye had a large collection of comic books. Disappointingly I found that I already had access to the majority of these comics and actually thought I could contribute. However, there doesn’t seem to be any easy way to add content but rather that the only thing a normal users can do is download content, so I couldn’t contribute anything.

What I have found though, is that there is a concerted effort to gather donations from the community. There were several occasions where there were multiple announcements daily on the discord asking for donations, or if users wanted to join one of the private sites they wanted, and even asking for people to show at the store.

Data archival is actually a big interest to me, especially in an error when content on the web is here one day or gone the next. There is probably more data lost in a day than was created in multiple decades in earlier centuries. I just don’t see the-eye actually as a entity that is actually arching anything, but rather that their interest largely lies in providing just enough data so that it can either be paywalled, or users can give donations.

Don’t take my word for it though, just spend some time on their website and you can see that they have numerous calls to action for a user to donate, but nearly nothing on their site which asks for a user to contribute actual content. If the user base that you are trying to cultivate is those who contribute content it stands to reason that uploading content would be trivial. Contrast that to imgur, however, which has an upload button displayed very prominently on their front page and you can see where their priorities lie. I know this may seem like comparing apples to oranges as one could make an argument for “quality” or data curation but even when you start searching the site more in depth there is a FAQ wherein the question is asked “How can I contribute?” and they ask you to donate rather than to upload.

I’m not trying to claim that I understand their model, but rather to relay that from my observations it doesn’t seem like they are really interested in users contributing anything.

As a disclaimer, the final straw for me was when I offered books that were behind a “library” on the OpenLibrary. I had written a scraper for the site, as there were numerous items which were now public domain but you had to borrow the books which essentially was a weird form of DRM. When I asked if I could upload them to the-eye the response was “why would you want to do that?” And it wasn’t until I had gotten angry, and honestly over reacted, that they claimed that there was no space on their servers for users to contribute. I left the community, and have not since been back. That was a couple of months ago. I have a friend who has claimed that there have been some changes made since then... But that seems to be hearsay, so I am not sure how much value I place on that.

However, there is a silver lining... I do agree that there is an awesome community on the discord, and in the communities. I think that if you are looking for a bunch of like minded individuals who are easy to talk to, and fun to be around that you couldn’t do much better than to join the Discord and chat with the people who are there. If you value nothing else then this site is absolutely essential and I think that everyone should consider joining.


You could instead contribute your digital assets to LibGen: https://news.ycombinator.com/item?id=21692222


Seconding the library genesis recommendation. There's a lot of people now mirroring those files, and uploading there will make those books easier to find for a wider audience than just the eye.


You may want to read this article...

https://www.vice.com/en_us/article/pa7jxb/archivists-are-try...

We (the-eye, like minds on reddit) are making sure libgen stays available. Further reading in a sub reddit I moderate here...

https://www.reddit.com/r/DataHoarder/comments/ed9byj/library...


Yeah, I've been uploading there for a while. I love libgen, and the-eye's efforts to help preserve sci-hub (they didn't have any seeders on many of their torrents) was one of the reasons I originally became interested in the site.


This hurts to read. Hey hackernews, I'm /u/-Archivist @-Archivist#7336 and run the-eye.eu. I'm going to reply to this and make clear why some people would take away some of these opinions after joining our community on discord.

> I joined this community a few months back hoping to contribute, but despite numerous offers for help I was turned down nearly every time.

This statement is entirely true, I don't know who this user is but often I'm contacted and offered data to host that I either already have an have chosen not to host or data that would be more at home at archive.org. I'm often short with people when responding to these messages due to the size of the community and daily influx of similar messages.

> There isn’t really any leveraging of the community within the discord or reddit or really anything that I can see.

On discord this statement is mostly true, we don't make broad calls for users to give us data as again often it's data we already have or simply data that isn't fitting for the site (not rare, available in many other places, etc)

However on reddit I've put myself in a position as moderator at r/DataHoarder and see many people tagging me on a weekly basis to add data to the-eye, weather that's a website that is going down and needs to be archived or a range of other content, more often than not I do take these queues and archive the data making sure it's preserved at the-eye or archive.org.

> I rarely saw the administrators ask for any help for archiving purposes, and most of the data that is added to the community has no rhyme or reason but largely seems to be at the whim of the administrators.

This is because we don't really need the help of the average user to do what we do, we're a very capable group of staffers that have connections and broad reach into other groups and areas of the internet where we would seek technical help should we need it. Be this contacts at archive.org, members of archiveteam (which often includes myself), etc.

> The-Eye was fulfilling a niche for me and that was comic book archival.

You say no rhyme or reason, this is somewhat true in the sense that data added isn't going to be what you're personally interested in 100% of the time, most recently my focus has been on books are rare video media in one form or another.

We haven't updated our comic collection on site in a good while, but we do archive millions more off site. A few 0day collections made it into a recent test collection too.

> https://the-eye.eu/hardwaretest/comics/

> However, there doesn’t seem to be any easy way to add content but rather that the only thing a normal users can do is download content, so I couldn’t contribute anything.

Over 90% of our life time this has been the case, why would we allow simply anyone to just upload random unchecked data to our servers? That seems like the worst idea ever and would result in the closure of our site very quickly.

Recently due to claims like yours however we have invested in a 32TB upload server to which users can request access to if they feel they can provide us with content which we otherwise couldn't obtain ourselves. Users have been uploading their own vhsrips and apple archive media content among other things which we will either host on site or make sure are uploaded to other archives around the internet for long term preservation.

> What I have found though, is that there is a concerted effort to gather donations from the community. There were several occasions where there were multiple announcements daily on the discord asking for donations

This is often how people attack us, when we're forced to ask for donations. It's very clear on all of our platforms that we run on donations. This is because we host a public facing website to serve content to the public, we don't need to operate this website to do what we do for preservation and only do so to offer fast access to the data we serve rather than just dumping everything to sites like archive.org on which the collections are disbanded and served at a much slower rate than we're capable of.

We own and operate our primary servers, this means we pay for our own hardware, bandwidth and hosting in colo at a datacenter rather than renting hardware from other providers. This comes at obvious costs, a drive dies? we pay for it, a router fails? we pay for it.

Our current bare minimum operating costs are $650/month. This covers only our primary server, domains and services like cloudflare, however this is far under our actual operating costs for all of the services we offer that require us to rent auxiliary hardware.

Without building an extensive list, this includes 4 servers at hetnzer each with over 24TB in storage and 12+ seedboxes at various providers for our torrent distribution network. These other servers are covered by myself and staff members out of pocket resulting in us operating at a personal loss to bring people our site and services, free of charge to 99% of those who download our content.

Bottom line, we hate asking for donations and only do so actively when 100% necessary, each time we do this is entirely on record and open to the public with explanation on our #news channel in discord. Furthermore we don't offer subscription based donations even though there has been push for it from our users, each donation made to us is at the will of the donor. This simply because the-eye isn't about profiting it's about making data available to the world.

I'm sorry that we have to take donations to provide multi Gbit/s access to content and we don't pay for everything out of our own pockets but be assured there is zero profit in running a service like ours, nobody is getting paid and the often exhausting time and effort we put into the project is for the love of the data and community alone.

> or if users wanted to join one of the private sites they wanted, and even asking for people to show at the store.

Yes we operate other sites/collections that are only available to donors as a perk of donating, these sites are curated collections taken from 100s of sources such as private trackers, offline collectors, etc. They are us giving back a little extra to those that support us, nothing we're obligated to do. If by store you're referring to our merch, yes, we sell merch to give something back to supporters. The margins on this are very low and you're free to look further into this yourself.

> I just don’t see the-eye actually as a entity that is actually arching anything, but rather that their interest largely lies in providing just enough data so that it can either be paywalled, or users can give donations.

Where are our paywalls? Or are you referring to the perks for donors mentioned above? If you paid any amount of attention to who I am personally and the work myself and my team do you'd realize just how silly statements like this sound and just how wrong you are. Though I forgive you, your interest is comic books.

> If the user base that you are trying to cultivate is those who contribute content it stands to reason that uploading content would be trivial. Contrast that to imgur, however, which has an upload button displayed very prominently on their front page and you can see where their priorities lie.

Okay now you're just getting stupid... imgur operate at millions in profit and are nothing at all to do with what we do. Ohh and due to that little upload button host one of the largest collections of child porn on the internet today.

I thought I'd be able to calmly reply to you but you're so far wrong and very jaded over my saying I wasn't interested and didn't need your help I'm going to stop here because this isn't a productive use of my time or patience.

To anyone else reading this you're welcome to join our community, download our content for free or just go about your day, thanks.


Oh okay.

I'mma let your response speak for itself.

Especially since...

> Though I forgive you, your interest is comic books.

lol...




Guidelines | FAQ | Support | API | Security | Lists | Bookmarklet | Legal | Apply to YC | Contact

Search: