Just tested it by searching for “spline”. This is great! Can someone elaborate on how it was made? Specifically, how it integrates with Google Scholar. Is that done client side or server side? If server side how come it hasn’t been blocked by Google seeing as Google don’t seem to like robots using their regular search function so my guess would be they wouldn’t like it for Scholar either. Perhaps it is proxying requests and would also pass on any CAPTCHAs presented? Still in that case I would expect all requests to get hit with a CAPTCHA. Perhaps it just hasn’t had enough traffic yet?
Likewise, the traditional publishers often respond to demands by funders to make research available by e.g. allowing researchers to share their work elsewhere, and often only after a year or so after publication [0]. This makes the barrier to do so higher, and makes the research less findable. It's not odd to expect that when initiatives like Unpaywall [1] make that research more discoverable, things like embargo periods will get worse.
A dataset at the scale of Scihub would have to pay someone to keep all the various PDFs online (which IIRC is around 70TB by now), which would mean a DMCA or N&T or similar would take down paid IPFS hosters.
I'm not sure how many people would willingly play IPFS hoster for 70TB datasets for free and while not giving a shit about authorities knocking on the door.
> If sci-hub is going to scrape publishers, they could put in a bit more effort.
+1, especially for the Onion site. Onion service supposed to be a primary mean to host uncensored websites instead of having to look for the latest domain name everyday, unfortunately it seems nobody cares about it. Most of the time I access it from my browser, the front-end proxy was malfunctioning, or the back-end Tor daemon has dead... Tor network itself do have capacity problem, but they could do much better than a broken front-end proxy... e.g. with Onion Balance.
At a glance it looks like it's really just a proxy, that was limited to scholar.google.com and mutates the page slightly (adds a header, sci-hub links).
I'd imagine that would be quite hard: many university libraries have their own proxies which make sure that visitors to the library are able to access the content that the library has paid for, and often modifies Google Scholar (with their cooperation, I believe) to list links to accessible versions of the content next to search results.
> Too much attention is a bad thing, Sci-Bay decides to stop service for a while. Sorry.
Apparently I was not wrong.
This could be developed as a browser plugin that would be much harder or almost impossible for Google to prevent. Well, a Firefox browser plugin, a Chrome browser plugin presumably they wouldn't allow.
The page's HTML is the API. It's pretty easy to download a web page, parse the HTML and then extract specific bits of information from it. The browser does the same thing on the user's behalf, which is why it is called the user agent.
The issue is regarding: this service (Sci-Bay) depends on Google Scholar, yet there's no public API for Google Scholar that it could leverage. If it's scraping Google Scholar results, then it's likely a ToS violation and unlikely to last long.
I believe people in academia are paid to write these books anyways, so they might as well not receive the royalties. As a prospective academic myself, I find it unethical.
I believe people in academia are paid to write these books anyways
While that does occasionally happen, it's definitely not true in general. The people I know who wrote academic books did so by taking a sabbatical from their university job and/or working evenings and weekends for the actual writing. Occasionally they can apply for a separate writing grant to cover their lost salary, but that's completely separate from their day job.
What I meant was their job is to author such books. Not that they should be paid separately for them. Depending on the country academicians might be underpaid, but that's an issue on its own.
What I meant was their job is to author such books
Except in many cases it isn't. Most of the people I know who have written academic books did so off the clock and on their own time. Sure the university let them use their university office and resources, and obviously much of the research the book is based on is research they'd already done as part of their job, but the actual writing time and any additional research they had to finance from sources unrelated to their university job.
Yeah, that's not really how it works. See dagw's sibling reply, and note that there's a continuum between academic monographs that would only ever be in university libraries, to popular science books, in between which is a large textbook market, and note that you can download all the above for free right now from libgen. And I am not moralizing. It's an amazing free resource! Transformative for people in nations at an economic disadvantage, and transformative for people in the first world too -- it would cost thousands of dollars or hours of inefficient library visits to be able to consult such a breadth of resources. I just don't know how exactly we/society should think of this.
I believe we should fund most of it through the academicians wage. And if that resource is deemed too low for them at its current value in a certain country, increasing it is a better solution than trying to add extra rewards for things like books.
I don't know how they are doing it, but Google Scholar does not have an API, and scraping is against their TOS.
> Don’t misuse our Services. For example, don’t interfere with our Services or try to access them using a method other than the interface and the instructions that we provide.
Despite this, there is scholar.py [0], which can extract files from Google Scholar, though it explicitly doesn't work around the rate limits.
or try to access them using a method other than the interface
Unless this actually exploits something and hacks into Google's servers to get to the content, which would be something quite different, it wouldn't really be distinguishable from someone manually visiting the site in a browser, volume aside.
IMHO the pervasive attitude today of somehow requiring permission or an explicitly sanctioned "API" to access what is otherwise publicly accessible data is rather troubling for the freedom and flexibility of the Web as a whole. It encourages walled-garden content models and centralisation.
I absolutely agree. If something is publicly accessible then the public should be able to use it as they see fit, from my viewpoint. (A HTTP response has already authorised you to copy the data to a machine. How can it be bound by a TOS that you need to access the original page to find?)
However, Google doesn't agree and the current court precedent doesn't either. So I tried to address the parent's concern from that viewpoint.
"See you later
Too much attention is a bad thing, Sci-Bay decides to stop service for a while. Sorry. Anyone who knows how Sci-Bay works and wishes this tool benefits more academics, please contact: info@sci-bay.org"
It allows for open-access of the epitome of human progress, which is something unavailable to low-income communities. It's copying data, not stealing from a museum, so it's not as if any journal is losing much, if anything.
There's no real ethical downside - allowing people who wouldn't normally have access to research helps to reduce inequality, and allows human progress to advance a lot quicker than if it weren't open. There are brilliant people of which have no access to research, who could - and would - more likely do something with it than not.
Given that we all pay for science, [0] from people in the lowest drudges of poverty, to people in the highest skyscrapers in New York, we should all have access to it. Imagine you paid taxes meant to improve roads in your area, but ended up going to a place hundreds of miles away from you. You'll never get to use these roads, and you'll be told you're unethical and unlawful if you even consider using them. It wouldn't be very fair, would it?
I never said the research was open-access, I said it allowed open-access to research. There's a difference, and although it's just positioning, it makes quite a difference in the meaning of a sentence.
One could easily come up with quite a few ethical arguments against Sci-Hub -- it's there to stick it to the "West". It has nothing to do with science, human progress, or anything as lofty. Basically, just another department of the Olgino Troll Factory.
That it lets you freely access some scientific materials regardless of whether the rights owner (or authors, for that matter) agree to it or not is just a minor side-effect.
She have strong pro-Putin view and tend to ban anyone who dont agree with her, even banned all Russian IPs from Sci-Hub last year because of some political discussions on social networks.
Though this doesn't change the fact project itself is very useful and obviously it's her own personality problems rather than some conspiracy.
She obviously doesn't share the NYTimes Western Standard World View, but to say that blocking all Russian IPs is pro-putin is a bit of a shortcut, no? The world isn't just "pro west" vs "pro Putin" and from what I can tell, Elbakyan has a nuanced standpoint.
I didn't wanted to explain here why exactly she banned it since I believe no one on HN will truly interested in authoritarian state internal politics, but it's important in this case. Few years ago Kremlin marked non-profit Dynasty Foundation [1] that was for years funding science and educational institutions as "foreign agent" and stated that that reason for that is foreign funding. Of course it's was complete bullshit because organization was funded by single person: Dmitry Zimin (founder of one of three largest Russian telecoms).
Real reason for that Kremlin action was funding of some politics-related book publishing and lectures. For non-profit "foreign agent" mark in practice bring regulation that almost impossible or too expensive to comply with. So Dmitry Zimin just decide to close non-profit.
Then most of people in Russian science community who still live and work in Russia have strong liberal views. Obviously they wasn't too happy about that since it's one one of few non profits in Russia that was actually funding science and education for real. Very few rich people in Russia ever donated anything to science or education so it's a big loss.
Elbakyan "nuanced standpoint" was such that Kremlin doing good job and since Sci-Hub users from Russia didn't agree she just decide to ban whole country.
Reducing "the West" to NYTimes seems to me as a much more severe oversimplification. Alexandra's viewpoint does not only contradict NYTimes, it contradicts basically the whole political spectrum of liberal democratic countries, except maybe radical left and radical right (die AfD and die Linke and the respective media in German context for example).
Of course her standpoint is not without nuances, and so is everyone's. But it is still not incorrect to call her anti West and pro Putin, and I believe she herself would not object this definition.
And of course that's not to diminish her positive role in protecting the freedom of humanity by maintaining SciHub. It is really thouht-provoking that otherwise non-free countries still contribute to common freedom by being less involved into copyright-protection system.
Her viewpoint contradicts pretty much everything that is good and decent in the world (and really, being a communist in XXI century is just as inspiring as being a neonazi).
Sci-hub is there to "expropriate the expropriators" in Lenin's words. That there is some wider benefit to the community is completely secondary.
> Sci-hub is there to "expropriate the expropriators" in Lenin's words. That there is some wider benefit to the community is completely secondary.
While you can see my opinion on her personality above I still think you going into extremes regarding the project itself. Keep in mind that she is just end up being face of the project, but there more people working on it, and maintaining Library Genesis too, and dealing with risk of getting into prison for the service they provide to everyone.
> but there more people working on it, and maintaining Library Genesis too, and dealing with risk of getting into prison for the service they provide to everyone.
So... one could go to prison for doing something that is explicitly illegal in most countries of the world, just so that the "face of the project" can preen about "standing up to the West", and fighting foreign enemies of the people, with the only risk to her being that Putin suddenly does a 180 and she's too slow on the uptake... Or one could, say, work on a viable long-term solution. Maybe setting up a wiki-like, peer-reviewed alternative publication scheme or something.
While the project does provide some benefit (which is not Elbakyan's goal), I just believe that the problem should be solved in an entirely different way -- you should start at the publishing side, not distribution side.
Yes they doing illegal things, her goals might be horrible or or awful and you might have right ideas. Problem is that all around the globe scientist need this problem solved ASAP and they need access to papers today, right now.
It's enough that few companies like Disney absolutely abused copyright. Yet we can live with it as long as it's just entertainment. Ok, they also put some horrible broken crap like DRM and DMCA and bunch of other things in our lives. Still it's not very important and might be after 10 years it's will be fixed somehow.
This is not the case for science. I can't even explain how wrong it could be to delay progress for years until "proper solution" is there. It's simply not acceptable. Sci-Hub might be illegal crutch, but it's important to have it.
I think good analogy for Sci-Hub would be Uber. It's might be absolutely hateful company for many here on HN that break every rule imaginable. Yet it's significally improved quality of life for millions of people and pushed taxi companies all around the globe to improve service.
PS: Also if you live in the US you might truly don't understand problems Sci-Hub solve, but there are people in ex-USSR who doing science on salary of $300 / month and they have no budget whatsoever on any papers. It's that bad. Yet some of these people still contribute.
Being from exUSSR, and knowing quite a few people who are trying to do science there, I would say that the whole Dynasty incident perfectly demonstrates what Sci-Hub (or Elbakyan at least, but she's Sci-Hub's public face, so same difference...) is about. Progress of humankind it ain't.
I also don't think this is the case when ends justify the means. But stealing is much easier than trying to change copyright laws, let alone setting up a proper alternative publishing system, so there we have it.
You'd have to ask her how she reconciles being a non-marxist communist, transhumanist, and Putin's stooge, but she manages somehow. There were examples posted here of her punishing scientists for speaking up in support of someone Putin had targeted.
As to why... I would hazard a guess that it's because she's neither a good person (far from it, by all indications) nor, really, a particularly bright one.
Obviously, not because of where she's from, but because she actually is anti-West.
There's an example posted below of how she is a Putin stooge. Blocking access to everyone in your own country because scientists (you know, ones she is allegedly supposed to help) because the ones who are actually involved in science supported pretty much the only foundation that funds science in Russia when Putin decided to close it...
Sure, yeah, that's who selfless altruists working for the betterment of humanity do, not Kremlin trolls. (insert sarcasm tags appropriately).
What exactly's your point? A "department of the Olgino Troll Factory" that directly benefits a vast portion of the world's population? That, again, only happened because of government—and therefore taxpayer—coin? That has no direct or indirect involvement in government, democracy, or harm of any living entity?
I think "directly benefits a vast portion of the world's population" is a vast exaggeration.
And when Russian scientists (real ones, not the ones who write fake dissertations for Putin's ministers) needed that benefit (science in Russia isn't exactly well funded) she yanked the access in the entire country because some of them dared to support someone Putin had targeted.
Sorry, don't see the upside or even much ethical excuse in supporting something that is demonstrably evil just because it might have a beneficial side-effect.
> I think "directly benefits a vast portion of the world's population" is a vast exaggeration.
How exactly does it not? There are more college-aged people than ever, and a good portion of them wouldn't have access to a substantial amount of research without Sci-Hub. Similarly, there are more pre-collegiate students than ever who are interested in research. Both of these groups directly benefit from having it. Indirectly, the rest of the world gets more progress and a better standard of living.
> Sorry, don't see the upside or even much ethical excuse in supporting something that is demonstrably evil just because it might have a beneficial side-effect.
Demonstratably evil? I'd love for you to clarify on that.
THat's still not what I would call vast. Vast would be far more mundane stuff, like providing clean water and medicine. Not to say that access to research publications is not important, but there are great differences in magnitude.
Demonstrably? I would say that being a communist, in this century, when you can't even pretend that "you did not know" like fellow travelers in 1950's did is a pretty good indication. Her stated goal of "standing up to the West" (which is far from perfect, but really much better than any alternative) and supporting strongmen like Putin (apparently she's also quite fond of characters like Saddam or Qaddafi, too) who do so, to the point of directly and intentionally hurting the very scientists that her project allegedly supports points in that direction, too.
I just really don't get all the hero worship on someone who, at no risk to herself (she's no Edward Snowden or even Aaron Swartz), wants to stick it to the very people who support her. That effort would be far better spent on setting a real alternative publication system.
One interesting thing about sci-hub is that it's actually in a researcher's best interests to have their articles uploaded to sci-hub and widely distributed. The more people that have access to an article, the more people can build off the ideas contained within, resulting in potentially more citations and advances that the original author can take advantage of.
So, you reading articles off of scihub and expanding on the contained work within directly benefits the author. Just make sure to cite the author if you yourself publish! (Which is actually much less difficult than you might think. Even a blog post can contribute to the field).
Authors are often allowed to share at least their preprints (i.e. submitted manuscripts that have not been peer reviewed yet) elsewhere, but this rarely happens in many academic disciplines. In fact, they are often required to do so by their institution, to deposit their work in its Institutional Repository, yet even that requirement is often flaunted.
(In my view, this is because the personal benefit is not that clear yet, especially if the work is just hidden in some long list of other research. Which is why I'm working on helping them use their research to raise their academic profile: https://medium.com/flockademic/the-holy-grail-in-open-access... )
Well, the research and the researchers are all almost entirely funded with public funds from around the world. The fact that they are forced for professional reasons to funnel all their work into a for-profit publishing industry for distribution is a historical anachronism that dies hard.
In a broad sense, there's only very weak analogy to music or film piracy in terms of the process of creative work. Everyone has already paid for scientists to do this science. Scientists and institutions have even paid the journals to publish their work in almost all cases, and they won't reap any reward if you read the work through the legitimate distribution channel. You only pay the publisher, not a starving researcher.
Personally, I'd feel very happy if someone read my papers by any means necessary. That was the whole point of writing them, setting aside the practical annoyance of simply trying to keep my career afloat.
Edit: The lifetime value of a single paper to the journal can actually be priced, since they started charging researchers for open access a few years ago (often in addition to other fees). This number is usually in the $2-5K range per paper, in my field.
I'm an academic. I've never met one who has objected to Sci-hub. Many of us use it, even to get papers we do have legit access to: journal publishers' access control systems are a pain in the arse.
Leaving aside the societal merits, our incentives are to get published, then get read. Sci-hub helps with the latter, and doesn't harm anyone except the publishers (who most academics cordially dislike anyway).
I imagine even the most socialist/communist/whatever is at least partly incentivised by earning a living too.
In fact, isn't that the problem, that quantity of publications (and citations) is used as a metric for continued employment? Those awarding research grants, etc., don't provide a free open-access publication as the place publication is required.
It seems governments could fix this in favour of the public quite easily; require publication in an open access journal of all research (papers and data) with any funding from the public purse.
We're very much incentivised by earning a living, but given that journals either pay us nothing or charge us for publication it's not particularly relevant to this discussion. You're right that volume of pubs and cites is the important metric, and piracy helps the latter without really affecting the former.
Governments and other major research funders are indeed beginning to do what you recommend and mandate various kinds of open access terms on publications. There are multiple options, some of which are reasonably favourable to the incumbent publishers (e.g. delayed OA release with an embargo) and some of which are not (e.g. it goes straight into our public database of funded research on day one). What the end game looks like isn't exactly certain at the moment.
Whereas a lot of OA options exist, they are quite scattered. Some universities have preprint servers that are more or less maintained, with varying up times. There are multiple open access options you have to pay for, if your grant or institution has funding for it.
You usually have to apply, and spend countless hours getting permission to spend the money, fill in forms, etc. It's all just too complicated.
Academics don't have time, either to find where the OA content is, or how to access it, and nor do they have time to organise their content to end up open access. It's much better for academics if everything is in one place, and trivially accessible.
Journals should be paying sci-hub to maintain a single repository where all articles are accessible for everyone, trivially, then switch to a model where universities pay only for the services journals are actually providing (in some cases, from what I can tell, none).
Google Scholar is doing a reasonable job at solving the OA discoverability problem, though at the risk of us becoming dependent on it.
The deposit problem is, as you say, a lot knottier. I personally suspect a linked network of institutional repositories with a twelve month embargo may end up as the end state of the current process, which I don't think is good enough.
Sci-hub provides a sociotechnical means of doing differently, and that itself justifies its existence, I think.
I'm preparing for a master's thesis (and then a research career), my university's library is rather scarce. Local uni's with superb libraries like Bogazici and Koc only allow graduate students from other unis. W/o sci-hub, I'd not have been able to read a single paper I've read in the past couple of years. Would have had to wait until I graduate, and had to expect that I made my way into a uni with a nice library, or at least be in a city where I can access to another uni with a rich library.
Information is produced mostly on tax money, and even when it's not, it's based on tech and info developed on tax money, and all of that is based on millennia of science one way or another the global society produced. I believe it's fair to think I and every other person "own" all of the scientific information already anyways. Even if you don't believe in that, I can't see how the other option is to pour mostly taxpayer money into cancerous organisations that sell what they have not even helped produce.
In favour: the fact that costs of access to research haven't plummeted with the rise of the internet as they have for other media, and that the traditional publishers are making profit margins of 30-40% (typically associated with luxury brands) indicates that the academic publishing market is not working as it should, and that the traditional publishers are making more profit off publicly funded work than you'd expect them to "deserve" based on the value they add.
Against, from a societal point of view: Sci-Hub is no longe-term solution, as it still depends on articles being published elsewhere. Furthermore, there is not yet a proper alternative to the traditional publisher's main contribution: credentialing of academics.
Against, from a point of view of the shareholders of the traditional publishers: if this leads to cancelled subscriptions, they lose money. A case could be made that Sci-Hub has contributed to e.g. most German university libraries cancelling their Elsevier subscriptions this year.
"So this is my argument that Sci-Hub can be ethical. Universalized it would destroy the system – but the system is bad and needs to be destroyed. And although this would break the law, a very slight amount of law-breaking might be a beneficial solution to inadequate equilibria that could be endorsed even when universalized."
The system needs to be reformed, and that is currently happening, peacefully. Perhaps not everywhere at once and perhaps not as quickly as you might prefer, but open access publishing has made great strides in just a few short years.
This doesn't liberate the millions (probably) of academic articles whose authors relinquished their copyright to big publishers, but it is resulting in new tensions like Germany and Elsevier butting heads (https://www.nature.com/articles/d41586-018-00093-7) and SciHub and exciting new legislation
No destruction or 'very slight law-breaking' necessary, nor mealy-mouthed 'inadequate equilibria' lies-to-self necessary either. Change is happening, the world doesn't need to go all Mad Max.
> but open access publishing has made great strides in just a few short years.
You mean SciHub. That's the thing, really; SciHub managed to do much more for the Open Access cause in one single move than the entire "legit" Open Access movement in more than a decade.
Tensions like Germany vs Elsevier are possible, because Open Access is now talking from position of strength - because everyone involved knows that the alternative is not "no access", but "SciHub" or "#ICanHasPDF".
Did you read the slate star codex article? Please don't miss-characterize it, it doesn't talk at all about going Mad Max in any way. Just wanted to mention this for anyone who might read your comment and has not read the article.
I was just about to post words to this effect. The article is quite clearly making an argument for a small amount of violation of laws (but not too much). The claim is that 100% adherence to laws could be suboptimal, as that would prevent us from escaping from local maxima in the utility function.
> I would want the laws to carry some force beyond just the barrel of a gun – a high trust society with consistent institutions is really important, and the more people follow the law without being watched the less incentive there is to create a police state. But I also wouldn’t want to live [where every law is enforced 100% of the time]...
Profit motive is holding scientific progress back. It's time to disrupt the status quo. It's fully possible to design a non-profit system that makes scientific publishing possible. Sci-hub may provide the structural perturbation required to fully dismantle the existing industry so that we can make a new one in the shell of the old.
The movie industry has it's fair share of copyright protectionism (DRM is popular) and same goes for gaming.
Elsevier is worse. They get paid by scientists so they publish their papers. They get paid massive amounts of money by universities (sometimes even the same that paid the scientist) to be granted access to research. Research largely funded by tax money. Research results that had already been paid for to even be published there.
There is a very good article in the SlateStarCodex if you're interested in a more in-depth article on it.
I read this quite often and frankly do not understand it. How is Elsevier worse than ACM, IEEE or Springer? I found them to be equally intolerable but still Elsevier gets routinely picked as the prime example...
Elsevier are disliked for litigation against scihub+libgen, IIRC? Also might be noteworthy that they have a long controversies section on their wikipedia page (including lots of things I'd never read about until now) while Springer and IEEE have none listed.
Yes. I work in CS in the field of Computer Graphics and (by association) Computer Vision and Robotics. Our top conferences and journals are by ACM. However there are a few journals by Elsevier and IEEE which are relevant. I do not have (legal) access to any of those three. However there are publishers who allow authors private/institutional hosting of their papers without paying additional open access fees. I would still doubt their legitimacy in tax funded research results...
I wouldn't think Hacker News would be in any trouble with this one. The link is to a noteworthy site, not to any infringing content; you'd have to take action and use a specific search term to find copyrighted content. And the implication of posting it is clearly that people should find it interesting and discuss it, not that they should use it to download articles.
