I find it really cool that this whole process didn't involve scientific publishers at all. Paper was submitted to Arxiv, there was Turmoil. Now a national lab is reviewing it, all without a Journal submission.
As an academic, I have not so secretly been enjoying the shit storm of revelations lately. Science got along fine for thousands of years without Journals and Conferences deciding what's "correct" and not. It's been a failed experiment. Journals started as a way to help distribute works to other researchers to improve upon the old method of just mailing it to one another. Arxiv is great for peer review. Your work is actually fucking available and not behind some pay wall, where your peers can... review it. Only thing would be better is if we were using OpenReview so we could track discussions, but I'll admit that could get messy real quick as anyone that's open sourced their research work will tell you (lots of questions like "I trained my model, how do I test it?" and "I'm getting a cuda out of memory error, how do I fix this?").
The thing that established journals do better than arXiv is clarity
You can upload whatever trash methods you want, but a normal journal will have at least one guy who tells you to wipe your ass and make your bullshit presentable in public, if only because they're expected to be gatekeepers for a minimum standard of supposed reproducibility.
In practice I'm not seeing much of a difference. Maybe it is just being in ML, where if you wait for conferences you're far behind. If a paper is that shitty, it is usually very apparent. Like if a paper isn't in latex you know... I mean there's a lot of garbage in conferences and journals too, I just haven't found it to be a meaningful signal.
And it is still silly that people call it "peer review." Peer review is not 3-4 randos briefly glancing my work in an adversarial setting who say my work is not novel because it is the same as some unrelated work that they didn't read either; peer review is the grad student building on top of my work, peer review is lucidrains rewriting my work from scratch, it is Ross Wightman integrating it into timm and retraining, it is the forks that use my work for projects (hobby or professional). Peer review is peers looking and reviewing. More peer review happens on Twitter than these conferences. You can say these conferences and journals are a form of peer review, but we gotta stop saying that just because something is a preprint that it isn't peer reviewed. That's just incorrect. Peer review is when peers review.
Sure, peer review is peer review. It depends on your peers.
I'm a chemist, who has considerable established competition. So our reviwers know the systems and studied the method in grad school and sweated over it as a postdoc and had to innovate for cash in a professional capacity while doing something other than what we were doing.
They'd excoriate for vague methods, or poor explanations, or general nonsense. You had to explain everything and make sure to refence the Big boys you were close to but definitely not stealing from.
I think ML is unformed enough that there isn't that sort of public stricture as a random field in chemistry. Feel lucky. We don't even get to hide behind beautiful LaTeX, because there's too much benchwork for anybody's boss to ever give a fuck about it.
> You had to explain everything and make sure to refence the Big boys you were close to
That sounds problematic
> there isn't that sort of public stricture as a random field in chemistry. Feel lucky.
It's pure noise over here. I don't feel lucky, I feel frustrated. I get 3 reviewers and 3 completely different reasons to reject, often nonsensical (not joking, recently was rejected because a broken reference link to the appendix). Then submit to another conference and it is another 3 reviewers with 3 different reasons to reject that are completely different from the previous ones. Often asking me why I didn't cite arxiv papers that were released after the submission deadline. ACs don't care as long as reviewers agree because they gotta keep that acceptance rate low. It's playing the fucking lottery except your graduation and career depends on it despite people frequently saying conferences don't matter they'll still critique on lack of top tier publications.
Your system doesn't sound great and also sounds frustrating, but I'll trade you. Regardless, I don't have faith in either system to significantly determine if a work is good (I do believe they can identify bad works, just not good works).
> > You had to explain everything and make sure to refence the Big boys you were close to
> That sounds problematic
Bingo, it is problematic. It's part of how "the Big boys" get their academic and compensation (investment in their firms) rewards. You have to play the publish-or-perish, paper rank game to get ahead and stay ahead in academia, and this leads to all sorts of problems. Authors don't want to question "the Big boys" because that leads to their papers not getting published because the "the Big boys" and their bootlicker wannabes are the reviewers and they will exact their tribute. Make it to "the Big boys" club and now you're a gatekeeper and now you're also responsible for perpetuating this system.
It's why Nature-style peer review needs to become a thing of the past.
I'm not saying that "popular" (for a value of "popular" that involves peers at large, not the public at large) peer review is / will be without problems. But it seems to me that it will -at least for a while- be less corruptible.
Yeah, this is a hill I will die on. I love researching, but once I grab my PhD I do not plan to push to journals/conferences unless it is requested in a job. It just holds no meaning and I'm tired of pretending it does. Perpetuating the system harms my fellow researchers, kills innovation, and just kicks the growing can down the road.
The other hill I will die on is that we shouldn't refer to journal/conference publishing as "peer review." This is one form of peer review, but there are MANY more. And as far as I'm concerned, 3 randos that briefly skim my paper in an adversarial setting (zero sum) looking to reject works barely constitutes peer review. Peer review is what happens when your peers read your work, test it, build upon it, replicate it, etc. We need to stop this language because it helps no one.
I love when this happens tbh, but it is rather rare. Often I can tell that a paper was rejected for dumb reasons and that gives me a good signal to actually read it.
> I get 3 reviewers and 3 completely different reasons to reject, often nonsensical
Could one problem with these kinds of reviews be that the reviewers have absolutely zero skin in the game? Like, you can write complete nonsense and nobody will blame you personally for it, you have no stake in the outcome of the submission/paper what so ever, and you're much more incentivized to come up with reasons to reject than to say "good enough". I realize the latter part can be good in theory, as it sets a high bar, but it also often feels like this goes awry when not kept in check somehow.
> Could one problem with these kinds of reviews be that the reviewers have absolutely zero skin in the game?
Actually, it is worse. They probably have skin in the game, and incentives to reject! Most CS fields publish through a conference system. This is a zero or one shot system (if you get a rebuttal phase, _if_ you want to count that). These conferences have acceptance rates that they "need to maintain" to keep their rankings. In other words, you are competing against all other papers being submitted to the same conference, not just papers with similar topics. Even if it is only a little, rejecting a paper actually increases the odds that your submission makes it through.
But yeah, you're in the right ball park. This is also why you will quite frequently see conference review guidelines blatantly violated and why you will see area chairs and metareviewers just not care. They all have incentives to encourage rejection, let alone be impartial.
I think the system works when the community is small and there is accountability among peers. Accountability creates a dampening effect on bad behavior. But at scale, you only need a few bad actors to setup a feedback system and to not just spoil the entire barrel, but the entire shipment.
Edit: I should also add that there's an additional negative incentive. You are not judged by how often you review, how many reviews you perform, or how good your review is (hard to measure). So reviewing ends up taking away time from the very limited time you have to do work that you are actually evaluated on. This is likely why reviews are so rushed. There's a feedback loop too, since many will see that others rush when reviewing their work so they get tired and end up rushing when reviewing the work of others. "If they aren't going to give my work their time, why should I have their work my time?" thinking grows.
>> I don't feel lucky, I feel frustrated. I get 3 reviewers and 3 completely different reasons to reject, often nonsensical (not joking, recently was rejected because a broken reference link to the appendix).
The people who review your papers in conferences and ask you why you didn't cite future arxiv papers are the same people who put their work on arxiv and cite each other's preprints. You can't rely on the process of "peer-review" on arxiv any more than you can rely on the conference peer-reviews because they're performed by the same people, and they're people who don't know what they're doing.
The sad truth is that the vast majority of the researchers in the machine learning community haven't got a clue what the hell they're doing, nor do they understand what anyone else is doing. The typical machine learning paper is poorly motivated, vaguely written, and makes no claims, nor presents any results, other than "our system beats some other systems". As to reproducibility, hell if we know whether any of that work is really reproducible. Everybody who references it ends up doing something completely different anyway and they just cite prior work as an excuse to avoid doing their job and properly motivating their work. The people who write those papers eventually get to be reviewers (by sheer luck), or sub-reviewers. They have no idea how to write a good paper, so they have no idea how to write a good review, either. And they couldn't recognise a good paper if it jumped up and bit them in the cojones.
I love to cite Geoff Hinton on this one:
GH: One big challenge the community faces is that if you want to get a paper
published in machine learning now it's got to have a table in it, with all
these different data sets across the top, and all these different methods
along the side, and your method has to look like the best one. If it doesn’t
look like that, it’s hard to get published. I don't think that's encouraging
people to think about radically new ideas.
Now if you send in a paper that has a radically new idea, there's no chance
in hell it will get accepted, because it's going to get some junior reviewer
who doesn't understand it. Or it’s going to get a senior reviewer who's
trying to review too many papers and doesn't understand it first time round
and assumes it must be nonsense. Anything that makes the brain hurt is not
going to get accepted. And I think that's really bad.
As to your very last point, it isn’t my “job”. It’s yet another task that I take on for no recognition, nor additional pay - as is much of academic life.
Jobs come with a lot of shitty aspects. Don't get me wrong, I generally don't enjoy reviewing either. But I put a lot of work into it because regardless of what I think, this has a significant effect on real people and their entire livelihoods can depend on this task. Especially those in their early career. One or two publications in a top tier journal can land them that internship or job which snowballs.
So I'd ask you do one of two things, either:
- Review a work with the diligence and care that you wish someone would give to you
or
- Don't review
I'd also appreciate it if you openly recognized how stochastic the system is and that if/when you become in a position where you need to evaluate someone, that you remember this and take it into consideration. It has a lot of value to you too, since if the metric is extremely noisy it doesn't provide you value to heavily rely upon that metric. Look for others.
I do see it as my job, and my responsibility. I also see it as my job to help more junior colleagues, and even to teach what I've learned to undergraduates. I don't want to do any of those things. I don't even want to write papers. I just want to sit on my couch, code, and discover new knowledge that blows my mind.
But, people pay me to do a job. It's not in my contract in any clear terms, but to do my job well I need to do all those things; and I like doing my job well.
> The people who review your papers in conferences and ask you why you didn't cite future arxiv papers are the same people who put their work on arxiv and cite each other's preprints.
I don't have a problem citing arxiv works. I mean I have no faith in the official system, so why wouldn't I? But this is a clear indication of an impossible bar to pass and demonstrates the ridiculousness of the system. It isn't just that there's a malicious or dumb reviewer, it is that the other reviewers don't call them out, the AC doesn't call them out, and the metareviewer doesn't call them out. The best case was that broken link person and the AC made them update their response, but this was in the response to my 1 page rebuttal (for 4 reviewers) where they got to say whatever they wanted and I had no chance of response. (Their initial review was literally 2 lines) I don't know how anyone can see this and not think that the system absolutely failed at every level. You're probably unsurprised to hear that this is a common occurrence rather than uncommon.
> the vast majority of the researchers in the machine learning community haven't got a clue what the hell they're doing
I'm well aware. __I__ have no idea what I'm doing, but at least I can tell you the difference between probability and likelihood or that tuning on the test set is information leakage. Research often requires venturing into the unknown and unexplored. That's fine. I don't care if we're all stumbling around in the dark. I do care when people are not just unable to admit it, but unable to recognize this. But that's the classic "tell a lie enough times and you'll start to believe it" situation.
I am in full support of that Hinton quote. This is the first I've heard it (or recall at least), but I often say quite similar things (in fact, just did in another thread). I do mean it when I say that our current system harms us and I'm confident that we won't get to AGI with this system.
> the problem is that peers in peer review lack expertise and knowledge and they can't do their job well.
I won't disagree with this point, but I believe that this is a systematic problem rather than an individual. The system encourages this behavior rather than stamps it out. So in that sense, I think people are doing their job very well. It is just that I don't think their job actually aligns with the intent of the job. Classic case of irony, that the group of people that highly discusses alignment is one of the worst at this. But I guess we shouldn't be surprised given that lately we've seen how unethical we've seen people who write about ethics are.
I do want to add one thing though. A good paper is hard to recognize. A bad paper is easy, but a good paper may be indistinguishable from a bad paper. This is the "paradox" of research and something people need to take to heart. That is if we want to align our job descriptions with our actual jobs.
>> Research often requires venturing into the unknown and unexplored.
Oh, in that sense I'm clueless too. We all are. What I'm pointing out though is that many people in machine learning research don't really understand what's the point of research, or even what's the point of a research paper. They copy each other's writing style and produce the same kind of low-information-content paper.
There's been a huge influx of new people to machine learning research in recent ish years. I think it started around 2012 and the ImageNet results. I did my Master's in 2014-15 (part-time) and I was surprised by the number of people in the machine learning class I took, and how many of them had nothing to do with computer science or AI, and were instead coming from backgrounds in business or more rarely science. At least I'd expect the people with the science backgrounds to know how to write a paper, but it seems most people who come to machine learning from, say, physics or biology, bring with them a facility with calculus and continuous mathematics, but not much in terms of scientific methodology.
Hinton hints at that in the rest of his comment I didn't quote:
What we should be going for, particularly in the basic science conferences, is radically new ideas. Because we know a radically new idea in the long run is going to be much more influential than a tiny improvement. That's I think the main downside of the fact that we've got this inversion now, where you've got a few senior guys and a gazillion young guys.
Well I guess the "gazillion young guys" need a bit of time to figure out how to write good papers, and review them.
>> I do want to add one thing though. A good paper is hard to recognize. A bad paper is easy, but a good paper may be indistinguishable from a bad paper. This is the "paradox" of research and something people need to take to heart. That is if we want to align our job descriptions with our actual jobs.
That's true, and part of the reason why reviewing is hard work. I still find it very hard to reject a paper. What if I'm the one who doesn't get it?
> (...) make sure to refence the Big boys you were close to but definitely not stealing from
Much as giving credit is important, this is unfortunately also how you end up with intro sections littered with references to the same ten papers everyone's read already just to back up some extremely vague/general statement.
I hate this. Honestly, I'd love to see short papers. Just cite what is relevant and not much else. But I see citations exploding. Look at this paper[0], it is 34 pages in total and 5.5 of those are the bibliography! It has almost 100 references! ~14 pages are images (not figures... images). Nearly an entire page worth of material is just the citations! Not the text referencing the citations, but the citations themselves. It is absolute chaos.
No, this isn't a survey paper. So it is serving no purpose other than greasing their peers.
I agree with the core of your message, and you are far from alone with this opinion.
The history of science is a sequence of profound platitudes, each time refining the prior ones. It is a part of awakening to pick the lowest hanging fruits first, just to get a foot in the door. While its not theoretically impossible to go straight from ignorance to -say- the postulates of quantum field theory, its incredibly unlikely to scale such a huge step at once, both for scientists at the frontier as well as each generation of students catching up. In this sense the profound platitudes are quasi necessary to build up an understanding of the world around us. Individually each outdated platitude may appear so very wrong when looking back with 20 / 20 hindsight, and most of these individual platitudes could plausibly have been skipped by scaling 2 or 3 steps at a time, but no one could scale all of them at once.
Literally respect and review mean the same thing. To view or observe anew (re-).
One may objectively claim that each step or correction of a prior platitude was an improvement of our worldview, even though in hindsight each step of correction also contributed the next erroneous stumbling block to be scaled.
Throughout history policy decisions (from household decisions, to court decisions, to budget decisions, to research direction decisions, etc...) humans sought ways to settle matters.
The desire for Finality is universal and justified. However the expectation that this universe came with a manual is not.
A core tenet of the scientific mindset is the recognition that this finality is a temporary illusion.
Selling this illusion of a recipe for finality is profitable and what journals and gatekeepers have gravitated toward, especially since information became dirt cheap to duplicate.
I have no problem citing, and actively encourage it. I do have a problem with someone rejecting my work because I failed to cite a work which would have been impossible for me to cite. The key is that it was released (as a preprint) AFTER my paper. Not only is it in bad taste to make concurrent works a requisite (happy to update btw), but it is ludicrous to require that I have a time machine.
This isn't about respect and me complaining about pomp and circumstance. This is me complaining about the frequency that I have been criticized for not having a time machine.
The point is that a system that allows people to reject works due to lack of access to a functioning time machine is not a system we should support.
Perhaps you confused me for xorbax, or you were still in the stream of consciousness of discussing with him: if you double-check which of your comments I replied to I think that you'll find it is not the one where you mention the demand for a citation that was simply non-existent.
I would appreciate if you re-read your own comment at the last link, and observe the second paragraph you wrote about peer review, and then re-read my correctly placed comment. I think you'll find my comment to make more sense, and in my opinion ties in nicely with your second paragraph. I was not tryihg to summarize you, merely appending my gripes with the supposed finality and atomicity of the Journals redefinition of "peer review".
In reality the real peer review is in the back and forth discussions within the community, spread out over many papers, and sometimes even over generations.
As an oversimplified example consider relativistic mechanics correcting Newtonian mechanics: if we anachronistically pretend Newton published his Principia in some Respectable Journal under the current implementation of "peer review", then it would have been accepted with apparent Finality under peer review, even though it would eventually be repudiated by relativity generations later; this artificial example should make anyone pause about the insinuated Finality of "peer review", effectively forcing one to recognize that the current concept of artificially muzzled "peer review" through the arbitrary selection of the reviewers by journal editors does not constitute true peer review. I'm not saying it's useless to have an initial screening, but then it needs a better name, perhaps "Quality Control" or "Publication Criteria" or "Discipline Specific Methods and Publication Criteria" etc... Before the modern incantation where reviewing by peers is made artificially atomic and pseudo-final, anyone in the field could provoke questions about specific claims. The illusion of Finality works discouraging: where-as in the past you would feel compelled to respond to any criticism, the current concept of "peer review" implies "it already happened since its published" and nothing forces an author to come up with a rebuttal for a criticism. While this may help prop up appearances of the scientific field with an aura of Finality of conclusions made in its domain, its really damaging to the scientific process.
Back to the comment confusion:
I've seen this happen to others when a sub-thread gets a lot of attention, resulting in a high rate of replies, where one of the authors gets confused which exact comment was being replied to. I would be lying if I were to claim it never happened to me ;)
I think you're right. There's a lot of comments I've responded to and the alignment isn't always clear who's responding to what, so I use context to fill in the gap. My bad. You are totally right about your comment making more sense haha. (I'm known to be an idiot, so people should question if you really want me reviewing your papers lol) But as your last line is suggesting, I think this is just a human thing and when a comment gets a lot of attention and someone tries to respond to a lot of different people. Too many trains of thought to juggle around in a format that isn't super explicit. It's why I like my code to have colors and group chats to have colored bubbles for identification. I need the aid lol.
> In reality the real peer review is in the back and forth discussions within the community, spread out over many papers, and sometimes even over generations.
This is 100% what I believe and I'm surprised others don't. Like how arxiv exists because people were already sharing papers outside the walls of journals. For some reason people are acting like this sharing means these are drafts rather than the actual publications themselves. As if arxiv and what's published are different versions. Maybe a lot of non researchers are participating in these discussions and assuming things?
I'm sure you know this, but others might not. Newton is the perfect example here because many of the works at these times were "published" years or even decades after the original work was done. Which is the whole Leibniz and Newton debate thing. I'd also highly recommend the podcast "Opinionated History of Mathematics". It's hilarious and well done. By a mathematician but for the general nerdy public
There are exceptions [1] but most journals don't expect the reviewers to even attempt to reproduce results, which makes sense given how specialized and expensive scientific experiments often are. As a reviewer on open code papers I would usually try to run the provided code, it didn't always work and that wasn't always addressed before publication. (I was also usually the only one who even tried.)
Usually peer review is more about making sure the work is novel and interesting, fits the journal's audience and doesn't have any glaring flaws. Not entirely unlike code review: if it builds, merge it, and we can address problems in a future PR. Those are basically the reviewer instructions you get from most journals IME.
[1] OrgSyn famously requires a reproduction from one of its editors lab before it accepts any paper,
It has a very high reputation amongst chemists, even if it's "impact rating" is low. High impact journals are not usually considered the most accurate.
I don't think you are really arguing against what the parent poster was saying. That is, I interpreted the parent commenter as saying that journals require that submissions at the very least be in a clear, understandable, "your paper must be at least verifiable (or falsifiable)" format, not that they actually attempt to reproduce the results.
(not OP) Verifiability/falsifiability are big words, mostly it is not clear what that means in a specific case. Crucially, that is not what journals/editors/reviewers do. They check if they find the contribution convincing, novel, and in line with the discipline's community standards, nothing more.
No, I think a half decent paper is expected to 3ither explain their methods or reference a paper that explains their method
You don't get to handwave away the instructions of your experiment in my mind. Maybe other fields are fine without that, but I would never write a paper that doesn't clearly explain how I made samples or reference a paper which does. To do otherwise is bad science.
It's not about a reviewer replicating it, it's about anybody replicating it in a year or 30 years.
> As a reviewer on open code papers I would usually try to run the provided code
You're only one of two people I've ever heard make this claim. Which I'm sure you're aware, but many people probably aren't. Fwiw, I'm often called diligent because I read the code (looking at main method and anything critical or suspicious. Might run if suspicious). Even reading supplementary materials will earn you that title (which is inane). According to this informal survey, ~45% of neurips read the supplementary material <13% of the time and less than a third always read it[0] (I'm in that third, and presumably xmcqdpt2).
> Usually peer review is more about making sure the work is novel and interesting
This is why I find p̶e̶e̶r̶ ̶r̶e̶v̶i̶e̶w̶[1] journal/conference reviewing highly problematic and why this system is at the root of our current existential crisis: the reproduction crisis. Reproduction is the cornerstone of science. And many MANY good works are not novel in the slightest. See the work of Ross Wightman (timm) or Phil Wang (lucidrains). These people are doing critical work in the area of ML but they aren't really going to get "published" for these efforts. Many others do similar work, but just not at the same scale and so you'll likely not hear of them, but they are still critical to the ecosystem.
But with your next point: if it builds, merge it; I'm all for. The system should be about checking technical soundness and accuracy, NOT about novelty and how interesting it is. Of course we shouldn't allow plagiarism (claiming works/ideas that aren't your own), but we should allow: replications, revisiting (e.g. old methods, current frameworks (see ResNet strikes back)), surveys, technical studies, and all that. Novelty is a sham. Almost all work is incremental and thus we get highly subjective criteria for passing the bar.
Which is probably why high impact journals are not considered the most accurate. Because they don't encourage science so much as they encourage paper milling, rushing, and good writing.
[1] We need to stop calling journal/conference reviewing "peer reviewing." Peer review is when your peers review. Full stop. This can come in many forms. Similarly publishing is when you publish a paper. Many important works come through open publishing.
> if only because they're expected to be gatekeepers for a minimum standard of supposed reproducibility.
There are decades of beta-amyloid papers that show that this is false. Like any for-profit entity, premium journals' primary purpose is to satiate shareholders - it is their feduciary responsibility to get away with as much as possible in the name of profit.
That's not to say that open access doesn't have equally as concerning issues: which of the two is better or worse is an extremely difficult call to make.
Triple integrals and sigmas over sets, instead of clearly stated limits, and that are not analytically solvable, is not "presentable". Scientists do this stuff on purpose to make their papers harder to read.
> expected to be gatekeepers for a minimum standard of supposed reproducibility
They don't enforce this at all. Otherwise they would demand code and training data be released for ML papers.
> Scientists do this stuff on purpose to make their papers harder to read.
Not just that, but if your paper has math that the reviewer doesn't understand they're more likely to think the work is good and rigorous. It's not like they read it anyways.
> and training data be released for ML papers.
Other than checkpoints and hyper-parameters, what do you want? The wandb logs? I do try to encourage people to save all relevant training parameters in checkpoints (I personally do). This even includes seeds.
Hyperparameters yes, but also the data used for training. I should be able to reproduce the checkpoint bit-for-bit by training from scratch. If their training process is not deterministic, also release the random seed used.
Oh yeah, that I agree. I'm kinda upset Google is frequently pushing papers with JFT and 30 different versions of it and making conclusions based on pre-training with it. This isn't really okay for publication. Plus it breaks double blind! I'd be okay if say CVPR enforced that they train on public datasets and can only add proprietary after acceptance (but you've seen my views on these venues anyways).
All ML training is non-deterministic. That's kinda the point. But yeah, people should include seeds AND random states. People forget the latter. I also don't know why people just don't throw args (including current iteration and important metrics) into their checkpoints. We share this frustration.
· By one side the Journal reclaim a high amount of money for being able TO TRY to publish on them (APCs fee), more if one want to be published for to allow free access, as to read the publication in such Journals require subscription, are pay walled.
· By other side, the reviewers of such papers, academics and researchers from other universities, don't receive economical compensation for doing the review, while the taken APCs fee its supposed for doing it? only in few cases the review is compensated by some programs of the reviewer's university or other 3rd parties. Then, what is for all those pay-walls and publish-fees?
Sounds like a big filter... for what? for who? Because can be seen it's not working just as a mere "quality" filter.
Being a paper published in one of those popular Journals, or being done directly in medias like arXiv, both sources needs the same amount of "grain of salt" about what is being read, until others replicate experiments, or contrast theories.
So IMHO, outsider point of view, from time to time I have to use Sci-Hub in fact, I understand if they decide to don't publish in such popular Journals. I consider absolutely legitimate when they do it in distribution channels like arXiv.
What I really would like it were exists an open research platform without pay walled papers, like happens in arXiv, and in addition, peer reviews through the platform were possible, as to read such reviews, and also, equally important, to have available debates about the paper, like if it were hacker news format.
> Sounds like a big filter... for what? for who? Because can be seen it's not working just as a mere "quality" filter.
To me it seems it keeps not only a wall around papers and journals. But also about whole academia which is probably an unwanted side effect since inside and outside of academia there seems to be consensus that knowledge transfer could be better. Also it seems everybody who can rather does research outside (AI/ML, Chemistry, Pharmacy).
I mostly agree with you. But I absolutely despise debates. They are more often than not worthless. They are based on charisma and how good you are at language, not facts. Discussion is the right word. I've often thought of pushing directly to OpenReview rather than just arxiv, but discussions can get off base quickly and novices can dominate (while academic Twitter is great, this happens there a lot). It is not our job as a researcher to communicate our ideas to laymen, that's the job of science communicators. Sure, it is great and a bonus, but please stop asking on my github how to do something that is in the pytorch intro tutorial. Read the tutorial or attend my class. Open discussions are nice in theory, but HN and Reddit are good examples at how these systems often get over burdened with noise. There's a catch-22 and so it is a tough call.
> […] and in addition, peer reviews through the platform were possible, as to read such reviews, and also, equally important, to have available debates about the paper, […]
science in the modern sense is far less than 1000 years old, and emerged in tandem with systems of publication which no did not use the contemporary review system but did have editors who decided whether or not to publish submissions. not defending the peer review process as it currently exists but it’s not a bizarre outlier in the history of science
Eventually somebody will say "what if arXive, but we had really smart people who were volunteer experts read this bullshit first and decide if it was good enough? It would be like arXive but way more efficient for the end user!"
That's not far from how it actually works. 90% of what I read is from arxiv, and before it has been accepted into a journal or conference. Most of the time that I'm visiting a conference website is to submit a paper or get the proper citation for the paper I'm writing.
Like I'm not sure what you expect, just dumb people to read arxiv? If that were the case then no one would be submitting works to arxiv. The reason works go there is because researchers are reading them. They then talk about the works to their peers, post on twitter, blog, or whatever. And guess what, that's also exactly how works that are published in journals and conferences get passed around because there's a million of them too and no way for people to sort them. No one is sitting around ranking papers. That process happens by citations.
Are you aware that the origin of Arxiv was as a pre-print archive? People read stuff on there because it was freely accessible drafts of published work, and in my field that is still the most common practice.
Yes, but I think you're being too strict with the definition of draft. It is not "draft" as in "first draft of my paper" it is "draft" as in "we submitted this to journal/conference and are just making widely available" or "here's the same copy that is behind a paywall, fuck jstor."
There's a reason most papers that end up in journals or conferences don't get a revision, or if they do only get minor ones. In general it is the same exact work which got sent to a journal or conference. So that is either the final draft or near. I do not want people to confuse this with "early" or "first" draft.
The history of arxiv is a few physicists getting annoyed that, like many of their peers, that sending per-prints (papers awaiting review by a journal) by email was burdensome. This was about just making access easier. It was about literally making peer review easier. Because let's be real, your colleagues reading your work closely is better review by peers than a few random people who are annoyed about having to review and know it is easy to just dismiss and reject.
That was one way, but it definitely was not the only way. A lot was people writing letters to one another and sharing papers by word of mouth. There will always be elitist groups, yeah, but we shouldn't support them. It's about doing good work, not gatekeeping. Reviewing is supposed to be critical, but that's not the same as adversarial. Science is about progressing human knowledge and we need to ask if this system is accomplishing that or if we're just caught it Goodhart's nightmare.
I'm not seeing good evidence that journals or conferences do any better of a job at this than pre-print servers do. They do have a slight edge, but that is not that great. The edge is because the default position is to reject and so identifying just a small percentage of good papers does affect this outcome, but that's not a meaningful signal. Have an extremely high false positive reject rate and a high false positive accept rate doesn't make the signal meaningful, it just means it is very noisy.
I don't give this very low signal any meaning because the truth is that good works filter to the top regardless of the journals. They rise because peers share them, not because they got the stamp mark from a journal.
I don't understand how pull requests would help with reviews. Peer review is generally looking at the methodology and analysis, not simple typos or wording that might be corrected by a pull request.
Also, I believe we should focus more on rigorous replication process rather than publishing new things from scientific researchers. Peer review is not enough to check the claims as it merely checks on reasoning of the science instead of experimental confirmation unless it is a very simple and short experiment. In other words, instead of directing funds to exploratory research, more money should be spent on multiple experimental verifications by independent researchers as I believe there are so many papers that need retraction due to unsubstantiated claims.
I guess you're just trolling, but in case you're serious, yes, being able to group together as scientists (university) and pay a single (or small amount) of entities to get all the scientific results, peer reviewed, has done wonders. Being able as university/funding agency to evaluate someone's scientific output without actually reading their papers (because other experts did it) allowed scaling. And to be honest, I do not know a single person how pays $40 for a paper, there are so many ways to get it for free (including just asking the authors). There is space for improvements, but "get rid of journals and conferences and hope scientists will somehow figure it out" is not a solution
The journals take works others have done (work often paid for with public money) and get unpaid volunteers to peer-review it. Next they require the authors to sign away their copyright to it. And then they turn right back around and sell it to very same universities where it came from for thousands of dollars a year.
And for what? So they can have have profit margins rivaling big tech? (Their major expenses are lawyers and server costs for pdf documents…)
And why do researchers publish there? Because they need to in order to keep their jobs, or to get funding. You need those high-impact-factor and prestigious journals. And so you sign away your research.
Most 1st world researchers don’t care or don’t know about the costs. But most of the world is not so lucky. Yes you could go around begging the authors for a free copy (but definitely don’t go to scihub or library genesis and download them for free, that would be illegal…).
The “space for improvements” starts with burning down the journals and scattering their ashes among the fields to fertilize new science.
Again, we had scientific progress (and problems with it) before the current system.
The only currency that matters is if your results can be reproduced, as as far as I'm aware the overwhelming majority of journals make no attempt at reproducing the results before accepting it. Hell, it doesn't seem like they even make an attempt at make string there's enough information provided to even attempt to reproduce it.
When I come across a paper that does any sort of modelling, I simply assume I won't be able to reproduce it if the code isn't included. They all seem to leave out important information.
> Again, we had scientific progress (and problems with it) before the current system.
Again, the scale and accessibility of science is different now and then. We had commute before steam engine, it doesn't mean we can get rid of all engine-based transportation now without consequences.
> The only currency that matters is if your results can be reproduced
This is extreme reductionism. LHC results cannot be replicated unless you have multi billion particle accelerator in your backyard. JWST cannot be replicated. Many other large scale experiments are nearly impossible to replicate. In fact, these are often considered "important" ones. And this not necessarily due to price, but rather combination of price, unique expertise and willing to spend effort.
You know who will be able to tell you if the experiment description makes sense and has chances to be replicated if you ask them? Peers. Peer review at will never worked (and there were attempts) and never will because nobody likes to do peer review. I'm yet to see any proposal for peer review without editorial invitations which has at least theoretical chances to succeed.
> I won't be able to reproduce it if the code isn't included.
So you do mean "reproduction", not "replication". This is very low threshold and definitely not "only currency that matters". If I send you the code that alters the results somewhere deep inside, you'll be able to reproduce them.
> Many other large scale experiments are nearly impossible to replicate.
Thankfully, the LHC is not what all of science looks like. There is a great deal of science that can be reproduced without needing billions of dollars.
> If I send you the code that alters the results somewhere deep inside, you'll be able to reproduce them.
But the current status quo for most papers is that you don't even have that. If you public put out code that manipulates data in this way then it can be scrutinized.
The alternative is to attempt to replicate the results, and waste months of time trying to guess what the authors did (since they won't tell you), and in the end never being able to reproduce it, not knowing if it was because key information was missing or because it was falsified.
That is, if you can spend the time attempt to reproduce the results, rather than publishing your own novel and revolution results. And if you'll find a journal who'll be interested in publishing your failure to replicate.
I've tried emailing peers for code and they would ignore me. I then got my advisor to talk to their advisor because I couldn't replicate their program and I got code sent to me. It was nowhere near what was explained in the paper and didn't give the results. Nothing happened. No retraction, no nothing. Just made it harder for me to publish my paper because it is very hard to beat results that were made up.
So what I'm saying is, I agree with your point here. The proof is in the pudding. Journals are convincing everyone that anything not in them is false and garbage but once garbage gets into them it creates and impossible bar to pass. Now repeat this for 50 or so years in a highly competitive space and what do you get? Well I think we all know the old saying: garbage in...
The LHC has multiple teams that will replicate an experiment. The facility is so big that you have different groups. You also have different detectors (such as ATLAS and CMS) where you replicate the experiments because each one has a bit of difference. And except for the very high energy stuff, other labs around the world can reproduce the results. Not to mention that particle physics has a 5 sigma significance threshold...
But for a lot of this, what happens is that they make the claim, they contact peers, send out the data, and have their peers confirm. You ever wonder why experiments like these have hundreds of authors[0,1]? That's because that shit is peer review. [0] has literally 8 pages of authors, which is in the format of first initial, last name. There's also 5 pages of university/lab affiliations.
Everything you're talking about here is done outside journals. And guess what, there are two versions of the arxiv work for [0], July 31st 2012 and August 31st 20122. Guess when it was "published". September 17th of that year. The announcement? July 4th! Btw, the journal is showing v2.
This is how big science is done. The replication and verification is happening intra-paper. You of course have to trust the machine, but that's true for everyone and a limit that can't be bypassed until a later date when a new one is built. Which frequently the first tasks are to confirm other observations, such as the JWST confirming exoplanets that were previously observed.
So to sum up: Replication happens, on big important experiments, outside journals
> [reproduction] is extreme reductionism
I'd still argue, with _aavaa_, that reproduction is the only currency that matters. Your experiment isn't worth shit if it can't be reproduced (i.e. it is indistinguishable from fraud)
> LHC results cannot be replicated
Most of the work is replicated, prior to publication.
> JWST cannot be replicated.
Observational data is confirmed. And JWST replicates its predecessors (an important part of its mission). (I'll even add LIGO in here and repeat the LHC and JWST comments)
> You know who will be able to tell you if the experiment description makes sense and has chances to be replicated if you ask them? Peers.
Everyone agrees with this. We are just saying the reviewers aren't necessarily your peers and the people you should be asking.
> because nobody likes to do peer review.
Lots of people like to do peer review and it happens all the time. If you're talking about reviewing for a journal, then yes I agree with you. But if you're talking about replicating work, building on, doing it to learn, or for fun, then this happens all the time. In fact, grad students do this professionally. How many works did you rebuild and try to replicate?
> I'm yet to see any proposal for peer review without editorial invitations which has at least theoretical chances to succeed.
You haven't dismissed __aavaa__ or my claims, only asserted they are wrong. If you would like to say why we are wrong, we're clearly open to hearing it and responding. But you've just asked us to trust you.
> So you do mean "reproduction", not "replication".
Reproduction is a form of replication. Replication is a spectrum. Reproduction is the minimum and with a lot between confirming experiments within a different context frame. But I don't think anyone is saying that just running your code constitutes reproduction. I'd say it is confirmation of your code. The source availability is about verification, as in I can probe it and look for errors, not so much reproduction.
And pre-print servers have even accelerated science. Because frankly this is a MCMC system because the nature of the game is everyone stumbling around in the dark.
I am a scientist and I'm all for "get rid of journals and conferences BECAUSE we already do figure it out." It isn't "somehow" we're already fucking doing it. We aren't going to journals and searching them, we go to arxiv. We read what our peers share with us. We read what comes out of other groups that we know that are doing similar work to us. Journals and conferences aren't helping us do science, they are helping us advance our careers because we live in a publish or perish ecosystem that quantifies our work based on the prestige of a conference who is prestigious because they reject 80% of works. Nowhere in here is a measure for the actual quality of a work. That is often a difficult task and can only be done by peers, which is already happening. Identifying bad works is easy but identifying good works is hard.
I'm sorry, but we already live in the system that you're suggesting is silly and it's been this way for centuries. Journals are just people capitalizing on our work and labor. Conferences at least put us in the same room, but they have no business claiming that they can accurately ascertain the quality of a work.
I do. Some of my peers do and some don't. It differs between fields and you're probably from CS (peer reviewed conferences) where progress is often faster than publication cycle, but it is not like that everywhere.
> they are helping us advance our careers
And it is important role which cannot be disregarded just because it doesn't match your idea of best possible promotion scheme.
> but we already live in the system that you're suggesting is silly and it's been this way for centuries.
Centuries? Can you elaborate? Peer review in journals as we know it is less than 100 years old.
> And it is important role which cannot be disregarded
I disagree. I think academia has fallen for Goodhart's law. While I would say that CS is probably on the worse side of things (especially ML), I don't think we're special in having critical problems. The publish or perish paradigm combined with the noisy signal of jorunaling is a big reason we are in the replication crisis. Because novelty is held in high regard while replication isn't of any concern. The irony being that novelty is quite rare and that the faster you publish the less novel works will be (in general. There are exceptions of course). You just can't do novel work fast.
> Centuries? Can you elaborate? Peer review in journals as we know it is less than 100 years old.
What I was referring to is that we've been just sending papers to one another for centuries. Uploading to arxiv is just an easier form of that (and was the explicit reason for its existence). That we scientists have been communicating with one another for centuries "figuring it out" without journals just fine. As far as I'm concerned, it is a failed experiment. It was okay in the beginning, had some good ideas, but then the rot grew and took over. It only works when everyone acts in good faith, and at scale only requires a few bad actors to spoil the barrel. Maybe I'm biased because of the problems of my field, but my friends in other areas still frequently hit problematic walls that just slow things down rather than improve them.
> I disagree. I think academia has fallen for Goodhart's law.
It may or may not, but my claim was that we need some mechanism to decide who gets a position and who doesn't, and it should be scalable for the current scale of science and beyond, i.e., at least hundreds/thousands of applications. Saying that journals are used for career decisions and this is not good (enough?) and thus doesn't need replacement mechanism is weird.
> a big reason we are in the replication crisis.
The stated reasons are the same for different branches of science, while replication crisis is far from that. Replication of 100-year-old experiments is also not always easy/successful. Publish or perish motivates people to cut corners, but I think it is far from being the main reason.
> What I was referring to is that we've been just sending papers to one another for centuries. Uploading to arxiv is just an easier form of that (and was the explicit reason for its existence).
(Almost) nobody reads arxiv (edit: I mean (relevant part of) arxiv as a whole, of course most people read specific papers on arxiv). In fact the snr of arxiv is so low, even compared to journals that just by filtering garbage you can get quarter a mil followers [1]. So instead of random experts of varying quality and motivation I'll get a single rando on twitter (no offense to A K, he does good job)? No thanks.
> It only works when everyone acts in good faith, and at scale only requires a few bad actors to spoil the barrel.
Citation needed? You could claim the same for science as whole, but self-correction turns to be extremely robust, even if slower than you may want.
> my claim was that we need some mechanism to decide who gets a position and who doesn't
I don't disagree. My claim is that the current mechanisms are nearly indistinguishable from noise but we pretend that they are very strong. I have also made the claim that Goodhart's Law is at play and made claims for the mechanisms and evidence of this. I'm not saying "trust me bro," I'm saying "here's my claim, here's the mechanism which I believe explains the claim, and here's my evidence." I'll admit a bit messy because a HN conversation, but this is here.
What I haven't stated explicitly is the alternative, so I will. The alternative is to do what we already do but just without journals and conferences. Peers still make decisions. You still look at citations, h-indices, and such, even though these are still noisy and problematic.
I really just have two claims here: 1) journals and conferences are high entropy signals. 2) If a signal has high entropy, it should not be the basis for important decisions.
If you want to see evidence for #1, then considering you publish in ML, I highly recommend reading several of the MANY papers and blog posts on the NeurIPS consistency experiments (there are 2!). The tldr is "reviewers are good at identifying bad papers, but not good at identifying good papers." In other words: high false negative rate or acceptance is noisy. This is because it is highly subjective and reviewers default to reject. I'd add that they have a incentive to too.
If you want evidence for #2, I suggest reading any book on signals, information theory, or statistics. It may go under many names, but the claim is not unique and generally uncontested.
> The stated reasons are the same for different branches of science, while replication crisis is far from that.
My claim about journals being at the heart of the replication crisis is that they chase novelty. That since we have a publish or perish paradigm (a coupled phenomena), that we're disincentivized from reproducing work. I'm sure you are fully aware that you get no credit for confirming work.
> (Almost) nobody reads arxiv
You claimed __aavaa__ was trolling, but are you sure you aren't? Your reference to Ashen is literally proof that people are reading arxiv. Clearly you're on twitter and you're in ML, so I don't know how you don't see that people aren't frequently posting their arxiv papers. My only guess is that you're green and when you started these conferences had a 1 year experiment in which they asked authors to not publically advertise their preprints. But before this (and starting again), this is how papers were communicated. And guess what, it still happened, just behind closed doors.
And given that you're in ML, I'm not sure how you're not aware that the innovation cycle is faster than the publication cycle. Waiting 4-8 months is too long. Look at CVPR: submission deadline in early November, final decision end of February, conference in late June. That's 4 months to just get Twitter notifications, by authors, about works to read. The listing didn't go live till April, so it is 6 months if you don't have peer networks. Then good luck sorting through this. That's just a list of papers, unsorted and untagged. On top of this, there's still a lot of garbage to sort through.
> the snr of arxiv is so low... that just by filtering garbage you can get quarter a mil followers
You and I see this very differently, allow me to explain. CVPR published 2359 papers for 2023 and 2066 for 2022. Numbers from here[0] but you can find similar numbers here[1] if you want to look at other conferences. Yes, conferences provide a curation, but so does Ashen and Twitter. The conferences cover many areas and don't tag or sort, and I know you aren't just going down that list and reading each paper title to determine if you should read it or not. I follow several accounts like Ashen. I also follow several researchers because I want to follow their works. This isn't happening because conferences are telling me who to listen to, this is because I've spent years wading through the noise and have peers that communicate with me what to read and not to read. Additionally I get recommendations from my advisor and his peers, from other members of my lab, from other people in my department, from friends outside my university, as well as emails from {Google,Semantic} Scholar. I don't think it isn't a well known problem that one of the most difficult things in a PhD is to get your network established and figure out how to wade through the noise to know what to read and not to. It's just a lot of fucking noise.
The SNR is high EVERYWHERE
And I'd like to add that I'm not necessarily unique in many of my points. Here's Bengio talking about how the pressures create incrementalism[2]. In this interview Hinton says "if you send in a paper that has a radically new idea, there's no chance in hell it will get accepted"[3]. Here's Peter Higgs saying he wouldn't cut it in modern academia[4]. The history of science is littered with people who got rocketed to fame because they spent a long time doing work and were willing to challenge the status quo. __Science necessitates challenging the status quo.__ Mathematics is quite famous for these dark horses actually (e.g. Yitang Zhang, Ramanujan, Galois, Sophie Germain, Grigori Perelman).
> Citation needed? You could claim the same for science as whole, but self-correction turns to be extremely robust, even if slower than you may want.
This entirely depends on the latter part. Yeah, things self-correct since eventually works get used in practice and thus held up to the flames. But if you're saying that given infinite time to retract a work it proof that the system is working then I don't think this is an argument that can be had in good faith. It is better to talk about fraud, dishonesty, plagiarism, how a system fails to accept good works, how it discourages innovation, and similar concepts. It is not that useful to discuss that over time a system resolves these problems because within that framework the exact discussion we are having would constitute part of that process, leading us into a self referential argument that is falsely constructed such that you will always be correct.
There are plenty of long term examples of major failures that took decades to correct, like Semmelweis and Boltzman (both who were driven insane and killed themselves). But if you're looking for short term examples and in the modern era, I'd say that being on HN this past week should have been evidence. In the last 8 days we've had: our second story on Gzip[5], "Fabricated data in research about honesty"[6] (which includes papers that were accepted for over a decade and widely cited), "A forthcoming collapse of the Atlantic meridional overturning circulation"[7] (where braaannigan had the top comment for awhile and many misinterpreted) and "I thought I wanted to be a professor, then I served on a hiring committee"[8] (where the article and comments discuss the metric hacking/Goodhart's Law). Or with more ML context, we can see "Please Commit More Blatant Academic Fraud"[9] where Jacob discusses explicit academic fraud and how prolific it is as well as referencing collusion rings. Or what about last year's CVPR E2V-SDE memes[10] about BLATANT plagiarism? The subsequent revelation of other blatant plagiarisms[11]. Or what about the collusion ring we learned about last year[12]. If you need more evidence, go see retractionwatch or I'll re-reference the NeurIPS consistency experiments (which we should know are not unique to NeurIPS). Also, just talk to senior graduate students and ask them why they are burned out. Especially ask both students who make it and those struggling, who will have very different stories. For even more, I ask that you got to CSRankings.org and create a regression plot for the rank of the universities against the number of faculty. If you even need any more, go read any work discussing improvements on the TPM or any of those discussing fairness in the review process. You need to be careful and ask if you're a beneficiary of the system, if you're a success of the system, a success despite the system, or where you are biased (as well as me). There's plenty out there and they go into far more detail than I can with a HN comment.
You can find dozens of proofs on the arxiv that P=NP and that P!=NP. What does peer “review” look like here? Most people won’t bother to write an article specifically rebutting one random incorrect proof.
Now somebody tomorrow posts a proof that P!=NP. Will people pay attention? If it’s a “big name” or somebody at a university with a good PR office then yes; otherwise no. Doesn’t seem ideal to me.
How many of those works actually hit the front page of HN? How many do you hear of? The reason it is little to none is because peer review happens. If someone posted that to HN it would likely be trashed very quickly and fail to make it to the front page. That's peer review in action. Now does shit get through sometimes? Yeah. But I'm not convinced this is at a meaningfully higher rate than any top tier journal or conference, if at all.
But as an experimental chemist who got a PhD and works at a national lab, if the twitch/Twitter streamers don't produce a semiconductor, it says nothing
Experiments and processes are hard and particular. You're expecting to produce a nanoscale material through macroacale processes. Chemistry can be like producing a microchip with a ballpeen hammer. "I hit it seventy times in thirty seconds in 3.8mL acetone and it just forms a single crystal structure. Do not hit it seventy-one or sixty-nine times."
Chemists are experts at clearly explaining their procedure while leaving out the meaningless keystone detail that will only take a year or two to suss out if you're a clever experimentalist.
And this isn't a clearly outlined procedure.
But, again, I think the best case scenario is to watch an ambitious amateur do materials synthesis and see it work less perfectly than he intended. Maybe he and they stick around for the other 5 or 10 or 150 weeks as he does the same thing again to make sure the problem is nature rather than technique.
If bet-placing is the fashion, shall we place bets about tenacity and thoroughness?
I wonder if there's a language barrier. I can't judge for myself--neither the substantive content nor the biographies--but hypothetically the principle investigators might not have strong (or any) English language proficiency, relying on a younger, less experienced peer to write the paper.
As someone who sucks at languages, I can appreciate that learning new languages does not come easy to everybody, or even to most people, especially in older age. If it seems like all reputable scientists have some English proficiency, that's a selection effect--within the English-speaking world in particular you're not going to bump into or even hear much about those who cannot speak or write English. AFAIU there's a substantial amount of high quality research in, e.g., Japan that is invisible to non-Japanese speakers. Japan's scientific establishment long predates English as a global language, unlike China; and unlike many European countries but similar to the United States there was never a widespread norm of learning foreign languages even among the professional classes, AFAIU. Though Korea is a younger industrialized country, I wouldn't take for granted that the principal discoverers have any English proficiency, especially considering that they would be late middle-age or older if they were working in the lab in 1999.
I dunno, I'd be suspect of the argument was that the directions were perfect in Korean and merely totally hidden from industrious Americans.
Japanese and Korean professionals speak English. There will be one researcher for whom explaining the language is a minor dalliance.
I work in a very minor polymer materials field. There are two Japanese researchers who cheerfully banter. If somebody had hidden some transformative work behind their language, even as a student, they would have talked about it straightaway, if only for the conversation and a bit of competitive intellectual football. Can the foolish American grad student reason around it? Let's see!
The twitch channel was packed with interested Koreans. If there had been the tiniest bit of confusion there would have been 200 translations from which to utilize the Translation Of The Crowd.
Well, Japan is the only mature industrial nation other than the United States where a majority of science papers are published in domestic journals[1][2], many of which are predominately or solely Japanese language[3], whereas international journals--including international journals in Japan--are invariably English.
Ofcourse the Japanese scientists you've met speak English. People who don't speak English wellenough are not generally going to participate in events or situations where they would need to rely on English-language speaking skills. That someone apologies for their bad English is, of course, irrelevant; if it were truly bad you wouldn't have encountered them. Likewise, Japan has many international science programs, forums, and conferences; and they'll be filled with proficient Japanese English speakers, but Japan is a huge nation so it says little necessarily about broader patterns.
I don't doubt that a majority of professional research scientists in Japan have some proficiency in English. Especially the top scientists; even in Japan "impact" would undoubtedly be heavily biased toward publication in prestigious international journals. But there's plenty of literature on the internet that describes the relatively poor English-language proficiency (especially conversationally or with verbiage outside that in their technical field) and poor English-speaking research community integration of the broader Japanese scientific establishment. See, e.g., "Japanese materials scientists' experiences with English for research publication purposes".
I first encountered this cultural dynamic when researching the tilting of the Millennium Tower. While friction piles weren't first used in Japan, Japan was where friction piles were first heavily researched and employed for large buildings, including skyscrapers. People kept saying friction piles were idiotic in earthquake-prone, sandy soils, but that's precisely the environment where they were first researched and from whence the global engineering community's confidence sprouted. However, AFAICT much of that research was published in Japanese, even through the late 20th century and (IIRC) even early 21st century. Was that Nobel-worthy science? No, but it was very useful applied science; and applied science which was highly consequential to global engineering and architecture, notwithstanding that much of the substance of it existed on the other side of a Japanese language barrier.
[3] E.g. many of the publications by the The Japan Institute of Metals and Materials are only or predominately in Japanese. Some domestic publications require summaries and tables in English but permit the text to be in Japanese; e.g. the Journal of the Society of Materials Science.
> People kept saying friction piles were idiotic in earthquake-prone, sandy soils, but that's precisely the environment where they were first researched
Friction pilings have been in use in NL for 100's of years.
There might be a body of historical purely Japanese scientific research, but I know some Japanese physicists who all speak English well enough, and have heard others on podcasts who do also. It would be serevely limiting to not speak English when that is the language of modern science. I don’t know how far back you would have to go to find monolingual scientists but I would guess several decades at least.
There are plenty of Japanese scientists who are _at least_ extremely uncomfortable in English. Plenty of people who struggle to write their own stuff in English and get it corrected a lot. Though I think that beyond a certain level you have to be interacting internationally to get anywhere in your career so you're gonna be forced along.
Plenty of people would learn Latin to consume some content, but doesn't mean they are proficient in it itself, would be another example.
Having said that, I think that in the research field that merely ends up with "very confusingly written papers with mistakes large enough to hide process errors" more than "cannot communicate any info in English". But there are plenty of people who get their doctorate and their only English-language output is their final papers in themselves.
I find it sort of implausible that somebody high in their field is totally ignorant in English, if only because so many papers are only published in English.
I've known so many researchers who apologize every other sentence because they say they've only read English and I would never had known otherwise.
Which is to say - it'd be wierd to say "well I don't know English and I don't know anybody who does oh well". But, anyway, the papers were published on aeXive in...
If you mean I am not aware of the ones that don’t speak English, not really, I have worked and gone to university in Japan and know a large number of people with whom I only speak in Japanese because their English skills are not good. Among working research physicists (not just Phds working on some company) under 45 I have found solid English skills as a rule, probably because international experience is important to their careers.
It’s really hard to see a language barrier affecting the sample synthesis instructions. It’s a few short paragraphs of “mix two things together and put them in the oven.”
It is amusing so many commentators cannot communicate novel materials science in Korean to the layperson. Where's your zero resistivity sample?
The paper is just a guide. Who would replicate likely needs to contact the authors to understand the implicit details of the recipe. That is process knowledge. There's a good reason world-leading lithography happens in that region. With luck, researchers and engineers will be able to refine and industrialize the process these coming years.
> if the twitch/Twitter streamers don't produce a semiconductor, it says nothing
If you're referring to Andrew McCalip, he's been very clear that he is not claiming to be doing a serious replication attempt, but he also does zero-g/orbital chemical synthesis for a living so its not quite fair to characterize him as a random twitch streamer.
Something I've also been thinking about is that we don't appear to have any data on the success rate. Maybe the samples of LK99 are the best that the team has been able to produce, but the process requires further refinement and experimentation.
1000 / 24y ~= 41/y, which after vacation and other things is probably closer to 1/wk. If all their experiments involve easy-to-synthesize substances (where easy == no more than 3 or 4 days) then it's not that intense, no. But you really have to be quite confident that this approach will pay off to put 2+ decades into it.
I suspect if they really did 1,000 experiments then they often did 2/wk because I don't think they ran all of them continuously over 20+yrs.
Speaking of which, can we see their notes on the other 999 experiments that didn't work out? That would be very useful, especially if LK-99 turns out not to be awesome but is good enough to hint that this approach is worthy of more research -- there's 999 compounds not worth testing, right? (Well, maybe maybe. If the physics of LK-99 come to be understood, then maybe some of the other 999 experiments might need to be re-examined.)
They didn't have a theory to drive them to doggedly pursue this system as an SC. Was it just the blip 20 years ago? Why wouldn't you wonder if your machine was shitty?
If I was them, its decades past believing this might be an empty rabbit hole. You're committed. If it's real, it's heroic. If not? It's a failure of rationality. But it's both simultaneously.
If they're right, I'm sure every notebook will be published. If not? We'll never see their fury or process.
A huge part of science is being a knowledgeable and rigorous experimentalist. You can do every great for years, but you still have to do things right 25 years in or you'll be unable to sleep
It's too complex to be perfectly predictable, like particle physics
But without instruments,you have to hope the nanoscale changes are slow and visible in solution at 1,000,000x larger scale
The fact that we can throw shit in a beaker and identify useful reactions is a hell of a thing. Our magic balls are the only thing that take it much beyond it's origins in alchemy.
Why should experiments wait to be submitted to some arbitrary academic body before trying to to reproduce them? This is what’s wrong with modern science, everything is academic.
The only thing valuable that academic journals and conferences bring to the table is the peer-review process. Which, to be fair, is needed.
In this case, the peer review is being conducted without the need for publishing venues. However, I don't think that is usually the case. Take a look at the countless number of ArXiv papers that lack any sort of peer review.
Somewhere I read that this is how things used to be in Einstein era (or maybe earlier than that). There was little bureaucracy in getting the idea out.
Richard Rhodes' monumental work on the A and H bomb mentions that from Thompson onward, "being heard" Was a problem. Juniors often struggled to be taken seriously, sometimes seniors ostentatiously "spoke for them" to have new ideas given credence and women were doubly disabled on the "taken seriously" front.
Right up until Murray Gell-Mann and beyond, speaking outside the current limitations to knowledge was hard. I don't want this to descent into AGW and antivaxx denialism, this is inside classic science but considering radical theory/paradigm shifts which were still testable propositions. New models are hard on early stage career.
Blinded peer review was partly designed to help some of that. In a narrow enough field it's impossible for reviewer and submittor not to know each other. There may only be 3-4 people who understand your niche fully. Rueben Hersh discusses that a bit in "what is mathematics really" (I think, could be another book of his, "the mathematical experience")
Rhodes discusses Michael Polyani's theory of science as old fashioned apprenticeship. Journeyman scientists publish reproducible, testable work. Theoreticians.. harder to test sometimes.
Well, the blinding is supposed to allow the juniors to speak up when the seniors are making mistakes, so it makes sense its only one way.
And while double blinding sounds nice in theory I'm not so convinced it is that useful in practice, because it requires the reviewer to play along and pretend they can't figure it out from the text alone: if they are able to do that reliably they can probably be trusted to keep an open mind anyway.
Reliable blinding of the author would mean having them consciously copy the style of others and avoiding citing their own previous work, which would be very hard in a small sub-field since they are by definition a sizeable fraction of it!
I don't see it this way. Juniors will be rarely appointed as reviewers by Nature.
Double-blinding gives unknown groups and juniors the chance to publish at top venues, which are incredibly biased towards big-name universities and big-name groups.
Yes. The system broke down. I don't know anyone who thinks it works how it should
I've had pretty hard bounces which were deserved and I know how to get work over the threshold, but some review feedback has been petty, passive aggressive ignorance, and suspiciously similar stuff pops up from time to time in "3 papers and your phd is done" which makes me wonder if copycats are getting softballed through for career development.
the rules behind length, word count and mark-up are pretty silly too. I've seen some rather odd latex tricks to compensate for length inside word count because typographical checkers were bouncing.
I don't do this for a living, barely act as maybe helpful co author these days: it's hard work.
I think in part this is because a lot of people started to be interested in the meta-metrics of scientific work and like any metric once you start tracking it you influence the system you are tracking. Publishers and various scientific actors then made things worse by making those very metrics (a symptom) a goal in its own right. That's what broke the system.
While I am very skeptical, when working with what seems to be a fluke, it is more important to test the claim then to test a reproduction.
Figure out how to send an independent lab's equipment and personal there or a sample of the substance to an independent lab that verify and if true. Also ensure that they can also do material analysis such as X-Ray spectroscopy & diffraction , and a battery of other stuff if a miracle did occur.
For all we know, this could be one of those accidents of sloppiness that introduces a particular containment that makes everything line up as a super conductor that nobody else will easily reproduce.
> For all we know, this could be one of those accidents of sloppiness that introduces a particular containment that makes everything line up as a super conductor that nobody else will easily reproduce.
Maybe we should call this the Hyde Phenomenon.
> Jekyll's involuntary transformations increased in frequency and required ever larger doses of the serum to reverse....Eventually, the supply of salt used in the serum ran low, and subsequent batches prepared from new stocks failed to work. Jekyll speculated that the original ingredient had some impurity that made it work.
I think all they have to do is send a sample to another lab to analyze.
Apparently, people feel it's pretty easy to make, but in any case, if they have a sample they can see if it has the claimed properties and analyze what it's made of.
> For all we know, this could be one of those accidents of sloppiness that introduces a particular containment that makes everything line up as a super conductor that nobody else will easily reproduce.
I get that terrible sense too - either the original studies were in error, or some one-off fluke makes it difficult to reproduce.
On the upside however, and I'd caveat this as not being fully in the know about the art of the possible in solid state labs, it seems that the material isn't that awfully difficult to produce given an appropriate lab and equipment. Hopefully we'll know one way or the other shortly.
why don't they just repro their own experiment? If they can do it twice, then it isn't a fluke. at worst it's something specific to their lab. if they can't do it a second time, then the issue is settled.
I'm sure it will take a lot of time and money to run everything again. but all of earth seems willing to give them whatever resources they need.
It doesn't particularly matter if it is reproducible, as long as they have 1 working superconductor that they can hold in their hand (Room temperature/pressure), its a nobel prize. Other people can figure out how to reproduce it since it is clearly possible.
The challenge is proving they have even that 1 sample.
If it was some kind of fluke and is not easily reproducible but the sample exists and is a room-temperature super-conductor then it becomes a reverse engineering problem.
Considering the magnitude of the discovery and the relatively easy steps to reproduce the process, they would have to be outrageously lazy to have put the paper out without doing it a second time. In absence of a statement that they only tried it once, I think it's fair to assume they at least verified they can reproduce it locally.
> On the other hand, he says, researchers at Argonne and elsewhere are already trying to replicate the experiment. “People here are taking it seriously and trying to grow this stuff.”
The submitted title has been heavily editorialized. That’s the only relevant part of the article, and that’s far from implying that there’s a concerted effort at Argonne.
> The submitted title has been heavily editorialized. That’s the only relevant part of the article, and that’s far from implying that there’s a concerted effort at Argonne.
What's wrong with the submitted title (other than being very narrow)? It just says they're attempting to replicate, which they are.
If anything I'd say that "taking it seriously" is stronger language, and the submitted title is slightly underselling it.
The article says many things. Cherry-picking one incidental detail to use as the title is editorializing, regardless of whether it is accurate. Do you think an equally good title might be "If you’ve ever had an MRI, you’ve lain inside a big electromagnet made of superconducting wire."?
I objected to shusaku implying that this headline was exaggerating.
I don't know how to make this clearer.
> Do you think an equally good title might be "If you’ve ever had an MRI, you’ve lain inside a big electromagnet made of superconducting wire."?
Well, if you really want to get into this, even though I acknowledge it's cherry-picking:
If that fact is what the submitter cared about, and this was the only page on the internet talking about that fact... it wouldn't be a terrible idea.
The HN guidelines aren't great here. You can make your own blog post about something and link that, but it will probably get replaced to the "original" link even if the "original" link had a completely different focus.
Sometimes you have to pray that whoever wrote the original splits it up into different articles themselves. Or that they write on twitter, so you can link to a specific tweet.
Sometimes a newspaper will put three completely separate stories into the same article, and trying to link the second or third story on HN risks the title getting replaced with something utterly unrelated.
Is the original title better? "A spectacular superconductor claim is making news. Here’s why experts are doubtful." Standard nothing title, "a thing happened." I zeroed in on the pithy part of the article, which is SOP on HN. The subtitle is just "skepticism abounds" which we already know. What some, including me, didn't know is that legit USG labs are studying it, and that's what makes it news.
Norman's complaints that lead atoms are too heavy do not seem consistent with the composition of other known superconductors. As far as I know, the most widely used cuprate superconductor is bismuth strontium calcium copper oxide (BSCCO), which was used for the world's first superconducting power line:
>What’s more, lead and copper atoms have similar electronic structures, so substituting copper atoms for some of the lead atoms shouldn’t greatly affect the electrical properties of the material
The lead (II) ions claimed have an even number of electrons. Copper (II) has an odd number. Or if copper (I) is present, then the charge itself is different. Again, this is just a very confusing argument to hear from a physicist.
>First, the undoped material, lead apatite, isn’t a metal but rather a nonconducting mineral.
The cuprate and iron superconductors are not metals either. In fact some are Mott insulators (materials with unexpectedly high resistance) under normal conditions.
There is a case for skepticism about LK-99, but it isn't this one.
One of the common threads seems to be that other scientists think that lk-99 is not a super conductor but just strongly diamagnetic.
As a non physicist i wonder if that is useful in and of itself? Skimming wikipedia it doesn't seem like there are that many strongly diamagnetic materials. Would discovering a new one still be a big discovery (just not earth shattering)
That depends on how the diamagnetism comes about. One way in which it could come about which would still make it a big discovery is if the material is locally superconducting but not globally. That would give you diamagnetism without superconductivity on a usable scale and might in turn open the door to modifying the material to increase the size of the superconducting regions. Pure speculation, obviously, the bigger chance is that if the material is 'just' diamagnetic that it is simply diamagnetic in the same way in which other materials are (all particles of which the samples are made are constructed such that all electrons are paired).
My understanding is that taking the paper at face value it seems like they’re saying the diamagnetism was an encouraging piece of supporting evidence beyond their other rationale for what they believe they’ve found. I’m not enough of a physicist (or one at all) to speak to that though so I guess I’ll just sit here wit ma popcorn for the next week with everyone else
I see. V=IR=0 means R can measure as 0, but they did measure small current. Presumably they know the voltage. That seems odd if they can’t measure the resistance accurately if their instrumentation is rated to measure that current.
It is not possible to determine that a resistance is exactly zero through direct measurement, because any kind of probes that can be used to contact the material will have a high contact resistance, which will vary from one probe contact to another.
Even when the resistance is measured with a 4-point probe, the only conclusion that can be reached is that the resistance is smaller than the experimental error.
However it should be possible to determine that the resistance is zero through various indirect effects, for instance by establishing a closed current in the material and verifying that it does not decrease in time.
Even such experiments must be done carefully because some materials with unusual magnetic properties may behave apparently in a similar way to superconductors in some experiments, e.g. both a magnetized ferromagnetic material and a superconductor ring carrying a current will have a remanent magnetic field that is constant in time and in both cases the magnetic field will disappear when the temperature is raised above a threshold.
Thanks that’s interesting. If it’s so hard to determine if a substance is superconducting, does that mean many substances that are apparently but not quite superconducting have similar applicability? I.e., if you need superconductors for X application, can you not just put the substance in question in that application and observe if it works or not? Sorry for the dumb questions.
Despite some criticism of the original work, they seem to have been very conscientious about making replication easy.
===========
Nadya Mason, a condensed matter physicist at the University of Illinois, Urbana-Champaign says, “I appreciate that the authors took appropriate data and were clear about their fabrication techniques.” Still, she cautions, “The data seems a bit sloppy.”
Yes this is an excellent thread. You have done some of the only on the ground reporting from Seoul. The takedowns you linked of the paper, CMTC giving it an F, are damning. There's one legit professor Kim from William & Mary who had a tiny part of this, but more and more, it looks like it's the output of a few cranks out of a "Q-Centre." The bulk of the work took place before Kim's involvement and he's listed on 1 of 3 papers so far. He's the only thread of legitimacy in this whole thing, and it's a thin one. The 5 other guys are just loony.
The reason the papers look like junk is that they were leaked by a rogue former team member who wants credit. The papers weren't supposed to have been published yet, peer review is still ongoing.
Having skipped to his analysis of the video, I find him pretty convincing.
He explains why the video is bunk using information provided by the lab, and basic physics. And then 'replicates' using his own sheet of metal and a magnet.
Also, he raises the fact that it's a commercial lab.
I've downgraded my level expectations a couple of orders of magnitude. But hey, let's hope for a miracle.
Just watching it. He brought up something I noticed also, showing the magnet moving the hanging copper plate with the material deposited on it... In like basic physics class you go over eddy currents induced by moving magnets, isn't that all we see there? I can't understand how this showed anything other than that?
Thunderfoot comment though seems ridiculous. It would be irrelevant because it is a ceramic??? lol. They proposed a mechanism how it works! Obviously this would just be the first step. That's like saying the first transistor was irrelevant because we don't use that design anymore.
Despite not being of Australian origin, I can understand how one could interpret his initial remarks as being tinged with sarcasm. In my personal analysis, however, I found his comments to be injected with a considerable amount of wit and humor.
as a non-australian but native English speaker, I found the video annoying because it's padded and the jokes/sarcasm weren't really all that funny. If you trim out his YouTube-OverReactions, I think the video goes down to like 1.5 minutes in my mind and still conveys the same information, perhaps even better.
I get probably this is a popular video maker in whatever sphere he's in, but as an outsider, it's difficult to get into the video because of what I attribute as the presenter's YouTube personality.
(also it very much so bugs me to see people using Chrome and see all the nags for notifications, the random extensions/features, etc. And any browser without uBlock origin throws me as it takes me a minute to realize why all the websites look so bad, but that's just a personal issue :) )
I've been wondering what the "hanging copper plate" demonstration would look like if you were just seeing eddy currents. I kind of assumed you wouldn't see much movement because otherwise that makes the whole demonstration seem dumb.
...and based on that demo, it really does seem like a dumb way to demonstrate your miracle material. Really doesn't inspire confidence.
Why is no one asking to analyze the superconductors they claimed to make? Everyone's talking about replicating it by following their recipe, but isn't the proof in the pudding?
To me the main takeaway is that there has had to be some advancement in this field over the past two decades if claims of sufficiently-close-to-ambient superconductivity are currently not dismissed outright.
Maybe we'll have maglev everything by the time I'm old and gray.
It's over for this compound I'm pretty sure, but it is cool to see the range of responses from professionals in this field. Some like Jorge Hirsch (superconductivity researcher and came up with the h-index) think it's a joke (and his work is cited by the Korean scientists), and then some guys at a national lab are trying to synthesize it.
Hirsch is also apparently a bit of a shitposter at age 70 and was banned for flaming people on arxiv last year.
> It’s not over until multiple labs fail to reproduce it. A definitive no will take months.
Yes. Some may remember the problems with cold fusion. I once went to a packed talk at Stanford where some physicists and chemists were trying to reproduce the effect. They mentioned that in the first attempts, they had radiation alarms set up in case the thing suddenly started emitting substantial amounts of radiation. After a while, it became clear that the effect, if it existed at all, was small, somewhere below twice background radiation, so it wasn't high risk.
They discovered that people moving around the apparatus affected the results; humans have a lot of water and are neutron reflectors.
Finally they did the experiment inside a "neutron cube", a box made of lead bricks, to eliminate stray neutrons. No neutron generation measured inside the cube.
They worked hard for months getting to a definitive "no".
Even if it doesn’t turn out to be a superconductor, it is at least strongly diamagnetic, so it opens avenues for future research either way. There’s a potential scenario here where this doesn’t turn out to be a superconductor, but it nonetheless leads to a superconductor further down the line.
The one wrinkle here is if the sample we saw actually was pyrolytic carbon and someone simply mislabeled a vial when looking for something to shoot the video with.
At the extreme end of big news, stupid things can happen (I've worked in a lab: people suck at labelling things they make).
If it is pyrolytic carbon, they've somehow significantly enhanced the dimagnetism of it, which is remarkable. Usually that stuff only barely floats, and only when it is very thin.
I forget the actual quote, but there is a saying that if an old esteemed scientist says something is possible, they are almost certainly right. If they say something is impossible, they may be wrong.
"1. When a distinguished but elderly scientist states that something is possible, he is almost certainly right. When he states that something is impossible, he is very probably wrong.
2. The only way of discovering the limits of the possible is to venture a little way past them into the impossible.
3. Any sufficiently advanced technology is indistinguishable from magic." A. C. Clark
You'll see a bunch of old scientists saying a legitimate revolutionary breakthrough is bullshit, but you didn't see the other 99 times they called bullshit correctly.
There’s some consensus that the results can be questioned based on:
- Lenz’s law applying to copper (Lenz's law - Wikipedia 7)
- Copper phosphide (“Lanarkite and Cu3P were uniformly mixed in a molar ratio of 1:1 in an agate mortar with a pestle”). Pretty high copper content.
-No Meissner effect just standard copper-related phenomena based on Lenz’s law.
