The higher-level problem is that there are tons of scientific papers with falsified data and very few people who care about this. When falsified data is discovered, journals are very reluctant to retract the papers. A small number of poorly-supported people examine papers and have found a shocking number of problems. (For instance, Elisabeth Bik, who you should follow: @elisabethbik.bsky.social) My opinion is that the rate of falsified data is a big deal; there should be an order of magnitude more people checking papers for accuracy and much more action taken. This is kind of like the replication crisis in psychology but with more active fraud.
Unfortunately as you spend more time investigating this problem it becomes clear that replication studies aren't the answer. They're a bandage over the bleeding but don't address the root causes, and would have nearly no impact even if funded at a much larger scale. Because this suggestion comes up in every single HN thread about scientific fraud I eventually wrote an essay on why this is the case:
(press escape to dismiss the banner). If you're really interested in the topic please read it but here's a brief summary:
• Replication studies don't solve many of the most common types of scientific fraud. Instead, you just end up replicating the fraud itself. This is usually because the methodology is bad, but if you try to fix the methodology to be scientific the original authors just claim you didn't do a genuine replication.
• Many papers can't be replicated by design because the methodology either isn't actually described at all, or doesn't follow logically from the hypothesis. Any attempt would immediately fail after the first five minutes of reading the paper because you wouldn't know what to do. It's not clear what happens to the money if someone gets funded to replicate such a paper. Today it's not a problem because replicators choose which papers to replicate themselves, it's not a systematic requirement.
• The idea implicitly assumes that very few researchers are corrupt thus the replicators are unlikely to also be. This isn't the case because replication failures are often due to field-wide problems, meaning replications will be done by the same insiders who benefit from the status quo and who signed off on the bad papers in the first place. This isn't an issue today because the only people who do replication studies are genuinely interested in whether the claims are true, it's not just a procedural way to get grant monies.
• Many papers aren't worth replicating because they make trivial claims. If you punish non-replication without fixing the other incentive problems, you'll just pour accelerant on the problem of academics making obvious claims (e.g. the average man would like to be more muscular), and that just replaces one trust destroying problem with another.
Replication failure is a symptom not a cause. The cause is systematically bad incentives.
Incentives are themselves dangerous. We should treat incentives like guns. Instead we apply incentives to all manner of problems and are surprised they backfire and destroy everything.
You have to give people less incentives and more just time to do their basic job.
> You have to give people less incentives and more just time to do their basic job.
That was the idea behind tenure, but then tenure became the incentive. Job security at a socially prestigious job with solid benefits is a huge incentive to fraud even to people who don't care about science, and then for people who do care about doing science and have been directing their entire adult lives towards that end, they face a cataclysmic career bifurcation: either they get a tenured academic research position and spend their lives doing science, or they leave science behind altogether and at best end up doing science-adjacent product development which at best best, if they join a pharmaceutical company or something like that, might sometimes closely resemble scientific research despite being fundamentally different in its goals.
Given the dramatic consequences on people's lives, fraud should be expected. Academic research should acknowledge the situation and accept that it needs safeguards against fraud as surely as banks need audits and jewelry stores need burglar alarms.
OP here. Perhaps I didn't explain it well, but I think the key is to de-incentivize bad behavior, and to make sure people publishing have some skin in the game.
Right now, it's the opposite. The system rewards flashy findings with no rigor. And that's a slippery slope towards result misrepresentation and downright fraud.
I think this hits the nail on the head. Academics have been treated like assembly line workers for decades now. So they’ve collectively learned how to consistently place assembled product on the conveyor belt.
The idea that scientific output is a stack of publications is pretty absurd if you think about it for a few minutes. But try telling that to the MBA types who now run universities.
You do need to incentivize something. If you incentivize nothing that's the same thing as an institution not existing and science being done purely as a hobby. You can get some distance that way - it's how science used to work - but the moment you want the structure and funding an institution can provide you must set incentives. Otherwise people could literally just stop turning up for work and still get paid, which is obviously not going to be acceptable to whoever is funding it.
I think it’s an interesting feature of current culture that we take it as axiomatic that people need to be ‘incentivized’. I’m not sure I agree. To me that axiom seems to be at the root of a lot of the problems we’re talking about in this thread. (Yes, everyone is subject to incentives in some broad sense, but acknowledging that doesn’t mean that we have to engineer specific incentives as a means to desired outcomes.)
I think there is some misunderstanding here. Incentives are not some special thing you can opt to not do.
Who do you hire to do science? When do you give them a raise? Under which circumstances do you fire them? Who gets a nicer office? There are a bunch of scientist each clamouring for some expensive equipment (not necessarily the same one) who gets their equipment and who doesn't? Scientist wants to travel to a conference, who can travel and where can they travel? We have a bunch of scientist working together who can tell the other what to do and what not to do?
Depending on your answers to these questions you set one incentive structure or an other. If you hire and promote scientist based on how nicely they do interpretive dance you will get scientist who dances very well. If you hire and promote scientist based on how articulate they are about their subject matters you will get very well spoken scientist. If you don't hire anybody then you will get approximately nobody doing science (or only independently wealthy dabbling here and there out of boredom.)
If you pay a lot to the scientist who do computer stuff, but approximately no money to people who do cell stuff you will get a lot of computer scientist and no cell scientist. Maybe that is what you want, maybe not. These don't happen from one day to an other. You are not going to get more "cancer research" tomorrow out of the existing cancer researchers if you hike their salary 100 fold today. But on the order of decades you will definitely see much much more (or much less) people working on the problem.
I meant to cover that in the last (parenthesized) sentence or my post. There will always be incentives in a broad sense, but it is not necessary to “incentivize” people via official productivity metrics. Academics used to figure out who to promote without creating a rush for everyone to publish as much as possible, or to maximize other gameable metrics. I don’t kid myself that there was ever a golden era of true academic meritocracy, but there really did used to be less of an obsession with silly metrics, and a greater exercise of individual judgment.
Einstein, Darwin, Linneaus. All science as a hobby. I don't think we should discount that people will in fact do it as a hobby if they can and make huge contributions that way.
Einstein spent almost all of his life in academia living off research grants. His miracle year took place at the end of his PhD and he was recruited as a full time professor just a few years later. Yes, he did science as a hobby until that point, but he very much wanted to do it full time and jumped at the chance when he got it.
Still, if you want scientific research to be done either as a hobby or a corporate job, that's A-OK. The incentives would be much better aligned. There would certainly be much less of it though, as many fields aren't amenable to hobbyist work at all (anything involving far away travel, full time work or that requires expensive equipment).
>>>• Replication studies don't solve many of the most common types of scientific fraud. Instead, you just end up replicating the fraud itself. This is usually because the methodology is bad, but if you try to fix the methodology to be scientific the original authors just claim you didn't do a genuine replication.
>>>• Many papers can't be replicated by design because the methodology either isn't actually described at all, or doesn't follow logically from the hypothesis. Any attempt would immediately fail after the first five minutes of reading the paper because you wouldn't know what to do. It's not clear what happens to the money if someone gets funded to replicate such a paper. Today it's not a problem because replicators choose which papers to replicate themselves, it's not a systematic requirement.
Yes but it only works if the field has consistently high standards and the team violating them is an outlier. In a surprisingly large number of cases replications are hard/impossible because the entire field has adopted non-replicable techniques.
> • Many papers can't be replicated by design because the methodology either isn't actually described at all, or doesn't follow logically from the hypothesis. Any attempt would immediately fail after the first five minutes of reading the paper because you wouldn't know what to do.
This is indeed a problem. But it can be solved by journals not accepting any paper that is not 1) described in such a way that it can be fully replicated, and 2) include the code and the data used to generate the published results
Just that would go a long way.
And then fund replication studies for consequential papers. Not just because of fraud but because of unintended flaws in studies, which can happen of course.
Journals can't police academia because their customers are university libraries. It's just not going to happen: journals are so far gone that they routinely publish whole editions filled with fake AI generated content and nobody even notices, so expecting them to enforce scientific rigor on the entire grant-funded science space is a non-starter ... and very indirect. Why should journals do this anyway? It's the people paying who should be checking that the work is done to a sufficient standard.
You’re right but it’s what journals are supposed to do - that’s why we pay. Otherwise we don’t need journals and everyone can just self publish to bioRxiv.
They'll only do it if the customers demand it. But the customers are universities, if they cared they could just fix the problem at the source by auditing and validating studies themselves. They don't need to pay third parties to do it. Journals often don't have sufficient lab access to detect fraud anyway.
I do think the journal ecosystem can just disappear and nobody would care. It only exists to provide a pricing mechanism within the Soviet-style planned economy that academia has created. If that's replaced by a different pricing mechanism academic publishing could be rapidly re-based on top of Substack blogs and nothing of value would be lost.
That's a tremendously expensive way to detect fraud. There's funding, but also the people replicating need to have at least near the expertise of the original authors, and the rate of false positives is clearly going to be high. Mistakes on the part of the replicator will look the same as scientific fraud. Maybe most worryingly, the negative impact of human studies would be doubled. More than doubled, probably- look at how many people claimed to successfully replicate LK-99. A paper may need to be replicated many time to identify unknown.
Maybe first we could just try specifically looking for fraud? Like recording data with 3rd parties (ensuring that falsified data will at least be recorded suspiciously), or a body that looks for fabricated data or photoshopped figures?
> My opinion is that the rate of falsified data is a big deal
Have anything that backs that up? Other than what you shared here?
I would be very interested in the rate on a per author level, if you have some evidence. Fraud "impact" vs "impact" of article would be interesting as well.
All of those examples have no relative meaning. If there are millions of papers published per year, then 1000 cases over a decade isn't very prevalent (still bad).
‘It is simply no longer possible to believe much of the clinical research that is published, or to rely on the judgement of trusted physicians or authoritative medical guidelines. I take no pleasure in this conclusion, which I reached slowly and reluctantly over my two decades as an editor of the New England Journal of Medicine.’ — Marcia Angell
0.04% of papers are retracted. At least 1.9% of papers have duplicate images “suggestive of deliberate manipulation”. About 2.5% of scientists admit to fraud, and they estimate that 10% of other scientists have committed fraud.
“The case against science is straightforward: much of the scientific literature, perhaps half, may simply be untrue.” — Richard Horton, editor of the Lancet
The statcheck program showed that “half of all published psychology papers…contained at least one p-value that was inconsistent with its test”.
The GRIM program showed that of the papers it could verify, around half contained averages that weren’t possible given the sample sizes, and more than 20% contained multiple such inconsistencies.
The fact that half of all papers had incorrect data in them is concerning, especially because it seems to match Richard Horton’s intuitive guess at how much science is simply untrue. And the GRIM paper revealed a deeper problem: more than half of the scientists refused to provide the raw data for further checking, even though they had agreed to share it as a condition for being published.
After some bloggers exposed an industrial research-faking operation that had generated at least 600 papers about experiments that never happened, a Chinese doctor reached out to beg for mercy: “Hello teacher, yesterday you disclosed that there were some doctors having fraudulent pictures in their papers. This has raised attention. As one of these doctors, I kindly ask you to please leave us alone as soon as possible … Without papers, you don’t get promotion; without a promotion, you can hardly feed your family … You expose us but there are thousands of other people doing the same.“
Yes; I watched the movie "Gaslight" recently, and the meaning that everyone uses has drifted pretty far from the movie's meaning. But it's probably hopeless to resist a meaning when it becomes trendy.
Although the title says "just discovered", obelisks were discovered a year ago. Obelisks are RNA elements inside bacteria inside humans. Here are some links for more information:
I discuss this a bit in footnote 1. Thousands of things were built on frames before computers, of course. However, IBM both built computers from semi-portable cabinets built around metal frames and also referred to the computer's units as "frames", including the "main frame".
Telephone systems used main distribution frames, but these are unrelated. First, they look nothing like a computer mainframe, being a large rack with cable connections. Second, there isn't a path from the telephony systems to the computer mainframes; if the first mainframes were developed at Bell Labs, for instance, it would be plausible.
There's also the Intel iAPX 432 "micro-mainframe" processor (1981) on two chips. (This was supposed to be Intel's flagship processor, but it was a disaster and the 8086 took over instead. The NYT called it "one of the great disaster stories of modern computing".) I think that microprocessor manufacturers had mainframe envy in the 1980s.
A rumour from my mainframe days was that Digital Equipment hired lacemakers from france to show people how they did it. This was wiring up the core memory planes for the Dec-10 (I have one, a folded 3 part card) which just barely squeezes into the mainframe class.
The guy who told me this was the Australian engineer sent over to help make the machine to bring back for UQ. He parked in the quiet side of the Maynard factory, not realising why the other drivers avoided it. Then his car got caught in a snowdrift.
A prior engineer told me about the UUO wire wrap feature on the instruction set backplane: you were allowed to write your own higher level ALU "macros" in the instruction space by wiring patches in this backplane. Dec 10 had a 5 element complex instruction model. Goodness knows what people did in there but it had a BCD arithmetic model for the six bit data (36 bit word so 6 bytes of six bits in BCD mode)
A guy from Latrobe uni told me for their Burroughs, you edited the kernel inside a permanently resident Emacs like editor which did recompile on exit and threw you back in on a bad compile. So it was "safe to run" when it decided your edits were legal.
We tore down our IBM 3030 before junking it to use the room for a secondhand Cray 1. We kept so many of the water cooled chip pads (6" square aluminium bonded grids of chips, for the water cooler pad. About 64 chips per pad) the recycler reduced his bid price because of all the gold we hoarded back.
The Cray needed two regenerator units to convert Australian 220v to 110v for some things, and 400hz frequency for other bits (this high voltage ac frequency was some trick they used doing power distribution across the main CPU backplane) and we blew one up spectacularly closing a breaker badly. I've never seen a field engineer leap back so fast. Turned out reusing the IBM raised floor for a Cray didn't save us money: we'd assumed the floor bed for liquid cooled computers was the same; not so - Cray used a different bend radius for flourinert. The flourinert recycling tank was clear plastic, we named the Cray "yabby" and hung a plastic lobster in it. This tank literally had a float valve like a toilet cistern.
When the Cray was scrapped one engineer kept the round tower "loving seat" module as a wardrobe for a while. The only CPU cabinet I've ever seen which came from the factory with custom cushions.
I heard a story of Seymour Cray doing a demo of one of the machines and it turned out there was a bug in some hardware procedure. While the customers were at lunch, Seymour opened up the machine, redid the wire wrap and had the bug fixed when they returned. (Note that many details are likely inaccurate as this is a 35-year-old memory of a second-hand story.)
I have searched once for the origin of the term "Central Processing Unit".
In his report, John von Neumann had used the terms "central arithmetical part (CA)" and "central control part (CC)". He had not used any term for the combination of these 2 parts.
The first reference to CPU that I could find is the IBM 704 manual of operation from 1954, which says: “The central processing unit accomplishes all arithmetic and control functions.”, i.e. it clearly defines CPU as the combination of the 2 parts described by von Neumann.
In IBM 704, the CPU was contained in a single cabinet, while in many earlier computers multiple cabinets were used just for what is now named CPU. In IBM 704, not only the peripherals were in separate cabinets, but also the main memory (with magnetic cores) was in separate cabinets. So the CPU cabinet contained nothing else.
The term "processor" has appeared later at some IBM competitors, who used terms like "central processor" or "data processor" instead of the "central processing unit" used by IBM.
Burroughs might have used "processor" for the first time, in 1957, but I have not seen the original document. Besides Burroughs, "processor" was preferred by Honeywell and Univac.
The first use of "multiprocessing" and "multiprocessor" that I have seen was in 1961, e.g. in this definition by Burroughs: "Multiprocessing is defined here as the sharing of a common memory and all peripheral equipment by two or more processor units."
While "multi-tasking" was coined only in 1966-09 (after IBM PL/I had chosen in 1964-12 the name "task" for what others called "process"), previously the same concept was named "multiprogramming", which was already used in 1959, when describing IBM Stretch. ("multitasking" was an improved term, because you can have multiple tasks executing the same program, while "multiprogramming" incorrectly suggested that the existence of multiple programs is necessary)
You've done a lot of interesting historical research there! I wanted to get into a discussion of "central processing unit", but decided my article was long enough already :-) The term "central processing unit" is unusual since it is a seven-syllable term for a fundamental idea. Even "CPU" is a mouthful. I think that the "central" part is in opposition to systems such as ENIAC or the Harvard Mark I, where processing is spread out through the system through accumulators that each perform addition. Centralizing the processing was an important innovation by von Neumann.
We will never get rid of the "von Neumann bottleneck", except for a relatively small number of niche applications.
The bottleneck consists in the fact that instead of having a huge number of specialized automata that perform everything that must be done to execute a useful application you have just an universal automaton together with a big memory, where the universal automaton can perform anything when given an appropriate program.
The use of a shared automaton prevents many actions to be done concurrently, but it also provides a huge economy of logical circuits.
The "von Neumann bottleneck" is alleviated by implementing in a computer as many processor cores as possible at a given technological level, each with its own non-shared cache memory.
However removing completely the concept of programmable processor with separate memory would multiply the amount of logic circuits too much for any imaginable technology.
The idea of mixing computational circuits with the memory cells is feasible only for restricted well defined applications, e.g. perhaps for something like ML inference, but not for general-purpose applications.
My mom was a programmer back in the 1950s. First thing in the morning, she would run a memory test. If it failed, she would slide out the failing tray of memory (100 words by 36 bits, and tubes), hand it off to a tech to fix, slide in a fresh tray, and proceed.
She had one CPU she worked on where you could change its instruction set by moving some patch cords.
A tax agency unified all its data from different agencies via X.25 and satellite connections. However, the process was expensive and slower than expected because the files were uncompressed and stored as basic plain-text ASCII/EBCDIC files.
One obvious solution to this problem was to buy an Ethernet network device for the mainframe (which used Token Ring), but that was yet another very expensive IBM product. With that device, we could have simply compressed and uncompressed the files on any standard PC before transferring them to/from the mainframe.
Another obvious solution was to use C to compile a basic compression and decompression tool. However, C wasn’t available—buying it would have been expensive as well!
So, we developed the compression utility twice (for performance comparisons), using COBOL and REXX. These turned out to be two amusing projects, as we had to handle bits in COBOL, a language never intended for this purpose.
The capture of all thought by IBM at these palaces was nuts.
Circa 2002 I’m a Unix admin at a government agency. Unix is a nascent platform previously only used for terminal services. Mostly AIX and HPUX, with some Digital stuff as well. I created a ruckus when I installed OpenSSH on a server (Telnet was standard). The IBM CE/spy ratted me out to the division director, who summoned me for an ass chewing.
He turned out to be a good guy and listened to and ultimately agreed with my concerns. (He was surprised, as mainframe Telnet has encryption) Except one. “Son, we don’t use freeware around here. We’ll buy an SSH solution for your team. Sit tight.”
I figured they’d buy the SSH Communications software. Turned out we got IBMSSH, for the low price of $950/cpu for a shared source license.
I go about getting the bits and install the software… and the CLI is very familiar. I grab the source tarball and it turns out this product I never heard of was developed by substituting the word “Open” with “IBM”. To the point that the man page had a sentence that read “IBM a connection”.
> C wasn’t available—buying it would have been expensive as well!
On the subject of expensive mainframe software, I got to do the spit take once of "you are paying how much for a lousy ftp client? per month!" I think it was around $500 per month.
The youngs have no idea. You used to have to pay for development tools. In high school, I would hand-assemble my 6502 assembler programs by writing them out longhand on graph paper, fill in the hex in the left columns than type in the hex in the Apple monitor to get the program running. Heaven forbid there was anything wrong with the code, the hand-assembling or the keyboarding, but I couldn’t afford one of the paid assemblers (in retrospect, I should have written one in AppleSoft, but hindsight is 20/20, and I don’t know that I was smart enough then to write an assembler anyway). Spending $500–1000 for a compiler in the 90s was typical. Now the kids whine about paying the relatively cheap fee for things like JetBrain IDEs.
Back about roughly 1978 to 1982, I wrote and sold a 6800/6809 disassembler ("Dynamite Disassembler") for I think $200 a copy. Sold enough copies to pay for computer equipment during my college days. I think we reduced the price to $100 or maybe $150 when the TRS-80 Color Computer came out with a 6809 powering it.
$200 back in 1980 is about $800 today. Amazing to think anyone would spend that much for a fairly simple tool.
We were running Oregon Pascal on a PDP/11-44 (later upgraded to a VAX 11/780) that cost thousands. To have access to Pascal for $49 was too good to be true. Kept thinking it had to be deficient somehow, but it wasn't.
The paradigm shift was underway right in front of us.
My CS 101 class in 1989 was all in Pascal and had to be entered via a IBM terminal and ran as a batch job on our school mainframe. There was no interactive feedback and you had to hike across campus to a basement of a building that had an enormous chain printer to get your greenbar paper output of your run to see if it 1) compiled and 2) output the right thing that the autoscorer checked when you flagged your assignment as complete.
I was lucky in that I had a Tandy 1000SX in my dorm room and I had Turbo Pascal (bought using an educational discount at the school bookstore). A hidden feature of Turbo Pascal was that it supported multiple comment delimiters, including the comment delimiters used by IBM Pascal (the assignments were also graded on comment quality). I was able to do all my class work locally, using interactive debugging, and thanks to a guy I met while working at a local computer shop that was the student IBM rep I got a file uploader and the phone number of hidden 2400 baud that it used so I could directly upload my code and then dial into the interactive terminal number and submit it.
I sort of felt bad for all the other kids in the class for the write/submit/walk/debug loop they endured, but not really.
> Another obvious solution was to use C to compile a basic compression and decompression tool. However, C wasn’t available—buying it would have been expensive as well!
I would have thought in the (IBM) mainframe world, PL/I (or PL/S) would have been the obvious choice.
What areas still use bipolar? Does a switching power supply use substantial bipolar? Does anybody still implement TTL or ECL?
Quoting you below...
"The most unusual circuit is the BiCMOS driver. By adding a few extra processing steps to the regular CMOS manufacturing process, bipolar (NPN and PNP) transistors can be created. The Pentium extensively used BiCMOS circuits since they reduced signal delays by up to 35%. Intel also used BiCMOS for the Pentium Pro, Pentium II, Pentium III, and Xeon processors. However, as chip voltages dropped, the benefit from bipolar transistors dropped too and BiCMOS was eventually abandoned."
I didn't realize that BiCMOS lasted so long. I thought it was only used on the original Pentium, but I really didn't look hard.
reply