Hacker News new | past | comments | ask | show | jobs | submit login
Baltimore plant ruins 15M Johnson and Johnson coronavirus vaccines (axios.com)
116 points by UncleOxidant on March 31, 2021 | hide | past | favorite | 94 comments



In other news, a developer had a test fail in CI/CD and the build wasn’t, therefore, pushed to production.


Alternate title:

Quality Control and Safety procedures (put in place to meet medical manufacturing regulations) prevent a bad batch of vaccines from entering the market and potentially harming patients.


Or the people who went to add ingredients to a AZ/Oxford batch found they were missing.


Agree with you Quality assurance guys saved and stopped the batch before it was out for general public for use, could have adverse effect on patients, if it was rolled out.


How do you make 15 million of anything without realizing there's an issue, let alone the most scrutinized thing on the planet right now?


I'm guessing because you're not really making "15 million of something". The process is more like making one or two (or ten) batches, which is then sub-divided into 15 million doses.

It'd be like the difference between burning dinner 7 nights in a row, versus making a single mistake during Meal Prep Sunday and ruining a weeks worth of dinners all at once. The latter is much more understandable.


It’s rare that an analogy is so easy to understand while remaining highly accurate. Nice.


Dose is 0.5 ml. Say the batch is equivalent in volume to the final doses (who knows, but it should be somewhere vaguely close). That's 7,500 liters.

Microbrewers often use fermenters larger than that.


The question should be more why would there have to be such large batches which can be SPOF for so many doses. Wouldn't it make more sense to make smaller seperate batches to derisk the process.

This is akin to me using a large 64 core server for an app, instead of say 4x16 . Sure, the larger server can easily handle the load, however the 4x16 setup is more redundant and avoids many scenarios where all users will be out of service

Just because there are high capacity machines capable of doing that much does not mean it should be used especially a With a new process trying to scale up


To use the same alarmism and a software analogy: sw developer destroys thousands of hours of CPU time with bad code!1 Yet every single time I push, the CI pipeline runs an extensive suite of automated tests, and I'd not have it any other way. Wait but CPUs aren't being destroyed by running tests, you say. Yeah but niche reporting is always atrocious and a mainstream writer can't be trusted to get things entirely right.

(If you're using a 4x16 server, whatever that is, to provide isolation in your system, you may want to talk to additional devops folks for other options.)


The production goal is hundreds of millions of doses this year, so I guess I wouldn't want to speculate too much about what batch size is sensible.


eventual goal maybe, Not for the month of March. 15M out of planned 20M ? . The scale out for any process starts small to try avoid this kind of issues.

Yes fuck ups happen at plant and in planning, and yes we are not experts in the field to comment in detail on what is best approach. However if you are looking at it as an admin, clearly risk was not well managed.


You might have smaller batches and smaller throughput, but some pesky people keep shouting that we need all the vaccines yesterday :).


It'll be one vat with the potential for 15 million doses... the corrupted mix never reached the individual dose stage.


I find it sad how quick the narrative jumps to the more absurd 15 million number.

Obviously, the mistake is bad. But this is something with a very high fan out. And the smaller number just doesn't get the clicks. :(


The batch time seems to be weeks/months, so there is some relevance to the dose count (it's wasted time in addition to the wasted ingredients).

And it's put other production from the facility into question.


But this is a natural consequence of mistakes in processes with very high fan out.

Obviously, it should be investigated such that were can reduce the chance is mistakes in the future. But piling on with blame will just convince folks not to try.


Given the $10 a dose revenue, I'd call it a failure in the process rather than a mistake. That it was possible is the bigger problem than it happening.

And then there's the much greater value of the doses once they have been administered.


And there will be an FDA investigation which will put further delays into new production.


And it's J+J's name that goes in the headline. It wasn't even their fault. At least they didn't somehow cram the AZ connection in there too, though I'm sure other outlets are working on it.


It's probably analogous to making a mistake in a single line that gets pushed to production - that then gets patched downstream into all 15 million customer's systems. Large systems moving in giant batches.


> Context: Workers at the plant, which had been producing Johnson & Johnson and AstraZeneca vaccines, conflated the ingredients between the two different types of vaccines, destroying 15 million doses of Johnson & Johnson vaccine, according to the Times.


Jesus Christ, this shouldn't happen in a certified cGMP (food) facility let alone a pharmaceutical plant.

HACCP would prevent this in any number of ways.


Errors like this happen in every single GMP and clinical facility.

What makes them different is 1) there are ample checks in place to prevent mistakes from ever leaving the facility, and 2) there is process that reduces the chances of having it happen in the first place.

But all of these places will have incident logs of mistakes like this, along with corrective actions that change the procedures and make it less likely to happen a second time.


> Errors like this happen in every single GMP and clinical facility.

They shouldn't happen in a 150yr old, multi-billion dollar pharmaceutical company.

How hard is it to restrict the ingredients available to the "chef" to the ones needed for the current product? Not hard at all unless you want to save a few dollars cutting corners.


Sounds like what epistasis is describing is a healthy QA environment with rapid feedback.

Some other comments have already pointed out that 15M doses is probably only one or a few batches. Combined with the fact that the procedure documents for this particular line are probably still warm from the laser printer, I'd say let them iterate a bit more to get the kinks out.

[ Procedures are not (should not be) engraved on stone tablets handed down from on high. If you try it that way, you are just as guaranteed to fail as when you had no procedures at all. ]

As to what went wrong at this facility (besides overpromising?); we'll have to see. I wish there was such a thing as open source procedures!


Being GMP isn't about eliminating all errors everywhere, it's about acknowledging the inevitability of error, minimizing error through conscious processes that stop those errors, and testing to assure that you made what you made.

In a situation where maximizing vaccine production sooner saves lives, pinning this on saving a few dollars doesn't make sense. Whatever they were doing cost them quite a few dollars in addition to the lives lost from slower vaccination.

I don't know their process, but suppose the "restricting the ingredients" phase happens. What if the ingredients were swapped at the time of restriction? Same problem happens.


> What if the ingredients were swapped at the time of restriction? Same problem happens.

Chef goes to find eggs as required by the recipe for his cake, and finds that there are none. Instead there is pepper, an ingredient not required by the recipe.


I think you mean it shouldn't happen undetected... these kinds of mistakes can and do happen. Everything from human error (oops, Billy put baking powder in the baking soda tank) to equipment error (the tank valve stuck open, adding 10X more of ingredient X than expected)


Can you describe in more detail how that process would prevent such an error?


Hard to do without knowing anything about this facility, but maybe just not store the ingredients for two different vaccines in the same place. Basically restrict the set of ingredients available to the person making the batch.

Or have the employee log what he is doing in real time BEFORE he does it. I am adding sugar, I am adding eggs, I am adding wheat flour. Then an alert is triggered when the system (which knows what product is being made) recognises that the ingredient doesn't belong.

This is a multi-billion dollar company... they could have the process live-streamed to 30 independent chemists each one watching the "chef's" every move (this is not a serious proposal).

This stuff is all hashed out when one writes up a HACCP process.


It's not the most scrutinized thing on the planet with the emergence of pro-vaxxers (the anti anti-vaxxers) who refuses to acknowledge anything bad about vaccines.


This is a bigger thing than anyone here realizes, for Emergent BioSolutions is a notorious company who's core competency is politics, not production, see one or more recent New York Times articles on them. This is a minor fowl-up compared to the sorts of things many of us have been expecting from this factory of their's.


I'm glad this was identified before people got injected with these and thought they were safe.


This should make some people sleep badly.

Assuming somewhat efficient allocation, a 30 day delay delivering 15 * 10^6 one-shot vaccines equals ~1100 deaths[0], plus many long term health issues for survivors.

Fingers crossed this f*ckup won't delay the lifting of the US vaccine export restrictions. Many people objectively need the vaccine much more urgently than the healthy young adults currently starting to be vaccinated in the US.

[0] rough extrapolation based upon current covid-19 mortality rates in Belgium, available at https://epistat.wiv-isp.be/covid/covid-19.html


Sure it's not great and the incident needs to be investigated but mistakes happen. We shouldn't expect perfection as a baseline. If nobody is willing to take risks, we wouldn't have any vaccines right now.


It bothers me that people have expectations of perfection. Of course there will be breaks in the chain the lead to doses being wasted. There are thousands or millions of ways for it to go wrong, and only a few ways for it to go right.

It is an astounding miracle of human progress that we can get to this point.


I was ready for the knee-jerk response to pounce and heap damnation on the plant for messing it up. But you have a very valid and insightful point.


I was ready for the knee-jerk hackernews response where people complain about deserved criticism.

I don't know how you people expect things to get better if this wouldn't happen.


There's a big difference between complaining and curiosity. Really, there's a whole host of ways to think and vocalize improving a system without complaining.


How easily avoidable was the error? If it was caused by gross negligence then it’d be fair to criticize those at fault.

We don’t know, but you seem to be assuming it wasn’t as the result of negligence.


Accepting that mistakes will sometimes happen does not absolve the need to prevent said mistakes from happening in the future. It certainly doesn't mean we can't or shouldn't be critical of the events leading up to those mistakes.


Exactly, particularly during an emergency where many procedures are being redefined in the interest of time and new processes learned.


This is a failure of leadership to ensure that a quality product is being delivered by the system they're responsible for.

I'd usually be pretty forgiving especially when there is a high degree of novelty.

Maybe this particular vaccine is novel and that benefit of the doubt should be extended in this case, I'm not sure though.

It's also possible this is common variation but we never hear about it.


Scenario:

An engineer who you pay $200,000.00/yr in salary pushes code to prod which brings your totally unique SaaS product down. Accounting says the incident cost $2,000,000.00.

Do you:

A) Fire the engineer on the spot.

B) Conduct a thorough investigation of your testing and deployment practices and implement changes to avoid such a mistake in the future.


Missing a lot of information to actually answer that question.

Was the engineer wilfully negligent of checks and balances put in place. Was he wilfully negligent because of undue pressure to deliver more in the same time and without any system changes. Did his mother recently die and he couldn't take time off to grieve properly.

There are many aspects that make up the analysis of a failure in quality, and that is the job of leadership.


Ok let me expand the choices then.

C) Fire the engineer's manager.

D) Resign.


I've a feeling that everyone has misinterpret the sentiment of my remark.

What i was trying to imply is that there's something to be reviewed here and it's leaderships responsibility to do so.

I'm feeling that potentially with covid and all it's a little too emotive and the whole comment was misplaced and doomed to degenerate into a slinging match.

Also there is a lack of detailed knowledge that precludes me from making more detailed statements.


Obviously, option A. Improves the morale instantly.


Proposition Joe had the connect on lockdown with the eastside and westside working together pumping out 15 million doses. Until young Marlo came in and put a stop to that. The game is the game, vaccines or not.


Well, I stopped looking at news for one week and the frothy right has entirely new sentences I don't understand.

Edit: folks stop upvoting this I was missing a cultural reference I didn’t remember.


It’s a reference to the HBO series The Wire.


Well now I just feel old.


The Wire came out almost 20 years ago.


Yes, and I watched it about 15 years ago and didn’t remember enough to recognize the reference.


[flagged]


Yes, a common and accepted genderless pronoun.


It has been commonly used and accepted in the case of unknown or unspecified gender, though it is rapidly falling out of favour for that case. In part because I think it's quite hard to say that it's truly 'genderless', it's embedding a 'default' into the language which perhaps is undesirable.


When has 'he' ever been used in a genderless context?


Apparently it started in the 18th century, so there's a lot of inertia to overcome to get people to change.

But she says it was from the 18th century onwards that people started using male pronouns when describing someone of a non-specific gender in writing and this marks the time when opinions on what pronouns should be used started to change.

"You might have a sentence like 'if a student comes to see the teacher, he must bring his homework', where he is supposed to refer generically to males and females," explains Dr Emma Moore.

https://www.bbc.com/news/newsbeat-49754930


I have been learning Italian. Don’t slam me for this if it’s wrong I am a beginner. But there are a number of contexts in Italian to which lei (she) is used. One for polite and respectful use (such as a general in the army, regardless of gender) or if the person’s gender is unknown.

Just thought I’d throw that out there.


Very formal/polite way to refer to people yes. But in Italian we also have “voi” and its use can be traced back centuries before “lei” started to be used.


Welcome to the English language.

It's technically correct to use He when the gender is unknown.

Provided by a sibling dead comment, see 2nd definition:

https://www.merriam-webster.com/dictionary/he


https://en.m.wikipedia.org/wiki/To_Be_a_Pilgrim is the first example that springs to mind.


[flagged]


Dominique (a non-gendered name) goes to school everyday. They take the bus.

Is this a proper English sentence?

(question from a non native speaker)


As a native english speaker I consider that valid.

The possible confusion is the fact that “they” could be singular or plural. AKA “singular they” [1].

Your second sentence could be referring to some other group of people who are not Dominique who instead of going to school take the bus.

[1]: https://en.m.wikipedia.org/wiki/Singular_they


"They" is a plural pronoun. Obviously.


I wonder if those vaccines were primarily going to be distributed locally. I live nearby, and my county's weekly covid update said they expect to have most people 16+ vaccinated by the end of April because we're supposed start getting an increased number of vaccine doses every week.


"The error should not affect Johnson & Johnson doses currently being delivered and administered nationwide, as they were produced in the Netherlands. ... The Times says the accident won't stop the U.S. from reaching President Biden's goal of having enough vaccine by the end of May for every adult, but in the global race against the virus, every dose counts."

Won't affect vaccine delivery for Americans. Difficult to say for the rest of the world.


It's good that it won't hurt the current schedule, but we could have had 15 million doses sooner.


Probably won't affect rest of the world given the US export ban is going to be around until second half of the year.


You could vaccinate two dozen countries to 100% with 15M doses.


If by locally you mean anywhere near the factory, no, for it just makes raw vaccine, which gets shipped to specialist firms that do "fill and finish."

A paper, pretty sure it was The Washington Post ran a hit piece on the company and how Baltimore was behind on getting vaccines or something, and they replied pointing out they're just doing this under contract, they have no say in the disposition of doses. Which of course is several levels above them.


Well, thanks EU for all the good doses!


You can have all our 30m Italian doses of AZ, as long as you don't insist on liability. We don't want them


This probably explains why only 5M doses of J&J vaccine were allocated for the week of April 5 as opposed to the 11+M that were anticipated.

Emergent was also supposed to be approved to ramp up the full supply chain but that hasn’t happened yet, could be related.


In the article it says that these doses were produced in a facility that wasn't approved by the FDA yet, so this has no impact on the availability of doses now. It might have an impact on the availability of doses in the future though.


Honest question: if the facility wasn't approved by the FDA then why was it manufacturing the vaccines at all?

And how could such vaccines be used anywhere if they were produced by an unapproved facility?


Because that's how good we are as a society at efficiently allocating capital and operating supply chains. Regulators have to sign off before you can administer it to anyone but you can ramp up your production in parallel with the approval process.

It's actually an incredible achievement.


Everyone has been gearing up production before approvals with the assumption they'll get approved for expediency sake.

You get approved? Great - We can immediately ship these 30 million doses while we start the next 30 million.

We don't get approved? Write off the produced doses, fix your process, and try again.


FDA is likely reviewing their processes and controls more than running a white glove inside of anything.

So those things would be the same for a batch that was in production as they would be for a batch started tomorrow.


Maybe the approval process includes a "must have met X criteria for the past Y months", so once you get approved, everything you've done in the past Y months counts as having been created in an approved facility. I'm just guessing though, I have no idea what the process is.


Basically this. You need four consecutive batches with consistent quality in a row for approval.


Maybe making them for another country?


It's funny to read it as J&J vaccine, while here in the Netherlands where the institute we all call it Janssen because that is the name of the institute. Im also in favor if the title reflected Janssen as well but I guess it's what it is as the article and the rest of America also calls it J&J vaccine. It's just a bit weird that it now gets the American name while they're just the parent company and didn't were involved so much in the R&D.


It's based on the name most well-known/respected in the target population. Janssen is well known and has a better reputation in the Netherlands. J&J is better known and has a better reputation in America. It's quite literally marketing.

See also Pfizer vs. BioNTech


They're not just the parent company, J&J owns the vaccine. It's their property.

The world knows the Johnson & Johnson brand. It's one of the top brands on the planet and has been for decades. Most of the world doesn't know Janssen. It's logical to call it the J&J vaccine accordingly, for marketing purposes it makes a lot more sense.

If the Berkeley West Lab subsidiary of J&J produced the vaccine, nobody would call it the Berkeley West Lab vaccine.


Vaccine providers(at least some) are referring to it as J&J/Janssen, or even just Janssen sometimes, here in the USA.


Same deal with the Pfizer vaccine, really. My family in Germany lives near where BioNTech is located, and everyone is very proud to call it the BioNTech vaccine.


I'm not in Germany, but I also get very frustrated when the Pfizer name is applied to what BioNTech created. BioNTech is going to be doing amazing things later on too, with their great tech platform.

I have a feeling that news sources are very guarded about introducing new names to the public, especially ones as difficult to pronounce as BioNTech (I've heard some great butchering of the pronunciation too).


May be its just easier to pronounce Pfizer than BioNTech?

Not everything needs to be distilled down to nationalistic undertones.

I am getting sick of this kind of discussion permeating on HN - It is extremely juvenile and damaging to the spirit of HN. In this regard, Reddit seems to be a lot better. Go on any vaccine related thread and there is absolutely none of this. There is so much more global cooperation observed on Reddit than on HN.


This has nothing to do with nationalism. I'm not German. I don't care about whether Germany gets credit for this. I care about BioNTech getting credit.

My complaint, beyond just the media, is that large pharma, like Pfizer, being fundamentally uninnovative companies that grow by acquisition rather than invention. And about small companies being denied the potential to grow because of an oligopoly of control.

Name brand recognition drives dollars, and in biotech the ability to raise money is just as much a hurdle as the chances of your trial failing.


I see your point, and that's a valid concern. I wonder how much damage public PR does to investors - who are supposed to be highly informed and not be pursued by public hype.


Entirely different in fact. Pfizer doesn't own BioNTech, they're business partners. J&J owns Janssen (for 60 years now), the vaccine belongs specifically to J&J.


Oh how I would like this to be an april fool's joke...

15M doses? We definitely could have used them (if they weren't messed up).




Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact

Search: