Hacker News new | past | comments | ask | show | jobs | submit login
I reverse engineered McDonalds’ internal API (twitter.com/rashiq)
845 points by swyx 40 days ago | hide | past | favorite | 420 comments

Just a note, the machines are not broken, they have shut themselves down for sanitary reasons. Ice cream, as a food that is very close to neutral in pH and without much in the way of preservatives, is a common source of listeria bacteria. To prevent serious health issues, the machine must be sanitized regularly. As I understand it, the process is involved, and any given McDonalds may not have a person with the time and skills to complete it. So the machine shuts itself down rather than produce an unsafe product. This is translated to "broken" when you order ice cream.

Here's some detail, as someone that has worked with those machines before (internally and externally).

In the good old days the soft serve and shake machines were taken apart every night and every piece was washed and bleach-sanitized by hand. Then it was reassembled in the morning. This would leave a lot of room for error and possible contamination in the process. And the sight of a shake machine hitting full pressure with a misaligned O-ring in the barrel is a lovely one indeed.

So the newer machines take a different route: they self-pasturize. The machine goes out of service and slowly heats everything up to a bacteria-killing temperature (including the dairy mix inside), then cools everything back down to freezing temperature to serve.

This process takes about four to five hours. But if you set the clock wrong, the machine will go into its cycle at some super-inconvenient time. There are other triggers that can force a self-clean but in general it's not "broken", it's just doing this process at a bad time for you.

I'm curious how this data is being sourced. The last I checked there was an effort to put cloud-based telemetry into every unit and have them report home constantly (both to the equipment maker and to McD), but that was never intended to be public data.

EDIT: Ah, I didn't RTFT. He's using the public ordering site to scrape it (and now the title has changed). But that does mean there's some working connection from the equipment to the McCloud now. That's cool. Or maybe a manager is doing it by hand.

I actually froze my hand inside an ice cream machine like this when I was a teenager working fast food. I hit a standby button instead of the off switch. (I was too tall to see the off switch and was never trained.) I spent two weeks in a burn unit while they healed my hand with magic synthetic sin grafts. Today you can’t even tell which hand got frozen in there.

Woah, how cold are these things that you would need skin grafts like that?

Back in the day I worked in a dairy company as an IT guy. When the RFID antennas would need a reboot I had to go to the deep freeze storage (-40 Celsius), walk in with a ladder, climb the ladder, switch off and then on (IT crowd style), wait to see the green light bleeping, and then climb down and leave as fast as I could. The whole thing would take 40-50 seconds (I was counting EVERY time). I couldn't wear gloves, because ladder/switch. I was wearing all other gear though. Touching anything (aluminum ladders get cold very fast at -40)(so are walls, switches, everything). I did this process a couple of times wearing just jeans and tshirt. In that "less than a minute" my clothes would freeze solid, the wool was hardning and it felt like wearing a plank. Same with jeans.

In other cold related story, I was in a "cold country", out in the open, -25 Celsius, my phone rang, my gloves didn't have the electro-thingie so I couldn't "slide to answer" so I too my glove off, answered, talked for 60 seconds. Then I went to a warm pub, and after 10mins I could bend my fingers again.

Yes cold can .... you up badly in seconds.

I used to walk to school in sometimes very cold weather (-35 Celsius). Once, in order to try preventing the inside of my nose to freeze, I put my scarf over it. The scarf froze very quickly and I ended up with ice around my neck... Now that I’m thinking about it, I wonder how people will manage masks this year with COVID in cold countries.

TL;DR: don’t put a scarf over your nose in very cold temperatures.

I'm Canadian and we get -35 temperatures for at least a few days every winter. This doesn't make sense to me. Of course you should put a scarf over your nose. At those temperatures , exposed skin freezes very quickly so you need to cover all your skin or you'll get frostbite.

I've never had a problem with a scarf freezing stiff like you describe. They build up ice crystals of course, but they don't interfere with air flow and are easily broken.

The biggest problem I expect from wearing a mask in the winter is that they'll get damp quickly because the cold temperatures, and perhaps having a scarf over top, will prevent moisture from evaporating.

I usually ride my motorcycle through winters in the upper midwest. No, it's not Canada, but at highway speeds in freezing weather it gets cold very quickly. I can't cover my whole face with cloth, or my breath will fog up and freeze over my glasses and visor. I apply vaseline over all of my exposed skin before riding. It does a great job of protecting my skin. I also do this when bicycling through extremely cold weather.

Heh.. many people commented on that. Perhaps you had a "single layer" that made it hard since the droplets/water from your exhale got stuck in the fabric, and together with the atmosphere's humidity created this effect.

In such cold climates I wear an ultra long scarf (2m-2.5m), I wrap my neck/jaw/ears 3-4-5 times (depending how tight) and I tuck it in my jacket. The outside of the scarf freezes, the inside remains warm. I also try to breathe by my nose, less humid.

Would a coronavirus survive in these conditions anyhow? Or, remain transmittable?

It's a virus not a bacteria so the proteins don't disintegrate. The probability of transmission outside is small as sensibly people limit the time outside, but the flipside is they stay indoors, in confined spaces together, which is perfect breeding ground for the virus.

Cold acts as a preservative for viruses. In labs which do experiments with viruses, they keep viruses in freezers for a looong time.

Last I heard the coronavirus survives a lot longer in freezing conditions than at room temperature.

Coronaviruses cause the common cold, which is more common when it's cold.

>TL;DR: don’t put a scarf over your nose in very cold temperatures.

Weird, as a Canadian who has been through many -35 days, I absolutely use a face covering, whether that's a scarf, neck gaiter, mask...Never had any problem like that.

A manager is doing it by hand. Their POS system has the ability to lock out individual items, and this information is available in real time to the server backing the mobile app (GMA).

The machines that I've seen don't have real-time telemetry, and are not networked.

Good info, thank you!

There was an old effort a long time ago where the machines were on a powerline-networking system (LONWorks) but that is dead and buried. The next-gen effort was ramping up a few years ago and I don't really know where it is now. The predictive analytics company they bought was a big part of it.

Meanwhile, Chick-fil-a is running Kubernetes at the edge for equipment, POS telemetry, and ML.


Their tech stack is far more bleeding edge than I would have envisioned. I wish they gave an idea of why they chose to use k8s at the edge. I can see how containers are useful for deployment, and the fail-over built into Kubernetes is really nice, but I'm shocked that's the simplest/best way to do it at the edge.

I would have expected some VMWare product, just because it's "enterprisey". Shame on me for judging.

Deployments and monitoring are probably another draw. Same tools server and edge side.

Another project to keep an eye on is AWS's Greengrass which has a bunch of capability overlap, but a bunch of new ideas for edge and fog IoT also. I had a mixed experience with it but at least some ideas bear borrowing.

Not sure if this is the same thing, but I recall reading a case study in 2006 on McD's cloudifying their kitchen equipment. The project was apparently a huge failure and was eventually stopped due to runaway costs.

Do you have a link to that anywhere?

The 2006 timeframe does fit with the LONWorks effort. The McD implementation used....powerline networking. Man was that a kludge.

I don't sorry, but it was probably a HBR case study.

Said another way:

  - You have a system whereby every instance has a 75%-79% uptime.

  - Each instance's 21%-25% daily scheduled maintenance window can (generally) be arbitrarily-scheduled by the ops team.
How do you near guarantee that there's always one instance available?

If you said:

  "manage a redundant cluster of at least 2 instances with non-overlapping maintenance windows" 
Then you were correct!


Q: What's the max number of ice cream machines found in a McDonald's location?

Q: What's the cluster size of the McDonald's ice cream monitoring system?

Q: How much money does McDonald's have?

Q: Why the fuck?

If you said: "1", "at least 3", "metric fuck-tons", and "I don't know", respectively, then you were correct!

The answer to #4 is probably that the goal isn't to always have ice cream. The goal is to run a profitable business.

Doubling the cost of the ice cream infrastructure to avoid a 25% downtime may not be a profitable venture -- especially if the downtime is scheduled to occur while the store is closed.

Shelf-space is in high-demand and those machines aren't cheap.

McDonalds is in the real estate business (letting buildings), fast food is just a side order.

Aren't the stores franchise's, with the individual stores owned by the relevant franchisee?

The operations are mostly franchised, but the corp owns many of the buildings and plots of lands that they are located on.


Also, you can use data to schedule the maintenance when the probability of someone ordering ice cream is low. That 21-25% downtime results in significantly fewer people wanting ice cream not being able to get it.

I worked there in the old days when I was in high school (late 90's), and one of the most disgusting stories I have about a job is when I went to refill the orange juice dispenser machine, and a giant swarm of flies were all over and in the top of the machine. In the orange juice, all along the sides of the inside, and flying around like a small swarm when I opened it.

I yelped and stepped back. A manager rushed over to me, slammed the lid shut, and told me to not tell anyone.

I quit a couple weeks later.

Fruit flies or house flies? The difference matters.

Is one worse than the other for some reason?

Fruit flies eat fruit and are harmless (if gross) to ingest. Black flies eat feces and rotting flesh and transmit all manner of diseases.

  In the good old days the soft serve and shake machines were taken apart every night and every piece was washed and bleach-sanitized by hand. 
I was a McD manager in my youth.

Not only were the internals fully disassembled and sanitized, they were left to soak in the solution overnight. All that was left behind was the cylindrical plastic chamber, sanitized and left to air dry overnight.

The messiest part was lubricating the O-rings with a petroleum jelly. Failure to remove all that when cleaning seemed like a good vector for microbes.

The worst job of all was cleaning the fryer vats.

Heh! I did both of those jobs as well. I still remember the large plastic tray you had to use to organize all the little parts and O-rings.

Did you ever have a clever employee that thought they could use the 5-gallon plastic buckets as a quicker way to dispose of the old fryer shortening? I watched that happen one Sunday morning. That was amazing.

Four to five hours, so let's say that happens daily and machines are out of commission 20% of the time.

Currently it's early afternoon and the site says that over 8% are out.

That's about halfway between "all overnight" and "all at completely random times".

Are almost half of the clocks set wrong? Assuming that the actual number of broken machines is relatively small.

The machines still get taken apart and washed by hand, along with some maintenance, every 14 days. So every machine will additionally be off for an afternoon every two weeks.

So it's basically a perpetual stew ice cream for 14 days? https://en.wikipedia.org/wiki/Perpetual_stew

Basically, yeah. They get refilled multiple times a day, and it's continually stirred, but only completely emptied during maintenance.

Picture of the mentioned 46yr old (beef) strew mentioned in the article: https://external-content.duckduckgo.com/iu/?u=https%3A%2F%2F...

I'm not sure what material it's on to keep it heated. But something like coal I assume.

I would imagine this is what any industrial ice cream production that's making pints for grocery store freezers would look like.

Are perpetual stews sanitary? I'm probably misjudging, but the idea of anything being out in the open for that long makes me uneasy. I guess it's kept at a simmer, so bacteria can't grow?

I'd be willing to wager that half the clocks might be wrong. It's something manager-on-duty can control, so Murphy's Law kicks in.

A standalone GPS NTP server is $750 retail. Commercial soft serve ice cream machines are at least $4000. Just building the machines with an autonomous GPS clock that doesn't need to be set, would double the revenue but only increase the upfront cost by 18%.

Since they need internet connectivity to phone home status to McCloud, I don't know why they wouldn't just run an NTP client to keep their time synchronized and accurate?

Moreover the $750 you reference must be a commercial product since you can do GPS on a Raspberry Pi for < $50 easily? Which product are you thinking works for this scenario? Think I've seen everything from $300 to $13000 solutions for this over the years.

I doubt these machines are internet-connected. Most likely is that the store manager notices the machine has gone into its cleaning cycle and manually marks the item on the menu as unavailable.

You could have a cesium time source in the thing but if the service tech or manager sets it wrong...you're still borked.

But exactly the same borked each and every day!

Or a G Shock with radio based time correction is $90. Not saying you could use the watch, but maybe the receiver is available.

And use some computer vision on the raspberry pi to read off the time? I love it! This is why they don’t let me work on hardware.

Can you provide some information as to how I could do a cheap GPS setup for a non GPS iPad in a similar manner?

I think you can do TTL serial on the headphone jack? Just hook a uBlox module up there (or use Bluetooth)

If it was being integrated, the cost would be more like 5 dollars, since the ntp unit could just be an esp8266 chip that connects to the store wifi to sync the time.

If they can't manage to set the time what makes you think they'll connect the Wifi?

(...although in fairness some of the other comments here suggest the things might already have some kind of int[er/ra]net connectivity in which case no extra hardware required)

Have them log onto the free public McDonalds WiFi on first boot, and auto-provision themselves to the correct private WiFi?

That sounds like an awful lot of work to not send NTP requests over WiFi. Then again, I'm sure someone will inevitably find a way to make these things spray ice cream everywhere over WiFi if they do. Or worse, mess with the internal temperature or cleaning settings to make them not safe to consume.

It's actually probably best they never touch the public wifi. I don't know why no one has mentioned ethernet yet, though. These machines are static, they just sit there. There's already tons of wiring going everywhere, so I would assume that it should be relatively easy to get a cable drop to the ice cream machine. Then you don't have to worry about any of this wifi security. You could do port level security, but that's probably overkill for an edge network.

I have actually been wondering lately why there isn't more of a market for some of these things that it seems like you could hack together. There has to be something I'm missing. You'll never be a unicorn, because most of these kinds of sensors have a limited scale, but it seems like a good return on investment for a small team.

Add a couple hundred/thousand to penetrate the roof for an antenna to pickup signal.

WWV might be a better system, but runs you into time zone issues. Do you need a valid SIM to pickup time from 3G?

I don‘t know about 3G, but I just checked for LTE [1] and the timing is broadcasted in SIB16, which is not encrypted, so theoretically anything could pick it up.

[1] https://www.etsi.org/deliver/etsi_ts/136300_136399/136331/11...

WWV/WWVH might not be a great idea, as I've read that there are rumblings about phasing it out.

You used to be able to pick up a time signal almost anywhere in CONUS with an FM receiver. It was encoded in the transmissions of PBS television stations.

This was back in NTSC days. Now that everything is digital, I don't know if it's still true.

The specification was called XDS and was developed by Sony to automatically set the clocks on their VCRs. PBS was a major participant but some other networks included XDS timecodes as well. Unfortunately late in the systems life reliability became poor (the encoder was not integrated with any of the other station equipment and was not being modernized) and I assume they all died out with the digital transition.

A better source today would be the time codes sent out by some FM radio stations with RDS encoders, to allow car radios to set their clocks. Unfortunately not that many radio stations do this and, once again, the time is not always all that reliable since the RDS encoder may not have any synchronization source itself.

The cellular network used to be an excellent source, CDMA cells required GPS time sync for TDMA reasons (well, CDMA reasons, technically speaking...) and broadcast a time code that is directly off of their GPS time source. Unfortunately, while GSM cells (and LTE) do broadcast the time, there is no guarantee made of precise synchronization as they don't broadcast a time code directly from their GPS source (not an expert in this field but I think the GSM/LTE time information comes from the possibly remote controller rather than the local radio hardware). Still, it would probably be good enough for this application.

GPS time sync is actually quite cheap to implement these days but tends not to work in these scenarios since a clear sky view is needed. WWVB is possibly on the way out. SNTP is probably ruled out less by the BOM cost of WiFi hardware and more by the deployment pain of having to get kitchen equipment configured for the corporate WiFi network.

Here's a fun idea: McDonalds presumably centrally controls the in-restaurant audio. Could they encode timestamps into the background music in a way that machines can cheaply recover? You wouldn't need high reliability, just enough for it to work once in a while. The old-ish Nielsen Peoplemeter system would be a model.

They need to load balance them so the world can have HA soft-serve. Run them in a Kubernetes cluster.

Chick-fil-a actually has been running kubernetes at all their stores for a few years.


> McCloud

That’s an awesome way to name their cloud computing stuff that fits the brand really well. Is this an official/semiofficial name?

Imagine McDonalds had invented AWS and called it McCloud.

When you order a VPS they would’ve asked you then: “Do you want to supersize your order? For $1 extra you can have a whole bucket of RAM instead of a small cup of RAM”

Quite a few providers have wizards that effectively boil down to this...

This may be the first time I've actually wished HN had /. style moderation, so I could give you a +1 (Funny) vote.

Jokes are usually frowned upon here, but that's a good one. Have an upvote from me.

With the tagline "There can be only one."

It's no crazier than an online bookstore doing it :)

Imagine KFC had invented it and called it Kentucky Fried Cloud

Then later rebranded it as "KFC"... the C is for Chaos.

Kentucky Fried Chaos Engineering

Imagine if Barnes and Noble had done it, and you could drive to one of their stores to quietly sip a drink and peruse the new machines they had provisioned recently.

All controllable from your Nook device as well.

It would definitely be networked with Happy Meal Ethernet.

There's an actual town called McCloud in California, next to Mt. Shasta.

All running on an old mainframe named Connor.

You should always substitute "clown" for "cloud" when talking about this stuff. Doubly so in this thread.

Hybrid clown setups are the best. All your data floats down here

Since you and the OP are here, I have a random question: were the blizzards ever made with liquid plastic ingredients?

It probably sounds insane, but someone from a plastics company spoke at my high school and said this was the case since it was easier to ship and needed no refrigeration. Now, decades later, I'm wondering if this was just complete nonsense or had an element of truth to it. I've been able to find nothing online.

Not my field, but edible gums (guar gum, or xanthan gum for example) is "plastic" (adj.). Gums are used more and more, AFAICT as a consumer; presumably to allow products to be bulked out with air.

Basically take anything homogeneous, add gum, add air, create a foam. Basically like an Aero bar but with way smaller bubbles.

You just cut a quarter (I'm guessing on this figure) off your ice-cream ingredients by weight and you can now advertise "scoops from the freezer"!

It's genius but just a more complex version of putting all the pizza toppings in the little window, or having the coleslaw tub be much wider at the top, or putting everything in a wide plastic skirt so the box is 20% bigger than it needs to be, ... packaged food sellers are scumbags.

This 'bulking up with air' is most evident if you go to the yogurt or cream cheese section and look for the Yoplait Whips or Philadelphia Whipped containers. It's the same product, just as a foam, and seems to be marketed as a healthier alternative where users choose a satisfactory portion by volume, not by calories or weight.

And while I think the whipped foods are stupid, I don't think that packaged food sellers are necessarily scumbags. They're merely people responding to incentives, just like everyone else. The food market has razor-thin margins and consumers are ill-informed. Neither consumers (who are hard to inform) nor regulators (who are captive to the multinational conglomerates) have been able to push back more strongly than the profit incentive, so the so dark patterns like those you describe are inevitable.

Plenty of dieters do this themselves with protein fluff/"ice cream" (protein shakes with ice and xanthum gum). Overweight people almost by definition eat by volume not by calories (ie they eat more than their body needs for caloric maintenance).

So these whipped products seem pretty useful.

Whipped cream cheese is much easier to spread, in my experience. I know I could whip it myself, but why not just buy it that way?

For me it's because I don't add xanthan gum, and I get a bulk discount on air.

Yup, also generally same cost by weight.

My kids eat a lot of creamcheese sandwiches. Block cream cheese just destroys sliced bread.

Essentially all cream cheese, whipped or otherwise, contains gums anyway. "Real" cream cheese is kinda grainy whenever I've seen it.

It's not entirely true that regulators haven't plushed back: in the USA, there's a minimum amount of cream required before the vendor can call the product "ice cream".

The products in that category that have been puffed up with air (and don't meet this minimum) get sold as "dairy desserts".

(I don't work on the food industry, so I won't be shocked if I got details wrong.)

Which is exactly why McDonalds menus list "cones" and "sundaes". Not "ice cream cones". It's a frozen dairy dessert that is not legally ice cream. To be fair, McDonald's version is superior to any other fast food frozen dairy desserts that I've had.

The bulking aspect doesn't bother me so much, what does bother me is that it's not as good and the only major ice cream brands left that aren't like this are Haagen-Dazs and Talenti.

I saw these yesterday:



I don't think I'd want to eat that

What the hell

There is no way i would eat something linke that.

> "having the coleslaw tub be much wider at the top, or putting everything in a wide plastic skirt so the box is 20% bigger than it needs to be"

These days the trend seems to be to reduce the packaging, at least here in the UK. "Same size - less packaging!". So it seems like reduced packaging and distribution costs, and perhaps fitting more onto store shelves, outweighs any extra sales from customers who are fooled into thinking the product is bigger than it really is?

I think there are two things, one is a preference with some consumers towards less packaging (given two products about the same cost and one looks like less packaging I'll tend to pick that) and the other is there's been a lot of talk about taxing single use plastics so I've wondered if they've been trying to alter the supply chains before that point.

"The future is plastics!"

I'm not a chemist but I've recently come to realize that the meaning of the term in the popular psyche has probably changed over the decades, taking on largely negative connotations, because of its association with a small subset of materials (albeit large by volume), most of which are either toxic, a pollution problem, or both.

AFAIU, in industry "plastics" is just a blanket term for moldable organic materials that chemists are constantly inventing. They're not necessarily sourced from petroleum, nor necessarily toxic. It's a ridiculously huge category of material, and plastics really were and probably always will figure prominently in our future. And I would assume that some traditional, even edible materials have been subsumed into the plastics category given the general scope of the term and the industry itself.

Former chemist, yeah this always bugs me. Laypeople use "plastic" to refer to mostly inexpensive injection molded parts, but it really includes everything from LDPE bags to glass fibre reinforced nylon powertools to Rayon fibers to thermoplastic elastomers.

Maybe you mean McFlurry (Blizzards were Dairy Queen). Basically the same thing, but not sure if whoever said that was specifically talking about Blizzards.

Ah yes, McFlurries. Got my wires crossed.

Yeah! Whatever you do, never eat pretzels! They dip them in oven cleaner before baking, then they sprinkle them with those crystals they use for removing ice from roads!

And dihydrogen-monixide figures heavily in their production. Nearly everyone that's ever died had consumed DHMO in the 24 hours leading up to their death!

I prefer "oxidane" for this joke, as most people haven't heard it, sounds poisonous, and dihydrogen monoxide is not wrong but it is irregular.

Can also try out "hydranol" or "dihydral ether".

DHMO is a very dangerous substance. For example, in gaseous state it causes skin irritation, severe burns even.

Lye dipped pretzels are a thing! I didn’t know that thanks!

A solution of about 4% NaOH is generally recommended. Submerge the pretzels for a few seconds just before baking. It does something to the outside layer of dough that results in the brown crust of the pretzel. In addition to the pleasant color, it also affects the flavor. Also, it makes it easier for the coarse salt to stick to the outside of the pretzels. NaOH immediately causes stains on clothes and wooden cutting boards and countertops. Obviously it's also bad for your skin and especially the eyes, so be careful.

Bagels are usually dipped in lye as well, it's how they get their bagel-y texture on the outside crust.

Sodium carbonate is sometimes used as an alternative to lye for those who don't want to deal with the safety aspect of handling lye, as lye is very caustic. Of course, lye still provides the best results.

> Lye dipped pretzels are a thing! I didn’t know that thanks!

Lye-pickled fish is also a thing (lutefisk)!

The best tasting soft pretzels are the ones dipped in a weak solution of lye, for sure.

Speaking about BK, the liquid they used to make the soft-serve was most certainly refrigerated.

Logically though, refrigeration based transport is not a concern for any fast food company. They use ingredients that absolutely must be transported cold, thus the cost of adding another menu item that must be shipped cold to that list is immaterial (even at scale).

> Mcflurry ingredients: https://www.mcdonalds.com/us/en-us/product/mcflurry-with-ore... (no mention of plastic)

That's actually incorrect, they list two plastics there. It's just that "plastic" isn't something that is actually necessarily toxic, and the media misuse the terminology all the time.

The two are:

> Cellulose Gum, Guar Gum

Cleaning these things was my second most hated job when I worked in the industry. But deep fryers are a far more dreadful task.

Worked at BK.

#1 hated task was cleaning the broiler.

#2 cleaning the shake machine

Someone should save the data over time and figure what is wrong with every machine. If this is made public and enough tech people tell the store every day to change example from 2pm to 2am, this will likely result in a better product for everyone.

At first I was gonna ask why does this not run on off hours, but then you mention bad clock, so the next question is, why is the % of bad clock so high, and why isn't the clock also cloud based or set in a way that can go wrong.

Next up, what about redundancy, do stores only have one machine?

McD is conservative when it comes to kitchen technology. With certain exceptions, franchisees determine when they buy new equipment and not the corporation. Most of it is designed for a 7-year capex cycle.

But to answer the question: no, the machines aren't net enabled and do not NNTP their clocks. But that's changing.

But I still wouldn't design my architecture to let a central server decide when a machine should go out of service. Think about what else you can fuck up that way.

Why are they off? Good question. Sometimes it's as simple as a store owner setting the heat cycle for 2pm instead of 2am.

Why are they not redundant? They're expensive, take up a lot of valuable space and a lot more valuable energy, they need to be filled with 2x the product, and the customer demand isn't that high at any given moment to warrant the expense in most stores.

> Next up, what about redundancy, do stores only have one machine?

From my (old) experience working at one, yes. The annoyance of having to maintain a second machine (especially when cleaning and maintenance involved a physical person cleaning things) was not worth keeping extra 9's of uptime.

Well again, looking at the graph, I'm seeing 10-20% down time, so it's not "just an extra 9"

I think you might find that in a certain way it is. 90% to 99% is an extra nine.


Or even zero 9s (80%) to one 9 (90%).

Cloud based clock.

Don't you mean NTP? We've had this technology for years! If it's connected to the internet, it's a pretty fundamental piece of pretty much any networked OS. You can ensure correct time this way.

NTP is fine but so is pulling time off a server header if it's phoning home to report status.

Not that it matters for ice cream, but NTP can compensate for latency better than HTTPx can.

It's also easy to implement since it will be integrated into the OS of the ice cream machine already.

GPS based clock is better no need for internet connection. Seiko and other company has been doing this for years.

NTP is typically backed by GPS.

If you have a datacenter full of computers, obviously you don't want to run a separate GPS antenna for every system. NTP allows systems to synchronize to GPS over a network.

Need a clear sky connection though.

NTP is what I meant, yes.

The clock probably resets when the power goes out or the fuse blows. They would need a battery as a backup to keep the time current.


Calling fast food workers monkeys won't make you look any smarter.


I've not heard that term used, and its bad. You shouldn't dehumanize people for being "cheap labor". Honestly, that guy flipping burgers is doing more real good for people than any number of highly paid software devs working on VC funded projects that will mostly end up in the trash.

I think it's more acceptable to make fun of people of your level of privilege and up (like the management who underpays) and less of those lower.

If a fast food worker at a better paying, better run chain or restaurant said it, it would be still a little mean (as intended), but not kicking down.

yup. They call it punching up. No one likes someone who punches down. lol

Did you work in Oakbrook? I was on a team that was trying to make a few smart restaurants. We got to about 50 before the project was canned but we were trying to put everything from fryer oil life to what the platens were cooking and HVAC all on a touch screen. It was a lot of fun.

I’ve worked for a few commercial equipment makers. Talked to Oakbrook and Romeoville but never been inside.

Every time McDs changes CEOs I feel bad because that means almost every current project is going to be shitcanned as the new person tries to make their mark. Are you still there now?

Haha, nah, the project was canned and I moved on to greener pastures.

Chick-fil-a is ahead of the game here, they recently put edge compute into every store

I suppose ice cream isn't high on the list of services they feel they need up and running all the time, but if the machine is determining this I would think there would be some level of predictability that would allow them to schedule sanitizing and etc.

It's funny that the machine tells them there's no ice cream. I visited my local library for a tour. They have an automated system that collects returned books, sorts them, scans and processes them, then puts them in boxes to be shipped to other libraries if they need to go there. Or puts them in carts and labels them with instructions about the efficient path to travel to return books...

The humans just take orders from the machine when there is a cart ready with books to be put on the shelf or boxes to be sent to other libraries.

Interesting times.

There's a great short story called Manna that builds on this idea: https://marshallbrain.com/manna1

This is such a great short story!

It's about ten years since I've worked with an ice-cream machine, but in my experience no matter the model they never forced you to try not to serve ice-cream.

They would enter their self-pasteurisation cycle, and the display at the front would tell you that is what it was doing, but it wouldn't prevent you from pulling the handle, if you so wished.

The problem, though, was that the barrel of the machine where the mixture was normally frozen was now at some 60 degrees C, it would be much much higher pressure than usual, and since the entire staff of a McDonald's tends to consist of teens who don't exactly take pride in the precision in which they carry out their jobs, at least once a night you'd have some poor kid blast a fine mist of hot sugary milk over themselves and the surroundings. Sometimes twice.

Sometimes I miss working at McDonald's.

> I suppose ice cream isn't high on the list of services they feel they need up and running all the time

Depends on the country, here it is hot 9 months of the year. People queue up for ice cream every day. McDonald's has mini stores only selling ice cream.

Maybe it's the same in hotter parts of the US (which the article is about), for example Florida.

I recently saw a video that actually showed the machine (allegedly) was sterilizing itself. The video showed all the ice cream in the hopper melted and soupy, presumably heated to kill bacteria in it. The video creator claimed it took some hours to complete, thus the common claim that the machine is "broken"

I can tell you from working at mcd's in the late 90's, nobody wanted to clean the machines or take care of them, so we would turn the machines off. The manager generally didn't care unless someone complained to corporate or the franchise owner.

But why "broken" instead of "down for scheduled maintenance"?

The latter implies continuance of service, the former leaves that open.

Because the teenage wage slave at the register doesn't give a shit about your fine distinctions.

When I worked that McDonalds the machine was almost always down simply because the night crew didn't give a shit and it was too much effort to clean. I worked mornings with the old folks because I was too uncool for the night shift.

hey now, most of the people I see working at McDonalds are fully grown wage slaves. Chick Fil A and In and Out seem to be the main purveyors of adolescent labor these days.

"Good sir, it appears that our ice cream machine is down for scheduled maintenance, can I interested you in a piping hot apple pie perchance?" - teenage wage slave ;)

If you tell the customer "It's down" or "Down for maintenance" you will absolutely get some who tell you "Just turn it on for me and give me my ice cream". If you tell them "It's broken" then you don't get that.

Even though it's a valid distinction, I'm not sure it's one that's valuable to the interaction being described.

It's a simple way of shutting down the customer. Broken: non-negotiable. "Understandable, have a nice day"

As these kinds of places attract the pettiest and most annoying customers anything else would make the customers try to bargain with you, even for stupid crap like that.

It's unfortunately the way the interactions between individuals and organizations seem to be trending towards.

The flip side is that when you have a non-standard request to a business (like a bank) or a government agency, it used to be that a low-level clerk would have some workarounds, or you could escalate to a manager. Now, they all have a convenient excuse: "the system doesn't let me do that". And it's true, because it's all SaaS software with inflexible business rules, controlled by some central team that neither you nor the clerk will ever be able to reach.

Yep, which is why I get pissed off at manual processes in which people are strict and exactly comply with the rules and forms. One thing wrong, nope, cannot accept this document. Well then in that case those people are no better than a computer and I welcome the day they get replaced with one.

^this is the correct answer!

“Down” used in that sense is computer jargon. “Broken” is universally better understood, at the small cost of some accuracy which doesn’t matter to the customer.

No, that's by no means originated with computers.

People have said "the phone lines are down" for years, meaning landline phone service was temporarily unavailable (often when no actual physical cables were knocked down off of poles, etc, though that might be where the use of the term originated in that case).

I'm sure there are plenty of other examples of using 'down' to mean the same kind of thing (all of which predated computer use being common in a regular person's life).

Iirc it goes back to telegraphs, the line been physically down would prevent them working and it’s hung around since.

Like hang up, when was the last time I hung up a phone, that style of phone went out of fashion here 30 or more years ago.

Ackshully it's from the even older phones with separate earpiece and fixed microphone. You'd really hang up the earpiece on these old ones.

You'd physically hang up wall phones, even when they had a combined microphone and speaker: https://images-na.ssl-images-amazon.com/images/I/515Mj67cq3L...

Yes, but what I meant was that the expression is likely even older than from the ones which went out of style 30 years ago.

It matters to me

Why? What are you going to do different if you are told it is broken versus down for maintenance? You're still not getting any ice cream?

The French existentialist Jean-Paul Sartre was sitting in a cafe when a waitress approached him: "Can I get you something to drink, Monsieur Sartre?" Sartre replied, "Yes, I'd like a cup of coffee with sugar, but no cream". Nodding agreement, the waitress walked off to fill the order and Sartre returned to working. A few minutes later, however, the waitress returned and said, "I'm sorry, Monsieur Sartre, we are all out of cream -- how about with no milk?"

Brings to mind the Hitchhiker's Guide to the Galaxy text adventure by Infocom where you start the game with an item called "no tea" in your inventory.

I don't get it.

Sartre believed that it was impossible to abdicate the responsibility of making choices, because to choose to do nothing is still a choice -- in other words, "nothing" is a kind of something. So one might humorously imagine that Sartre would also think that "no cream" is a kind of something that could be different from "no milk."

Excellent! I once went to a local burger place - they have a menu full of the various types of burgers, all loaded with everything. My kids are super picky, so I asked for a burger, with nothing except the burger and the bun.

Server looked confused - but which burger did I want? It became apparent that while I could ask for a preset item and ask for removals, I could not actually go fully off menu. So I had to ask for "the cheese burger, without the cheese" (as somehow distinct to say, the bacon burger, without the bacon). I laughed at the ludicrousness of that, but now I see they were channeling Sartre! In retrospect it would have been a truly beautiful absurdity if see had come back and said "sorry, we have no cheese, do you want no bacon?"

Like the scene from Five Easy Pieces with Jack Nicholson. He asked the waitress for an omelet with a side of toast. She says they don't have a side of toast, so he asks for a chicken salad sandwich, on toasted wheat bread, and hold the mayonnaise, the lettuce and the chicken salad.

I get the feeling that it's also easier for the cashier to put it into the point of sale / ordering system. There are buttons for specific menu items, and then buttons to remove things from them, but no buttons to build a burger from scratch.

I remember working at the drive thru for Wendy’s. I learned all about some clever orders that saved you $0.50-2.00 by ordering things a certain way. I can’t remember anything specific, but I remember one time asking, “so you want an X?” And they came back with “no, put it in exactly how I told you.”

In the ice cream situation, broken and down mean the same thing. In the Sartre anecdote, milk and cream mean the same thing.

No, it was a logic error on the part of the server. Since he requested without cream, their lack of cream was irrelevant.

One implies neglect or incompetence, the other implies a limitation of a system.

Some people would look down on a franchise or a restaurant or a business that had super high failure rates. Like if the comedy section of Netflix went down every 3 times you tried to visit that section. As opposed to Netflix having some system level limitation requiring them to perform some magical dance and perform some spell every few days to keep the server room from exploding.

You would perhaps give Netflix more leeway and not feel as much disdain and switch to a competitor if you knew that the comedy section being down every once in a while was some legitimate limit the engineers had found and not just a shoddy organization. But if you just got some nginx 500 error page randomly you would be like 'wow this site is hot garbage'.

Sir, this is a McDonald's.

This is McDonald's we're talking about. No one walks into one expecting spit-and-polish and Michelin-starred service.

Further if they told the customers that it was sanitizing itself they would assume that there would be an issue with the hygiene.

"down for scheduled maintenance" is phrase that's very common in the tech world but sounds weird in a restaurant.

Given the "entitled customer" incidents I've experienced, seen, and heard about, "scheduled maintenance" sounds intentional and like something "you shouldn't be doing when I want my ice cream", where "broken" sounds like something the guy behind the counter can't help.

I'd be willing to bet that "broken" is just less likely to start a fight with an enraged customer whose day was very slightly inconvenienced by their inability to consume extra junk food.

> Ice cream, as a food that is very close to neutral in pH and without much in the way of preservatives, is a common source of listeria bacteria.

Former Blue Bell Creamery CEO was recently charged with seven counts of conspiracy and fraud for not taking proper actions in regards to ice cream based listeria in 2015.

But their listeria just came from being licked?

No. It came not following standard practices to prevent contamination. Multiple plants had contamination and three people died.

> Ice cream, as a food that is very close to neutral in pH and without much in the way of preservatives, is a common source of listeria bacteria.

I know very little about food science so apologies in advance but curious as why they don’t add preservatives themselves to the Ice cream and why other pH neutral foods McDonalds do don’t require as much care as the ice cream?

> Ice cream, as a food that is very close to neutral in pH and without much in the way of preservatives, is a common source of listeria bacteria.

How come the the high amount of sugar in ice cream doesn't help?

Would it be any different if it was a non-dairy milk like soy?

That effect kicks in at higher sugar concentrations, e.g. a 2:1 sugar:water simple syrup, which can have some shelf stability. I don’t think soft-serve mix would be even close.

You need for there to be little enough water in the solution that cells with permeable membranes will be desiccated through osmosis. This can be seen in a common household science experiment, where you remove the outer shell of an egg by leaving it in vinegar for a while, then submerge it in corn syrup to watch its further transformation.

Sugar itself is food. It's the fact that sugar draws water out of the medium that it retards some kinds of bacterial growth (not botulinum for example).

Sugar is pH-neutral, and is not itself a preservative (it's pretty much the opposite).

How hard is it to devise an ice cream machine with a simpler sanitation process?

Considering McDonalds is on the forefront of food automation tech, probably harder than you think.

Well, not always on the forefront, but they're the best at scaling it up reliably once they get it working. They test the living hell out everything they ship.

How about some redundancy then? Two machines instead of one.

I would have imagined someone in technology (or at least enough to read HN) would understand the hidden complexities of things they're not intimately familiar with

Well I don't, hence why I'm asking...

Pasteurization via heating the entire machine to a high temp and holding for some time is likely the simplest you can get. No need for cleaning solutions, flushing, etc. Simply heat the thing up for a few hours.

As for duration, that's a function of temperature and desired safety margin. The higher temp you go to sanitize, the faster you can sanitize, but if they're heating the ice cream mix as well, there's likely a limit to how high they can get without damaging/altering the mix. So the other variable is time, and that's probably what leads to the multi-hour sanitize cycle.

Oh I see so it's a feature, not a bug. :)

Original author here: the phrasing placing an order was probably wrong - mcdonald's lists the products they're currently out in their api. I'm just querying for those.

No ice cream was harmed in doing this.

> I'm currently placing an order worth $18,752 every minute

Do you have to have a cart in order query for broken machines? If not, why was it mentioned in the tweet?

Honestly, this confused me. The number immediately made me think he was submitting fake orders for that amount. I don't know to read it any other way.

My guess is that there is every possible combination of items that is added to the cart, and then the API response returns what items are unavailable/out of stock.

Honestly, I'm not in love with this post for HN. It seems like something done and communicated for twitter clout and is devoid of interesting details.

It helps to say something outrageous like that to attract more attention.

yeah you have to actually put the order into the cart in order to get the availability of the product

Because it's hilarious to quantify and state. Why ask ridiculous questions?

Because it's a meaningless number. If he is actually placing an order for that amount every minute, it would be a useful number (and he would be an asshole for making so many fake orders and filling their system with garbage metrics). If he placed ice-creams in the cart for 8000 stores, then he should give that number. The dollar value is a completely pointless number in this case, and was done purely for clickbait.

Should be simple enough to filter his IP address out unless he proxied his requests right?

Because it's one extra feather on the scales of increased cynicism about what you see online. Clickbait culture is toxic

Legitimate question, how and why is clickbait culture toxic?

I am not creating it or defending it. I am just a little confused what you mean by that exactly.

Not original commenter but I think jackric already alluded to the increased cynicism that develops from clickbait. Personally, when I see so many ppl giving up honest and nuanced statements/opinions for hyperbole in order to gain views/likes, it's disappointing.

Raises the noise in the signal-to-noise ratio of a browsing experience. I may pass over reading a useful truthful article because I'm trained to dismiss its title as misleading.

That’s one word to describe it.

Personally I find it to be in spectacularly bad taste.

I appreciate the amusement and humor, but you might want to be careful with phrasing on something like this.

I think researchers and people who implement interoperation on the Web don't want to run afoul of federal law, nor accidentally give misleading guidance to people new to their kind of work, nor invite new rules that could be overly restrictive.

Nice to see you on the Frontpage, what an amazing project!. I remember you introduced me to HN 7 years or so ago at some MUN meeting hahaha.

Not sure where this data is coming from but for months now I've been ordering ice cream through the McDonalds app only to find out when I get there that the machine is broken and that they can't cancel or refund the order, which ultimately results in a stressful cash refund. It's happened multiple times at various locations around Manhattan. Eventually I'll just use this method to get 30-day cash loans on my credit card.

Take it from someone who has spent a few summers troubleshooting, fixing, flushing and cleaning soft serve ice cream machine. If they are not working one day do not go back there and buy ice cream.

There are few employees who are trained to maintain and clean those thing at McDonald's. If they get gross they usually just turn them off which cuts the cooling and then it gets very very nasty.

Getting a bellyache is the bear to hope for.

Disclaimer. I didn't work for McDonald's and it was two decades ago.

It still puke a little if I even see people eating it

3 hours I'm a closed kiosk at 31C cleaning out machines is a nightmare. The smell is horrible. But they didn't allow opening the flaps and letting air conditioning since it would be optically and stinkwise off putting for customers


Well if you keep doing this and your credit card gives like a 5% cash back on restaurant purchases...this is virtually free money?

Until the one time it does work, and you end up with 500 ice cream cones.

Yeah, now you have 500 ice cream cones so it's still a win:)

It's not free. That cash refund isn't refunding the merchant their processing costs for the credit card. And that's what's paying your 5% cash back. The store is losing a pennies each time that happens and some of them are flowing to you.

Well, yes, but it's also them not refunding via the original payment processor and them letting the order go through despite being unable to fulfil it. I don't want to make a point that this is the moral thing to do, but the error is with McD here.

I'm not making any moral judgement. I'm just explaining where that money is coming from, and that it's not "free". Also, issuing a refund would still likely incur fees with the processor. They may dodge fees if they are within the window to void the transaction instead. Depends on their processor and contract.

I don't think anyone was under the impression that it's 'free money' to everyone involved.

It's 'free money' to the person it happens to (original poster).

Sure. But there is no actual thing as "free" money, in the sense that it all comes from somewhere. And honestly, probably nothing is truly free. Even sunlight is the result of spent nuclear potential of the Sun. Unearned is maybe a more precise adjective to use in these scenarios.

But again, the point behind my comment was to explain where the money was actually coming from. I feel like these responses keep trying to apply a moral-judgement reading to my comment where none was intended. If one didn't know the details behind merchant credit processing, they didn't know where that cash back was coming from. And the answer is that the merchant still lost money in that transaction.

>But there is no actual thing as "free" money

Yeah I get that. I feel like my reply might not have properly explained itself.

Nobody is applying a moral judgement to your comment (that I've seen at least), we're pointing out that yes, we know there is no such thing as free money in a grand sense. For most people on this board merchant fees aren't some unknown thing. We know this, and obviously McDonalds takes a tiny hit for the fees every time they refund a card transaction with cash.

It's free money to the person it happens to is what the commenters were saying. McDonalds refunds my card payment with cash, I get the credit card points and my money back. It's free money. To me. McDonalds taking a hit is irrelevant to me as I did not lose anything, I only gained. Hence free money. The context is important.

If this is true you're also earning points for your purchase. I don't know why they wouldn't be able to refund an app-based order though beyond ignorance, considering I've had it done before.

Worst. ATM. Ever.

Starbucks is really bad at this as well. I used to sit at a counter next to the pickup shelf, and probably 40 times a month just during times I was there, a mobile order would come through for unavailable items.

McDonald's mobile ordering has multiple contradictory and nonfunctional answers on how to get a refund, and I'm not sure any of them actually work. I'm going to try a chargeback soon.

From creator[1]: “I reverse engineered mcdonald's internal api and I'm currently placing an order worth $18,752 every minute at every mcdonald's in the US to figure out which locations have a broken ice cream machine”

[1] https://twitter.com/rashiq/status/1319346264992026624

They were just taking the opportunity to make their work more hype worthy. Typical marketing garbage.

Hard to imagine that lasts long...

It seems the first explanation given wasn’t entirely accurate (or is at least misleading), as they continued roughly 50 minutes later:

> to clarify how this works: mcdonald's keeps track which locations have a broken machine, I'm merely querying for those - no order gets executed, no ice cream is actually wasted

Maybe in reality, the order is created, but not actually “placed” in order to query the above info? Or maybe the first tweet was simply clickbait...


What I’m understanding is that the order is placed, meaning it’s in checkout. But he’s not actually fulfilling it, meaning he didn’t ordered/paid for ice creams for real.

Someone on Twitter also mentioned ice cream orders don't actually get filled until you arrive at the restaurant (which does make sense), providing an extra fail-safe for this.

I’m not sure if it’s a regular request, or so regular that it’s automatic, but in France, if you order an eat-in meal with ice cream, they don’t make the ice cream until you come back to the counter with your receipt and request it.

Don't make joke a subcategory of clickbait, hacker news. come on now

If your "joke" is structured to sound sincere and be misleading, it's clickbait.

If your joke is structured to sound sincere and be misleading it's called a joke... that's literally what a joke is

Uh, no, many jokes count on the audience being in on it.

Clickbait for sure

I'm curious why the order has to be worth that much? Couldn't just be an order for one ice cream product?

I'm guessing his statement was unclear. It's probably (1 ice cream product) * (number of US locations) = $18,752

One ice cream product per store worldwide, presumably.

it is an order of one across every location.

It's funny that this is possible. An altruist could create an order for "give a cheeseburger to that one guy who hangs around outside every McDonald's".

If you never pay for and complete the order, nobody gets a cheeseburger.

For reverse engineering, I highly recommend learning burp suite. (This is a bit like saying "Just learn vim!" in terms of learning curve, since the curve is basically a wall, but it's well worth it.)

E.g. I was curious how TPUs actually work -- the low-level protocol, built on protobufs. But burp suite is only good for analyzing HTTP. No worries, a couple plugins later and I was good to go: https://twitter.com/theshawwn/status/1315638127772405761

You'd be shocked the amount of information you get when you merely look at the traffic. Chrome devtools is nice, of course, but it's not an intercepting proxy; you can't (easily) capture every request, modify it, replay it, see how the webserver responds to malformed input, etc. Burp repeater makes all of that super easy.

Unfortunately the power is hidden behind a rather unfriendly UI, with some unfriendly defaults (the very first thing everyone wonders is, "why did burp cause my website to freeze?" answer: it starts up in intercept mode, so that's actually a sign that things are working; but you have to turn off intercept mode to actually get it to do useful things / see a bunch of requests).

Emacs users might feel at home, at least. :)

AFAIK, reverse engineering APIs isn’t illegal. [1]

There are some (VC backed) start ups that are open about doing it. Teller API is an example. [2]

[1] “Reverse engineering generally doesn't violate trade secret law because it is a fair and independent means of learning information, not a misappropriation. Once the information is discovered in a fair and honest way, it also can be reported without violating trade secret law.“ https://www.eff.org/issues/coders/reverse-engineering-faq

[2] “We reverse engineer these apps to discover their secret API contracts and then implement clients for them.” https://teller.io/

> AFAIK, reverse engineering APIs isn’t illegal. [1]

No, but unauthorized access to a remote server can be. Just because something is unsecured doesn't give one the right to access it. Anyway I would not take that gamble personally and risking indicted federally for computer fraud.


I'm not a lawyer and neither are you.

In some jurisdictions I think they basically have a blank cheque to get you on any misuse (https://www.legislation.gov.uk/ukpga/1990/18/contents)

What in the world is Teller? Are they an actual company with an actual product? The authentication demo on their website is the exact same as Plaid Link, down to the wording. There isn't even a sign up link or pricing page, but they've somehow raised 4 million dollars [1].


The biggest problem, presumably just like Plaid, is that its institutionalized phishing tricking users into giving their online banking details to some 3rd party.

This. Plaid is very convenient in what it does, but please, just let me OAuth with my bank like literally any other organization.

I've known about Teller for some time now but I have always thought that they seem shady... I don't have an issue with them reverse engineering bank APIs, I just don't get why, how and to whom they would want to sell that.

Any small change in any of these APIs completely ruins it for all of their clients. Not even sure if they have a product or service either but again, also not sure who would want that.

Placing bogus orders using a reverse-engineered API, however, is likely illegal. Especially at the speed and scale at which he is purportedly doing it.

> AFAIK, reverse engineering APIs isn’t illegal

It doesn't matter whether or not it is illegal. What really matters is whether some prosecutor somewhere thinks it is illegal, and dislikes you sufficiently to prosecute you–or sees enough political benefit from it to make it worth their while. "Hacker indicted for sending McDonald's $18,752 of fake orders every minute". It might be misleading (or even just plain false) but it makes a great headline. Even if they end up losing the case, they can still make your life hell for months on end with their criminal charges.

As the saying goes, "You might beat the rap, but you can't beat the ride"

Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact