Hacker News new | past | comments | ask | show | jobs | submit login
Edge Computing at Chick-fil-A (medium.com)
570 points by ayanai 8 months ago | hide | past | web | favorite | 379 comments

I work in midtown NYC near a Chick-fil-a location. Used to be a line down the block, went with a friend yesterday thinking we caught the line at a short time but turns out they were using smiley employees with ipads coordinating orders and that they were serving arguably more people than before. Very impressed with how seamlessly they've integrated tech in their brick and mortars in contrast with the countless "cashless" restaurants that just add cruft, inefficiency, and customer resentment.

I have a friend who used to be a shift manager at Chick-fil-A. I can't remember exactly what he called that process... It was like meet and greet or walk and talk, but it's surprisingly efficient for the drive thru. He said it cuts down the average wait time for customers around 15% at his location.

I don't know what it would be like at NYC since it's not a drive thru.

It's crazy efficient. When you walk in someone with a tablet takes your order, and by the time you get to the cash register your food is ready. Works so well because the employees are able to catch customers at the end of the line.

Here's "Line busting" the drive thru 68+ cars deep. It's astonishing the WiFi goes out that far.


FWIW, also an edge approach, with Prometheus etc.

Slightly off-topic, but what's the reason that Americans all tend to line-up at the drive thru? I was in LA, there was a huge line at the drive thru, so we parked, went inside and got our orders really quickly.

The comfort of still being in your own space. For example, listening to your own music, no excursion for your kids to belabor, precisely don't have to park and get out.

Yeah, I think I would rather wait 10 minutes in my car than 5 minutes in line. I'm sure other people may feel differently. And something about the drive-thru still "feels" faster, even if it's not. I think it's just the idea that you get to pull out and be on your way after you've gotten your food.

At McDonald's at least the drive through is prioritised over walk ins. That said if the queue is still long it might be quicker to walk in. But inside if orders come at the same time they do the drive through order first.

I guess it also stops people from leaving. Since once you made an order you're more invested to stay and pay for it. But if you were in a long line you might get fed up and go somewhere else (or get a phonecall to do something etc).

True, but I've generally found the line moves fast even when the line is out the door. The main bottleneck in most fast food places are putting your order in at the cash register so if that's removed it speeds things up a lot.

They figured this out at McDonalds in Moscow. Kudos to CFA, queue management at QSR are otherwise seriously lacking.

It's called "Face to face"

Chick-fil-a does not run the iPad software but it does integrate with their systems. The same company who sells their POS software runs the iPad software and I believe the Chick-fil-a One app.

They've been doing this for several years at least (employees taking orders of people in line). I think since before the iPad was released.

For sure. Meant it more as a response to contents of the article that they've managed to break down their business processes into a subset of (dare I say) microservices that have allowed them to pinpoint weaknesses in efficiency and improve upon them - ie. the waffle fries example mentioned.

On the other hand when reading the article I kept thinking of the poor sod who has to cook a new batch of waffle fries because “The Edge” told him to.

Why do you consider the cook "a poor sod"? Because he/she is working a job and making a living? I imagine you didn't mean much by it but understand that just because everyone isn't employed in what many consider desirable professions doesn't mean their work is not important or that they don't take pride in what they do.

Not meant to be a rant, just wanted to make the point.

I actually interpreted the comment a bit differently. I read it that the employee who used to have some autonomy on when to make adjustments is now doing it because the computer, which they (perhaps) have no control or insight into, is telling them to. I'm sure there's more nuance and people who would view that instruction differently. But to the comments direction, I'd assumed it was alluding to a dystopia of sorts where we do more and more things "because Computer™ said so".

Yes, that’s exactly it.

In hindsight, “sod” wasn’t the right term to use (that’s what you get for posting at midnight) but the point I wanted to make was the removal of meaning, autonomy and human interaction when not only are you doing a repetitive job, but how you’re doing it is dictated by a machine. The movie Brazil came to mind, although I’m sure there are better parallels to make.

You'd be amazed how much satisfaction you can get out of learning to keep a steady stream of perfectly-crispy fries going. It's quite fun on a busy day - hectic, but strategic, and rewarding when you can stay on top of things _and_ produce minimal waste in the process.

In some sense, the automation of demand prediction takes some of the fun out of gaining that intuition yourself. On the other hand, it is nice not to be the queue blocker with orders piling up behind you because you weren't prepared for a sudden burst of traffic...

Okay, looking at it as the computer helping you stay in the flow is certainly a different perspective.

It might be looking back on my time in fast food with a slightly more romanticized lens than most would probably give it. :)

There’s a short-order chef game my kids play on the iPad, you can practically see the endorphins pumping when they manage to serve all the customers in time.

The human brain is definitely wired for this sort of thing.

Me too probably. I loved my fast food job in high school. That said, I'm glad I'm not still doing it.

From my days in those kinds of jobs I'd welcome it. I won't have to go count how much we have on hand. I just drop whats listed and move on to something else. My problems would usually come from "a system" that were missing large steps or prerequisites. "Start frying 1 batch of fries and other in 4 minutes" except the fries weren't cut the night before so both fries wont go in for another 20m. Or the food in the fridge is no longer fresh enough to serve, so we're down some food for the dinner rush.

Ultimately “the edge” will just do it for him :-P


These got installed, but adoption was low due to cost, they were tied into the POS.

What!? I LOVED my fast food job as a high schooler. It was fast-paced, and good work. I likely would have enjoyed getting instructions from a computer, lol. I certainly wasn't some poor sod.

> They've been doing this for several years at least (employees taking orders of people in line). I think since before the iPad was released.

Wendy's used to do this around 91/92 (clipboard, not iPad)

I think they ditched it with the smoking sections.

Quite a few stores do that when they get busy.

I live near the busiest chic fil a in the country. If they didn't have the iPads it would be impossible to serve all the people.

That is strange to have a cashless restaurant. If anything cash is preferred. I know of a few places that only accept cash. And in the last week my debit card chip has been malfunctioning so I have to pay cash for everything until my new one arrives. I even attempted to use nfc today. That is a shit technology and was a waste of time. The POS terminals simply don't work.

If you want to see the actual hardware used at the restaurant, check out the previous blog post: https://medium.com/@cfatechblog/bare-metal-k8s-clustering-at...

Spoiler: It's a stack of Intel NUCs.

Spoiler: It's a stack of Intel NUCs.

Funny how the world has changed.

I got my first taste of real computing from a guy in the next town who would buy dead PDP's from local McDonald's restaurants, fix them up, and sell them on.

He always said the hardest part wasn't replacing broken parts, it was that invariably the machines would always have soda spilled in them, no matter where they were stored. Cleaning the insides was his most time-consuming task.

One of the advantages of an Intel NUC is that it probably has a lot more horsepower than an old PDP/11. It's probably also a smaller target for wayward beverages.

Used to work for a very large POS system provider, and I can confirm that soda spills make repairs take 10x longer.

I'd like to hear the story of why a McDonalds would need a PDP. Weren't they...expensive? They would be at the heart of university's IT department, or running a steel mill. Stuff like that. But until PCs came out, at a restaurant I can only picture registers and a dumb terminal hooked up to a distant mainframe.

The logistics improvements from early computers were massive. If you compare the equivalent price nowadays it's not worth it compared to other options, but going from paper to computing wasn't just an incremental improvement but allowed for entirely new abilities.

It wasn't like having an account to crunch all the numbers on staff full time was cheap. Especially when you'd have to have 3-4 accountants just to have 24 hour service, regardless of the work load

McDonald's corporate has or used to have a royalty fee for each item sold. That's a lot of tracking of sales info in the early days...

I spilled beer all over my laptop keyboard once, so I stripped it down and cleaned down the whole thing with isopropyl alcohol. It was a really annoying task, I can't imagine how annoying it would be with soda in a PDP.

I used to work in a hospital and the worst problem was keyboards full of blood. I would rather clean up soda all day long. :)

I'm just trying to understand, I see three of those machines, inside a pretty nice rack, with the switches, routers, a keyboard, I would assume some fairly high quality hard drive / RAID type of thing etc. then the blog post claims "We run our Edge infrastructure on commodity hardware that costs us, ballpark, $1000/restaurant. ". Just the intel nuc I can find on newegg alone costs $349 each, with no memory or harddrive installed, retail. I know they are likely getting amazing volume discounts but it still seems amazing that whole stack could possibly be only $1000 each.

This is Caleb (SRE on this project). Here’s a similar NUC we use ... this one is like $300... just an example of what we use; https://www.bhphotovideo.com/c/product/1316113-REG/intel_box...

Caleb, I'm working on a project with a similar architecture (on-prem services). I'm curious what your auth-pattern looks like as it's something we've struggled with. Mostly the balance between being convenient and being secure. It seems like each site needs an api key to access your cloud. Do you have an auth-pattern that would prevent the key from being readily available to an attacker who got access to the machine? Or do you kinda just say if they get access to the machine, then it's game over? Also with shoddy networks it is difficult to be confident that key-rotation will happen successfully. Happy to get your thoughts. Cheers.

Just curious (since I work in edge computing products) the deploy here seems rather complicated... What are your feelings on the matter and is there a market for simpler deployments?

I'd assume it's 3x$349 that they're referring to in the article, since most places these days already have some networking equipment (probably including a racked switch someplace) in order to provide guest wifi, connect POS terminals, etc. So that portion isn't directly tied to this effort.

Intel NUCs come in a wide price range (for example on Newegg the Celeron-based NUC is $130)

This is Brian -- I wrote the article (I suspect I will be typing that a lot tonight). We do clusters with 3 nodes per restaurant, so at full scale (when we have rolled out to every restaurant in the chain) that will be ~6000 nodes, and growing by new stores * 3 going forward. This will support an estimated 100k IoT "things" of various types in the next year and a half to two years (estimate)

Hey Brian, I have been investigating deploying a very similar stack to what you guys are using now. How are you handling onsite load balancing? Is it a simple round robin type load balancer at the router level?

Also is Highlander open source? I don't see a link to it in the article.

We will open source highlander eventually, but it’s not quite there yet.

We use a VIP that the NUCs share... ie; one of the three will always have a VIP, and if it dies another NUC grabs it. This is a poor man’s load balancer in that sense, because we only have the NUC hardware onsite;


We are also looking at metallb

Volume pricing maybe? 2,000 stores, so around 3,000 units.

NUCs are surprisingly well-built little machines. I have one in my car, and I give them my absolute seal of approval. I've heard of people running little VMware clusters on them too.

It's actually suprising that people think they're not "normal" computers. I provisioned a few of them and they're little beasts. Most in use are I7, 16-32gb ram, Dual NVME. The Skulltrail nucs are extremely powerfull and on the lower end the atom based units are solid too. I also ran a nuc in my car for a while, mostly stumbling radio / wifi spectrum as I drove around, but it was perfect with its relatively low power requirements.

This is Brian -- I wrote the article. Super cool use for a NUC! I'm going to follow the links and check out what you did some more.

But is it real, 6-core 3GHz+ i7? Or is it this mobile crap Intel has been pushing lately?

What is it running in your car?

I have a custom piece of software that powers a display in the car, syncs and plays my music, and logs GPS. It also can manage audio and video recording and is intended to handle some future features[1] I have planned, so I wanted to over-spec rather than under-spec.

[1] It's the same software that runs my home automation system, so an inevitable feature is my car and home units interacting, I wrote half of a piece of navigation software before I stopped, and the entire interface is intended for voice control so I need to throw that in there at some point too. I intend to work on a CAN interface to connect to the car, but I've had a few roadblocks to getting started on that project.

Have you documented any of this or written on your home automation endeavors? I'm looking to "Tony Stark" my house up a bit (i.e. voice control, some touch enabled panes in the kitchen/office) but want a bit more control than something purpose built, off-the-shelf while also being a bit more insulated than Alexa or Echo. The NUC has been on my radar as a device that I may go with as the brains of the operation, and I'm just curious to read what design and implementation patterns others have taken.

I get asked to do more of a "write-up" a lot, but it's hard to figure out how to format or frame it. I have a fairly large pile of opinions about how one should go about doing this, but it's hard to concisely organize my thoughts on it into a coherent write-up.

The code is here: https://github.com/ocdtrekkie/HAController but I don't know if I'd really recommend others use it. The main perk to me is that it's designed around what I want and use (I tried a lot of alternative options before rolling my own), so unless you also really like Visual Basic code and have a brain ordered bizarrely similarly to mine, you may want to start somewhere else.

I'd love to chat about what I've learned along the way or what ideas you have (because I might borrow them!), if you want to chat elsewhere hit me up at inbox (at) jacobweisz (dot) com

What's the power draw like?

I once left the reading lights on; car was supposed to have turned it off when the doors locked, but somehow didn't. I found myself calling AAA in the morning to get a jumpstart.

Also curious, does GP have some way to shutdown gracefully when the car turns off, and reboot fast when it turns on? You could potentially handle this with a laptop and some battery-settings but not aware of an external battery solution for the NUC.

I currently use a laptop power adapter for the cigarette lighter, which is only on when the car is. The NUC's BIOS is set to turn it on when the power comes on, but I power it off manually before turning off the car. This is not ideal.

There is an external battery solution for the NUC, and I bought it! http://www.mini-box.com/NUC-UPS But I haven't switched to it yet. The big upside is that it will gracefully shutdown my NUC after I power off the car. But if I use the cigarette lighter for power, it will lose the ability to turn itself on when power is present, because the batteries are always present. NUC UPS supports using a different power connector which can turn on the PC on ignition, but I haven't had it installed in my car as of yet as that's a little bit more work, and I'd need some help from someone who knows more about the car's electrical system.

As far as fast power-on, it's a pretty high end i5 NUC with an NVMe SSD. It boots pretty darn fast, and my software takes less than a second to load once the OS is up. The slowest part of boot is that I don't want my location history easily steal-able, so it's encrypted, and I have to key in the code to unlock the machine. (I'm looking at a security key or similar to replace this step in the future.)

As a note, I'm more or less specced out what's involved in a solar power setup on the roof of my car to independently power my computer all/most of the time when the car is off and otherwise charge a secondary battery off the car's inverter, but there's no reasonable or sane reason to do it. ...But I thought about it.

Thanks. Curious, where do you stash the NUC in your car? I imagine it's insulated and not exposed -- otherwise a hot day or a very cold day might kill it.

You don't want insulated and hidden, you want perforated and airy (while still being discreet).

The NUC-like hardware that goes into tanks have chassis consisting of a mesh cage to allow maximum airflow since the cabin temperature alone can exceed 100F.

Actually, it's pretty important for the NUC to get good airflow so it doesn't overheat; insulating it would be a bad idea! I haven't had any temperature-based issues with it, in part because most of the time the car is running I am in it, so I keep the temperature at temperatures I am comfortable in. And the NUC's temperature range is pretty wide as well, which is good, because weather here ranges from -10 F to 110 F.

Probably the only real concern I'm afraid of is condensation when heating up the car on a cold day, but it hasn't been an issue so far, perhaps because the computer is not near a window.

It should be pretty easy to hook up a microcontroller (think Arduino nano) to "press" the power button when the power from the lighter comes on.

Since I'm not a hardware guy, I try to avoid assembling my own gadgetry as much as possible on this project. I use consumer grade home automation modules, standard computer parts where possible. (Everything in my car is largely interconnected with USB.) And generally, experimenting with the NUC UPS aside, I try to avoid parts that may be hard to replace. Which is to say, I don't have time to build, test, and fix a random Arduino gadget. ;)

I've been planning to make some custom-cut USB cables and a 3D printed part just to make the setup of my display a lot cleaner... and I haven't done either of those and it's been a year or so since I planned to.

Nice machines, fast enough, VESA mount to the back of a monitor, and easy to upgrade. They've basically replaced Mac minis and iMacs in our labs. Have not had one bit of trouble with any of them.

If they had OOBM of some kind I would ditch all my hardware at home and replace it with them.

Why do you have one in your car?

they've kind of a pain in the butt for us... For example, it would be nice for them to have remote admin, more than one NIC, not freak out when the HDMI is not plugged in, etc.

I use my NUC without HDMI plugged in, and I assume Chick-fil-A does as well. ;)

They do actually make a dual NIC NUC, but it's a little overkill for most uses: https://www.amazon.com/Intel-Machine-NUC8i7HVK-Radeon-Graphi...

On the other hand, my NUC has plenty of USB 3 ports, and an Ethernet dongle will run you about $7.

Hmm we did buy NUCs about two years ago so I suppose it's possible things have improved

Interesting as it seemed that smarter edge/iot nodes would be like ARM/RaspPi, yet it seems it may get even as high as (or just straight to) Intel nuc.

This is Brian -- I wrote the article. For what its worth, we considered trying to run our clusters on an array of cell phones since they would natively support connectivity fallback and pack a real effecient resource punch in a small footprint. One of our engineers came up with that idea. With the challenges compiling on ARM and the relative ease of moving forward with x86, we went that way to start. We tried really hard to balance the "ideal" with the short term "MVP", and I think we landed in a decent spot with the NUCs. They give us enough power to run the things we need in the short term, and we have some capacity for horizontal scale in the future as our needs increase. We tried to think "cloud native" at the edge as much as we could without truly being a cloud.

"Kubernetes for Android" would definitely raise a lot of eyebrows.

Raspi cannot be used for a semi serious server application due to microsd card write wear issues. All other methods of attaching external storage by USB bus to an rpi are not reliable or suitable for this sort of purpose. Nowhere near the longevity of something with a native sata or m.2 SSD interface and a real SSD.

There's also the added complication of the fact it's an ARM based device so in some cases you might run into compatability issues, or testing your container locally becomes problematic.

It's definitely getting better though

"Raspi cannot be used for a semi serious server application due to microsd card write wear issues."

As an end user, I run a personal authoritative DNS server that has small RAM requirements. The RPi (or other SBC) boots to a mfs mounted root, then mounts all directories as tmpfs. Then I remove the SD card.1 As such, the logs for this server, which are automatically rotated and do not exceed 5M in total, are written to RAM.

1 I only use the SD card to boot. The only files on the card are a bootloader, a bootloader config and two kernels, each with an embedded filesystem. If updates are necessary, I make them to one of the kernels at a time. The other is the backup. The bootloader and bootloader config lets me specify which kernel to boot.

Does that remain true if you grossly over-provision the SD card? Use a 64GB card for 1GB of data and you have a lot of wear leveling space.

The write lifespan in DWPD on any sd card is a tiny fraction of the expected lifespan of even a cheap $55 consumer SSD.

Part of the problem is that even if you only write 1GB per day the card is highly likely to reuse the same sectors over and over.

Don't most modern SD cards contain some wear leveling logic?

I did a bunch of research on this a while back and the conclusion I found was: yes, but buy name-brand better rated cards to avoid cheapo cards that do not do wear leveling or do it very badly.

It's nowhere near as good as SSD drives, but it's better than just a naked flash chip... unless I'm wrong.

Yes, but it's not enough.

Even the brand name cards aren't worth shit; you need to buy "industrial" cards.

Source: tried to get a root filesystem to work on name brand SD cards for a high availability product.

Hey if you see this can you answer one more question:

How much abuse was your high availability product delivering to these cards in terms of writes? Was it something like a video recorder, database, cryptocurrency, or some other application that did large amounts of write I/O?

I ask because we're about to ship something that uses SD cards, but the I/O is very low. It's a network appliance and doesn't do anything locally that is high write throughput.

We weren't beating up the SD card with heavy writes. It was for a robotics controller, so just sort of normal, low throughput logging.

We even saw SD cards screwing up in RO mode in fact, from major brands. Trust me, you want to swap out for an industrial card in your BoM.

Have you tried things like jffs2? Do they help?

jffs2 runs on raw flash. SD cards have an abstraction layer in front of the flash to make it look like a normal block device and jffs2 isn't usable.

...the card is highly likely to reuse the same sectors over and over.

One would have expected this to be an area of focus for RasPi developers? Surely some driver could be updated to avoid this?

The Linux kernel in raspbian treats the SD card like a normal block device. I'm not aware of any special optimization for write wear leveling at the OS level, or on the board's sd card controller.

SD card wear-levelling is usually supposed to be handled on the controller inside the SD card. SD cards aren't like having raw access to flash, you basically write over SPI and the controller in the SD card decides where to actually put it.

This talk about SD cards is really great and explains some of this:


Fantastic talk, thanks for the link!

Not without making the card unreadable in other systems.

And even then I'd be surprised if you could game physical sector offsets by playing with the virtual offsets you have access to.

Working with RaspPi and Docker has been surprisingly difficult. Docker's support for multi-arch is poor at this point, and also applications that are I/O heavy seem to bottleneck at the network.

On a related note, Chick-fil-A using Deep Learning models for object detection with MXNet to track how long fries have been waiting: https://www.youtube.com/watch?v=3Uuq_cX8b1M

This seems like such an over-engineered solution.

It would've been much simpler to have a plate with multiple pressure sensors and a multi-color LED per spot. When you set the fries down on the sensor it activates and the LED turns green. After X amount of time the LED can turn yellow, meaning that it's becoming stale. Finally, after X+N time has passed and it's no longer fresh it can turn red. Removing the fries turns it off. Aside from being way cheaper, I'd conjecture that this would be much more reliable. I'm pretty sure I could get a prototype of this up and running over the course of a weekend or two.

If you wanted to get really fancy I guess you could also track the room temperature and moisture levels and use that to get a better guess of how long a group of fries will remain fresh. Although I don't know if environmental factors like these have enough impact on fry freshness to be worth taking into consideration.

Anyway, it looks like they were just doing this for fun and learning, so I guess it doesn't really matter.

This was a good learning experience for our engineers at our Innovation Center at Georgia Tech in Atlanta. It may bear fruit in the form of a useful solution in the future, though. What is unique to Chick-fil-A is 'volume'. We do a lot of sales in our restaurants, so anything we can do to try and make our team members lives easier is important to us. We want them to enjoy their jobs and we want to do the best we can to consistently create high quality food experiences for our customers. Our teams in restaurants are the heroes, but we are trying to use technology to help them do what they do. <thumbs up>

The computer-vision based version means you just take the first fry off the rack. Yours means the employee needs to assess the LEDs each time (which may only take a second, but is repeated hundreds if not thousands of times a day), reach more for the yellow one in the back, etc. That's a lot more sensors to break down than a single camera, too.

Doesn't look like they actually use that. It seems like more of a research project that they wanted to see if it was feasible.

Or more like a fun summer intern project, really. I'm sure it was a cool learning experience but things like that are rarely ever deployed.

My life is complete.


Unrelated but I've done a good amount of nonprofit work and everytime I've had to ask for food donations chickfila has given food free of charge.

I see people like this the way I see my Mormon family. They usually try to be very kind and I value that.

At the same time, they harbor beliefs which are incredibly damaging to both me and those I love, so I keep a healthy distance while understanding that they mean well.

Don’t mistake their kind acts for open-mindedness or acceptance, though.

Out of curiosity, in what way is Chick-fil-a not open-minded?

I understand that you disagree with the CEO, but how do you get from there to the conclusion that he is not open-minded?

Because he claims that homosexual relationships do not deserve the same state recognition as heterosexual relationships.

Wait, so in order for a person to be open-minded, they have to not only honestly consider other ideas and perspectives (which is what being open-minded actually means) they also have to agree with your perspective?

A person's open-mindedness is independent of whether or not somebody else agrees with them. Otherwise, by definition nobody - including you - is open-minded, since there are people that disagree with you. It's completely nonsensical.

It's the fact that they are unable to honestly consider it because of their dogma rather than the fact that they don't agree.

I'm fascinated by your stance - genuinely fascinated, I'm not trolling you, I'm serious.

So to recap, the CEO of Chick-fil-a is part of a group of people ("they") who are actually unable - as in, incapable - of honestly considering certain viewpoints? How did you reach this conclusion, and how did you come to be so certain that this CEO is in that group?

Is your understanding that he gave no serious thought to his opinion (i.e. it was just imposed on him), or that he thought about it lightly but lacks critical thinking skills, or something else?

Is there any possibility that he simply has a different opinion than you, and that's all there is to it? Like, he educated himself like you did, thought about it like you did, etc. but just reached a different conclusion than you? Or is the only explanation that he is flawed/limited in some way?

Apologies in advance if anything I've said comes across as demeaning, I'm genuinely trying to understand. Thanks!

I was raised in a cult. I know what it’s like to be incapable of considering others’ views because of dogma because of my experience.

I am assuming that his Christian perspective is similar to the one I had. My experiences with Christians since have reinforced this expectation.

Does that help?

Ok, I appreciate your willingness to share.

FWIW it's virtually impossible to determine the open-mindedness of someone from afar, and to make assumptions about their ability to even consider other ideas seems like a way to insulate oneself from difficult ideas - by declaring that someone is incapable of an honest evaluation of your perspective, it prevents your perspective from being scrutinized (which, ironically, could lead to being close-minded if you're not careful).

A parting thought: most people (a) can be reasoned with if not attacked - you can find common ground and have a good discussion with nearly anybody (b) on every "interesting" social/political issue there are intelligent and thoughtful and just really great people with nuanced and reasonable perspectives all across the spectrum - the extremist nutjobs on the fringes are wildly outnumbered by pretty normal people

So as a random dude on the internet, just a friendly warning that if the above strikes you as untrue (e.g. if you believe large chunks of people are intolerant or incapable of being reasoned with, or if you see a difficult social issue as pretty much black and white and can't understand why so many people don't get it), then there's a good chance that you've been duped and have been sucked into a form of modern tribalism.

Have a great day!

>My belief is so self-evident that anyone who comes into contact with it must agree with it, therefore if someone doesn't agree with me it must be because they haven't come into contact with it, and the idea is everywhere so therefore if they haven't come into contact with it they must not be open-minded!

Would just like to say Chick-fil-A should publish more like this. They are absolutely knocking it out of the park with tech. My wife and I are friends with a few operators (they don't have "owners"). I recall them, several years back, trying to get me to quit consulting to come work with them and open more stores. "Food service, ugghhh, no offense" I said. A friend whipped out her phone and showed me their really great in-house app that showed detailed analytics for her store (everything you could imagine) and tools for for doing purchasing and much more we didn't have time to get into. This is circa 2010 if I recall. For reference the iPhone SDK dropped in late 2007 and these folks were already way onboard with mobile. I was impressed.

I don't know why, but for some reason the juxtaposition of traditional, brick-and-mortar brand images with tech blogs about building modern application technologies is amusing to me. It's seeing two completely different sides of the same organization. There's 0 condescending sentiment behind it, it's just like "wow, that was unexpected."

I'm in Atlanta and have had a few discussions with people at Chik-fil-A (CFA) about joining on of their teams. I have a good friend that just moved from Intel to CFA working on business analytics. It seems like a really great place to work. And yes, whenever I mention to someone that I might be working on machine learning problems at CFA I always get a puzzled look in return. :)

Their food court is amazing. You can have steamed salmon and a Spicy Chicken sandwich at the same time. Also free to your family.

> machine learning problems at CFA

This is prime deep fried learning pun territory...

Alex Smola already nailed this in 2015 with deep-fried convnets, which replaces fully connected layers with fft-accelerated algorithms.

CFA has long meant control flow analysis in PL.

I worked for Nordstrom for several years and whenever I describe their technology department to another developer they're always pretty shocked. It was pretty shocking to me when I started there too.

I'm curious about Nordstrom, I know nothing of their infrastructure.

I was surprised too, when I saw their name in the Devops Handbook; apparently they were one of the pioneers of Devops. Talking with a previous employee, it sounds like they really reinvented themselves from an IT department to a little tech oasis within the company.

You can find some pretty awesome work on "traditional" companies. There are some pretty cutting-edge projects where I work too :) (but way more legacy migrations and boring financial CRUD apps).

I think one thing that doesn't always occur to programmers who haven't experienced is that really boring problems can be really interesting to work on.

(And really interesting problems can be really boring to work on, and really interesting companies can have a lot of cruft and technical debt under the hood.)

Their tech stack is similar to what you'd expect from any large tech company. I can't speak much to the server side of things (I worked on the iOS app) but I know they had everything ranging from older monolithic .NET apps to Node and Go micro services.

Their GitHub page (https://github.com/Nordstrom) doesn't have much in the way of internal projects, but it has lots of forks that will give you an idea of some of the tools they used.

Nordstrom also does polymorphic React with server-side rendering via .NET [0]. I've always been impressed with this part of their stack.

[0] https://libraries.io/github/Nordstrom/React.NET

Latest commit e56c9be on 24 Apr 2015 - wonder if it's still in use internally?

Actually they're a bit of a pioneer in Serverless computing:


We did a case study on Nordstrom recently in the CNCF, check it out: https://www.cncf.io/case-study-nordstrom/

Here's a recent talk from KubeCon/CloudNativeCon: https://www.youtube.com/watch?time_continue=5&v=xZO9nx6GBu0

To add to the curiosity, how is Nordstrom doing anyway, compared to other mall-type retail? I'm wondering if this forward-thinking in tech is allowing them to successfully fight "new" ecommerce such as Amazon

As far as retail companies go, they’re doing well. Stock isn’t doing great, but as far as I know they’re still quite profitable.

Being forward thinking in tech has helped them a lot with regards to being able to move pretty fast on new things. At the same time, a website as old as theirs has a LOT of legacy.

Nordstrom is doing pretty well for their online stores. I have to order things there for family every so often

What's interesting to think about is how, in the lead up to the dotcom bubble/bust, I would think execs at tons of traditional retail/brick and mortar/consumer brands would have probably been like shit shit and then put more IT-focused organizational architecture components in place, which probably lead to many interesting technical solutions to many of the problems they faced (if they did it the right way). It would be cool to see different branching that lead to different technical solutions for common problems.

Unfortunately, as this was before the OSS boom that came later, we probably will never see these solutions out in the open.

Recently at work someone used JS tooling from their Github org and I was like what!? Found a lot of good stuff there.

Seeing down thread they were big DevOps adopters changes my mind of how I saw them as stodgy. Too bad I dislike pretty much all they sell!

Ha, y'all were an AWS/clickstream example at the Nashville Analytics Summit yesterday, granted it wasn't a new deck so maybe you've heard that before.

Which session was that? The Kinesis/Lambda one?

The 4 hour workshop/hello world for AWS, it was in the intro deck on why you should be doing analysis and what not. Probably preaching to the choir for most of the room.

It's almost like being down the street from 2+ major technology companies might have rubbed off on them...

Word of mouth tells me they are underpaying people for Seattle-area CoL

Most retailers don't have the challenges of scale that we have and I think thats why its weird to see tech innovation in a restaurant. That said, we need it to scale and are grateful to the k8s community for making what we are working to do possible!

I enjoyed reading this post, but came away with more questions.

They talk about providing a service for developers to deploy stuff easily, but I'm wondering how that works in practise.

I've very little experience with Kubernetes, the author mentions they run 'mini' kubernetes clusters at each restaurant. Does that mean they have to deploy software/container updates to each restaurant one by one? Or is it abstracted above that level, where they can see ALL of the restaurants as one "big" cluster?

Hey! Caleb here... I'm the lead SRE building the clusters on the restaurants.

Each individual restaurant gets it's own cluster (we aren't fully deployed yet). There are too many network latency challenges and too much immaturity around federated clusters to take the route (unfortunately).

We currently use gitops, which houses all the configs per cluster in a single repo per restaurant (CRAZY amount of corgis!)... we call that Atlas (the repos) and we made a little pod called Vessel that polls and applies configs in those repos.

We're almost done building something called Fleet that will generated and manage those repos (Atlas) at scale. Ie; a UI where we can say "send this version to 10% of restaurants" and it will regenerate and deploy those configs to all the appropriate Atlas repos, which will then get pulled down by Vessel.

We tried doing all this with Helm but failed miserably. Maybe it was us? But templating vs gitops... the choice seemed obvious.

Wouldn't Spinnaker be suitable for something like this?

Caleb, we’re Armory, W17. This does indeed sound like an interesting use case for Spinnaker, as Jacques said. Happy to chat with you about it. DROdio@Armory.io (we are commercializing Spinnaker — http://www.Armory.io )

Your blog post turned up in slack today (congrats). My particular flavour is Concourse but I nodded all the way through.

What were the main issues you faced with Helm?

The primary challenge was just reasoning with the template and using Helm at scale... ie; what exactly did we deploy on those hundreds of varying clusters?

Other issues included; tiller would sometimes become unstable... version mismatch issues between helm local and roller... lack of a clear, outage free canary deployment... we even found cases where helm would not cleanup after itself during a deployment and retain previous config settings within k8s.

For me, Helm caused more problems than it solved. Pulling in packages always seems good, but as soon as you want customization, you're back to merging in the (relatively) straight-forward yaml files from the chart. Also, instead of Helm's templates (which get crazy complex and unintuitive), a simple tool like Kustomize[0] is very straight-forward and allows per-environment configuration. Finally, tiller pods do present (yet another) security risk for Kubernetes.

[0] https://github.com/kubernetes-sigs/kustomize

We should have a Medium post about this soon, but we built a set of tools called Fleet, Atlas, and Vessel. The simple premise is that Fleet generates code for 1..n clusters. By code I just mean yaml files. Atlas is a git repo per cluster that holds the declarative state for that cluster. At any time it should be able to be applied to the cluster and result in the ideal state. And then vessel runs in each cluster and does a (I am oversimplifying a bit) git pull and kubectl apply -f. We can use these tools to do canary deployments (generate x percent this way, y percent that way in ingress rules) across a wide range of clusters. The goal here is really to minimize configuration drift and keep a large number of fairly ubiquitous (but not always so) clusters in order.

They call their "cloud": Cloud-Fil-A.

Par excellence.

Hot standby is Cloud-Fil-B

We will have an order of magnitude more cloud datacenters than AWS! :)

Do you also have a CDN (Chicken Delivery Network)? (Stolen from a guy in a slack channel.)

Indeed they do... that's their core business.

My coworkers found this one quite amusing.

    Since we have a highly available Kubernetes
    cluster with data replicated across all
    nodes ..
This sounds like one of the more cutting-edge bare-metal kubernetes setups I've heard of!

Curious what the kubernetes implementation / configuration looks like for the setup they've described. Particularly in terms of HA-cluster k8s configuration, networking solutions integrated, and replicated storage.

Considering there is no AWS / GCE / Azure block-storage primitive to build on, it'd be cool to understand the choices and tradeoffs made in this scenario.

Is all data being stored on something like HDFS or CockroachDB? Or perhaps the k8s minion slave nodes launch containers with Ceph volumes.

Well, it’s a 6 man team so most of our trade offs were for expedience, not the worlds greatest architecture. We cheated and sync’d data using a HA MongoDB setup amongst the cluster. RKE for K8s clustering was a life saver (nothing easier at this point on bare metal IMO), although RKE can be brittle at times.

Hi Jay -- this is Brian (I wrote the article). There are constant tradeoffs. I am leery to say we are the most sophisticated bare metal setup. I think all of us that are close to the work would love for many parts to be much more sophisticated. That said, the way we handle distributed data right now is using Mongodb. We have a cluster that spans the three nodes that we have in a restaurant. The primary use case is persistence for our MQTT topics. Most of this data is what I will call "mildly ephemeral". It isn't needed for the long term, but we do `hope` to maintain it long enough to exfiltrate it to the cloud and deliver it to any necessary recipients. If we were to have a global cluster outage, it is not the end of the world. Over time, it probably will be. Back to work! :)

Sure, they have the biggest cloud in the US, but it's down on Sundays.

Kidding aside, do you guys do maintenance on Sundays or is that forbidden by company policy?

This is Brian -- I wrote the article. We are closed on Sunday but its not dogmatic in nature. We want to make sure that everyone at Chick-fil-A has a great work/life balance. Occasionally we need to do some work on a Sunday but we try to architect and engineer in a manner that supports business-hours-friendly deployments, or scheduled deployments over night to protect that. This is truly a first-class design consideration. In my 14+ years at Chick-fil-A, I have only worked a handful of Sundays for any reason. Its a tribute to leadership at all levels and a great company culture that values people (and their need for times to rest).


I've always found CFA employees to be kind, courteous, and respectful.

They’re really reasonable about it... you do what you need to do.

That being said, it’s really REALLY nice to mute pagerduty for a whole day every week.


I've always found CFA employees to be kind, courteous, and respectful.

That's irrelevant. Their work generates profits which are used to actively undermine my civil rights.

I happen to agree with the views they have stated in the past. Marriage (defined as being between one man and one woman) is one of the most foundational institutions in the world and is worth defending.

My marriage is just as valid as anyone else’s. And it’s incredily offensive to suggest that my marriage is somehow an attack on marriage itself. Your view is bigotry in an old, familiar form, and can only be defended based on religion, which has no role in limiting civil rights within our secular society.

Having worked with some of CFA's competition and comparing their IoT architectures, I'd say these guys have their heads on straight. Kudos to them.

I've never been to a retail establishment with anything close to the credit card speed of CFA. The cashier can swipe and grab receipt in one smooth and quick motion. It's so fast you feel like it's broken or slight of hand.

I wonder how they do it or why other companies can't or don't do it. CFA stores are so much busier than even an average McDonald's, maybe the efficiency gains are just worth more.

No idea how they handled the chip card shift.

From what I understand, merchants can do a quicker swipe (basically record the card # and transaction amount without waiting for authorization), but the merchant fees are slightly higher because of the possible higher fraud/chargeback rate. It's up to the chain to decide on what to do.

Perhaps CFA can do it even better since they could possibly keep a database of bad cards at the edge. Remember those books that cashiers used to thumb through to verify if a card was stolen?

Or keep a Bloom filter of hashes of repeat customers’ cards on the edge to accelerate their experiences.

The goal is to detect stolen/deactivated cards. Not sure how that helps.

If it's a repeat customer, at the same location, then you can avoid contacting the payment processor gateway, and just record the transaction on the local cluster for later payment processing. The expected value of the loss if someone stole their card, and used it at the same restaurant that they frequent as a regular, after the card was canceled, and the restaurant gave the thief a chicken sandwich, and now corporate needs to eat that cost because the asynchronous CC transaction doesn't go through when they finally run it... is absolutely miniscule.

The one place where CFA IT seems to fall down is at the drive-through.

I used to be able to pay for my order with the order taker on his/her iPad using my Apple Watch.

For the last six months or so, I have to go all the way to the pick-up window, and they have to get a manager to unplug a card swipe terminal from the front counter, plug it in near the window, log in to the terminal, and then hang the actual terminal out of the window so I can tap my watch.

Very strange. Hopefully it's only a problem at the one CFA in my geographic area.

I'd send that feedback through our feedback form... our product teams definitely need to hear that; https://www.chick-fil-a.com/Customer-Service/Contact

Costco is the fastest I've come across. Haven't tried CFA as I usually do drive-thru.

Most places have convoluted the whole card process. It's slower than cash now...

Same. Costco's card readers are super fast. They also just started accepting tap to pay (Apple pay) nationwide.

Petsmart was like the before the chip change over. It would typically take a few hundred MS to get approval and print the receipt.

I have noticed that some chipotle cashiers print the receipt before the card goes through, the CFA cashiers could be doing something similar.

To add another data point, the quick-accept only happens under a certain amount, like $25. Anything above that might trigger a signature.

No major issue requires a signature since April. Some places still ask for one because there was no real need to update software over it.

Thanks! We appreciate the compliment! :)

Sounds like it would be fun to be on your team, but my work lies on the other end of the equipment chain. I do OEM work for the fryers and grills, and boy could they use a kick in the pants from the chains to get their act together.

We think of our Edge Computing environment as a “micro private cloud”

We call local clouds "fogs."

Aren't most edge computing reffered to as fog-computing? Or is it just in academia

Hey this is Brian -- I wrote the article. I have heard the term a lot over the years. I think you could call it that -- the idea being its the "foggy" area between the user and the cloud. The reason we like "micro private cloud" is that 1) its a tiny environment with limited cloud properties (no elastic scale) but 2) we do try and provide a platform with cloud-like services for our internal developers, and we like the cloud paradigms being applied to how we think about compute at the edge, in restaurants.

Interesting - why? (this is Caleb, one of the authors and lead SRE for the project)

Interesting - why?

Because more than one person on the team has a background in meteorology, and to them a localized cloud event is "fog." I think it started as a joke, but it just kind of stuck.

It helps that our events that require localized cloud infrastructure are sporadic, just the way real fog is. And just like real fog, we don't always know where it will be. Once or twice a year we have to deploy about 150 devices to a different location in the desert. Then as quickly as we arrived, we pack up and move on.

That is epic, and i am stealing that name.

What would "smogs" be considered in the local cloud computing world?

This is a pretty cool example of cybernetics ala Stafford Beer - constant information and reaction to streamline production. I'm excited to see this be brought in to more production (though maybe it has, and most places aren't as upfront as this). It's truly a shame the Chilean experiment in the early 70s didn't work out.

On a side note, it's interesting that Kubernetes is the word from which cybernetics is derived, meaning "governance".

with articles like this and Mac'd from this YC cohort it seems like an increasingly interesting time in the restaurant/technology space.

Hey this is Brian -- I wrote the article. It really is an interesting time, especially when you have the scale that Chick-fil-A has. While traditionally a fairly archaic industry (POS registers, ERP at corp, thats about it), the volumes and challenges of scale have created many challenges that solid engineering can help solve.

the domain is interesting, but I worry about the long-term effects of maximizing efficiencies (read: automation) forpeople who rely on jobs in that industry.

The whole debate on if automation is a good thing or not has always been pointless to me.

It's like trying to un-discover nuclear weapons, once it's out there you can't go back. Things are going to get more efficient, more automated, and those jobs will be lost. It's not a matter of "if", but "when".

What we need to do is to figure out ways to cope with the change. Find tasks that these workers can do that allows them to earn a living, experiment with universal income to maybe one day not need everyone to work, fund education programs so these people can learn to do something different, provide some safety nets so that these people have time to cope with the changes.

We need to adapt as society, not pretend that this technology doesn't exist.

> What we need to do is to figure out ways to cope with the change. Find tasks that these workers can do that allows them to earn a living, experiment with universal income to maybe one day not need everyone to work, fund education programs so these people can learn to do something different, provide some safety nets so that these people have time to cope with the changes

yes to this.

A myth about manufacturing jobs is that they all went overseas. Reality is that most of them were replaced by robots and automation.

Low skill, low knowledge, repetitive work is key target for automation.

Fast food workers wouldn't have large downstream effects. Automated truck driving would though. There are likely entire towns right off of highways that depend on humans passing through daily - restaurants, repair shops, etc. If automated trucks don't stop anymore, a town could literally be a ghost town.

Hey this is Brian -- I wrote the article. I do want to add that Chick-fil-A isnt looking to replace jobs with "robots". We really want to assign our restaurant team members to tasks that add maximum value to customers, which are generally service oriented (iPad in Drive Thru, personal service in the dining room, etc). Our automation will help our team members be more effecient and enjoy their jobs more rather than replace them.

Of all places I doubt CFA would replace workers, having nice, helpful people is the second awesome-est thing there (chicken and waffle fries tops that).

I can see the automation and analytics helping staff appropriately and offer more options for efficiency and waste reduction.

I think you hit the bullseye here, because the tasks you highlighted simply aren't done at most other QSRs. They are novel ways to apply human labor to fast food (at least, novel within their competitive space).

> A myth about manufacturing jobs is that they all went overseas.

Partially true, many of them went to Mexico. Import of partially completed goods, as well as automation of assembly, continue to drive overall manufacturing output in the USA. But many jobs have been exported. A recent high-profile example: http://www.nydailynews.com/news/national/carrier-plant-movin... and a study discussing overall impact to jobs: https://www.epi.org/publication/heading_south_u-s-mexico_tra...

I may be alone on this one, but if Chick Fil A automated most of their workers, I would stop eating there. Part of the reason I go there is I am always impressed at just how good they are at customer service. There isn't another fast food restaurant that even comes close that I know of.

I hate talking to machines, I hate talking to automated voice menus instead of getting actual assistance from a human being. I hate ordering stuff online and dealing with slight one off cases. The google Ai assistant had me cringing for weeks. I especially have a deep hatred for self checkout lines. They can all burn in hell.

Why? I think that they move on to another service industry job such as customer service or something. I don't think working in a fast food restaurant is building the sort of skills that making a horizontal jump to another industry is outlandish.

I think that they move on to another service industry job such as customer service or something.

That's just something people think so that they don't have to feel empathy for people who lose their jobs to automation. It's kind of an internal excuse.

The reality is that not everyone is re-trainable. And if someone could easily get a job in customer service, or some other marginally better situation, they already wouldn't be working at a fast food joint.

In addition, many fast-food jobs are temporary/seasonal. A high school kid can get a six-month stint at MacDowell's to pay for car insurance over the summer. A call center isn't going to take him in and train him knowing that he's going to bolt come September.

Recently, I was contemplating something regarding customer service jobs and automation. I was at a fintech data conference listening to the work being put in to analyze customer support call log history to understand where prospects are in a pipeline or improve retention.

If a system can pinpoint a prospect is close to onboarding, and customer service rep is a great closer, the software could pass the prospect off to the "closer". The same could be said for keeping a customer who is on the fence from leaving. All of the sudden, a highly skilled closer has data and metrics attributable to the company's bottom line, and in theory, someone like this should become a high priced asset.

In some instances, could automation actually greatly increase the salary dispersion in certain low wage fields?

> I think that they move on to another service industry job such as customer service or something

Right, they can compete with all the other displaced workers for the #1 position most likely to be outsourced.

Hell, many Targets don't even have a customer service desk anymore. They demolish them and assign the workload of both sales and returns to a random cashier.

A lot of people are not qualified for skilled work and will never transcend entry-level fast-food jobs. If you take away those tasks that keep them productively occupied, it might be a good time to start buying stock in industries related to prison management.

There are certainly negative consequences of automation, and many existing jobs will disappear.

A key feature of the times we live in is that knowledge is ubiquitous. If an idea becomes known and understood to be feasible, and it serves a useful purpose, you can't hide it from existence. If your restaurant doesn't capitalize on these tools to improve efficiency, another one likely will. This might mean that your restaurant gets be out-competed by a better product elsewhere AND your workers will be out of a job.

Trying to freeze a system that works at a given time is becoming more and more difficult as we advance our knowledge and technology. This paradigm of rapid evolution has been playing out since the industrial revolution, and seems to be accelerating with time.

Maybe it ends badly for us in the macro sense, we'll see. I feel for the folks that can't evolve and get left behind, but I'm not sure how to help them. Resisting the change probably won't work.

We would probably be okay, if it wasn't because we are also looking at soonish fully automatic trucks, which will kill all the jobs in the trucking industry, plus all the jobs at the trucking stations, mechanics etc, plus the retail collapse.

I guess if you aren't in one of those areas, and you have a business that hires a lot of low income people, times will be good, but like you I don't think the rest of the market can take up the slack; there are only so many people who are willing to pay for laundering.

From other retail IT I've been involved with, these guys could make an incredibly successful side business doing this to pretty much any retailer.

However when I see that retail stores are using dialup to do batch uploads of inventory and transactions I wonder if their central IT systems would be remotely ready for something like this.

You hit on a great point -- Edge infrastructure buffers outbound internet traffic but does not replace the need for a good circuit.

Pretty interesting, but good lord I've never seen such buzzword-density in an article that's making real sense

This is Brian and I wrote the article, and I am taking this as a massive compliment ;)

Interesting that they use Kubernetes exclusively for edge computing resources (as opposed to their cloud architecture). Never really considered this approach before.

If you decide to play around with it, we use RKE to build and manage the k8s environments locally.

We use K8s in the cloud as well! Our entire control plane for the IOT activity runs there. The rest of the engineering teams have a mix of solutions, but many are moving to K8s as well.

When it comes to Chick-fil-A, there's also the regressive side in addition to the modern and traditional sides.


Your comment is essentially "the owners hold certain values I disagree with regarding sexuality, therefore I'm surprised their technical team is any good."

I don't see how that follows.

Political values are important to point out. Companies create these tech blogs as a recruitment vehicle. You should know if the company you're considering applying to is funding bigotry (or "wholesome family values" if that's your angle).

Like IBM or Facebook?

..who get ripped on HN for every reason possible as well (and rightfully so).

> 'Bigotry'. noun : intolerance towards those who hold different opinions from oneself.

Surely castigating the owner of Chick-fil-A for advocating his personal beliefs is the truest example of bigotry here.

There is no world in which pointing out the political beliefs espoused by a corporation should be considered an example of bigotry.

Of course neither of those things is actually bigotry. This "oh I'm suffering" crap is disgusting. If we actually cared about suffering our freely elected government wouldn't be causing so much of it.

Euphemisms like "advocating for his personal beliefs" wouldn't fly, if say, the owner was donating to organizations that fought to make interracial marriage illegal again.

We'd call that what it is: bigotry.

I love seeing this argument. It's like you people think you can raise any arbitrary instance of hate/intolerance/bigotry onto a criticism-proof pedestal simply by brandishing a dictionary. Like, who are the real bigots: Nazis, or the nasty people who have the gall to disagree with their racist, antisemitic, etc. views? Is the answer not obvious? Does the question send you scuttling back under your dictionary?

I'm fine with a bit of pedantry but ultimately in the real world we need to show a little more intellectual maturity.

You're missing the point. I'm not concerned about the word. I'm concerned about the definition - those quickest to cry bigot are often guilty of acting like one themselves.

Case-in-point: Twitter CEO Jack Dorsey being forced to apologise for eating at Chick-fil-A during LGBT Month. What kind of bizarre intolerant nonsense is that?

Another case-in-point: Boston and Chicago mayors claim "Chick-fil-A values are not [our] values" and refuse to let them open branches in those cities.


> I'm not concerned about the word. I'm concerned about the definition

Come again?

It seems like you're the one who's missing the point, anyway. You didn't address anything I wrote in my comment. Par for the course, I guess. Just imagine I wrote the exact same comment again, and reply to that if you feel like it.

The paradox of intolerance[1]: being tolerant of those who are intolerant leads to less tolerance. It is completely right to call out intolerant views and actions and work towards eliminating them.

[1] https://www.wikiwand.com/en/Paradox_of_tolerance

The obvious problems with 'the paradox of intolerance' being that anyone who can convince themselves of another's intolerance now has moral license to behave intolerantly to an unspecified degree. Of course, exactly this sort of thing happens all the time; notably, it was used to rationalize Google's firing of James Damore--Damore's memo allegedly contained "dogwhistles" so he deserved being fired, slandered by the media, and mobbed on social media.

If it wasn't problematic it probably wouldn't be called a paradox.

I'm inclined to divide intolerances into two types: intolerances based on what somebody thinks, and intolerance based on what somebody is. I'm further inclined to model intolerances as a directed graph where nodes represent a group of one of those two types, e.g. conservatives and men might be represented by individual nodes of those two types respectively. Edges represent an intolerance, e.g. there would be an edge pointing from Klan members to black people. The former node would be of the first type, the latter of the second.

Where there's a cycle in this graph we are seeing a mutual intolerance of people with different ideas. An example might be people who think the Earth is flat and people who think it is round. Who is right? It's impossible to say without diving into the issues themselves.

    flat earthers <=> round earthers
Because what you think is independent of what you are, nodes of the second type cannot have outgoing edges. Incoming edges represent base intolerances of the sort that have historically been very harmful: racism, sexism, homophobia, and so on. Obviously these edges represent a different type of intolerance because they are based on innate attributes that their targets have no control over. To a liberal person, they are essentially definitionally bad. To an illiberal person, they're fine. Great, even, depending on who you ask.

Note that I'm talking about liberalism in the general definitional sense, not in the modern U.S. politics sense (i.e. "the left").

So, how about this graph taken from real life (the <=> represents two edges, one pointing in each direction):

    anti-racists <=> white racists -> non-whites
Deletion of the rightmost edge (i.e. deletion of white racism) means that we also have to delete 'white racists' or the implied meaning of the graph is paradoxical. It then follows that we have to remove 'anti-racists'. Now we have no intolerance at all—all that's left in the graph is `non-whites`, enjoying their existence without harassment, and all it took was the elimination of racism. You could also remove the 'white racists' node, with similar meaning and identical effect.

However, deletion of 'anti-racists' leaves the racism intact. I'd argue that this form of graph appears in many places where there is an illiberal group intolerant of a type two group and a liberal group locked in mutual intolerance with the illiberal group. The inverse setup doesn't really appear because, as I noted before, edges pointing to type two nodes represent a sort of intolerance that is anathema to liberals.

I'd also argue that there is a strong implication that these sorts of edges are the most important to remove—because, as demonstrated, doing so causes the rest of the graph to collapse. They are the "root cause", if you will, of many webs of interconnected intolerances.

It should be obvious that this model denies the assertion made by many modern American 'conservatives' that intolerances are all equivalent and relative, "the tolerant left", etcetera. I would say the water is most muddied by the fact that our 'left' and 'right' do not map cleanly onto liberal and illiberal. It's easy and sexy—but invalid—to apply the same political label to someone who thinks gay couples should be able to enjoy the same legal benefits as heterosexual married couples, and someone who thinks all white people should be killed.

P.S. Damore is a moron and his pseudoscientific screed was legitimately offensive to his co-workers. That's fine justification for him to be fired from his job at a private company. As for the rest of it, we live in a culture of growing illiberalism and zeal for generating outrage. It's a bummer.

I think you missed the point of my post, which was that this paradox allows anyone to abuse anyone else if the abuser can convince themselves that the abused is really 'intolerant'. As in the Damore case, a certain political group didn't like his message, so they claimed it contained hidden white/male supremacy messages, which was sufficient to justify his termination and abuse (contrary to your ad hominem post-script); the mob patted themselves on the back citing this very paradox.

I think you missed the point of mine. Doesn't matter.

A lot of people made a lot of politicized fuss about the Damore thing, but the simple fact is that his document contained offensive generalizations about women and if I was a woman I would not want to work with him. He's free to say whatever he wants; as I said before, though, this is a fine justification for termination.

A lot of people from all sides of the spectrum made it a political issue, because of course they did. The left said Google didn't act quickly or harshly enough and the right acted like Damore's right to speak freely was being taken from him. I'm not one of those people and I'm not interested in discussing the issue in terms of 'ideologies'. Ultimately, as I've said before, what he did does not have to be viewed through a political lens for it to be a legitimate justification for termination.

edit: I said "if I was a woman I would not want to work with him." Actually I would not want to work with him regardless.

> I think you missed the point of mine.

I may have. It sounded like you were highlighting the paradox itself, while I was intending to highlight the way the paradox lends itself to abuse. Perhaps ironically, your comments about Damore and the memo are exemplary--the memo contains zero offensive generalizations; it's been picked over and no one has produced an 'offensive generalization'--only phrases that allegedly contain offensive hidden messages. And because of those hidden messages, the mob pats itself on the back for helping to suppress Damore's "intolerance" (by the way, the same mob was rationalizing the New York Times hiring Jeong despite her four-year montage of overtly racist tweets). Again, I don't want to focus too much on any one example, because there are so many others and I don't want to steal attention from the broader phenomenon.

> I may have. It sounded like you were highlighting the paradox itself, while I was intending to highlight the way the paradox lends itself to abuse.

My intent was to illustrate that not all intolerances are only relative degrees of the same thing (as claimed in the popular 'conservative' talking point); some are categorically distinct, and in a web of interconnected intolerances the two types that I identified interact in asymmetrical ways.

I mostly just wanted to write down an idea that occurred to me on my walk home from work. I don't expect something that long to play well in an internet forum. It's meant as an interesting abstraction, not an airtight position.

> Perhaps ironically, your comments about Damore and the memo are exemplary--the memo contains zero offensive generalizations; it's been picked over and no one has produced an 'offensive generalization'--only phrases that allegedly contain offensive hidden messages. And because of those hidden messages, the mob pats itself on the back for helping to suppress Damore's "intolerance" (by the way, the same mob was rationalizing the New York Times hiring Jeong despite her four-year montage of overtly racist tweets). Again, I don't want to focus too much on any one example, because there are so many others and I don't want to steal attention from the broader phenomenon.

Have you read this: https://blog.ycombinator.com/ask-a-female-engineer-thoughts-...

It's quite literally the only piece of coverage I've seen of the Damore saga that I mostly agree with. Otherwise I think I've said what I have to say about him and his document. If you want to project politics onto me, I can't stop you, but I don't think it's productive. My opinions on this issue aren't political.

> My intent was to illustrate that not all intolerances are only relative degrees of the same thing

Ah, I see. I agree, though I wasn't espousing the "popular conservative talking point" (which I've never actually heard before), so I'm not quite sure why the conversation took that particular turn.

> Have you read this

I have, but I don't really want to dissect that here and now.

> If you want to project politics onto me

I don't want to, and I don't think I did. :) Sincere apologies if I offended.

Your very own link lists other major philosophers and thinkers (including Thomas Jefferson) who disagree with Karl Popper's conclusion on paradox of intolerance! There is _no_ universally agreed upon answer to the paradox of intolerance; I do not know why people keep pretending that there is.

(I will also observe that, where the article notes: "Popper asserted that to allow freedom of speech to those who would use it to eliminate the very principle upon which they rely is paradoxical.", it clearly applies to those who wish to (quoting from your post) "...call out intolerant views and actions and work towards eliminating them".)

That's very intolerant of you to cry bigot about people who support LGBT rights.

Totally bizarre intolerant nonsense.

No, not really, since his personal beliefs involve advocating for others to suffer.

He suffers nothing by permitting LGBT people to live as they wish.

You should probably not rely on Google's fairly crappy dictionary definitions.

The quoted definition of bigotry as "intolerance toward those who hold different opinions from oneself" is from the New Oxford American Dictionary, from Oxford University Press (the OED people). If that's what Google is using, I don't think I can rely on your definition of "crappy."

Yep, it's a crappy definition that does not reflect current usage. It doesn't even reflect the definition in the OED, if you want to check there. 'Bigotry' is not limited to difference of opinion. Nobody uses it like that, you can take a look at the quoted usage examples here, which are from current publications.


Ahh, yes. Always useful to continuously cherry-pick or alter definitions to tailor words that have an objectively negative moral implication to apply to exactly the standards you want.

Just because some publications use the word in that way does not mean the definition of the word should be shifted.

I’m sorry but this is a pointedly stupid comment. I’m doing the exact opposite of cherry-picking - I’m saying the definition is much broader than the inaccurately narrow one posted.

The OED is a historical dictionary; the Oxford Dictionary of English and the New Oxford American Dictionary are explicitly meant to reflect more current usage. At any rate, I think this is getting awfully pedantic; the definition you linked is "obstinate or intolerant devotion to one's own opinions and prejudices," which seems similar enough that I'm not really sure what the point of this argument is. Given that I'm fairly sure the point of this was whether Chick-Fil-A's executives and in-the-company's-name giving exhibits bigotry, by any dictionary anyone's linked to, the answer is "yes."

The point is that the definition posted is narrow and bad. This isn't some obscure technical point, a debate about the proper usage of 'less' and 'fewer' or whether 'literally' literally means what it says in some odd case. All you have to do to convince yourself of its badness is read a newspaper or magazine or listen to a news report. What exactly would 'racial bigotry' mean if the posted thing was the only definition of 'bigotry'?

As to pedantry, you're the one who started arguing this clearly lousy definition is not lousy, brought up the OED, decided the OED is no good, finally acknowledged the existence of a broader definition but then somehow decided that it's, in fact, as good as identical to the narrow bad definition. I'll be the first to acknowledge I have a habit of getting involved in some deeply silly threads but I'm left utterly flummoxed by this Cirque-du Soleil-level display of mental gymnastics.

I just hope the authors of that lousy definition aren't holding your cat hostage.


Bigotry does not just mean 'intolerance towards those who hold different opinions from oneself.' It's a silly definition, not what I want it to mean.

Language does not operate on the basis of what the speaker _wants_ it to mean.

I don't understand what ghost windmill you are tilting at here. At all. It certainly doesn't seem to be anything I've actually said.

I have a pretty personal beef with the idea that language can simply be contorted to supposedly mean whatever someone feels like it should mean, as heading down that road is basically abandoning meaning and communication altogether, and is wholly incoherent.

Yeah but... I didn't say any of that.

> It's a silly definition, not what I want it to mean.

You wrote this intending to say, "I'm saying the dictionary definition is silly, not that bigotry should mean what I want it to mean."

Instead, the sentence says, "The dictionary definition is silly, [because it's] not what I want it to mean."

That's fair, I likely misinterpreted it that way.

No, that was not what he said. He said to keep in mind the opinions of the company owners, which I imagine encourages one to not purchase from them.

I do not see how one can claim a criticism of their engineering follows from his statement.

For those coming to the thread later: the parent comment pointed out a certain fact[1] about the corporate activities of Chick-Fil-A that doesn't directly relate to the contents of the article.

The basis was the observation that such articles are,in essence, a "work-for-us" ads, and in that vein, informing the readers of other, broader actions of a corporation is relevant, especially when it pertains to ethics.


The poster said nothing of the sort.

To make the point more directly for you, I think and I’m guessing the poster thinks we ought not to normalize certain kinds of discriminatory corporate behavior by just looking away and cheerfully admiring their oh-so-efficient tech stack.

I never made any statement conflating views of ownership with technical team skills. That's absurd.

This whole post and thread are an effort to recruit engineers and accumulate good will within the HN community. Pointing out that the company is rotten is extremely important, as is anything else that can undermine the project of manipulating talented, powerful hackers into working on the project of generating profits that power an evil social agenda.

Hey, this is Brian and I wrote the article. I have worked at Chick-fil-A for over 17 years. I will volunteer time to take a phone call with anyone that is interested in hearing my first-hand experience with Chick-fil-A and its culture over the years. Just reach out on LinkedIn if you are interested and we will schedule some time.

I'm sorry, but whatever positive experience you've had as a member of the organization does not simply negate the negative effect the ownership has on our society.

Came here for this.

Imagine being so dead inside you feel perfectly fine, even proud of working for Chic-fil-A, lol

This was not helpful. Sometimes people take jobs just to survive; I know I have.

Yeah, me too, but at least I wasn't writing Medium articles about how cool it was that I was generating revenue to be funneled directly into conversion therapy youth camps.

This person[0] seems to have found a way of engaging the CEO in dialog. I, personally, think this is the best way of discussing differences.

[0] https://www.huffingtonpost.com/shane-l-windmeyer/dan-cathy-c...

We also have an open dialog with Kim Jong-un; doesn't mean we should applaud the people that work and support him or feel sorry for those workers. They're traitors to the human race, in just the same way that Chick-Fil-A workers are directly responsible for what the company does.

Really? You're equating the "workers" of CFA to Kim Jong-un?

Nothing changes with attitudes like that.

No I'm equating the CEO Dan T. Cathy to Kim Jong-un. This guy's a scumbag who loves nothing better than commanding the world to follow his hypocritical commands, and his staff are personally responsible for supporting this behavior by going along with it. I can't say I have any respect for him, his company or the people who work for him. The US should simply outlaw his antisocial policies because that's the right thing to do.

HN loves to talk about how CS needs ethics courses - this is a great example :|

If you're going to complain about the ethics related to Chik-Fil-A, it would probably be better to start with the fact that they raise and kill a billion chickens every year rather than that they have donated $120,000 to organizations like the Salvation Army (example from above article).

Eh, regardless of my personal thoughts on the topic, the ethics of eating animals are a lot more controversial among the HN crowd than LGBTQ rights.

Chik-Fil-A is not an arbiter of LGBTQ rights. Their small 6-figure donations to organizations that focus on other issues do very little in the grand scheme of things.

Do you really think Chik-Fil-A is directly causing more suffering by donating to a few charities in amounts this small than by raising in captivity and then killing billions of chickens? I see no way to reasonably argue that.

Chick-Fil-A is just the service provider, if not for Chick-Fil-A people would get their chicken elsewhere. If they were raised in a place that felt less like captivity, that would probably be worse for the environment, in terms of using more natural resources per pound of chicken produced. There are trade-offs to everything.

Chick-Fil-A isn't really in the meat business, they're in the restaurant business, and given that they aren't in a very tiny niche, their menus feature meat. They sell plenty of soft drinks and fried potatoes which reduce the amount of meat as a percent of the total calories.

Finally if you're going to get people excited about animal rights, it's probably better to mention cows than chickens. To me, Chick-Fil-A's ads suggest eating chicken as a lesser of two evils, without ever saying that eating chicken is evil. We also don't have a separate word for chicken meat like we do for cow or pig meat, which I think serve as euphemisms.

You may well already know this, but the separate words for meat and animal are likely because of the Normans; the richer French speakers cooked and ate the animals, so they got French names; the poorer "English" raised the animals and called them by their English/Germanic names.

We do have a word with a French root for chicken, by the way: poultry. As to why the separate name didn't catch on, I can only speculate that it's because chicken are cheap to keep and so the English may have eaten them themselves.

Not all languages have separate words for meat - German perhaps being the most famous. It's slightly disconcerting to walk around a supermarket and see schweinefleisch being advertised.

> Finally if you're going to get people excited about animal rights, it's probably better to mention cows than chickens. To me, Chick-Fil-A's ads suggest eating chicken as a lesser of two evils, without ever saying that eating chicken is evil.

I posted this article to our company #random channel and that was the first reply- better chicken than beef!

Most effective altruists would agree with you (and thus disagree with a possible takeaway from Chik-Fil-A's advertising) that eating chicken is better than cows, as cows provide significantly more meat than a chicken, with ratios being up to 50 chickens per every 1 cow, as far as quantity of meat for normal portions go. In other words, someone eating only chicken may kill X chickens, but someone eating only cows may kill 1/50X cows. Of course there are other variables involved like weighing the differences between these animals, their treatments, lifespans, etc., but it is nonetheless a point with notable weight. Although this isn't quite the area I was getting at, just that the most significant things should be looked at first, all else constant.

pretty sure the ads are just a joke to not eat hamburgers, but to eat chicken whether it be nuggets or sandwiches or whatever which is essentially the menu for Chick-fil-A.

The "other issues" that organizations like Focus on the Family are concerned with are just a reprehensible as their anti-LGBT bigotry.

Again, regardless of my personal beliefs:

If I thought Chick-Fil-A shouldn't be supported for either their treatment of animals or for their anti-LGBTQ corporate giving, and wanted other people to stop supporting the business, I would focus on the latter in my messaging, because I suspect it would be more effective on the audience. Attempting to draw conclusions about which of their behaviors I personally think cause more suffering from this is not a good use of your time.

If your audience cares significantly about the charitable donations and not at all for animal suffering, and your goal is to only to convince them to be against the company, then sure, go for it.

But any effective altruist should have their goals set higher than this, at actually reducing the insane amount of suffering that's created. It does very little good if everyone stops supporting Chik-Fil-A only to flock to another chicken company, which will be the same in almost every way, possibly minus some donations, possibly one which treats animals even worse. Effective altruism is a very different beast than simply appealing to or signaling the crowds' most popular views on ethics in order to persuade.

At least we aren't quite to the point of HN articles praising the cutting edge technology being implemented at Raytheon and Northrop Grumman

Not... really...

Chik-fil-a is only a useful example in so far as how well the ethical throughlines match things we've seen many times before, the media has taught us to care about, and how much time and effort political interest groups have spent highlighting those to the public. Ethical thinking on this topic will not be engaged, people will take the opinion their political affiliation and chosen memes have predestined them to take, and they'll act like it's serious thinking.

Parent poster said "Imagine being so dead inside you feel perfectly fine, even proud of working for Chic-fil-A, lol." Substitute in Facebook, Google, Apple, Microsoft or Twitter, and you might be approaching an interesting statement.

Where the ethics gets engaging are the day-to-day business model at any currently big or hot software company. Only a small subset of people have the background to comprehend the issues. And the people saying "let's stop and think about the consequences of this" gets steamrolled by the status quo that has money to make. That's where all the good examples live.

But it's a bit harder to come to terms with when the focus hits close to home. So we have whipping boys like Chik-fil-a. "At least I don't work THERE"

I agree that there are many important ethical questions that are bypassed in the day-to-day of those companies for sure, but at the same time I think there is an attitude here of "who cares what happens with the company's money once it leaves my area, they are solving interesting problems and the people working for them are doing intellectually engaging work".

Imagine feeling the bizarre compulsion to turn every discussion about technology on HN into yet another tedious political argument.

The company themselves chose this political stance, not the community around them.

No, the grandparent's political stance is to be outraged at anybody he/she disagrees with and has an unhealthy obsession to blurt it out on discussions that aren't political.

Why is their stance deemed as political and not humanitarian?

All discussions are political. The only question is whether or not the politics at hand affect you.

And to quote Chick-fil-a's CEO: "Jesus had a lot of things to say about people who work and live in the business community … Our work should be an act of worship. Our work should be our mission field." So he sees everything they do as political as well.

Not political. Religious. For those of us who are Christians, our politics are driven by our religion, not the other way around.

The Heidelberg Catechism sums it up well:

> Q. What is your only comfort in life and in death?

> A. That I am not my own, but belong — body and soul, in life and in death — to my faithful Savior, Jesus Christ.


HN won't let me reply to your next comment, but regarding:

> a reflection of their religious principles or their political ones

This is actually a common misunderstanding and what I'm trying to explain. "Religious principles OR political principles" means there are actually (at least) TWO categories of ideas a person holds, with specific items (homosexuality, death penalty, etc.) falling inside one of those categories.

That's a secular worldview, and it isn't how Christians think. We have one category: God. God is sovereign, he owns everything including me. As Thomas Aquinas said "all truth is God's truth". When we're being consistent (key caveat), we approach all issues as "what does God want me to do?". Political activities follow from there. We often disagree about those secondary conclusions (what God wants from us regarding issue X), but not the core starting point (ultimate allegiance is to God alone).

> This is actually a common misunderstanding and what I'm trying to explain. "Religious principles OR political principles" means there are actually (at least) TWO categories of ideas a person holds, with specific items (homosexuality, death penalty, etc.) falling inside one of those categories.

> That's a secular worldview, and it isn't how Christians think

And yet literally the previous comment you say

> Not political. Religious.

So which is it?

I was responding to scope of the statement:

> So he sees everything they do as political as well.

Obviously we have political views, but they're a only subset of religious views. By analogy, geometry exists and is a legitimate field of study. But if you said "all mathematicians think about is geometry" that would be false because it's too narrow in scope. If he's consistent in his Christian worldview "he sees everything they do as religious as well" would have been true.

Was funding a lobbying group to reinstitute the death penalty for homosexuality in Uganda a reflection of their religious principles or their political ones?

There's nothing Christian about the board at Chick-Fil-A

> All discussions are political.

Nonsense. A certain subset of our society insists on making everything political.

Most of us don’t like it because it ruins otherwise interesting conversations.

Or... your privilege allows you to ignore the political consequences of most conversations.

What do you think politics is? Like an actual definition.

> Or... your privilege allows you to ignore the political consequences of most conversations.

What political consequences are there to most conversations?

Once again, define 'politics'.

Guidelines | FAQ | Support | API | Security | Lists | Bookmarklet | Legal | Apply to YC | Contact