
Edge Computing at Chick-fil-A - ayanai
https://medium.com/@cfatechblog/edge-computing-at-chick-fil-a-7d67242675e2
======
max_im
I work in midtown NYC near a Chick-fil-a location. Used to be a line down the
block, went with a friend yesterday thinking we caught the line at a short
time but turns out they were using smiley employees with ipads coordinating
orders and that they were serving arguably more people than before. Very
impressed with how seamlessly they've integrated tech in their brick and
mortars in contrast with the countless "cashless" restaurants that just add
cruft, inefficiency, and customer resentment.

~~~
beckler
I have a friend who used to be a shift manager at Chick-fil-A. I can't
remember exactly what he called that process... It was like meet and greet or
walk and talk, but it's surprisingly efficient for the drive thru. He said it
cuts down the average wait time for customers around 15% at his location.

I don't know what it would be like at NYC since it's not a drive thru.

~~~
raiflip
It's crazy efficient. When you walk in someone with a tablet takes your order,
and by the time you get to the cash register your food is ready. Works so well
because the employees are able to catch customers at the end of the line.

~~~
burgreblast
Here's "Line busting" the drive thru 68+ cars deep. It's astonishing the WiFi
goes out that far.

[https://www.youtube.com/watch?v=TIQWyYQ0EYg](https://www.youtube.com/watch?v=TIQWyYQ0EYg)

FWIW, also an edge approach, with Prometheus etc.

~~~
donkeyd
Slightly off-topic, but what's the reason that Americans all tend to line-up
at the drive thru? I was in LA, there was a huge line at the drive thru, so we
parked, went inside and got our orders really quickly.

~~~
always_good
The comfort of still being in your own space. For example, listening to your
own music, no excursion for your kids to belabor, precisely don't have to park
and get out.

~~~
flamtap
Yeah, I think I would rather wait 10 minutes in my car than 5 minutes in line.
I'm sure other people may feel differently. And something about the drive-thru
still "feels" faster, even if it's not. I think it's just the idea that you
get to pull out and be on your way after you've gotten your food.

------
timdorr
If you want to see the actual hardware used at the restaurant, check out the
previous blog post: [https://medium.com/@cfatechblog/bare-
metal-k8s-clustering-at...](https://medium.com/@cfatechblog/bare-
metal-k8s-clustering-at-chick-fil-a-scale-7b0607bd3541)

Spoiler: It's a stack of Intel NUCs.

~~~
ocdtrekkie
NUCs are surprisingly well-built little machines. I have one in my car, and I
give them my absolute seal of approval. I've heard of people running little
VMware clusters on them too.

~~~
086421357909764
It's actually suprising that people think they're not "normal" computers. I
provisioned a few of them and they're little beasts. Most in use are I7,
16-32gb ram, Dual NVME. The Skulltrail nucs are extremely powerfull and on the
lower end the atom based units are solid too. I also ran a nuc in my car for a
while, mostly stumbling radio / wifi spectrum as I drove around, but it was
perfect with its relatively low power requirements.

~~~
uncreativebrian
This is Brian -- I wrote the article. Super cool use for a NUC! I'm going to
follow the links and check out what you did some more.

------
thomasdelteil
On a related note, Chick-fil-A using Deep Learning models for object detection
with MXNet to track how long fries have been waiting:
[https://www.youtube.com/watch?v=3Uuq_cX8b1M](https://www.youtube.com/watch?v=3Uuq_cX8b1M)

~~~
TheAceOfHearts
This seems like such an over-engineered solution.

It would've been much simpler to have a plate with multiple pressure sensors
and a multi-color LED per spot. When you set the fries down on the sensor it
activates and the LED turns green. After X amount of time the LED can turn
yellow, meaning that it's becoming stale. Finally, after X+N time has passed
and it's no longer fresh it can turn red. Removing the fries turns it off.
Aside from being way cheaper, I'd conjecture that this would be much more
reliable. I'm pretty sure I could get a prototype of this up and running over
the course of a weekend or two.

If you wanted to get really fancy I guess you could also track the room
temperature and moisture levels and use that to get a better guess of how long
a group of fries will remain fresh. Although I don't know if environmental
factors like these have enough impact on fry freshness to be worth taking into
consideration.

Anyway, it looks like they were just doing this for fun and learning, so I
guess it doesn't really matter.

~~~
uncreativebrian
This was a good learning experience for our engineers at our Innovation Center
at Georgia Tech in Atlanta. It may bear fruit in the form of a useful solution
in the future, though. What is unique to Chick-fil-A is 'volume'. We do a lot
of sales in our restaurants, so anything we can do to try and make our team
members lives easier is important to us. We want them to enjoy their jobs and
we want to do the best we can to consistently create high quality food
experiences for our customers. Our teams in restaurants are the heroes, but we
are trying to use technology to help them do what they do. <thumbs up>

------
ryanmarsh
Would just like to say Chick-fil-A should publish more like this. They are
absolutely knocking it out of the park with tech. My wife and I are friends
with a few operators (they don't have "owners"). I recall them, several years
back, trying to get me to quit consulting to come work with them and open more
stores. "Food service, ugghhh, no offense" I said. A friend whipped out her
phone and showed me their really great in-house app that showed detailed
analytics for her store (everything you could imagine) and tools for for doing
purchasing and much more we didn't have time to get into. This is circa 2010
if I recall. For reference the iPhone SDK dropped in late 2007 and these folks
were already way onboard with mobile. I was impressed.

------
Kagerjay
Unrelated but I've done a good amount of nonprofit work and everytime I've had
to ask for food donations chickfila has given food free of charge.

~~~
stochastic_monk
I see people like this the way I see my Mormon family. They usually try to be
very kind and I value that.

At the same time, they harbor beliefs which are incredibly damaging to both me
and those I love, so I keep a healthy distance while understanding that they
mean well.

Don’t mistake their kind acts for open-mindedness or acceptance, though.

~~~
dbrueck
Out of curiosity, in what way is Chick-fil-a not open-minded?

~~~
hogu
[https://www.forbes.com/sites/clareoconnor/2014/03/19/chick-f...](https://www.forbes.com/sites/clareoconnor/2014/03/19/chick-
fil-a-ceo-cathy-gay-marriage-still-wrong-but-ill-shut-up-about-it-and-sell-
chicken/#4532740d2fcb)

Anti gay marriage

~~~
dbrueck
I understand that you disagree with the CEO, but how do you get from there to
the conclusion that he is not open-minded?

~~~
stochastic_monk
Because he claims that homosexual relationships do not deserve the same state
recognition as heterosexual relationships.

~~~
dbrueck
Wait, so in order for a person to be open-minded, they have to not only
honestly consider other ideas and perspectives (which is what being open-
minded actually means) they also have to agree with your perspective?

A person's open-mindedness is independent of whether or not somebody else
agrees with them. Otherwise, by definition nobody - including you - is open-
minded, since there are people that disagree with you. It's completely
nonsensical.

~~~
stochastic_monk
It's the fact that they are unable to honestly consider it because of their
dogma rather than the fact that they don't agree.

~~~
dbrueck
I'm fascinated by your stance - genuinely fascinated, I'm not trolling you,
I'm serious.

So to recap, the CEO of Chick-fil-a is part of a group of people ("they") who
are actually _unable_ \- as in, incapable - of honestly considering certain
viewpoints? How did you reach this conclusion, and how did you come to be so
certain that this CEO is in that group?

Is your understanding that he gave no serious thought to his opinion (i.e. it
was just imposed on him), or that he thought about it lightly but lacks
critical thinking skills, or something else?

Is there any possibility that he simply has a different opinion than you, and
that's all there is to it? Like, he educated himself like you did, thought
about it like you did, etc. but just reached a different conclusion than you?
Or is the only explanation that he is flawed/limited in some way?

Apologies in advance if anything I've said comes across as demeaning, I'm
genuinely trying to understand. Thanks!

~~~
stochastic_monk
I was raised in a cult. I know what it’s like to be incapable of considering
others’ views because of dogma because of my experience.

I am assuming that his Christian perspective is similar to the one I had. My
experiences with Christians since have reinforced this expectation.

Does that help?

~~~
dbrueck
Ok, I appreciate your willingness to share.

FWIW it's virtually impossible to determine the open-mindedness of someone
from afar, and to make assumptions about their _ability_ to even consider
other ideas seems like a way to insulate oneself from difficult ideas - by
declaring that someone is incapable of an honest evaluation of your
perspective, it prevents your perspective from being scrutinized (which,
ironically, could lead to being close-minded if you're not careful).

A parting thought: most people (a) can be reasoned with if not attacked - you
can find common ground and have a good discussion with nearly anybody (b) on
every "interesting" social/political issue there are intelligent and
thoughtful and just really great people with nuanced and reasonable
perspectives all across the spectrum - the extremist nutjobs on the fringes
are wildly outnumbered by pretty normal people

So as a random dude on the internet, just a friendly warning that if the above
strikes you as untrue (e.g. if you believe large chunks of people are
intolerant or incapable of being reasoned with, or if you see a difficult
social issue as pretty much black and white and can't understand why so many
people don't get it), then there's a good chance that you've been duped and
have been sucked into a form of modern tribalism.

Have a great day!

------
siruncledrew
I don't know why, but for some reason the juxtaposition of traditional, brick-
and-mortar brand images with tech blogs about building modern application
technologies is amusing to me. It's seeing two completely different sides of
the same organization. There's 0 condescending sentiment behind it, it's just
like "wow, that was unexpected."

~~~
RandallBrown
I worked for Nordstrom for several years and whenever I describe their
technology department to another developer they're always pretty shocked. It
was pretty shocking to me when I started there too.

~~~
Aloha
I'm curious about Nordstrom, I know nothing of their infrastructure.

~~~
RandallBrown
Their tech stack is similar to what you'd expect from any large tech company.
I can't speak much to the server side of things (I worked on the iOS app) but
I know they had everything ranging from older monolithic .NET apps to Node and
Go micro services.

Their GitHub page
([https://github.com/Nordstrom](https://github.com/Nordstrom)) doesn't have
much in the way of internal projects, but it has lots of forks that will give
you an idea of some of the tools they used.

~~~
tomnipotent
Nordstrom also does polymorphic React with server-side rendering via .NET [0].
I've always been impressed with this part of their stack.

[0]
[https://libraries.io/github/Nordstrom/React.NET](https://libraries.io/github/Nordstrom/React.NET)

~~~
voltagex_
Latest commit e56c9be on 24 Apr 2015 - wonder if it's still in use internally?

------
djhworld
I enjoyed reading this post, but came away with more questions.

They talk about providing a service for developers to deploy stuff easily, but
I'm wondering how that works in practise.

I've very little experience with Kubernetes, the author mentions they run
'mini' kubernetes clusters at each restaurant. Does that mean they have to
deploy software/container updates to each restaurant one by one? Or is it
abstracted above that level, where they can see ALL of the restaurants as one
"big" cluster?

~~~
coffeesn0b
Hey! Caleb here... I'm the lead SRE building the clusters on the restaurants.

Each individual restaurant gets it's own cluster (we aren't fully deployed
yet). There are too many network latency challenges and too much immaturity
around federated clusters to take the route (unfortunately).

We currently use gitops, which houses all the configs per cluster in a single
repo per restaurant (CRAZY amount of corgis!)... we call that Atlas (the
repos) and we made a little pod called Vessel that polls and applies configs
in those repos.

We're almost done building something called Fleet that will generated and
manage those repos (Atlas) at scale. Ie; a UI where we can say "send this
version to 10% of restaurants" and it will regenerate and deploy those configs
to all the appropriate Atlas repos, which will then get pulled down by Vessel.

We tried doing all this with Helm but failed miserably. Maybe it was us? But
templating vs gitops... the choice seemed obvious.

~~~
swampthinker
What were the main issues you faced with Helm?

~~~
coffeesn0b
The primary challenge was just reasoning with the template and using Helm at
scale... ie; what exactly did we deploy on those hundreds of varying clusters?

Other issues included; tiller would sometimes become unstable... version
mismatch issues between helm local and roller... lack of a clear, outage free
canary deployment... we even found cases where helm would not cleanup after
itself during a deployment and retain previous config settings within k8s.

------
jedberg
Sure, they have the biggest cloud in the US, but it's down on Sundays.

Kidding aside, do you guys do maintenance on Sundays or is that forbidden by
company policy?

~~~
uncreativebrian
This is Brian -- I wrote the article. We are closed on Sunday but its not
dogmatic in nature. We want to make sure that everyone at Chick-fil-A has a
great work/life balance. Occasionally we need to do some work on a Sunday but
we try to architect and engineer in a manner that supports business-hours-
friendly deployments, or scheduled deployments over night to protect that.
This is truly a first-class design consideration. In my 14+ years at Chick-
fil-A, I have only worked a handful of Sundays for any reason. Its a tribute
to leadership at all levels and a great company culture that values people
(and their need for times to rest).

------
nvahalik
They call their "cloud": Cloud-Fil-A.

Par excellence.

~~~
coffeesn0b
We will have an order of magnitude more cloud datacenters than AWS! :)

~~~
nvahalik
Do you also have a CDN (Chicken Delivery Network)? (Stolen from a guy in a
slack channel.)

~~~
deskamess
Indeed they do... that's their core business.

------
jaytaylor

        Since we have a highly available Kubernetes
        cluster with data replicated across all
        nodes ..
    

This sounds like one of the more cutting-edge bare-metal kubernetes setups
I've heard of!

Curious what the kubernetes implementation / configuration looks like for the
setup they've described. Particularly in terms of HA-cluster k8s
configuration, networking solutions integrated, and replicated storage.

Considering there is no AWS / GCE / Azure block-storage primitive to build on,
it'd be cool to understand the choices and tradeoffs made in this scenario.

Is all data being stored on something like HDFS or CockroachDB? Or perhaps the
k8s minion slave nodes launch containers with Ceph volumes.

~~~
coffeesn0b
Well, it’s a 6 man team so most of our trade offs were for expedience, not the
worlds greatest architecture. We cheated and sync’d data using a HA MongoDB
setup amongst the cluster. RKE for K8s clustering was a life saver (nothing
easier at this point on bare metal IMO), although RKE can be brittle at times.

------
joezydeco
Having worked with some of CFA's competition and comparing their IoT
architectures, I'd say these guys have their heads on straight. Kudos to them.

~~~
ballenf
I've never been to a retail establishment with anything close to the credit
card speed of CFA. The cashier can swipe and grab receipt in one smooth and
quick motion. It's so fast you feel like it's broken or slight of hand.

I wonder how they do it or why other companies can't or don't do it. CFA
stores are so much busier than even an average McDonald's, maybe the
efficiency gains are just worth more.

No idea how they handled the chip card shift.

~~~
joezydeco
From what I understand, merchants can do a quicker swipe (basically record the
card # and transaction amount without waiting for authorization), but the
merchant fees are slightly higher because of the possible higher
fraud/chargeback rate. It's up to the chain to decide on what to do.

Perhaps CFA can do it even better since they could possibly keep a database of
bad cards at the edge. Remember those books that cashiers used to thumb
through to verify if a card was stolen?

~~~
btown
Or keep a Bloom filter of hashes of repeat customers’ cards on the edge to
accelerate their experiences.

~~~
joezydeco
The goal is to detect stolen/deactivated cards. Not sure how that helps.

~~~
btown
If it's a repeat customer, at the same location, then you can avoid contacting
the payment processor gateway, and just record the transaction on the local
cluster for later payment processing. The expected value of the loss if
someone stole their card, and used it _at the same restaurant that they
frequent as a regular_ , after the card was canceled, and the restaurant gave
the thief a chicken sandwich, and now corporate needs to eat that cost because
the asynchronous CC transaction doesn't go through when they finally run it...
is absolutely miniscule.

------
reaperducer
_We think of our Edge Computing environment as a “micro private cloud”_

We call local clouds "fogs."

~~~
d0100
Aren't most edge computing reffered to as fog-computing? Or is it just in
academia

~~~
uncreativebrian
Hey this is Brian -- I wrote the article. I have heard the term a lot over the
years. I think you could call it that -- the idea being its the "foggy" area
between the user and the cloud. The reason we like "micro private cloud" is
that 1) its a tiny environment with limited cloud properties (no elastic
scale) but 2) we do try and provide a platform with cloud-like services for
our internal developers, and we like the cloud paradigms being applied to how
we think about compute at the edge, in restaurants.

------
rileyphone
This is a pretty cool example of cybernetics ala Stafford Beer - constant
information and reaction to streamline production. I'm excited to see this be
brought in to more production (though maybe it has, and most places aren't as
upfront as this). It's truly a shame the Chilean experiment in the early 70s
didn't work out.

On a side note, it's interesting that Kubernetes is the word from which
cybernetics is derived, meaning "governance".

------
dpeck
with articles like this and Mac'd from this YC cohort it seems like an
increasingly interesting time in the restaurant/technology space.

~~~
bradleyankrom
the domain is interesting, but I worry about the long-term effects of
maximizing efficiencies (read: automation) forpeople who rely on jobs in that
industry.

~~~
matt_s
A myth about manufacturing jobs is that they all went overseas. Reality is
that most of them were replaced by robots and automation.

Low skill, low knowledge, repetitive work is key target for automation.

Fast food workers wouldn't have large downstream effects. Automated truck
driving would though. There are likely entire towns right off of highways that
depend on humans passing through daily - restaurants, repair shops, etc. If
automated trucks don't stop anymore, a town could literally be a ghost town.

~~~
uncreativebrian
Hey this is Brian -- I wrote the article. I do want to add that Chick-fil-A
isnt looking to replace jobs with "robots". We really want to assign our
restaurant team members to tasks that add maximum value to customers, which
are generally service oriented (iPad in Drive Thru, personal service in the
dining room, etc). Our automation will help our team members be more effecient
and enjoy their jobs more rather than replace them.

~~~
matt_s
Of all places I doubt CFA would replace workers, having nice, helpful people
is the second awesome-est thing there (chicken and waffle fries tops that).

I can see the automation and analytics helping staff appropriately and offer
more options for efficiency and waste reduction.

------
Yhippa
From other retail IT I've been involved with, these guys could make an
incredibly successful side business doing this to pretty much any retailer.

However when I see that retail stores are using dialup to do batch uploads of
inventory and transactions I wonder if their central IT systems would be
remotely ready for something like this.

~~~
uncreativebrian
You hit on a great point -- Edge infrastructure _buffers_ outbound internet
traffic but does not replace the need for a good circuit.

------
_bxg1
Pretty interesting, but good lord I've never seen such buzzword-density in an
article that's making real sense

~~~
uncreativebrian
This is Brian and I wrote the article, and I am taking this as a massive
compliment ;)

------
barbecue_sauce
Interesting that they use Kubernetes exclusively for edge computing resources
(as opposed to their cloud architecture). Never really considered this
approach before.

~~~
coffeesn0b
If you decide to play around with it, we use RKE to build and manage the k8s
environments locally.

------
psychometry
When it comes to Chick-fil-A, there's also the regressive side in addition to
the modern and traditional sides.

[https://hornet.com/stories/chick-fil-a-anti-lgbtq-
donations-...](https://hornet.com/stories/chick-fil-a-anti-lgbtq-donations-
two/)

~~~
unit91
Your comment is essentially "the owners hold certain values I disagree with
regarding sexuality, therefore I'm surprised their technical team is any
good."

I don't see how that follows.

~~~
stickfigure
Political values are important to point out. Companies create these tech blogs
as a recruitment vehicle. You should know if the company you're considering
applying to is funding bigotry (or "wholesome family values" if that's your
angle).

~~~
factsaresacred
> 'Bigotry'. _noun_ : intolerance towards those who hold different opinions
> from oneself.

Surely castigating the owner of Chick-fil-A for advocating his personal
beliefs is the truest example of bigotry here.

~~~
pvg
You should probably not rely on Google's fairly crappy dictionary definitions.

~~~
chipotle_coyote
The quoted definition of _bigotry_ as "intolerance toward those who hold
different opinions from oneself" is from the _New Oxford American Dictionary,_
from Oxford University Press (the OED people). If that's what Google is using,
I don't think I can rely on your definition of "crappy."

~~~
pvg
Yep, it's a crappy definition that does not reflect current usage. It doesn't
even reflect the definition in the OED, if you want to check there. 'Bigotry'
is not limited to difference of opinion. Nobody uses it like that, you can
take a look at the quoted usage examples here, which are from current
publications.

[https://www.merriam-webster.com/dictionary/bigotry](https://www.merriam-
webster.com/dictionary/bigotry)

~~~
Fellshard
Ahh, yes. Always useful to continuously cherry-pick or alter definitions to
tailor words that have an objectively negative moral implication to apply to
exactly the standards you want.

Just because some publications use the word in that way does not mean the
definition of the word should be shifted.

~~~
pvg
I’m sorry but this is a pointedly stupid comment. I’m doing the exact opposite
of cherry-picking - I’m saying the definition is much broader than the
inaccurately narrow one posted.

------
walrus01
One thing I question is what kind of WAN link a single chicken fast food
restaurant can get, or is economical. A lot are in places where the only last
mile ISP options will be adsl2+ over copper or docsis cablemodem. I don't see
it being a good use of money to spend $600/mo to pay some local ISP for a
dedicated 100Mbps line on a 5 year contract.

But then again it's highly likely that what they do is not bandwidth intensive
and will work fine on a 5Mbps x 1.25 Mbps DSL connection.

~~~
reaperducer
_One thing I question is what kind of WAN link a single chicken fast food
restaurant can get, or is economical_

At this point, I don't think it's a problem for CFA.

CFA doesn't try to go into every little town in the nation like McDonald's.
It's expansion is very deliberate and only seems to go into very high value
locations, where internet presumably isn't a problem.

Moreover, landlords fall all over themselves to land a Chick-fil-a. Almost as
much as they do to get an Apple Store. A landlord will make business-grade
internet happen, if it means getting a CFA.

~~~
abraham_lincoln
Meh. Unless you personally own a franchise, or work for corporate, this is
pure conjecture.

The article did mention if a failed node is detected, a next-day restore
'disk' is delivered to the store, so obviously that is faster/more reliable
than random ISP.

~~~
Kagerjay
Having a chickfila greatly increases realestate value for the landlord though,
more so than other restaurant chains like McDonalds.

~~~
abraham_lincoln
That doesn't mean the local cable company will put in $100k of poles and coax
to your restaurant.

Also, McDonald's corporate owns all their real estate, franchisees, too.

~~~
Kagerjay
Im not 100% sure its all of them (malls and airports come to mind) but yes its
a vast majority.

True I didn't consider the cable investment either. Normally the landlord pays
for this but I imagine chickfila probably pays for part of the bill. That I
dont know though

~~~
walrus01
For an ISP, if it requires a costly outside plant build to reach a premises
for a business circuit, it will usually be rolled into some combination of an
upfront connection fee (NRC) and monthly rate (MRC) to recoup the costs and
have some positive ROI over a 3 to 5 year term. If it's something like a multi
tenant office building they may gamble and take a calculated risk to lose
money on the first customer over 5 years (example: $800 NRC and $500 MRC that
comes nowhere near xoveinft the cost of the build), with a new expensive fiber
build, and hope the sales team can sign up more tenants ASAP.

------
georgeburdell
I'm going to guess the surprising (to the layperson) technical savvy of CFA
has some relationship to Georgia Tech

------
benburleson
This was posted in a devops-related slack not too long ago and caused quite a
stir over CFA's lack of, uh, diversity. I know it's technically off-topic and
frowned upon here, but I believe the fact that they make their beliefs such a
big part of the company that it deserves to be pointed out.

~~~
oh_sigh
In what manner is CFA non-diverse? Are you talking racially or belief wise?

If it's the latter, would you class the big tech companies as equally non-
diverse, since Christians are frequently treated with derision there?

~~~
benburleson
To put it nicely, CFA is extremely unfriendly to the LGBTQ community.

------
jorblumesea
That level of infrastructure investment is really playing the long game,
especially for a brick and mortar business. I often feel like "traditional
businesses" see technology's disruption effects but not the positives they can
bring. Other businesses could benefit from this example.

------
forkLding
Not that surprised, a fully automated restaurant is every restaurant chain's
dream, reduces labour costs and logistical accuracies go up at real-time
speed.

------
TickleSteve
Can anyone explain what the actual problem is they're solving using 'edge
computing'?

I get the feeling this is a completely unnecessary use of technology.

~~~
coffeesn0b
Here's the abbreviated reason;

Internet connectivity for restaurants stink.

So we put our compute at the edge.

Trust me... as an SRE I would much prefer we shove it in the cloud and call it
a day, but alas... no such luck.

~~~
TickleSteve
Yeah, but what compute? what are they computing?

The driving need for edge computing is low-latency. There just isn't the need
for it in a restaurant.

I do hope they're not doing real-time control of kitchen equipment via
something in a container, containers in Linux are nowhere near deterministic
enough.

This strikes me as someone who is familiar with only web technologies but
unfamiliar with control systems attempting to create a control system....
completely inappropriate tools for the job.

~~~
coffeesn0b
Could you define what is not deterministic about containerized computing?

Examples of things we are controlling are timers for food, cameras that can
recognize and track food items, drying machines that will automatically
trigger and fry fries.

~~~
TickleSteve
Linux is general is not a real-time system, in that you cannot get
deterministic response time from it.

For example I'm guessing it doesn't matter if one of your timers are a second
late due to Linux deciding to swap a process out or prioritise some upgrade
check but imagine if your time constraints were a lot tighter.

Similarly if you're attempting to track food on a conveyor belt and Linux
decides to prioritise the file indexer on your filesystem... oops, a lettuce
has been thrown on the floor.

Real-time operating systems exist for a reason.

~~~
coffeesn0b
Our tolerance requirements for synchronicity are broad enough that we can
tolerate blips like this... at the end of the day we are automating away some
simple human interactions (ie; fry the fries or track the food), we aren’t
performing surgery with these systems.

So far we’ve been satisfied with Linux (Ubuntu 18.04 to be exact) and it’s
overall capabilities.

Also, don’t forget that cost at scale is a big factor... doing things
perfectly but expensively is not profitable at 2k locations.

~~~
TickleSteve
The point about real-time systems is not the your tolerance of 'blips' its
about analysis of your worst-case timing requirements. This is what real-time
control systems are all about.

Sure, you may be able to tolerate a 1 second 'blip'... how about ten seconds?
a minute? do you know for sure how long Linux may put your process on hold
for? Linux provides _no_ guarantees at all.

It seems to me that what you are really creating is a traditional control
system with web technologies and using the hype-of-the-day (i.e. edge
computing) to justify it... with the corresponding increase in overhead and
cycles.

This does not seem like a good solution to me, the only novelty here is the
inappropriate use of technologies. i.e. you've managed to dig a tunnel with a
spoon.

~~~
samlewis
Seems like a very appropriate use of technology to me?

If they can get "good enough" results using Linux and running with
technologies that the average developer is more likely to know, is it worth
the investment in running a RTOS? Especially given running a RTOS will make
any higher level app integration more difficult?

The benefit of running with Linux outweighs the risk of "lettuce on the
floor", imo. Yes, there's probably overhead in terms of clock cycles, but
there's also probably dramatically less overhead in terms of developer
resources.. and when you're running off commodity hardware you can guess which
overhead will cost more!

~~~
TickleSteve
It would also be 'cheaper' to build a bridge using lego....

Correctness of the solution is important, otherwise we're just playing at
being engineers.

~~~
nielsole
To quote you:

> At some point, you will realise that your job is not to create an idol to
> the software-engineering gods but rather to have a working product.

> Does this product work? if so, its doing its job.

[https://news.ycombinator.com/item?id=17824782](https://news.ycombinator.com/item?id=17824782)

------
sergiotapia
Chickfila is very much a top-to-bottom ownership company. They own every step
of the process in the cooking, from the machines, to servicing, to the
logistics behind maintenance.

It's really exciting to see a company like this just want to own everything
they do, and not reach out for 3rd party tools and SaaS and such.

~~~
Kagerjay
They don't own everything, their logistics and servicing is through Trimark,
granted many of those employees there only work on Chick-fil-a. But yeah
Chickfila has always been one of the most high tech restaurant chains out
there, its not common knowledge either. They have one of the most unique
kitchen setups in the industry as well.

------
gammateam
Is this use of "Edge Computing" a term that anybody else uses when they are
not talking about an always online distributed network in datacenters?

"I have some Raspberry Pi's in my basement and at my vacation home"

VS

"I'm keeping bare metal close to the action with our uniquely scalable Edge
Computing implementation"

------
gammatrigono
What data does CFA actually /need/ to collect in their restaurants? Orders and
supply chain metadata? Store traffic statistics? Cleanliness and user ratings?

I hope it isn't Bluetooth or WiFi (or worse yet, GSM/LTE) metadata from their
customers.

~~~
mh-
_Our hypothesis: By making smarter kitchen equipment we can collect more data.
By applying data to our restaurant, we can build more intelligent systems. By
building more intelligent systems, we can better scale our business._

~~~
PascLeRasc
Yeah, I wish the article had more detail on the data and how they want to use
it instead of the same overview of containerization you could find a thousand
other places.

~~~
xc100
Best overview of containerization is here (the video parody is actually funny,
perhaps NSFW):

[https://zwischenzugs.com/2016/04/12/hitler-uses-docker-
annot...](https://zwischenzugs.com/2016/04/12/hitler-uses-docker-annotated/)

------
sjg007
Pretty neat.

------
marv3lls
Should I read this, after Chick-fil-A's "THING" about LGBTQQIAA people?
_cough_

~~~
loco5niner
I've always found CFA employees to be kind, courteous, and respectful.

------
ForrestN
Sorry to interrupt the eager participation in this engineer recruiting effort
(wow, iPads, cool!)

Don't forget that the people who run Chic-Fil-A support a toxic version of
Christianity that, among other things, holds that God himself is going to
punish the United States because I'm allowed to get married.

(Also, in case you're a few years ahead of the current western ethical
consensus, they torture and kill a massive, genocidal number of chickens every
year in order to better monetize related public health crises.)

Maybe don't work for them if you value modern society.

~~~
loco5niner
I've always found CFA employees to be kind, courteous, and respectful.

~~~
dang
You've repeated this often enough in the thread that it's actually a kind of
trolling. Please don't do that.

[https://news.ycombinator.com/newsguidelines.html](https://news.ycombinator.com/newsguidelines.html)

~~~
loco5niner
Ah, good to know, thanks for the heads up.

