
How Linux Saved A Fast Food Giant. - megablast
http://therealedwin.com/2010/05/17/how-linux-saved-a-fast-food-giant/
======
kogir
The real issue here has nothing to do with Linux or Windows. Rolling out
updates to production without testing them first is a fail, not matter what OS
you're using.

If they'd tested the McAfee definitions on even one of the POS terminals
before rolling them out to all the restaurants everything would have been
fine. They could have even automated having one restaurant get the definitions
one or two days before all the others so only a single location was affected.
Or better, set up staggered updates within each restaurant, so they all stay
up and at worst lose a single POS terminal.

~~~
bad_user
> _Rolling out updates to production without testing them first is a fail_

Deploying McAfee in the first place to all those terminals is a painful
process by itself, yet many people do it because of the "extra" security it
provides. When you can't trust something like McAfee for your production, then
you're doing something wrong, and that's not the lack of testing upgrades to
McAfee ;)

IMHO, the WTF here is that they've used Windows in the first place for what
are in essence dumb terminals that help them manage drive-through orders ...
for that you can always deploy a highly customized and minimal Linux OS, with
most of the ports blocked and with a minimal number of services running, with
reduced capabilities and enhanced with Linux SE if you're a security freak ...
making the area of attack for any potential threat a lot smaller.

Then you wouldn't need a piece of shit like McAfee to handle your security ...
all anti-viruses lately are pieces of shit IMHO.

> _The real issue here has nothing to do with Linux or Windows_

I think it does. On that terminal they could've had a small Linux image
providing a ssh server, whose purpose would be to boot into it in cases of
emergency. And you can control a Linux instance a lot more efficiently
remotely. Try doing that with Windows.

------
a2tech
From my time working at McDonalds back in highschool I can tell you they had
solved this problem already. The head machine in the back office was running
AIX, and all the POS terminals netbooted a linux variant from it. There was
also a secondary server which no one touched that did the most critical job of
receiving, tracking and reporting orders. Windows on a dumb POS terminal? That
just sounds like a straight up bad idea to me.

~~~
count
How long ago was that? In 1999/2000 they were running everything off of a
MSDOS app on a Win95 desktop PC...

~~~
a2tech
2001-2002. This wasn't a corporate owned store either-just a small franchise
operation. I think they had 5 stores.

------
hoggle
Grml is another Debian based Linux distro for rescue scenarios like this. The
small version weighs in a little more at 100MB versus PLoP's 62MB though.

If your deployment environment doesn't offer NFS/SMB/CIFS you can fetch your
images via HTTP from your PXE booted kernel too.

Regarding data centers - it can even boot over serial in connection with HP
iLO and IBM RSA/AMM out of the box.

<http://wiki.grml.org/doku.php?id=rescue>

<http://grml.org/changelogs/README-grml-small-2010.04/>

PLoP seems to be a great tool for similar cases, interestingly both are
Austrian distributions:

<http://www.plop.at/en/ploplinux.html#pxel>

I've got a grml usb stick with me at all times (now even supports multiboot
32/64bit!). I just recently recovered a relatives' WinXP after a seemingly
failed automatic update and did a dll rollback in a couple of minutes.

It's a great tool plus it shows you how to use the mighty zsh properly!

~~~
qjz
Tiny Core Linux is another candidate at only 10MB (or its X-less variant,
Micro Core, at 6MB). It includes a capable terminal server for PXE booting
with support for loading extensions via NFS/TFTP/HTTP:

<http://wiki.tinycorelinux.com/tiki-index.php?page=Netbooting>

It's well documented and easily customized:

<http://www.tinycorelinux.com/>

~~~
hoggle
I love busybox really makes sense in regards to size, thanks man!

------
cstuder
The most interesting line was this:

> If I had not flirted with Linux during college, where would we be right now?

I wonder how much in support just relies on good luck. Someone happily trying
out stuff several years back on his own. Curiosity saved the system.

~~~
Andys
This is HUGE in the IT world.

Corporate support is based on knowledge bases built up by customers "trying
stuff out" and reporting in when it doesn't work.

~~~
eru
Too bad they probably don't hear as often about stuff that does work.

------
shadowsun7
> After cleaning up the processes and adding several easter eggs, it was time
> to pilot it at 10 local restaurants.

I admire his sense of humour. Easter eggs in the face of calamity indeed.

~~~
eru
I guess he had a bit of downtime, when he had to work for other people.

------
kaptain
At first I wasn't sure if this was a rant against Windows because the author
kept using POS. Then I realized that POS also meant point-of-sale.

~~~
eru
What does POS mean besides point-of-sale?

~~~
infinite8s
piece of shit

------
whalesalad
I wonder why these die hard Windows fans stick with it when they realize what
kind of things you can do with Linux.

~~~
noonespecial
I wonder about the mindset that allows one to accept running MacAfee on a
freakin' _Point of Sale_ device is something that just has to be done?!

~~~
andrewf
If credit card data is going to pass through the device, anti-virus software
is mandated by PCI-DSS.

~~~
noonespecial
Hmmm interestingly, the "unix exemption" seems to have been dropped from 1.1
to 1.2 of DSS document.

Part 5.1 used to include this note:

 _Note: Systems commonly affected by viruses typically do not include UNIX
based operating systems or mainframes._

Consumer grade MacAfee that auto updates is still probably a very poor choice
for this application as they should have a test bed in the lab that they beta
_all_ changes on before they roll out to thousands of machines that likely
need a truck roll if something goes wrong.

~~~
count
You can run McAfee on linux now as well (mostly for scanning email and CIFS
shared files, I assume?). PCI DSS and DoD IA controls require it.

Enterprise grade McAfee still sucks, it's just a more manageable suck. Despite
that, it still blew up with the bad DAT file - those are the same between
product lines.

------
patrickk
A previous hotel job I had used a touchscreen Windows-based POS. One day, for
whatever reason, I accidentally held down the 'submit transaction' button for
half a second too long. The terminal went apeshit and crashed.

Several hours later, after hours of phone calls, hundreds of orders written on
paper and totted up by calculator, a confused-looking technician managed to
get it working properly again.

What a mess that was.

------
aaronblohowiak
PXE is radsauce, it is netboot for the 21st century

~~~
Andys
Booting off iSCSI is all the rage now. Any server with an Intel 1Gbe network
port can be made to boot off iSCSI after flashing with special firmware from
Intel.

~~~
aaronblohowiak
Woah, that is cool! I should hang out with more SysAdmins.

------
yosho
I'm guessing POS means Point of Service but every time I read it, I think
"Piece of Shit" terminal, which I suppose is applicable as well.

------
wendroid
TRWTF is that they didn't prepare for a catastophic failure such as this one.
If you have your POS getting auto-updates then why haven't you considered the
possibility that it will go wrong with "WILL NOT BOOT" as a consequence.

It should simply be a matter of phoning the manager and tell him :

"Don't worry, just insert the disk that says "RESCUE DISK" on it, reboot and
it will roll back the machine to last week".

~~~
noonespecial
Since we know that the machines could PXE boot, they should have just had a
server in the back preloaded with a known good image that would clone onto the
POS automatically if started via PXE.

Then the fix would have been a matter of simply swapping the dhcp entry for
the pos devices on the server with one that had the pxe directive. The pos
would have caught that during its next reboot and restored itself.

~~~
count
What happens when that system is Windows and gets dorked up by McAfee too? :)

