
Sysadmin war story: the network ate my font - atsaloli
http://verticalsysadmin.com/blog/sysadmin-war-story-the-network-ate-my-font/
======
24gttghh
Yeah? Try dealing with a printer that has to have a font file sent to it with
an lp command from cron every 15 minutes or else checks wont be formatted
correctly. AND there is a physical device (either USB on the printer or inline
on ethernet in front of the printer) which has the MICR fonts stored.

And then the printer you are migrating the MICR device to has decided it
doesn't understand the PCL in the font file anymore.

This is 2017 and I just dealt with this today.

~~~
notgood
The engineering time dealing with that printer is probably more expensive than
just buying a simpler/better one.

~~~
24gttghh
Close. To be fair the printer is _supposed_ to support PCL5, and this is the
first time I've witnessed this happen.

~~~
kevin_thibedeau
If this is so mission critical why not get a proper Postscript printer with an
onboard filesystem for font storage. It doesn't make much sense to run a
business on consumer junk.

------
gumboshoes
This sort of stuff happened all the time when I worked in tech support for ad
agencies. Printing font problems seemed to be 99% of the job, some days. PDFs
helped — but only of the person making them remembered to embed the fonts.
Usually you would simply make an EPS (encapsulated Postscript file), which
would work most of the time — as long as the receiving printer had sufficient
memory for huge print files. Can't tell you how many thousands of times art
directors who should've known better would just send Quark or PageMaker or
(later) InDesign files without the fonts. In fact, there were (are?)
preflighting programs designed just to solve this exact problem. They'd look
at your files and determine: Are your fonts there? Are your images there? How
will this print? etc., etc.

~~~
semi-extrinsic
There are indeed PDF preflight tools to this day. E.g. on Linux you can run
`pdffonts` and check that everything is embedded. I believe it's still very
common, to the point that my local university's PhD thesis print delivery
checklist includes it.

~~~
noir_lord
Scribus (DTP program for Linux I love) has it built right in and it's good.

They also pride themselves on the quality of PDF output (and have full (iirc)
support for PDF-A).

------
cbhl
This leads to an obvious question: "why did the programmers not always just
embed the font?".

And it turns out that the answer will probably be, "well, we asked the
lawyers, and you need rights from the font owner to do so, and it makes files
bigger, and most stuff is in whatever Microsoft Office uses as their default
font this year anyway."

And so we're torn between "yes, of course the designer who built the font
should get paid" and "a sane default became a checkbox because of US copyright
law".

~~~
MBCook
Maybe because PDF has been around a long time and it used to be that embedding
even a basic font would have been a MASSIVE increase in file size?

~~~
KekDemaga
Why don't PDFs just render the text and store it as some form of vector image
file.

~~~
kps
A font _is_ some sort of vector image file.

~~~
yellowapple
Not always. Bitmap fonts are a thing.

------
izzydata
I love it when you are troubleshooting and question how it ever worked in the
first place.

~~~
_cereal
Sounds like a schroedinbug:

[http://catb.org/jargon/html/S/schroedinbug.html](http://catb.org/jargon/html/S/schroedinbug.html)

~~~
lokedhs
If someone had not heard of that one before, it should not be confused with a
Heisenbug, which is approximately the opposite thing.

[http://catb.org/jargon/html/H/heisenbug.html](http://catb.org/jargon/html/H/heisenbug.html)

------
joehan
I just spent all day yesterday debugging why my web app's font-
awesome/material design icons weren't showing on IE while they were showing on
chrome, ff, and edge. Turned out that to be the corporate policy to disable
font downloading on IE.
[https://www.stigviewer.com/stig/microsoft_internet_explorer_...](https://www.stigviewer.com/stig/microsoft_internet_explorer_11/2017-03-01/finding/V-46505)

~~~
0x0
Ouch.

To be fair, fonts really do pave the way for malware into your kernel -
[https://googleprojectzero.blogspot.no/2015/07/one-font-
vulne...](https://googleprojectzero.blogspot.no/2015/07/one-font-
vulnerability-to-rule-them-all.html)

------
fit2rule
I once (as a junior developer) had to write some software to format a massive
text file for printing to an IBM line-printer (1403, model N1) .. this (1M)
text file had to be checked a few times before it was printed, and that meant
opening it up in an editor and verifying the data.

I got pretty sick of opening this file after half a day, because it had tons
and tons of CR/LF's and my editor at the time rendered these with strange
characters on the screen .. and I didn't like that, as a junior guy, so I just
replaced the LF's with 00's. For some reason, this just worked fine in my
local environment, and I was able to validate the data in the file before
sending it off to the spool for printing, later in the afternoon.

About 3 minutes after I closed the job, the building got a fire alarm, and we
all had to exit. Apparently there had been smoke detected in the operations
room, where the printer was located, so the Halon systems went off, and we
went into full-blown "Ops Reset" mode.

After an hour of hanging around the parking lot, I was called in by the head
honcho's in the Ops Room, sat down in front of the printer, and told to
explain myself.

Well, turns out, I was responsible. The lack of LF's in the text file meant
that the printer was printing - as fast as it could - every single line on the
very first character position .. and after a few minutes, the printer simply
caught fire.

Oh man, since that day (mid-80's), I've eschewed any job that requires me to
deal with printers, and I've been anti-printer ever since. ;)

------
lfx
Love those stories. Read some similar ones in the past. First time when
debugging - I'm looking into the cache.

~~~
atsaloli
Thanks for the kind words!

------
vehementi
The darkest ring of hell would be tech support for wireless (bluetooth?)
printers

~~~
ianai
In a casino full of smokers and devices interfering with those signals.

------
bluedino
We had a printer that did checks and kept the signature on a USB drive, at
some point the printer started printing a distorted signature - it ended up
using the BMP signature and not the PNG file. Changed out of the blue
somehow... both images were fine but it had a bug in how it decoded the BMP
files

------
AnimalMuppet
> 508 Resource Limit Is Reached

The network ate my webpage...

~~~
atsaloli
Update: should be good now. The engineer logged in and killed a couple of
stuck processes. In 2017, folks.

------
dba7dba
> Turns out the printer had a cache for fonts and was using the font cached
> from the earlier check image which included the font!

Dang cache.

------
Top19
I upvoted simply because the title was too good not to.

~~~
atsaloli
Thanks!! The network ate the font is actually how the problem was reported.
:-) That was a fun break from not breaking the HP-UX server running the 24/7
factory and from babysitting various Linux web app server issues at the colo.
:-)

------
apetresc
I usually love these kinds of war stories, but this one was anticlimactic.
This would have been the first instinct of anyone who's ever tried to open a
PowerPoint they got over e-mail – "oh, one printer shows the wrong font? I bet
the font's not bundled, and one machine just happens to already have it."

~~~
icebraining
I thought of that, but I have to say I wouldn't have thought about the
printer's font cache. I didn't even know they had one. Then again, I try to
avoid printers as much as I can :)

~~~
atsaloli
Same! You live, you learn. Thanks!

------
yonatron
It's "pored over", not "poured over".

------
sneak
500 mile email comment in 3... 2... 1...

~~~
athenot
I always enjoy re-reading it.

[http://www.ibiblio.org/harris/500milemail.html](http://www.ibiblio.org/harris/500milemail.html)

~~~
beautifulfreak
The Daily WTF has a trove of similar stories.
[https://thedailywtf.com](https://thedailywtf.com)

