
The 500-mile email (2002) - folz
http://web.mit.edu/jemorris/humor/500-miles
======
Twirrim
This one comes up ever 3-4 years or so in sysadmin communities, and I read it
every single time. because it's worth it.

It's one of those things that I highly doubt would have occurred to me to have
even checked, or given even a moments thought to, under normal circumstances.

~~~
AceJohnny2
I was looking for another famous sysadmin story, where the guy who also
happens to be a top Linux developer (so maybe Alan Cox?) rescues a deeply
broken Linux system where even glibc is no longer accessible by manipulating
inodes in a running process. Or something.

Over the years, my Google-fu has failed me. Any clue? :)

~~~
HarryHirsch
[http://www.lug.wsu.edu/node/414](http://www.lug.wsu.edu/node/414) is what you
are looking for.

~~~
AceJohnny2
That's pretty damn close (and impressive!), but I don't think that was it :)

For one thing, I'm pretty certain the story involved recovering a deleted
file's inode via some other running process's FDs.

------
danbruc
Another email incident at Microsoft worth reading [1].

[1]
[http://blogs.technet.com/b/exchange/archive/2004/04/08/10962...](http://blogs.technet.com/b/exchange/archive/2004/04/08/109626.aspx)

~~~
MrBuddyCasino
Ha. There is no explanation of why the mailing lists were named "Bedlam"
though, and I doubt non-native readers know what it refers to. To quote
Wikipedia [0]:

"Bedlam may refer to:

Bethlem Royal Hospital, London hospital first to specialise in the mentally
ill and origin of the word "bedlam" describing chaos or madness"

[0] [http://en.wikipedia.org/wiki/Bedlam](http://en.wikipedia.org/wiki/Bedlam)

~~~
bevacqua
I'm a non-native speaker and I know what Bedlam means. Thanks to Ultima Online
and Diablo :)

------
IgorPartola
If only every bug report that I received had been processed by a
geostatistician... Usually I get a "hey, I can't get X to work". One of three
responses from me usually fixes it: "Is your computer on?", "are you online?",
and "try hitting refresh".

I am actually surprised the sysadmin in this scenario thought it was a bad
thing that the statistics department did their research and presented a well
documented error.

~~~
mcguire
Well, technically, the geostatistician (Did I spell that right?) was doing
research that was orthogonal to the actual problem and its symptoms. In this
case, the results were sufficiently odd that they sort of pointed in the right
direction, but I've been sent off on wild goose chases by people skillfully
applying their own particular set of skills before.

On the other hand, there's the word document with nothing but a screen shot
showing half of a useless error message.

------
copperx
I didn't know about the units program. Is there any resource out there that
lists these little *nix utility programs?

~~~
motters
That's the thing about unix-like systems. No matter how much you have learned
there's always some command you don't know.

~~~
creshal
paste was my most recent "holy shit this saves so much time" discovery. I
blame it on the not quite intuitive name.

~~~
nchelluri
What do you use it for?

~~~
pwg
One use is converting columns of numbers into math strings for bc.

Example (contrived):

    
    
        $ seq 10 20 | paste -s -d +
        10+11+12+13+14+15+16+17+18+19+20
    
        $ seq 10 20 | paste -s -d + | bc
        165
    

Or converting columns of strings into regex 'or' clauses for searching
(contrived example again):

    
    
        $ cut -f 3 -d , something.csv | paste -s -d "|"
        a|b|c|d|e|f
    
        $ egrep "$(cut -f 3 -d , something.csv | paste -s -d "|")" another_file
        ... result lines appear here ...

~~~
father_of_two
seq has the -s flag which voids the need of the paste for that command:

    
    
      $ seq -s + 10 20
      10+11+12+13+14+15+16+17+18+19+20
    

But I agree that the paste is very useful.

    
    
      # a few random samples for an IN SQL statement
      $ shuf -i 1-500000 -n 5 | paste -s -d ,
      371492,250061,266669,455846,295852
    
      # we can even get PI
      $ ( seq -s + -f '4/%g' 1 4 100000 && seq -s - -f '4/%g' 3 4 100000 ) | paste -s -d - | bc -l
      3.14157265358979523735

~~~
schoen
There is a faster way to get pi in bc:

    
    
      echo '4*a(1)' | bc -l
    

(That is, 4×arctan(1)=4×π/4=π.) But your way is truly an awesome use of Unix!

------
fbnt
Shouldn't this account for a round trip, and the speed through copper (~ 2/3rd
of the speed of light)? That would lower the radius to much more than 500
miles.

~~~
dantillberg
I had this thought when reading this before as well. I imagine that the "3
milliseconds" they determined from testing was a _typical_ number, maybe the
median/mean, and that the actual timeout varied considerably depending on CPU
load at that particular moment. Add in a number of retries for the server to
attempt sending each email, and the effective timeout might have been a few
milliseconds more... or at least it _must_ have been, because `(2 * 500 miles)
/ (2/3 speed of light)` works out to about 8 milliseconds (where the 2X is for
the round trip, and 2/3 is a rough multiplier for the speed of light traveling
in either copper or optical fiber).

~~~
ErrantX
The FAQ answers this question. Basically; it was a long time ago, and the
point of the story isn't in the detail. :)

[http://www.ibiblio.org/harris/500milemail-
faq.html](http://www.ibiblio.org/harris/500milemail-faq.html)

~~~
lloydde
And this is why we can't have nice stories.

I felt for the author as I got deeper into the faq, and recognized this
pattern of cynicism, then decided the author was so generous and thorough, not
out of obligation (make the emails stop!), but because that is the type of
detailed person he is -- and good at dinner parties too!

~~~
ceequof
Writing stories for a technical audience is tricky. I've been doing it for
going on 10 years now, and I'm still not very good at it.

A critical rule, however, is to omit detail, (a reader is unlikely to question
an explanation they make up themselves) and most importantly, _to omit details
you know to be wrong_. (It is impossible to nitpick a statement that is never
said)

    
    
      An odd feature of our campus network at the time was that it was 100%
      switched.  An outgoing packet wouldn't incur a router delay until hitting
      the POP and reaching a router on the far side.  So time to connect to a
      lightly-loaded remote host on a nearby network would actually largely be
      governed by the speed of light distance to the destination rather than by
      incidental router delays.
    

He knew this was largely wrong, and didn't really improve the story, yet he
said it anyway. It should have been summarized in a single sentence, leaving
out all the problematic assertions that the slashdot trolls leaped on.

~~~
chatmasta
Reminds me of the old saying, "it's better to stay silent and be thought a
fool, than to open your mouth and confirm it."

It's also a reason why short business emails are better than longer ones. You
can always go more in depth. It takes skilled restraint to touch on only the
most relevant details without losing the larger point.

~~~
emodendroket
I don't think he's confirmed he's a fool so much.

~~~
chatmasta
Yes, sorry! Apologies if it sounded that way. The author is definitely no
fool. If anything he proved the saying wrong. :-)

------
bpp
Another of the 10,000 here - this is such a delightful story.

Also just discovered the "units" conversion program and disappointed that the
default Mac library has only 586 units. And shockingly there don't seem to be
compatible libraries out there.

~~~
girvo
`brew install gnu-units` should do it :)

Edit: You'll then want to run it with `gunits` rather than `units`

~~~
oaktowner
Now I know where the rapper got his name.

------
andrewchambers
I was so happy to discover that units command line program, then i realized
that Google already does this, it just wasn't as fun.

~~~
tim333
yeah and units on the mac terminal doesn't recognise "3 millilightseconds"
whereas Google works for "0.003 light seconds to miles"

~~~
chris_b
works exactly as in the blog post in fedora.

~~~
lucb1e
And Debian.

For Mac, try brew install gnu-units and then run gunits.

~~~
TreyHarris
As I wrote in the FAQ, I decorated my own units.dat (units.lib in some
implementations) with lots of stuff because I like easily editing units.
(Nowadays I use Emacs Calc, but I still add a bunch of my own units, like the
binary prefixes like mebi, gibi, etc.)

I suspect that a lot more can convert millilightseconds to miles out-of-the-
box now at least in part because of the popularity of this story over the past
13 years.

------
tsaoutourpants
Forgot to account for the difference between traditional speed of light (in a
vacuum) and speed of light traveling through copper of fiber. :)

~~~
sampo
And the time it takes to make a round trip.

------
vog
Better link that contains more headers (showing the email's date, and linking
to a FAQ):

[http://www.ibiblio.org/harris/500milemail.html](http://www.ibiblio.org/harris/500milemail.html)

------
th0ma5
This always reminds me of the email around the world:
[http://phrack.org/issues/41/4.html](http://phrack.org/issues/41/4.html)

~~~
fabulist
Thanks for a good read. Its strange to think about a time when there were a
myriad of incompatible networks, and their different capabilities could be
exploited.

------
anonfunction
Since I've seen a few comments about units not having lightseconds so here are
a few ways to add the missing unit if you don't have it.

1) Add this line under the lightyear definition in /usr/share/misc/units.lib
(or wherever `man units` says the standard units library is under the FILES
section)

    
    
        lightsecond lightyear / 365.25 / 24 / 60 / 60
    

2) If you're on a mac and use homebrew just `brew install gnu-units` and then
run `gunits`

~~~
chernevik
I did #2, then:

sudo mv units macunits sudo ln -s $(which gunits) units

~~~
lloydde
or use brew install option --with-default-names and put your homebrew at the
start of your path.

------
Aissen
Damn statisticians. They do know their job quite well.

~~~
nchelluri
It was a seriously accurate bug report. If only all users were so thoughtful.

~~~
nashashmi
> If only all users were so thoughtful.

But then it sent him off in a direction not worth going. He literally started
to map out how far emails would go if they succeeded. The whole time the error
was in the timeout instead.

~~~
bdevine
The mapping was done for the purpose of reproducing the error though. I
wouldn't call that a wasted effort.

------
Scarbutt
Who though it was going to be a TTL issue before finishing reading the story?
:)

~~~
mobiplayer
You probably mean something else (RTT?) but definitely not TTL, which is a
completely different thing :)

~~~
motoboi
TTL is involved when dealing with routed networks. The farther the destiny,
you normally get more hops on the way. If the starting TTL is low, you won't
reach the destiny. So, TTL values cause problems like this, although the
radius wouldn't be so precise. Damn statisticians!

~~~
mobiplayer
Hehe yeah, I know what's TTL and how it works :) hence why I was pointing out
(as politely as I could) that the previous comment was wrong.

But I tend to agree with your conclusion. Damn statisticians! :)

------
kissickas
Why do I get:

> unknown unit 'millilightseconds'

Is this one of the embellishments that just makes the story more entertaining?

~~~
kaishiro
Not an embellishment at all.

Via 'man units': "The conversion information is read from a units data file
that is called 'definitions.units' and is usually located in the
'/usr/share/units' directory."

Via definitions.units (L. 223), you can see the milli- prefix:
[https://gist.github.com/anonymous/f06769de95e0c7f9e658#file-...](https://gist.github.com/anonymous/f06769de95e0c7f9e658#file-
definitions-units-L223)

Via deifnitions.units (L. 1060), you can see the lightsecond unit:
[https://gist.github.com/anonymous/f06769de95e0c7f9e658#file-...](https://gist.github.com/anonymous/f06769de95e0c7f9e658#file-
definitions-units-L1060)

Maybe check it for completeness?

Edit: Spelling

~~~
anonfunction
Some distributions only support lightyear so adding this line to your units
file (which you can find with man units) will give you support for
*lightseconds:

lightsecond lightyear / 365.25 / 24 / 60 / 60

~~~
kaishiro
Nice!

------
tlrobinson
I see this story every so often, and it's good one, but haven't thought to
verify it. Has anyone else?

~~~
ryan-c
This FAQ was posted in the comments of a previous posting:
[https://webcache.googleusercontent.com/search?q=cache:http:/...](https://webcache.googleusercontent.com/search?q=cache:http://www.ibiblio.org/harris/500milemail-
faq.html)

------
bontoJR
Absolytely a good reading. Sometimes this kind of readings can help in a
complete different problem. Sometime happens you are dealing with another
problem, then you remember this story, and you figure out what's wrong because
there're some similarities. I remember to have fixed a problem with Postgresql
remembering a story about Unicode and Postfix, different domain, but similar
problem.

------
carlesfe
That was great out-of-the-box thinking, and I wonder if that could be used as
one of these job interview questions:

Q: "Your email server for some reason is only working for addresses within 500
miles of the server. What may go wrong?"

And let the candidate think logically and reach some sane answer, even if not
100% accurate (i.e. check routers first, connectivity, DNS, timeouts...)

~~~
iopq
That's one of those interview questions that tests for someone reading hacker
news and pretending that they figured it out all on their own...

~~~
rmc
"culture fit"

~~~
DaFranker
Translation: "Tunnel Vision" and "Lack of diversity"

------
ilaksh
If you're a sysadmin and someone brings in a consultant who gets root access
and upgrades the whole OS to a new operating system which then almost takes
out email.. wouldn't that be a problem?

If I were the sysadmin and that happened, I would need to have a meeting with
some people. What's the point of being a sysadmin if he operating system is
randomly going to be completely changed without someone telling you?

I have a fair amount of built up rage. This seems like one of those situations
where it is actually your responsibility to rip people a new one.

------
t27
A perfect answer to the YC application question - "Tell us something
surprising or amusing that one of you has discovered" :)

------
laex
I tried the 'units' program on OSX. It seems that it does not recognise the
'millilightseconds' unit.

~~~
JulianMorrison
Try one L in mili?

~~~
laex
Didn't work.

~~~
laex
@girvo suggested installing gnu units, which does the trick.

> brew install gnu-units > gunits

------
kowdermeister
I'm wondering how many hits that email address got at the bottom of the page
:)

~~~
anonu
Was this just a clever way to let people know he was looking for a job?

------
mborsuk
Every time I read this I am reminded of units(1) util, which is super useful
and I always forget about and revert to Google. But yeah, that connect timeout
to 500 mi correlation is fun too.

------
nathancahill
1 year ago:
[https://news.ycombinator.com/item?id=8176458](https://news.ycombinator.com/item?id=8176458)

2 years ago:
[https://news.ycombinator.com/item?id=6296201](https://news.ycombinator.com/item?id=6296201)

3 years ago:
[https://news.ycombinator.com/item?id=3802458](https://news.ycombinator.com/item?id=3802458)

4 years ago:
[https://news.ycombinator.com/item?id=3114391](https://news.ycombinator.com/item?id=3114391)

5 years ago:
[https://news.ycombinator.com/item?id=1293652](https://news.ycombinator.com/item?id=1293652)

6 years ago:
[https://news.ycombinator.com/item?id=385068](https://news.ycombinator.com/item?id=385068)

7 years ago:
[https://news.ycombinator.com/item?id=123489](https://news.ycombinator.com/item?id=123489)

~~~
Twirrim
Maybe the submitter is one of the ten thousand
[https://xkcd.com/1053/](https://xkcd.com/1053/)

Bet there are a few more that will find this submission too.

~~~
emilioolivares
Ha, I'm one of the ten thousand for both the XKCD and this post. Lucky me!

~~~
golergka
You mean, for the whole XKCD? Not just this issue, but the whole thing? I envy
you so hard right now. (Unless you were going to be productive today).

~~~
qznc
Productivity? Overrated.
[http://beza1e1.tuxen.de/lore/](http://beza1e1.tuxen.de/lore/)

------
ai_ja_nai
old but gold

------
elchief
Can we get a nice "HN Classic" tag to put beside annual stories like this? I'm
fine if stories like this pop up every year, actually.

------
kazinator
> _And also being a good system administrator, I had written a sendmail.cf
> [...]_

Say what? Nobody writes a sendmail.cf from scratch, unless they are crazy.

> _... that used the nice long self-documenting option and variable names
> available in Sendmail 8 rather than the cryptic punctuation-mark codes that
> had been used in Sendmail 5_

Good system administrators stick to conservative, portable subsets of
configuration and scripting languages, rather than bleeding edge stuff.

When they deviate, they have a clear plan. They document their choice to use
something new and shiny, and they keep it separated from the default system
configuration.

Since SunOS came with Sendmail 5, the upgraded Sendmail 8 should have been
installed in some custom location with its own path so that it coexists with
the stock Sendmail, and is not perturbed if the OS happens to upgrade that.

A good syadmin would stick that in some /usr/local/bin type local directory,
and not overwrite /usr/bin/sendmail.

The consultant was not wrong to update the OS. People have reasons to do that.
The consultant should have consulted with the sysadmin, of course. But even in
that event, it might not have immediately occurred to the sysadmin what the
implication would be to the sendmail setup.

~~~
gabeio
> Say what? Nobody writes a sendmail.cf from scratch, unless they are crazy.
> The point moreover was that he had a custom version of the config file (not
> just default).

~~~
kazinator
Yes, sites have necessary customizations in sendmail.cf. These do not have to
be rewrites that use shiny new syntax.

My biggest problem with the author was not that he uses his admin blunders as
a basis to call himself a good sysadmin, but that he assumed that the stats
people were idiots who don't know anything about `puters or networks.

I was not surprised by the 500 mile claim. It strikes me as obvious that the
500 miles has to do with some combination of network topology and propagation
delays, those being approximately the same in every direction.

Yes, networking does work "that way": farther places take more time to reach
than nearer ones, broadly speaking. (Of course, it's faster to reach something
12,000 km away with no packet switch in between than something 50 miles away
with switching. That doesn't eliminate the generality.)

It was also obvious why they didn't report the problem instantly; you _cannot_
instantly know that mail isn't reaching beyond 500 miles without gathering
data and correlating to a map, which takes time. Instantly, you can only know
data points like "I can't mail to users@example.com". You know that if a stats
person gives you a number, it was based on data, and not just a couple of data
points. The head of the stats department isn't going to give you a number that
isn't factual and backed by science. Of course stats people pride themselves
on their data analysis; they are not just going to relay a couple of data
points with no analysis attached.

~~~
mfringel
Your superior knowledge has been noted by the Ministry of Statistics. Thank
you for supplying this data point.

