
The case of the 500-mile email (2002) - libeclipse
https://www.ibiblio.org/harris/500milemail.html
======
curtis
The 500-mile email story seems to be the Hacker News equivalent to Reddit's
SR-71 speed check story [1]. It keeps coming back again and again because
everybody loves it.

[1]
[https://www.reddit.com/r/SR71/comments/2dpmw7/the_sr71_speed...](https://www.reddit.com/r/SR71/comments/2dpmw7/the_sr71_speed_check_story/)
(and many others, too numerous to reference)

~~~
okket
[https://xkcd.com/1053/](https://xkcd.com/1053/)

~~~
rconti
This new XKCD comic is quite amusing! I read back through several of them and
they're so true to my life experiences!

------
throwaway-emc2
[throwaway account] A couple of years ago, my company (SEA) was bought by a
private equity-based company that was assembling a portfolio of similar
companies. The IT group got taken over by a competent yet overworked group in
their main office (PHX.)

The remote IT group wanted to move everyone's computer to their windows VMs
that were in their colo in PHX, which were managed (and presumably logged) out
of a central management panel. From their point of view, this was a great
solution. I was trying to get them to put a separate VM bank up locally,
because of the speed of light delay (at least 15ms) that would be small but
annoying.

I had convinced the remote IT people that this would be a good idea, and also
the [PHX] CTO apparently thought it was a good idea, but my local 'location
boss,' who used to run our company decided that I was just doing it to
increase my power. (which was untrue, because I would be going back to just
ENG and not IT after the transition.)

So, my boss cancels it and throws me under the bus to management at our new
home office. It takes a couple of weeks, then the terminals that log into the
remote hosts get installed. The speed of light is one of the constants of the
universe, so the problem I mention immediately happens.

My location boss gets annoyed by the slow performance on his computer and,
remembering that I'd said something about slowness, storms into my office and
starts demanding that I fix the computers. I try and calm him down and tell
him that there's no way to fix this without moving the terminals & vms closer
together.

He calls up the remote IT folks and they agree with me. He, at his next trip
down to HQ, starts slagging on me to anyone who'd listen. Various people at
the new home office warn me about this, but there's really nothing I can do
about it.

After some hilarity, I'm transferred to a project that was designed to fail so
there can be paperwork justifying laying me off.

So that is _another_ way the speed of light can have real-world consequences.

------
alekratz
> And I was fairly certain I hadn't enabled the "FAIL_MAIL_OVER_500_MILES"
> option.

If this actually was a config option, I would bet anything the default would
inexplicably be set to "true".

~~~
komali2
It's like this weird setting in Ubuntu's network driver config file,
"11n_disable." I won't pretend to know much about it but every time I install
Ubuntu on a machine, wifi internet speeds are slow until I set the above to
"8". To me it seems like literally a "go faster" switch.

~~~
rincebrain
Only 8, and not 1?

Because, at least on my system, looking at iwlwifi...

parm: 11n_disable:disable 11n functionality, bitmap: 1: full, 2: disable agg
TX, 4: disable agg RX, 8 enable agg TX (uint)

So 8 is only aggregated TX, while 1 would be aggregated TX and RX. (It does,
however, also inexplicably default to 0 on my system.)

~~~
komali2
Oh neat, lemme try that out then.

~~~
rincebrain
I'd be curious to learn what it does for you, because my intuitive
interpretation would say that 1 would be "disable all", but that doesn't
explain why "enable agg TX" would be an improvement unless agg RX is worse
than the baseline...which is probably a bug.

e: Oh, apparently this is a pathological re-use of an existing module flag to
force something on that they disabled due to instability in some cases, see
also:

[https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/lin...](https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=205e2210daa975d92ace485a65a31ccc4077fe1a)

So I'm sorry, but my assumption about =1 is not valid, as I assumed that all
of the bits (1,2,4,8) were mutating the same thing, not "one bit does
something VERY DIFFERENT".

------
yoav_hollander
Fun story - never heard it before.

BTW, I think information travels over "normal" networking gear (e.g. fiber
optics) at about 70% of the speed of light (which is why high-frequency
trading tends to move to over-the-air microwave networking to shave a few
nanoseconds - see [1]). But I guess the story still works, at the resolution
it is told.

[1] [https://en.wikipedia.org/wiki/High-
frequency_trading](https://en.wikipedia.org/wiki/High-frequency_trading)

------
YesThatTom2
I was a friend of Trey back then and I remember him explaining this and
thinking "wow, that's some good debugging!"

You'd be amazed at how many people tell Trey that this story is fake. Ugh.
There were plenty of witnesses.

Trey made news recently during the flood of negative United Airlines stories:

[http://www.newnownext.com/how-united-airlines-almost-
ruined-...](http://www.newnownext.com/how-united-airlines-almost-ruined-a-gay-
couples-honeymoon/04/2017/)

------
stevefeinstein
TIL: units is a really useful utility command that I had forgotten even
existed.

~~~
late2part

      You have: 100 millilightseconds
      unknown unit 'millilightseconds'
      You have: ^D
    

Apparently I need to add more units to achieve this level of usefulness.

~~~
dmacedo
Something tells me you're on OS X, and like myself, you'll find you mostly
want everything replaced with GNU binaries:

    
    
      $ brew install gnu-units
      $ gunits
    

Also see: [https://apple.stackexchange.com/questions/69223/how-to-
repla...](https://apple.stackexchange.com/questions/69223/how-to-replace-mac-
os-x-utilities-with-gnu-core-utilities)

------
nashashmi
A lengthier discussion:

[https://news.ycombinator.com/item?id=9338708](https://news.ycombinator.com/item?id=9338708)

------
krylon
Reading this kind of debugging story is kinda like watching a horror movie.

It's very interesting and fun to read if you know the kind of problems you
have to deal with are tame and furry in comparison.

But if I actually had to troubleshoot a problem like this, it would give me
nightmares for weeks.

------
jessaustin
Statisticians make really good bug reports!

~~~
DamonHD
Maybe in this case, with n=1, but we'd need a better sampling of impoverished
departmental heads to support your thesis.

------
wmichelin
I'm a simple man, I see the 500 mile email story, I upvote.

------
orangepenguin
I thoroughly enjoyed this story, but wonder if the details have changed a bit
over time, perhaps? If light can travel ~558 miles in 3 milliseconds, it seems
that mail sent more than 250 miles would have failed (because RTT)?

Also, I'm not sure how big a difference it makes, but signals propagate at
less than the "speed of light" (vacuum) when the medium is copper.

------
js2
I've read this story dozens of times and this is the first time SunOS jumped
out at me. Around 2001 or so there was a Solaris release (2.7 maybe?) that had
a bug in a particular patch release where sleep() returned instantly. That
would have exhibited all sorts of interesting symptoms such as this story.

~~~
shakna
I had the joy of re-engineering a terrible in-house web server because of that
bug.

It had used the system sleep to pause execution between the fake threads in
their awful program. Suddenly everything was hitting race problems, instead of
about 10%. Not something I really understood why it was going so bad, so I
just dumped the inhouse implementation of threads, and took up the Solaris
one, which was a joy by comparison.

~~~
mikeash
Sleep() returns early if a signal is delivered to the program, so doesn't your
code need to handle this case anyway? Of course, this is super rare, so
incorrect code could survive for a long time on a correctly implemented
system.

~~~
geocar
You should always handle your errors, but people trust in exceptions and "what
works" often enough that we have a culture that doesn't.

[http://geocar.sdf1.org/close.html](http://geocar.sdf1.org/close.html)

------
danschumann
"And I was fairly certain I hadn't enabled the "FAIL_MAIL_OVER_500_MILES"
option."

I laughed.

------
sdoering
Already really often times submitted. Last time was (I believe) 24 days ago:

[https://news.ycombinator.com/from?site=ibiblio.org](https://news.ycombinator.com/from?site=ibiblio.org)

[Edit] Typo

~~~
sjcsjc
I love this every time I see it. Newcomers may not have seen it so reposting
doesn't seem a problem. And 24 days ago it was totally ignored in the new
submission lottery, so again reposting doesn't seem a problem.

~~~
stefanpie
It was my first time reading about it; it was fantastic.

~~~
cyphar
If you haven't read it before, there's also "The Story of Mel" which is quite
a bit older (1983) but is also a very fun read.
[http://www.catb.org/jargon/html/story-of-
mel.html](http://www.catb.org/jargon/html/story-of-mel.html)

------
reaperducer
Stories like this remind me of the days when the internet was young, and fun,
and silly, and best experienced in its green-on-black usenet glory. (418
Error: I'm a teapot, for example.) The internet just doesn't seem fun anymore.

~~~
chrissnell
It's not that the Internet has lost its fun--it's just too easy to find it
these days. You appreciated everything a little more when you had to get on
your bike and ride a mile across campus to the campus computing center just to
read that silly USENET post on a 80x24 serial terminal. Now you can read it on
your watch or your glasses or yell at your home stereo to read it to you.

I used to wonder what it would be like to time-travel to ancient times and
show a gun or a car to a Neanderthal. I think it would be just as crazy to
show 2017 me to 1993 me.

