
We can't send email more than 500 miles (2002) - alanpage
http://web.mit.edu/jemorris/humor/500-miles
======
thyrsus
I've had enough years to become wiser, become a fanatic for configuration
management, and get over the embarrassment: I'm the consultant that screwed
things up. Some background: the Stat department was running a variety of
systems besides the Solaris workstations, and there was, within UNC-CH, a
separate support organization that was cheaper and more comfortable with
Microsoft products where Stat was sending their support dollars. When that
organization needed Unix support, they called my employer, Network Computing
Solutions, and I showed up.

There was effectively no firewall at UNC-CH at the time (something something
academic freedom something something), and the Stat Solaris machines were not
being regularly patched. Uninvited guests had infested them, and it appeared
the most likely entry point was sendmail - at the time, it was the most
notorious vulnerability on the internet. Since my preference to wipe and
reload was unacceptable - too much downtime and too many billable hours - the
obvious thing to do was update sendmail. The rest is history.

~~~
sillysaurusx
Can I get your autograph? The 500 mile email story has been a story since I
was a teenager.

Thank you for the comment. It was delightful to hear your take on the (mildly
apocryphal, but highly enjoyable) tale.

~~~
sah2ed
How about a joint autograph from both the protagonist—Trey Harris, and the
antagonist—the consultant to the Stats department :) ?

Trey Harris also on HN [0].

[0]
[https://news.ycombinator.com/threads?id=TreyHarris](https://news.ycombinator.com/threads?id=TreyHarris)

~~~
techdragon
Yes. Seriously... I’m willing to pay for the shipping and everything. I’ve
told this story countless times to people over the years. This and the
“OpenOffice can’t print on Tuesday” bug are two of my favourite
troubleshooting stories.

~~~
franky47
HN thread for the OpenOffice story:

[https://news.ycombinator.com/item?id=8171956](https://news.ycombinator.com/item?id=8171956)

~~~
dang
Also
[https://news.ycombinator.com/item?id=11717010](https://news.ycombinator.com/item?id=11717010)

and
[https://news.ycombinator.com/item?id=22157430](https://news.ycombinator.com/item?id=22157430)

------
surround
Source:

[https://www.ibiblio.org/harris/500milemail.html](https://www.ibiblio.org/harris/500milemail.html)

FAQ about the story:

[https://www.ibiblio.org/harris/500milemail-
faq.html](https://www.ibiblio.org/harris/500milemail-faq.html)

Past submissions:

[https://hn.algolia.com/?dateRange=all&page=0&prefix=true&que...](https://hn.algolia.com/?dateRange=all&page=0&prefix=true&query=500%20mile%20mail&sort=byPopularity&type=story)

[https://hn.algolia.com/?dateRange=all&page=0&prefix=true&que...](https://hn.algolia.com/?dateRange=all&page=0&prefix=true&query=http%3A%2F%2Fweb.mit.edu%2Fjemorris%2Fhumor%2F500-miles&sort=byPopularity&type=story)

(It seems I can’t find a search query which includes both urls)

~~~
surround
2015 Comments from the original author:

[https://news.ycombinator.com/threads?id=TreyHarris](https://news.ycombinator.com/threads?id=TreyHarris)

------
drtz
I love that this took a perfect storm of having a statistician and sys admin
both bent on finding the cause of a weird intermittent problem in their own
way.

This could have happened a million times where the story was a lot less
interesting:

"Hey, I'm having weird intermittent problems sending email."

"Hmm, we're using the wrong version of Sendmail. All fixed, case closed."

------
abhgh
I used to collect these kind of stories:

When I flush my toilet my computer reboots:
[http://www.techtales.com/tftechs.php?m=199712#66](http://www.techtales.com/tftechs.php?m=199712#66)
(the first story on the page)

If I buy vanilla ice-cream my car wont start:
[http://www.netscrap.com/netscrap_detail.cfm?scrap_id=501](http://www.netscrap.com/netscrap_detail.cfm?scrap_id=501)

A specific cargo routing crashes system: [https://www.jakepoz.com/debugging-
behind-the-iron-curtain/](https://www.jakepoz.com/debugging-behind-the-iron-
curtain/)

Tape-drive failure only within large print jobs:
[http://patrickthomson.tumblr.com/post/2499755681/the-best-
de...](http://patrickthomson.tumblr.com/post/2499755681/the-best-debugging-
story-ive-ever-heard)

Interplanetary debugging with the Mars Rover: [https://www.eetimes.com/the-
trouble-with-rover-is-revealed/#](https://www.eetimes.com/the-trouble-with-
rover-is-revealed/#)

~~~
photonic277
In my intern days some time around 10 years ago, a PI at the NASA GRC facility
told me about a problem of this flavor an old grad student of his had.

The guy was working on an optical sensor in a light-tight lab. Every morning,
he came in, calibrated the sensor, and performed measurements. All morning, it
held calibration with negligible drift. But when he came back from lunch, each
time, the calibration had drifted off.

Could it be related to the time of day? He tried taking his lunch an hour
earlier and an hour later. Each time, the calibration was rock solid until
right after lunch.

In spite of protocol, he tried eating lunch in the lab, no one else in or out.
Before lunch: good calibration. After lunch: bad calibration.

He tried not eating lunch at all. That day, the calibration held all day.

How could an optical sensor have any concept of whether its user had eaten
lunch? It turned out, it only had to do with the lunch box. The sensor was
fiber coupled, and it was sensitive to changes in transmission losses
generated by changes to local radii of the patch chord. Every morning, the
grad student set his lunch box down on the lab bench, nudging the fiber into
some path. After eating, he’d replace his lunch box on the bench, nudging the
fiber into a different path.

After that, the fiber was secured with fixed conduit, and lunch boxes no
longer entered the lab.

------
wuster
Best part of reading this is coming away having learned the existence of units
the CLI. How did I spend 20 years on the shell and not have needed or
discovered this?

~~~
ornornor
Be aware that currencies are stuck with rates from several years ago and don’t
update.

~~~
duckerude
Running `sudo units_cur` does the trick for me.

    
    
      $ units
      Currency exchange rates from FloatRates (USD base) on 2020-05-12
      $ sudo units_cur
      $ units
      Currency exchange rates from FloatRates (USD base) on 2020-07-09
    

(GNU units, packed by Debian)

~~~
throwaway8941
In case anyone else needs this:

    
    
      # systemctl edit units-currency-update.service
      
      [Unit]
      Description=Update units(1) currency rates
      
      [Service]
      Type=oneshot
      Nice=19
      ExecStart=/usr/bin/units_cur
      
      # systemctl edit units-currency-update.timer
      
      [Unit]
      Description=Update units(1) currency rates
      
      [Timer]
      OnCalendar=daily
      AccuracySec=3h
      Persistent=true
      
      [Install]
      WantedBy=timers.target
      
      # systemctl daemon-reload
      # systemctl enable units-currency-update.timer

------
cbhl
Previously:

2018:
[https://news.ycombinator.com/item?id=18675375](https://news.ycombinator.com/item?id=18675375)

2015:
[https://news.ycombinator.com/item?id=9338708](https://news.ycombinator.com/item?id=9338708)

~~~
anamexis
A few more:

[https://news.ycombinator.com/item?id=12778904](https://news.ycombinator.com/item?id=12778904)

[https://news.ycombinator.com/item?id=21578606](https://news.ycombinator.com/item?id=21578606)

[https://news.ycombinator.com/item?id=16384158](https://news.ycombinator.com/item?id=16384158)

[https://news.ycombinator.com/item?id=16285093](https://news.ycombinator.com/item?id=16285093)

[https://news.ycombinator.com/item?id=12997568](https://news.ycombinator.com/item?id=12997568)

[https://news.ycombinator.com/item?id=3802458](https://news.ycombinator.com/item?id=3802458)

[https://news.ycombinator.com/item?id=123489](https://news.ycombinator.com/item?id=123489)

[https://news.ycombinator.com/item?id=385068](https://news.ycombinator.com/item?id=385068)

[https://news.ycombinator.com/item?id=2701063](https://news.ycombinator.com/item?id=2701063)

[https://news.ycombinator.com/item?id=14676835](https://news.ycombinator.com/item?id=14676835)

[https://news.ycombinator.com/item?id=1293652](https://news.ycombinator.com/item?id=1293652)

[https://news.ycombinator.com/item?id=10305377](https://news.ycombinator.com/item?id=10305377)

[https://news.ycombinator.com/item?id=12126564](https://news.ycombinator.com/item?id=12126564)

[https://news.ycombinator.com/item?id=6296201](https://news.ycombinator.com/item?id=6296201)

[https://news.ycombinator.com/item?id=3427263](https://news.ycombinator.com/item?id=3427263)

[https://news.ycombinator.com/item?id=17591084](https://news.ycombinator.com/item?id=17591084)

[https://news.ycombinator.com/item?id=8176458](https://news.ycombinator.com/item?id=8176458)

[https://news.ycombinator.com/item?id=3114391](https://news.ycombinator.com/item?id=3114391)

[https://news.ycombinator.com/item?id=19321973](https://news.ycombinator.com/item?id=19321973)

[https://news.ycombinator.com/item?id=13667317](https://news.ycombinator.com/item?id=13667317)

[https://news.ycombinator.com/item?id=13082438](https://news.ycombinator.com/item?id=13082438)

[https://news.ycombinator.com/item?id=10734108](https://news.ycombinator.com/item?id=10734108)

[https://news.ycombinator.com/item?id=22065142](https://news.ycombinator.com/item?id=22065142)

[https://news.ycombinator.com/item?id=19749556](https://news.ycombinator.com/item?id=19749556)

[https://news.ycombinator.com/item?id=9081869](https://news.ycombinator.com/item?id=9081869)

[https://news.ycombinator.com/item?id=7892556](https://news.ycombinator.com/item?id=7892556)

[https://news.ycombinator.com/item?id=17602158](https://news.ycombinator.com/item?id=17602158)

~~~
akeruu
Yes, this story gets posted a lot, and many of us might know it, but at the
same time, I like to think about the ones that didn't. They will learn
something new today. XKCD said it better than I could:
[https://xkcd.com/1053/](https://xkcd.com/1053/)

~~~
dredmorbius
Genuine perennial favs are worth repeating every so often --- a year or two's
interval seems reasonable, and is vouched by HN.

A marker of aging for me was seeing, a decade or two after I'd first read them
in the local paper as a callow youth, repeats of previous features, by topic
if not the actual text. Eventually the thought occurred to me that perhaps the
versions I'd remembered were themselves not original.

People tell, and repeat, and embellish, stories. Sometimes because the
young'uns and whippersnappers and new arrivals haven't heard them yet.
Sometimes because they're just damned good stories and we enjoy the retelling.

~~~
divbzero
Yes, I’ll continue upvoting this classic when it pops up in the years ahead.

------
abeppu
The ending of this makes it sound super clean. 3 ms * speed of light => ~560
miles. "It all makes sense!"

But ... isn't the speed of light through fiber actually like 2/3 of the speed
of light in a vacuum? And that fiber isn't going to be laid out in a straight
line exactly to the destination. So I think really there must have been a fair
bit of uncertainty around that ~3ms to abort a connection.

~~~
comex
The FAQ addresses this:

[https://www.ibiblio.org/harris/500milemail-
faq.html](https://www.ibiblio.org/harris/500milemail-faq.html)

~~~
frandroid
Since the story has been posted umpteenth times, this is what should be posted
these days. Very entertaining, thanks!

~~~
divbzero
Yes! Thank you GP this too was my first time reading that follow-up FAQ.

------
xook
Truly this is a worldly gem. Thank you for submitting this. :)

It's easy to forget that, even though transmissions still travel at near
lightspeed, it still takes more than an instant to reach its destination, even
digitally. I should keep this in mind, I think.

------
xwdv
IMO even though this has been posted a bunch of times it’s important to repost
these sort of campfire ghost stories so future developers can think more
creatively about strange errors.

~~~
mensetmanusman
Is there a book written about these campfire programming stories?

This one and the microsoft bedlam one are some of my favorites.

~~~
xwdv
Not sure, but there’s plenty of stories out there that one could probably be
compiled.

------
martyvis
Only the other week we were doing some testing on a new HCI (hyper-converged
infrastructure) I'm doing the network for. At the end of the test period, we
were having some storage sync issues. Everything seemed to PING ok, except my
colleague happened to notice large jumbo frames over 8000 bytes were getting
dropped. I double checked that we hadn't inadvertently changed network
configuration. It was only by chance we had another test looking at
transceiver signal levels that a customer engineer saw an alarm on RX level.
It was then we remembered one test was to remove a module. I then noticed some
error counts. We shutdown that particular link until we could visit the site.
Sure enough, that fibre wasn't quite clicked in anymore. There was enough of a
bridge across the fibre air gap for shall packets, but just wide enough so
large packets statically couldn't be corrected enough to work.

~~~
lostmyoldone
Made ne think about the precision required for some errors to occur. Have had
sort of similar things occur, and where it's almost impossible to reproduce it
when you try!

As a hobbyist sound engineer, usually regular cables are the first I check,
but maybe I should extend that to fibre?

Interesting error nevertheless, and honestly, checking fibre cabling for those
kind of errors would probably be a bit lower on my list, unless I saw a lot of
tranciever errors.

------
rogierhofboer
A while back, Andreas Zwinkau collected this kind of stories. Please post him
if there is anything not yet listed.

[http://beza1e1.tuxen.de/lore/index.html](http://beza1e1.tuxen.de/lore/index.html)

------
noisy_boy
If somebody is tracking apt/yum downloads of packages, they might see a sudden
spike for "units". Just installed and it is a nifty little useful tool.

------
ryanmarsh
Love this story.

I'm certain it will continue to be reposted to this website until ceases
operations or the heat death of the universe whichever comes first.

------
PlasticTank
Love this story, funnily enough it was the first story I ever read on HN on my
first day of work as a junior developer.

------
noisy_boy
Edit: Found it - its /usr/share/units/definitions.units (on Pop!_OS, so
probably same on Ubuntu/Debian).

The FAQ[0] mentions:

> units on SunOS doesn't know about "millilightseconds."

> Yes. So? I used to populate my units.dat file with tons of extra prefixes
> and units. And actually, I think I was using AIX to run units; I don't know
> if it knew about millilightseconds. Take a look at the units.dat shipped
> with Linux these days. It definitely knows about millilightseconds.

I tried locate for units.dat but couldn't find it. Anyone knows where is it?
Not keen on running a system-wide find.

[0]: [https://www.ibiblio.org/harris/500milemail-
faq.html](https://www.ibiblio.org/harris/500milemail-faq.html)

~~~
hackerNoose
units -U gives you the location of definitions file

------
bobblywobbles
Can someone please explain to me the POP reference? I do not understand what
this author means by that.

I also would like help understanding what $ units gives? The command looks to
be "units", but where do the numbers he entered in come from? I would
appreciate this extra context.

~~~
sonofgod
$ is the shell prompt; he's not typing it. "3 millilightseconds" is the
distance light travels in 3 milliseconds, the time a "zero" timeout would take
to actually timeout. (This comes directly from the definition of the
lightsecond: how far light travels in one second) "miles" is what he wants to
see that distance converted to. Turns out it's 558 miles; one mile is 0.00179
of 3 millilightseconds.

------
davidkuhta
Hat tip to @nfriedly from back in 2011:

FYI units on OS X doesn't recognize millilightseconds, but you can do this:

    
    
      You have: 3 lightyears / 365 / 24 / 60 / 60 / 1000
    
      You want: miles
    
       * 559.21802
    
      / 0.0017882113

~~~
jxy

        % units
        586 units, 56 prefixes
        You have: 3 lightyear*millisec/year
        You want: mile
         * 558.84719
         / 0.0017893979
    

or you can patch up the library file

    
    
        /usr/share/misc/units.lib

------
MasterYoda
Haha, this was interesting. I posted the same story with more or less the same
title a couple of month ago [1], and no one saw it and no comments. This time
it got almost 1000 points and a lots of comments. What I think is interesting
is how the same thing can get so different traction. Wonder what factors it is
that makes a thing get traction and not?

[1]
[https://news.ycombinator.com/item?id=22164691](https://news.ycombinator.com/item?id=22164691)

------
rcarmo
This story makes me smile every time it comes up. It's fascinating how many
arbitrarily coded limits we keep breaking as we make our tech go faster
without re-assessing the original assumptions :)

------
ip_addr
Does anyone know of stories similar to this? I would like to read more.

~~~
Diederich
I don't have time to type it up more completely, but back in the early to mid
90s I administrated, among others, some AT&T 3B2 servers. (
[https://en.wikipedia.org/wiki/3B_series_computers](https://en.wikipedia.org/wiki/3B_series_computers)
)

There was an odd situation where some of the systems were unable to make
connections to other systems ....some largish distance away. It was fairly
variable, but some systems were almost always unreachable, and some were
occasionally reachable.

Long investigation, but in summary, this happened because the default packet
TTL was, for some reason, set to a fairly low value in a minor kernel update.
I simply increased that number in the kernel, recompiled, and all of the
problems went away.

------
staycoolboy
The one thing that throws a wrench in this story for me: Lattes.

Latte's in 1994? In North Carolina? No way. Maybe on the West Coast, but I
moved to Cali in 1989 and they were a rarity until the mid-late 90's. There
were only 425 starbucks in the US in 1994 (from their site). The "fancy
coffee" craze was just a blip on the radar in the mid 90's but gaining
momentum.

;-)

~~~
function_seven
> _The "fancy coffee" craze was just a blip on the radar in the mid 90's but
> gaining momentum._

 _Friends_ premiered in 1994, with The Central Perk being a major set piece of
the show. I mean, yeah, its New York City and not North Carolina, but college
towns anywhere are going to be early in trends.

A latte in 1994 seems plausible to me. I remember getting them from a Gloria
Jeans in my local suburban mall around 1990 or so.

You're not wrong about the shape of the trajectory, but all throughout the 80s
the coffee shop/latte trend was slowly building steam (heh) before it went
hockey stick in the mid-90s.

~~~
staycoolboy
Oh yeah, that's right. Good point! I just illustrated how unhip I was in the
early 90's.

------
henpa
Since I've worked with linux email servers (sendmail, qmail, postfix, exim,
etc) practically my whole profissional life (since around 1997, but used BBS
since 91 - I'm 41 now), this story really amused me and got my attention! I
_love_ this kind of email debugging! LOL

------
darekkay
There is a popular GitHub repository for similar debugging stories:
[https://github.com/danluu/debugging-
stories](https://github.com/danluu/debugging-stories)

------
walrus01
In network engineering circles, this one is second only to the RFC for IP over
avian carriers.

~~~
vidarh
There was also this "implementation" of IP over avian carriers:

[https://www.blug.linux.no/rfc1149/](https://www.blug.linux.no/rfc1149/)

------
doebi
Soundtrack for this story:
[https://www.youtube.com/watch?v=tbNlMtqrYS0](https://www.youtube.com/watch?v=tbNlMtqrYS0)

------
bawolff
I've heard this story before, but i didn't realize it was as recent as 2002.
It feels like something from a much earlier bygone era, like the early 90s

~~~
wh1t3n01s3
In fact it is from 1994 -1997, as explained in the FAQ linked above

------
failrate
In a way, Hacker News acts like a Leitner deck.

------
kinduff
This is one of my favorite bugs ever. Loved to read it again.

------
panosfilianos
Debugging software is not unlike detective work.

------
dang
Anybody know the original year?

------
optimaton
This is wild!

------
netmonk
So old! DUPE

------
coronadisaster
just get it in the pipes

~~~
glitchc
Tubes, alright?! The Internet is a series of tubes!

[https://youtu.be/f99PcP0aFNE?t=127](https://youtu.be/f99PcP0aFNE?t=127)

~~~
coronadisaster
lol, thanks for the correction.

------
simonebrunozzi
I suggest to change "mail" to "email" in the title.

~~~
dang
Ok, we've e'd the mail above.

