
Ask HN: Are there forgotten servers out there? - forgottenacc56
I was wondering if there are &quot;forgotten&quot; servers out there. No longer doing anything, but still up and running.<p>Is there any way to even know?
======
owenwil
Yes - in my network admin days back in 2007, we were decommissioning our
datacenter as we'd virtualized everything and as we were gutting the room we
found our PABX/Voicemail server under the tiles, happily running away with
OS/2\. Nobody ever knew where it was and it had been installed 13 years
before. The floor was on a UPS and generator so the power had never been
disconnected.

It was happily chugging away, running our shitty phone system, and hadn't been
restarted in 10+ years.

Edit: This also happened to me at a utility company I worked at. We had a
server that ran some critical calculations for us but nobody could recall
where it was physically located. It's way worse now that everything's
virtualized - the cruft just sits there for years until you suddenly run out
of resources and start looking closely.

~~~
Havoc
>nobody could recall where it was physically located.

ah yes, the good old "responds to pings but where is it?"

~~~
mozumder
Does the ping protocol have any capability to provide location information?
Because it should if it doesn't..

~~~
jlgaddis
The usual method is to get the device's MAC address, use that to track
down/identify the switch port it's connected to, then physically trace the
network cable that's plugged into that switch port.

~~~
edwhitesell
Or, if the hardware supports it, eject the CD/DVD ROM drive.

~~~
wmil
Do servers have onboard speakers these days? Can you make them beep in a
distinct pattern?

~~~
jlgaddis
Many/most servers (or the ones I've dealt with the most, anyways) have an LED
that can be activated remotely -- i.e. through IPMI, ILO, etc. -- but I can't
remember what's it called at the moment. It can be very handy when you're
remote and need to be able to direct someone (physically present) to a
specific server.

------
cddotdotslash
At a place I worked we had a service that relied on a cron job running a
script every hour. One day someone wanted to know where the server was so he
could connect an external hard drive to it and copy some files in. Not a
single person knew where the server was or even what it's IP address was.
Since it had been setup years ago, everyone who had worked on it had since
left the company and no one ever documented it. So somewhere either in our
office or in one of three public clouds we used was a server happily running a
script every hour that no one could locate or stop. We eventually found it
when we moved offices and a super old Dell box in a tiny dusty closet was
unplugged and the script stopped running. Now I always document the physical
locations of services too.

------
pi-rat
Oh, Yes! In a previous job we developed VoIP servers (pbxes). One day a
customer started experiencing some really weird problems and I was asked to
debug it.

I logged into the server and started analyzing traffic, turned out that
traffic on a upstream VoIP switch didn't always match the traffic leaving the
server I was debugging. It was as if a identical system was getting and
responding to parts of the traffic. - and after some more debugging I
discovered that there was a older identical system online somewhere in their
server room. Years ago - all services had been stopped, the system backed up
and migrated to a new server. One day there had been a power loss, and when
the servers rebooted, the old system everyone had forgotten about launched
it's previously stopped services - causing the customer all sorts of weird
VoIP problems.

------
SNACKeR99
I set up one of these deliberately in younger days...

As a bored government employee in the early 1990s, I become fascinated with
the WWW. I was a network admin (NetWare 2.15c), and there was a fat, unused
internet pipe and several unused phone lines. I started to mess around with
Linux (Slackware kernel 0.97 I think), and after two weeks I had it talking to
a Hayes modem. Voila, instant ISP! A little while later I installed Slirp, and
it became my personal dial-up connection for many, many years.

Before I left that job in 1995, I moved the server (headless) to a broom
closet (wrong of me, I know, I know). Knowing gov't culture, no one would mess
with something like that. It was up and running at least until 1998 or so, at
which point I moved to another country, and when I moved back, I no longer had
the dial-up number. I like to think it is running to this day.

------
quesera
There are many (possibly apocryphal) stories about servers lost in ceilings,
locked closets, etc that just kept on serving, sometimes critical services.

I think we as an industry have gotten much better about this. In the old days,
as small minicomputers and micros expanded into less and less technical
businesses, wiring standards and server room design guides were not well-known
or followed.

Often, people just sort of winged it. Some employees are more naturally
methodical, have better memories, and are longer-tenured than others. Also,
many of the stories are from universities, where long-term thinking isn't
guaranteed (but embarrassing stories have always been popular to share!).

A quick DDG finds this story from the University of North Carolina:

[http://www.bradleymonk.com/wp/weve-lost-
server-54/](http://www.bradleymonk.com/wp/weve-lost-server-54/)

USENET archives would be a great source for more of these.

------
olalonde
This guy's server: [http://bash.org/?5273](http://bash.org/?5273)

<erno> hm. I've lost a machine.. literally _lost_. it responds to ping, it
works completely, I just can't figure out where in my apartment it is.

~~~
iamcreasy
What to do in this cases? What are the options?

~~~
secure
Disconnect cables from each switch and see when the ping replies stop :).

~~~
huhtenberg
Unless it's on a WiFi connection, in which case things get substantially more
complicated :)

~~~
bdcravens
Typically you can login to the router and see machines attached, and kick/ban
MAC addresses.

~~~
rifung
That still won't help you find where the machine is located though. With
cables you can presumably just follow the cable when you realize disconnecting
one stops the pings

~~~
jevinskie
Wardriving software like kismet will show your the RSSI of the AP. Is it
possible with stock hardware to determine the RSSI of connected devices? Then
it is just the hot/cold game. I'm sure actual radio triangulation/direction
finding would be possible as well but I'm not sure how difficult that would
be.

------
miahi
This week I got a support call from one customer: "hey, we have a maintenance
scheduled for our Dell servers, is it ok if we shut down the two pre-prod
servers?" "what pre-prod servers?" [all the non-production servers were
virtualized about 4 years ago] "well, there are two servers here, with pre-
prod labels and running Linux; no applications are running on them"

So I guess 4 years ago somebody migrated the servers to a virtual environment
and then just forgot about them. The IPs were migrated to the VMs and these
servers were left without an IP address on the network connections, so apart
from going to the data room and checking every machine (what they did during
the maintenance) there was no way to find them. They were still up and running
after 4 years.

~~~
justizin
Auditing your racks periodically is good for many other reasons. RackTables
and similar tools are invaluable!

~~~
alinspired
Thank you for mentioning it, didnt know about the tool!

------
Thriptic
Oh god yes. A few years ago some labmates and I discovered that a post doc had
set up an entire computer cluster of 30 workstations / associated
infrastructure, colocated it at a data center nearby, then left without
leaving any documentation. Everyone that knew the cluster existed left soon
after, and so labmates that arrived later were running simulations on their
personal laptops while this cluster sat idle. We discovered it only because my
PI received a large bill for rack space and remembered it existed.

------
perlgeek
Sure, happens now and then.

I work at an ISP / housing / colocation company, and occasionally hardware
goes missing (nobody knows where it is anymore, it's not where it's supposed
to be). Maybe some of them are broken, some might be stolen, but I'm fairly
sure others are still running, not serving any purpose.

And from time to time we stumble over some virtual machine (or even phyiscal
server) where nobody knows anymore what it's supposed to do; the standard
procedure seems to be firewall it off, wait for a few weeks or months to see
if anybody complains, and if not, shut it down, maybe archive it.

~~~
simon_vetter
Same experience here. Servers get ordered, racked, provisioned and left
powered on waiting for a sysadmin to make use of them... which quite often
doesn't ever happen, because team priorities change over time, people move
on/get fired, entities go through reorgs, etc.

They sit there idling and unattended, burning power and disks, until some
script kiddie finds whatever default root password was used or how to exploit
some random apache/ssh flaw.

At that point the possibilities are endless: bitcoin miners are quite
unnoticeable in most environments, but DDOS/spam zombies, proxies, bittorrent
seedboxes, botnet C&C, "warez" and http servers serving drive by exploits are
fairly common.

Protip: ask your datacenter provider to power your servers down (be it VMs or
dedicated gear) after racking them up. Powering them back up when you _really_
need them will only take you a minute and you'll save big on power, bandwith,
security and peace of mind.

------
stevewilhelm
My father has an email account hosted by Verizon with the domain @gte.net.
General Telephone & Electric merged with Bell Atlantic in 2000 to become
Verizon.

My father's @gte.net email delivery has recently become spotty. After hours of
phone calls with Verizon, no one at any level of support can seem to find the
old GTE mail servers.

Some Googling for smtp.gte.net found this IP address, 207.115.153.29, which
seems to respond to pings but not smtp traffic.

------
jfuhrman
I am sure there are tens of thousands if not more. Server admins are very risk
averse and don't tend to turn off machines just to check if someone's using
it.

And in large organizations, it's hard to keep track of who's using what and if
they still need it.

Imagine working at a hospital or a bank, emailing people about a server, and
everyone says it's not being used. Then you turn it off and something critical
gets broken at 3am a week later resulting in an emergency. Who gets blamed,
you or the people who forgot the server was being used? The people who set it
up may not even be working there anymore.

------
dvirsky
Years ago I shut down a start-up, and as part of some business partnership, we
had a physical server hosted in some big company's datacenter. They forgot
about it, and I didn't have time or incentive to drive over there to fetch one
old machine. So it just kept running there for a couple years, at first just
continuing to serve my failed company's website, and then doing nothing, other
than serving as a personal download proxy for me. That lasted for about 2-3
years until they finally found it and shut it off, so I took it home.

------
thearn4
We came across one at work (NASA) a few years back, a server running in a far
off room that no one remembered anything about. It had probably last been used
by a grad student at least a decade ago, but was still up and running and
doing, well, something. We eventually had it removed and turned the room into
a collaboration space.

------
makeitsuckless
My 1996 homepage is still online under its original URL. Which likely means
the system hosting it is still online. Except the original ISP no longer
exists, nor does the ISP that bought the original ISP.

Next year I'm going to find that system on the 20th anniversary of that page.

~~~
frik
My first websites are still somewhat online too, as Archive.org and/or the
Archive Team managed to backup the site :)

------
driverdan
I've always wanted to intentionally setup servers in hidden places just to see
how long they stay online. For example, putting a Raspberry Pi in the back of
a rack in a server closet. Maybe run a Tor relay on it or something like that.

------
jason_pomerleau
Back in the late 80s / early 90s there was a BBS in my hometown (Windsor,
Ontario, Canada) called GB Hotel. It wasn't very popular, but if all the other
BBS lines were busy, it was something to do.

The owner of the BBS, handle "Kilroy" as I recall, hadn't logged into his own
BBS in _years_. Nor anyone else's, for that matter.

We used to joke that he'd probably died, and Mom and Dad just left his
Commodore 64 sitting there, wondering why their phone bill was twice as much
as their neighbours.

------
threeseed
Seen it a few times. Company acquires another company or an employees leaves.
And then you find out years later during an office refit that the PC under
their desk was running a critical business process.

~~~
engi_nerd
Once I was an intern at an air force base. In my cubicle, stuffed into a
corner, was an old SGI Octane workstation. It was absolutely covered in dust
and looked neglected. Its fan was very annoying to me. One day, I unplugged
the machine because I figured that it was completely purposeless. Less than 2
minutes later, someone I had never seen was standing at my desk, berating me
for turning off a server that ran a set of critical data reduction processes
for the nearby wind tunnel.

After that I added a sign to the machine. "DO NOT TURN OFF UNDER ANY
CIRCUMSTANCES." And made sure I wore headphones to block out the sound of the
fan.

~~~
Havoc
Should've added a note saying what it does too seeing how you did the old
"power it down & wait for someone to get angry" already.

~~~
engi_nerd
Yes, I should have. But I don't remember if I did or not. This happened in the
fall of 2005.

------
ForHackernews
[http://www.theregister.co.uk/2001/04/12/missing_novell_serve...](http://www.theregister.co.uk/2001/04/12/missing_novell_server_discovered_after/)

 _One of the university 's Novell servers had been doing the business for
years and nobody stopped to wonder where it was - until some bright spark
realised an audit of the campus network was well overdue._

 _According to a report by Techweb it was only then that those campus techies
realised they couldn 't find the server. Attempts to follow network cabling to
find the missing box led to the discovery that maintenance workers had sealed
the server behind a wall._

------
eleusive
I know of one interesting story here. Early on in (what would eventually
become a very successful) tech company's life, they acquired some collocated
space for free thanks to a friend (call him Brad) of an early employee.

Eventually Brad left that particular DC operator, leaving the startup with no
inside contact - but the servers stayed up for years to come (and the company
was never charged for the resources).

These machines were eventually "replaced" with newer hardware in a nearby
facility, but the DC operator has no idea where these old machines are
physically located within the facility (thus cannot remove them), so they
remain active to this very day, sitting idly by...

~~~
jon-wood
If its in a DC can't they track down the MAC of whatever is responding to
their IP, and then trace it to a switch port?

------
lcmatt
Previous company I was at had a small server setup to run a few automated
tasks every morning. It must have been running for at least 10 years before a
new employee asked where it was physically located.

They eventually found it covered in dust in a small utility closest. None of
the fans were working either (other than the CPU/PSU) yet it still chugged
along doing its job.

------
phamilton
I'm always fascinated by StorJ.

[http://garzikrants.blogspot.com/2013/01/storj-and-bitcoin-
au...](http://garzikrants.blogspot.com/2013/01/storj-and-bitcoin-autonomous-
agents.html)

TLDR; a SaaS system that not only auto-scales, but does so autonomously. When
it has excess funds, it buys more VMs and provisions them with itself. If
cashflow is low, it shuts some down. Add in the ability to create accounts
(using bitcoin as currency) and you could very quickly lose the paper trail
and have a SaaS owned by the Æther.

------
butterfi
There's the story about the novell server left running in a walled-off room:

[http://www.theregister.co.uk/2001/04/12/missing_novell_serve...](http://www.theregister.co.uk/2001/04/12/missing_novell_server_discovered_after/)

------
effhaa
Not really "doing anything", but related: I had to set up a voip server for a
test project in 2003, which was supposed to be terminated half a year later.
Being a test project, I was advised not to document anything formally, just to
place it in the rack and ignore everything else. I left this job soon after.

12 years later, the phone number still is working, even though the companies
phone system has been relocated to another room and was replaced by another
vendor. No idea how this happened.

~~~
perlgeek
> No idea how this happened.

It could very well be that somebody migrated all the relvant phone numbers off
the test system, and chose to preserve those numbers where they didn't know
what they were needed for. Might be cheaper than risking losing functionality
that somebody depends on. (Especially if the one doing the migration didn't
know it was an abandoned test system).

------
frik
I heard a company forgot about their third datacenter (second backup), as
several sys-admins left the company. It was found in an old bunker below the
basement of a building.

~~~
spydum
i work for a pretty large enterprisey-corp and this is very close what
happened to us. Early 2000's we had a DR setup for a specific platform at a
3rd party hosting provider. Mid-2000's was major transition of infrastructure
folks.. the DR site being completely out of date and never used, was abandoned
and forgotten. They continued paying the bill for the 3rd party for no joke, 4
years until someone realized that it was useless hardware.

------
snowpanda
Intriguing question; I would think that a certain percentage of servers might
have survived their owners post-mortem. Of course there could be many
additional reasons for abandoned servers.

------
olh
AWS instances on the free tier that starts to charge your credit card after a
year.

~~~
znpy
Oh, I must pay attention to that.

------
wanda
We have several gorgeous AS/400s still going strong, quietly gathering dust
while they wait for the next time we need to test something on them.

I think these servers will outlive me.

------
thenomad
Well, every six months or so I review my various VPSes and I usually find at
least one I'd forgotten about, so unless I'm unusually forgetful I'd say "yes"
:)

------
facepalm
Apart from normal servers, lots of embedded devices probably also have
integrated servers. There must be many, many of them idling away somewhere.

~~~
Raed667
MOTES have an average lifetime of 5 years. Unless they are linked to a power
grid and stable internet. Most of these are either
broken/frozen/unreachable/out of power.

We actually study this special case in Embedded systems engineering.

------
rbanffy
Right after I was hired by a large portal part of a much larger telco, we were
decommissioning one of our data centers. In one of the planning meetings there
was a discussion about an undocumented class C range that nobody knew what
was, yet had some significant traffic. I never knew what that network was or
what was running there. As far as I know, someone may have lost their spam
relays or their phishing hosts.

Another fun story happened years before. There was a large power outage at the
data center of another large Brazilian portal. Three days later, someone calls
our office, asking if we remember what kind of hardware the server was running
on. The machine didn't boot and they needed to get to its console.
Unfortunately, nobody knew where the machine was physically located or what
did it look like. In the end, it was found inside a Cubix chassis, an early
blade-like machine.

------
bowlich
I took over a job at a marketing firm once who had lost their senior developer
about a year before. They had been running on luck and a whatever quick fixes
their front-end developers could hack together when anything went down.
Needless to say, no one had any idea where the servers where or even how many
servers we had. Took a while to track them down.

Almost a year after I had thought that I had tracked everything down we had an
odd error start cropping up on our sites where the domains on a couple older
sites wouldn't resolve on just some computers. After digging through it we
found that there was some name-server that I had missed but we still didn't
know where it was until a thunderstorm killed out office router. In the
process of replacing it we tracked one cable to a closet in the basement
where, lo, the mystery name server was sitting.

------
weinzierl
Many years ago I did a job for a big corporation with a big IT department.
Servers were in the data center of course and to get one you had to fill out a
lot of different forms. When the project was on fire we were allowed to bring
in our own machine and use it as a server.

Later, when the project ended no one bothered to go through the whole formal
process to get the machine out of the premises again. The machine was just
left there connected to the network.

In the months after the project ended different departments moved into the
building. Different people with different tasks. Yet the machine stayed
connected. I guess no one had the courage to shut down a machine they didn't
know about.

For a year or so I still used to check occasionally if I still could log into
the machine. It disappeared eventually but unfortunately I don't remember how
long exactly it lasted.

------
frik
Maybe some remember an UK website with "celebs" in the URL, in the late 1990s.
(black background with small stars, website used frames)

It was probably the largest image database back then. That website had
thousands of photos from every female celebrity categorized by name. The
website owner collected all celebrity images from public news groups. One day
the updates stopped and everyone wondered what happened. Many months later the
website vanished (ca. 2002) and I read in the news that a friend found the
dead body of the owner and no one paid the bills. As usual a domain grabber
replaced it with a scam website shortly afterwards. I remember only a few
frame-pages were backed up by Archive.org - though I forgot the URL of the
website. Does anyone remember that former rather famous site?

------
ChuckMcM
Of course there are, the question though is how "forgotten" and what is a
server. There are web sites out there running on servers where they haven't
been changed or updated in years. Presumably they are forgotten in one sense
or another.

------
neals
I've got one, that I set up for a client but who decided to not need it. Still
running and paid for. I imagine this happens a lot.

Edit: Now that I think of it, I 've got 3 for 2 different clients.

------
RexRollman
The reason I enjoy these kind of stories is the uptime. I know it is not a
real indicator of anything but I enjoy seeing machines that have been running
for years. (I used to frequent a website were people actually tracked their
uptimes).

One of my favorites was this one, on ARS:

[http://arstechnica.com/information-
technology/2013/03/epic-u...](http://arstechnica.com/information-
technology/2013/03/epic-uptime-achievement-can-you-beat-16-years/)

------
iMerNibor
A few years ago when I upgraded to a new more powerful box at my hosting
provider I noticed half a year later the old one (which I had whiped and done
some testing on after transfering) was still chugging along, serving the silly
403 page I set up. After a few days of talking to the confused support it got
shut down.

------
jjawssd
Relevant [http://bash.org/?5273](http://bash.org/?5273)

------
namplaa
I have a forgotten server happily chugging away somewhere, it still send me
mail sometimes. it might be lonely.

------
viraptor
Yes, it happens. From the small company to a corportation, those can be: vms,
old servers, racks of servers...

Having a perfect CMDB for hardware and following all procedures step by step
could identify / prevent cases like that. But in reality, in normal
environment it's going to happen.

~~~
chris_wot
I'm curious - what IS the perfect CMDB software?

~~~
devonkim
My belief is that the only way to make the perfect CMDB software is for your
company to write it itself. Every CMDB system winds up being obliterated into
twisted machinations of the original vendor's object model to the point where
the vendor can't even help their customer anymore. I've witnessed CMDB
implementation projects at least a dozen times and have seen 0 actually
succeed at what the customer really wanted to get out of it (because people
are sold unicorns and puppies in the ITIL model that no vendor can actually
deliver because it's theory ware). Almost all CMDBs or equivalent system of
records database has been RDBMS based in my experience on the enterprise IT
side. One exception is that Facebook has their own that's not from some random
acronym vendor used with clients being pluggable such as from Chef in place of
the provided ohai.

~~~
jon-wood
My first proper software development job was to consolidate the 7 different
CMDBs that the company had - they were an ISP/hosting provider who had done a
rapid succession of acquisitions, all of which came with their own database of
hardware.

Some of the data was so bad that we ended up having one guy who's job was to
physically audit what hardware was _really_ in the data centres, and then
attempt to reconcile that with the CMDB data.

------
moru0011
I'd assume like 20% :-)

------
c22
I used to administer a website that was running on a server set up by the
original site owner at a hosting company he worked at. The site kept running
for 5 or 6 years after he stopped working there.

------
codeonfire
Yeah, a department I worked in had over 1000 vm's up with nothing installed.
That's what you get when you have high turnover, huge cash flow, and
management that didn't care.

------
yeezul
Somehow this thread reminds me of Wall-e.

------
tylermauthe
I wonder what the Carbon Footprint of all this unused compute power is?

