

Ask HN: I want to put my desktop computers in the cloud. - zissou

Something occurred to me recently: I currently pay 5 different companies to run code/programs/databases/scrapers/etc for me (AWS, Linode, Google, Dropbox, PythonAnywhere). While I'm ok with the costs I accumulate across the 5 companies because they all solve the problem I want them to solve, I can't help but stare at 6 desktop computers that I have around my place and think: why am I not just using these to do [almost] everything?<p>On one hand, the answer is simple: my Comcast 15mbit connection is far too slow. The other quick answer is security: I do not transmit personally sensitive data, so the normal privacy issues old-school security engineers have with the cloud aren't there for my use case.<p>Ok, so that got me to start thinking about how to put my capital to productive use. The batch processes and scraper type work could easily be done in house. 4 of the 6 desktop computers I would like to put to work are actually fast by current standards (8GB+ DDR3, 2 are i7 2700k, etc). I mean, if I were to rent the same machines "in the cloud", it would cost me a significant chunk of cash.<p>Ah, but that leads to the other cost: energy. Getting estimates of those costs is pretty simple to do. So, this is where I started thinking about how to actually solve my problem [which is more of an itch than a problem]. Upgrade my Internet? Or, more interestingly, asking friends who have faster Internet already if I can stuff a desktop somewhere in their house and just set up SSH access to it.<p>I'm beginning to ask friends if they'd be willing to let me subsidize their Internet/utility costs if they'll let me put a computer in their house.<p>Has any one else done this before? Any input or discussion here would be useful. Thanks.
======
oz
patio11 (as usual) said it best. There is a _lot_ more going on behind the
scenes of a professional datacenter than you are aware of, and even if you
were aware of it, you might not be able to replicate it.

The hardware they use is different. While you use desktop-class drives, they
use enterprise class drives with faster platters, bigger buffers and better
disk controllers.

While you use Marvell/Broadcom NICS which crash if you torrent too heavily,
they use Intel server-class NICS which support TCP/IP processing offload.
Which means they process network traffic in _hardware._

While you connect everything with an 8-port NEXXT switch you got for $20, they
connect everything with $20,000 Cisco / HP switches. While you have a 15 mbps
asymmetric connection with dynamic IP from Comcast, they have multiple 1gbps
feeds from Level3 and Global Crossing. They run BGP between themselves and
their upstream providers, so their IP addresses don't change even if those
providers were wiped off the map.

Most importantly? They have huge teams of _highly paid, highly qualified_
engineers building and monitoring these systems. Oh; and they _still_ have
outages.

Progressing is futile.

I'm a former sysadmin for a Big-4 accounting firm. Take it from a fellow
Pythonista: your time is _much_ better spent at the '>>>' prompt than at the
'root@localhost#' prompt.

------
mechanical_fish
"Fast by current standards" is probably irrelevant if you're running a
scraper: You'll be bandwidth-limited, and if you solve that you'll likely be
disk I/O-limited or RAM-limited.

Getting back to the overall question: Yes, lots of us have done this before;
fifteen years ago every geeky college student had your proposed setup.

The other responses are correct: From a business standpoint, you're crazy. Not
only will you find that bandwidth and power cost you more at home than they do
at a datacenter, but your inconsistent, unreliable homebrew infrastructure
will consume maintenance time and replacement parts at a rate that slowly but
surely eliminates all "savings". It's hard to appreciate that, because you've
got little experience at keeping network infrastructure running long-term, and
even less experience at billing yourself for your own work at market rates, so
failure events feel rare and inexpensive to you. But I assure you that NOCs
don't buy all that expensive capital equipment for no reason.

Having said that, I get the impression that you don't really care about uptime
or reliability; you're running spare-time hobby projects. If so, then the
question is whether assembling your own homebrew "distributed datacenter" out
of PCs in various closets is educational. My feeling is that it isn't, not
anymore. The ability to run production-quality systems on cheap consumer
hardware is not a valuable skill and is no longer worth practicing - not when
production hardware can be rented by the hour for pennies. You can practice
installing Linux and tuning the LAMP stack on Linodes or AWS boxes and get all
the same sysadmin skills, but in a realistic production environment. You can
practice debugging disk throughput problems, or networking problems, or DNS
headaches on real cloud infrastructure for a few bucks a day. Who would play
with a toy bulldozer if a real bulldozer was available to rent for a dollar
per hour?

(The counterargument is that homebrewed "infrastructure" is a great toy
_because_ it breaks all the time and you're forced to practice lots and lots
of debugging. After all, _I_ learned a lot of stuff this way - I had to; the
"cloud" didn't exist. But my gut feeling is that this kind of "practice" feels
more valuable than it actually is.)

But of course, hobbies are hobbies, and far be it from me to argue you out of
your own happiness. If you just can't resist the fun of turning your home
computers into a compute cluster, go for it! It's a better way to spend your
time than MMORPGs.

------
patio11
_why am I not just using these to do [almost] everything?_

Because their true cost once you factor in your time is substantially higher
than racked infrastructure and because the reliability which you would achieve
with that setup is not sufficient for professional purposes.

