

Making Facebook’s software infrastructure more energy efficient with Autoscale - otterley
https://code.facebook.com/posts/816473015039157/making-facebook-s-software-infrastructure-more-energy-efficient-with-autoscale/

======
bhouston
Vfx companies have software that turn off renddr farm machines in periods of
low load. I helped write software that did this backin 2006.

Basically, we found it was possible to shut down machines in periods of low
load and then use "Wake On Lan" to start them up once load picked up again.

I am unsure if the on-off power cycling reduces machine longevity.

Might be something worth exploring at Facebook.

~~~
mschuster91
> I am unsure if the on-off power cycling reduces machine longevity.

Possibly the disks are the most vulnerable components. I wonder if it would be
possible in software to shut down the CPUs and have only disks + network
running...

~~~
donavanm
A bit late to the party on this one. Check out cpu "c states" and things like
intels "speedstep." Modern cpus can reduce/shutdown power to individual
packages and cores. This can reduce power consumption from a hundred tdp to
tens of tdp.

The downside is latency associated with changing state. Depending on the
change it can be hundreds or thousands of micros to go through these states.
On a server workload this can introduce huge latency outliers as a request
blocks on a core to wake up.

~~~
mschuster91
I was more after "shut down _everything_ except the disks" to have the
absolute minimum running and the disks spinning to reduce wear on the engines.

If you're shutting down, you have massive latency anyway but if it's possible
to at least save all the power not required for keeping disks up that 'd be
great.

------
praseodym
This is one of these things where virtualisation can help even more. For
example, VMware can dynamically put servers in standby mode when demand is low
and power them up again when needed:
[http://www.vmware.com/products/vsphere/features/drs-
dpm](http://www.vmware.com/products/vsphere/features/drs-dpm)

~~~
SEJeff
You know that would be trivial to do with bare metal and out of band
management cards like Dell Dracs, IBM RSA cards, HP ILOs, or generic IPMI
BMCs, right?

Virtualization doesn't really add much of anything for that specific problem
other than increased context switching and slightly lower performance.

Disclaimer: building this type of thing (on bare metal) is a chunk of my day
job. I see it as unbelievably trivial. In fact, the same ideas are behind
Rackspace's "OnMetal" initiative:

[http://www.rackspace.com/cloud/servers/onmetal/](http://www.rackspace.com/cloud/servers/onmetal/)

~~~
devicenull
Getting IPMI controllers to do what you want is anything but trivial :)

~~~
thrownaway2424
Seriously. A person who would advocate using IPMI at scale has either never
owned an IPMI card or has never worked at scale or both. The just don't work,
and they erase whatever power savings you're trying to achieve.

~~~
wmf
Real servers can't be bought without IPMI and AFAIK the BMC cannot be turned
off, so it's probably not worth worrying about BMC power if there's nothing
you can do about it.

~~~
thrownaway2424
Sure, but as you're aware facebook, google, et al don't buy "real servers",
they buy servers that actually meet their requirements. That's why "real
vendors" like HP have missed the boat on selling millions of servers into the
cloud.

~~~
wmf
Speaking of Facebook specifically, the evolution is interesting. They replaced
BMCs with the reboot-on-LAN hack but then their next motherboard version had
BMCs again. It would be interesting to hear the story behind that.

------
meatmanek
You could probably get a similar effect by using HAProxy's "balance first"
algorithm, which chooses the first available server with an available
connection slot (as defined by maxconn). If you did this, you'd want to set
maxconn pretty conservatively.

