
OnMetal: The Right Way To Scale - philips
http://www.rackspace.com/blog/onmetal-the-right-way-to-scale/
======
ggreer
This sounds pretty cool, but I have a few suggestions:

1\. I want to know how much these things cost. Even a line like, "Under
$x/hour." would be useful.

2\. The blog post says that the memory and compute types have no disk. This
confused me at first, but then I saw on
[http://www.rackspace.com/cloud/servers/onmetal/](http://www.rackspace.com/cloud/servers/onmetal/)
that they have 32GB boot drives. You might want to add that to the comparison
table in the blog post.

3\. Has any thought been given to letting users provision OnMetal servers in
specific cabinets? Sometimes users want to have servers next to each other for
better network connectivity. Other times, they want them in different cabinets
for increased reliability.

Lastly, kudos to Rackspace for building this. Most people don't realize how
virtualization can hurt performance. Even without noisy neighbors, a modern
Xeon still takes >500 cycles to switch between VM and host.

~~~
jayofdoom
1) I don't think prices have been announced yet.

2) They do have 32GB SATADOMs, but these are intended for boot media and
configuration only. They are not intended for anything more than boot media.
Think something like an SD Card in a solid-state machine.

3) Yes, I want this. No, we don't do it currently.

FWIW, we didn't build all this. Ironic existed long before we came around, we
added a deploy driver and set it all up and exposed it for customers. We'll
have a blog up shortly with exactly how we're running this in production.

The coolest part of all this is something nobody has mentioned here yet --
this is all on Open Compute hardware. When I managed servers, I always wanted
access to the cool hardware and scale that companies like Facebook and Google
had. Hopefully with OnMetal, we're going to give our customers the cool
hardware I always wanted.

Edit: As a note, I'm a Racker and I work on the team that built this.

~~~
_Adam
Cool! Can you talk about some unexpected challenge you faced getting this
rolled out?

~~~
jayofdoom
Honestly? Using 512GB of ram is pretty hard. I learned during some of this
that you can actually get tmpfs to be CPU limited if you try to write to it
too quickly.

~~~
mbell
> Honestly? Using 512GB of ram is pretty hard.

It's extremely easy. Insert 512GB of data into postgres, there, done.

------
kator
Finally the pendulum swings back toward reality.

Virtualization is interesting for many problem sets but it does add overhead
and it has it’s costs. All of these technologies are cost -vs- reward and for
too long we’ve been drinking the virtualization cool aide not realizing the
real costs of those configurations.

Last year I did a super low-latency high qps project on a cloud box and I was
lucky to get 37k qps out of the box. I had the same exact box reloaded as bare
metal and was able to get to 165k qps. Mind you this is latency in the
neighborhood of 10ms and really extreme qps loads. That said the
virtualization layer is not free, it’s useful as are many technologies and
it’s an awesome tool for the right problem set.

I’m happy to see Rackspace coming out with this, I asked them for something
like this years ago and am excited to see they’ve figured out a way to do it..

Like many have said.. Let’s see the pricing. But before you compare make sure
you are really comparing apples to apples.

------
dmytton
This is an interesting announcement for a few reasons:

1) It's true that VMs generally provide bad performance for high i/o
applications, particularly databases. There are various ways to mitigate this
such as using high IOPS drives like AWS EBS PIOPS or SSD backed storage from
Google/Digital Ocean.

2) This is using OpenCompute and is all open source and going to be released,
so they can take advantage of the efficiencies of that hardware architecture.

3) It gives you quick access to physical servers connected to your cloud
environment. Benefits of flexibility/scalability with the benefits of
hardware. This seems to be how they're differentiating against Softlayer who
have bare metal servers available via API, but they're billed monthly and take
a few hours to provision (which is still pretty impressive).

However, there is no pricing announced - this will be key.

Also, will this start to eat into their managed services perhaps? Anticipating
disruption?

~~~
x0x0
ime non-vm servers are an enormous win over ec2, and probably vms in general.
A previous employer moved a large data processing pipeline from ec2 to
softlayer; we saw costs fall from ~$100k/mo to high $40k/mo while tripling the
throughput for roughly a 6x perf boost.

The one area that this announcement doesn't address, however, is contended
networking. Rack local servers with a 2G (or ideally 10G) full port-to-port
simultaneous switch did amazing things for our app's performance. When you're
looking at whole-app performance, you're sensitive to contention on all of
cpu, io, and networking. Hadoop is particularly sensitive to contended i/o,
but secondarily sensitive to networking.

edit: s/big data/large data/ \-- people who say bigdata are tools

~~~
jayofdoom
As we talk about here: [http://developer.rackspace.com/blog/how-we-run-ironic-
and-yo...](http://developer.rackspace.com/blog/how-we-run-ironic-and-you-can-
too.html), each instance is provisioned with redundant 10Gbit network links
with minimal network over-provisioning in each cab.

~~~
x0x0
oh awesome; did I miss that in the article? Reading is hard...

------
oasisbob
I think this is a neat development, but doesn't this post overstate the case
for reduction in complexity?

I hope a Racker will correct me if I'm wrong, but:

\- With the exception of the boot device, you're dependent on non-local
storage over the network.

\- You're not going to get dedicated networking gear

\- OnMetal is still dependent on the virtual networking layer (OVS?)

I can appreciate having more bang for the buck, but there are enough shared
components here that I would be very hesitant to assume that that MTBF crosses
the imaginary line where you can eschew fault-tolerant architecture patterns.

~~~
jayofdoom
For the network, you're getting 2x 10Gbit connections, configured in an HA-
Bond, and plugged into a real switch. There's no current dependency on a
virtual networking layer.

~~~
oasisbob
That's good to know. How and where are ACLs handled?

~~~
russell_h
Engineering manager on OnMetal here.

Our first iteration, you'll get two networks: "ServiceNet" is our intra-DC
network, which is unmetered (ie, free bandwidth) and "PublicNet" which is the
internet and will be billed just like we bill for other cloud server
bandwidth. Collectively you get 10Gb/s across these, which you can break down
however you want. To you these just show up as VLANs which will be configured
on build.

We're working on hooking these boxes up to our "Cloud Networks", which are
per-tenant software defined networks. If this sounds like a fun project, hit
me up, contact info is in my profile.

------
chx
The thing here is OpenStack and the beefy specs. We have seen other companies
allowing you to deploy bare metal, SoftLayer and Ubuntu MAAS jumps to mind. I
have been preaching bare metal for years now; it's great to see it's spreading
(again).

------
rdl
I never understood why people didn't do this before -- I've wanted great
application-accessible APIs to PXE and autoconfiguration for a while.

Even better if it's standardized across competing hosting providers, or at
least multiple physical locations.

OpenStack can do this -- makes a lot of sense.

I personally like provisioning hardware with a hardware-virtualized hypervisor
anyway, and running admin/keys/monitoring in another VM or the dom0 while
running the application in a domU, but I hate multi-tenancy, especially across
organizations, but often even across functions on the same server from the
same organization.

Pretty excited by this; would love to set it up in a colo cage.

------
twotwotwo
A key point of the post is that inconsistent VM performance is a problem that
scaling startups have to do a lot of work to deal with, and some people here
are scaling or have scaled startups, so...what are y'all's experiences?

------
ilaksh
Many people are still going to need or want to be really careful about
selecting the best value. So the question is, if you rent say an 8GB VPS from
Rackspace, Digital Ocean, Linode, or something similar, how inconsistent is
the performance actually? And given that measurement, will the presumably
somewhat more expensive "on metal" be a good value?

I'm not even considering EC2 because I don't trust the reliability and I know
that its not a good value compared to the others. EC2 only makes sense to me
if you really need to take advantage of features they have like the private
networking or DynamoDB etc.

------
stcredzero
I'm looking into developing simplified MMOs. I don't care about SSD. I do care
about latency and noisy neighbors. OnMetal sounds attractive, but it also
sounds like it will be costly.

I think there's a market opportunity for simplifying the implementation of
online multiplayer games, especially on the back end. Right now, there's a lot
of painful barriers and some apparent pent up demand.

------
jonathanoliver
Are there any other providers out there that have anything like this? For
example, SoftLayer has "Bare Metal" servers, but I'm not sure if there's a
virtualization layer.

~~~
berns
Storm (Liquid Web) also has a bare metal product.
[http://www.stormondemand.com/servers/baremetal.html](http://www.stormondemand.com/servers/baremetal.html)

------
kordless
These things have Teeth!

------
seunosewa
I think this is an excellent idea, combining the extreme flexibility of the
cloud with the high performance of dedicated servers.

------
ddorian43
Cant they throw at least a hdd on the HighCpu/HighRam servers ?

And would it good to have a HighRam/HighIo server ?

~~~
jrl2
Would like to see HighRam+HighIo too and that's definitely what I'd want to
use for a database server with lots of data, but I'm guessing there is a
constraint like space inside the chassis that forces a choice between PCIe
flash or lots of RAM. Hopefully someone from Rackspace will comment.

------
akurilin
Would be very interesting to have a side-by-side feature comparison with
SoftLayer.

------
mattbillenstein
Would love to see some pricing...

