

Ask HN: Buying servers? - slackerIII

I might be buying 10-20 servers in the next few months, and I'd really appreciate any advice folks have about getting the best deals.  I'm mainly looking for some fairly reliable 1U boxes with 8 cores and 16-32 GB RAM.  Disk is not that much of a concern.<p>Can I get significantly better prices if I start talking to a sales guy vs going over the web for places like Dell and HP?  Are there any white box vendors you guys would recommend?  Has anybody learned any lessons or had a great experience with a vendor they can share?
======
mmmurf
You can save a lot of money by going with the sales guy. Just be sure he knows
that you are growing and you'll be placing a big order every few months.

First go to the Dell website and price out the equipment. Then when you're
speaking to the sales guy, tell him you priced it out on the website and it
came out to about 88% of the actual price the site gave you.

Then tell him you've heard that talking to the sales guy is the way to start a
business relationship and that you were hoping he could quote it out for you.
Tell him you're ready to make the purchase decision immediately but the
pricing has to make sense. Tell him the pricing on the website looks like it's
marked up quite a bit.

It may take him a day or more to generate the quote.

Be aware that he has tremendous room to wiggle on some items, but not on
others. Last order I placed the salesman could discount heavily some items but
not others. You don't really care as long as the overall price makes sense. So
when you eventually go over the quote with him, be sure to focus on your
"budget" for the equipment, and not on where the specific pricing comes from.

You should also have a simultaneous dialog like this going with a rep from any
other companies you'd buy from.

To the salesperson, the equation is effort vs time. If he knows you're ready
to buy soon, then he has every incentive to make you a good deal b/c in a day
the sale will be made and so he'll be willing to take a small commission in
exchange for a quick sale. He has some constraints about how he can price
items, but has a lot of flexibility on price.

If you're buying 10 to 20 servers you should be looking at least a 25%
discount off of the website price. The only reason they publish those prices
is to give the salesperson a starting point to work down from (or to sucker
naive shoppers into paying full price via the web).

You want to come across as decisive and ready to buy at all times, if only he
can solve the "issues with the pricing".

Frankly Dell's sales process stinks, and it is annoying to have to work with
some of the salespeople there. But I've saved a substantial amount of money
over the web prices by negotiating with them, so it's worth it.

You may also be able to save money by modifying some of the configurations
slightly, b/c Dell lets salespeople discount based on the need to move
inventory.

Don't be afraid to tell the salesperson it's OK if he needs to ask permission
from his manager to make the deal, but that you're eager to get the order
placed today. Tell him he can put you on hold while he speaks to the manager.

Good luck!

~~~
mmmurf
Oh forgot one thing. You should expect to get free shipping on an order that
size. When it gets to the last negotiation then you can say "Ok, I'll do it if
you can get me free 2 day shipping on it".

~~~
slackerIII
Great information -- thanks!

------
kvogt
We've tried several vendors at Justin.tv, but we always end up circling back
to Dell. Pricing is great, spare parts are readily available on Ebay, and
they're quality machines. Definitely get in touch will their sales guys and
plan your purchase for the end of the month, or better yet, end of the
quarter.

You're making the right choice on 1U 8-core boxes. We've done the math and
there's no better combination of price, performance, density, and power
consumption out there.

-Kyle

~~~
e1ven
I'll add another vote for Dells-

a) Their servers rack very nicely, particularly when you're using Dell racks.
No tools are necessary, which saves a Lot of time when you're setting
up/moving the machines.

b) The management tools are much better than you get from
SuperMicro/offbranded machines- You can generally find a hardware fault
quickly from their boot CDs, rather than diagnosing manually.

c) As Kyle mentioned, the parts are ubiquitous- Dell still carries parts for
most of their machines, but even if you can't find it from them, you can find
it from another vendor.

d) Because Dells are so common, it's easy to find other's experiences with a
software/hardware combination. <http://en.community.dell.com/forums/>

e) Support. Don't underestimate this when choosing what vendor to buy from..
With one of out offbrand vendors, a replacement drive took at least 2 weeks to
be shipped out and arrive.. If you pay for the Dell premium support ($200ish
per server), they'll drive a tech to the spot within hours.

They're less than ideal on storage pricing- It's often cheaper to buy drives
elsewhere.. Further, their live reporting tools (Check HD status, etc), aren't
as good as they really should be.

I'd certainly call up a sales rep and have a chat with them. They'll usually
give you a discount if you're buying any quantity- If they hesitate, ask if
they'll throw in free gold support. They'll normally do that to save a sale,
since it doesn't cost the rep very much.

-Colin

------
johnm
(A) Lease vs. Buy: If you can swing it, lease them instead - tax benefits,
etc.

(B) Do you really need 8 cores per box? I.e., you can get used dual-dual
machines a lot cheaper than 50% of the price of the dual-quad machines.

(C) If you know what you want and are buying new then going with a white box
builder is significantly cheaper than buying from Dell/HP/etc. I've been
pretty happy with ASA especially since they've delivered/picked-up/serviced
machines directly to/from/in our colo but there are plenty of decent vendors
and they most end up using SuperMicro-based systems anyways.

(D) Do you really need 1U? Why? Power/cooling issues are putting more
constraints on things than size at many places and a 2U is often cheaper, more
flexible, fixable, etc.

(E) Really knowing exactly what you want can make a big difference in talking
with the salespeople. Buying at the end of the quarter is a good thing if you
can swing it.

(F) More ram is more better and 32GB is the current sweet spot in terms of
lots of memory for a good price. Pay particular attention to the motherboards
that you're considering as there can be a price premium at the various steps
depending on if you need the flexibility to upgrade the memory a lot later.

(G) You mention that disk isn't much of a concern. Is that because you just
want lots of compute nodes or because you're putting everything into a NAS/SAN
or what?

~~~
slackerIII
I was thinking 1U because I don't need a lot of disk and it gives me more
options about how many I can put in each cage. What kind of power limitations
have you run into?

As for the disk, I just don't have a lot to store.

~~~
johnm
If you don't need to cram a lot of boxes into a tight space then 2U will
usually give you better airflow and cooling.

Also, when those on-motherboard NICs die, you can more easily/cheaply add
additional NIC cards to the 2U boxes.

In terms of power, that's actually the main limitation in most colo's these
days. Some colos are down at 15 amps per rack, many seem to be in the 20 amps
per rack, and some you can get up to 30 but you will pay for that one way or
another.

Even at 30 amps per rack, for the kind of machines that you're talking about
you are still likely to be less than about 20 machines per rack. I just spec'd
a build out of the same horsepower & ram but with more juice for e.g., FC
cards to a SAN and the safe limit was 16 machines in a 30 amps per rack colo
and 8 machines in a 15 amps per rack colo.

Shopping around for power information can be tricky because some of the colos
like to play marketing games to get you to think there price is better than
others but the power charges can eat you alive if you need more than what they
allocate in their base charges.

------
brianm
So, some lessons learned...

1\. lights out management matters

2\. ensure your OS is happy on the hardware ahead of time

3\. if you are expecting to make regular purchases, with growth for example,
tell the vendor that

4\. always negotiate

5\. dell actually makes good servers now

6\. look hard at the power density you get wherever your servers will live and
factor cost of rack space into server cost numbers -- if you have low power
density, slightly pricier servers with lower power reqs may win out

------
lsc
Hah. I know this one. Ok. first, are you willing to use a wrist strap, to buy
an anti-static mat and verify you are grounded?

<http://prgmr.com/~lsc/luke_opterons.jpg> (my setup)

If you are not, stop right now and take someone else's advice about how to get
a good deal out of dell and HP (me, I can't afford dell and hp prices... my
business model cant take that kind of markup on the ram. Dell and HP have OK
deals on the base server, but you get screwed if you fill it with ram.)

Ok, you have the wrist strap and the dissapative mat? Ok, now first the ram.
If you want 32GB ram, remember that 4GB modules cost twice as much per
gigabyte as 2gb modules. this means you really want a board with 16 ram slots.

Second, the hardest part of assembling a computer is fitting the motherboard
into the case and designing the cooling. Me, I buy SuperMicro SuperServers
where that sort of thing is alrealdy done. It probably costs me $200 extra
over some homebrew solution, but it saves me time and gives me a better
finished product.

Here is the computer I settled on:

<http://supermicro.com/Aplus/system/1U/1021/AS-1021TM-T+.cfm>

Now, just search around for the lowest prices. if you are buying 20, you can
call salesjerks, but my experience has been that they spend a _lot_ of your
time (time that I can bill out for close to $100/hr) and almost never give you
a better deal on parts (remember, the salesguy wants to get paid, too.)

I got the aformentioned case from newegg for around $1200.

Next, do the same with CPUs. I ended up buying quad-core opteron HE chips at
1.9ghz for $260 or so each on newegg again.

Now, for ram, I very rarely beat the price on shop.kingston.com. They have a
nice thing for finding the ram that fits your motherboard, and a good
warranty.

the 2 servers in that photo? they take 1u of rackspace, around 350 wats power,
and the whole ball of wax, for 2 servers, each with 8 cores and 32GB ram and
2x1TB disks, came in at under $5000.

But again, if you are unwilling to use anti-static equipment, buy the servers
from a reputable vendor, and never open them.

------
brk
Yes, you will get better deals by talking to a sales person, ESPECIALLY in
about 2 weeks (as the quarter and year are coming to a close and people are
trying to make their numbers).

Make it clear that you know the value of what you are after and are ready to
make a purchase as soon as you find the right deal.

------
bbhoss-synsol
I like Sun servers, they can be expensive, but you can get some good deals if
you are accepted into SSE (Sun Startup Essentials). For me one of the best
things about them is the ILOM, which is badass as far as I'm concerned. Also,
their N1 System manager lets you manage and provision a bunch of servers from
a single location. It will also keep track of the environment on all of them,
but thats nothing you couldn't build with SNMP traps etc. Also, I must mention
that if you pay for the support, it is awesome, I had a problem with system
drives on a Friday, and they overnighted it and I had it Saturday. They also
don't charge you unless you don't send the defective part back, with THEIR
free packaging and shipping. It was also nice of them to include 3-4 3M anti-
static wrist straps with my shipment of new hardware :)

------
ezmobius
Supermicro servers have served me very well. All of our clusters at
engineyard.com are built on supermicro 1u's with 32gig ram and 8cores. Highly
recommended.

~~~
Freaky
What's their LOM like? I can't say I've been impressed with cheapo IPMI cards
which (attempt to) piggyback off the on-board ethernet.

~~~
lsc
yeah, the IPMI kinda sucks... more than DRAC, even (which is worse than HP
ILO)

I use SuperMicros but I just redirect console over serial (they can do this
even for the BIOS setup, nearly all 'server' boards can) then I hook the
serial to another server with a $5 USB to rs232 dongle (or to a terminal
server in the case of racks with a lot of servers)

then you want a remote rebooter. You want that even with DRAC, 'cause DRAC
isn't particularly reliable.

------
wehriam
Amazon EC2 has a large VM if you don't want to purchase or lease hardware yet:

"Extra Large Instance 15 GB of memory, 8 EC2 Compute Units (4 virtual cores
with 2 EC2 Compute Units each), 1690 GB of instance storage, 64-bit platform"

$0.80/hour - about $730/month.

~~~
lsc
if you leave that on all the time, and plan on keeping it for more than a
month, that's a horrible deal.

I can build a super micro server w/ 32G ram and 8 cores for around $2500, and
I get to keep it. Heck, I'll rent you one of mine and host it for $512/month.

Of course, if you only need it during the day, or temporarily or something,
ec2 is an awesome deal.

~~~
wehriam
I won't dispute that in many situations owning machines is the most economical
way to go. But keep in mind:

* EC2 servers can be deployed quickly and cheaply - you don't have to overspend on hardware to meet an upper bound of predicted demand. The original poster needed between 10 and 20 machines - if he only needs 10 but purchases 20 he wastes $25k.

* Co-location fees are included

* Features like elastic IPs, elastic block storage, and security groups make managing machines easier.

~~~
lsc
you have a good point. It's hard to find a co-lo that doesn't have excessive
fees for 'not knowing exactly what you will need for the next two years' - I
myself have gotten in trouble with bandwidth contracts.

My point is just that at $700 for 4 cores/15G, you are better off paying for a
server if you need at least two and stay more than two months. (I pay under
$2500 for 8 cores/32G ram) Even if you buy a dell and have to pay for the more
expensive 4GB modules ('cause all the dells I know of only have 8 ram slots)
and pay for the 'sales guy tax' I imagine you aren't more than twice that,
unless you get silly on the CPUs, so even then you are talking only 4 months
to make your investment back.

You do need to be careful about co-lo contracts, though, they are
professionals (by which I mean they are really good at screwing you.) so watch
out, and always assume the worst of any document you sign.

There are places that let you go month-to-month, especially on the low end.
He.net usually lets you get full racks on a month to month basis, with a
reasonable ($500) setup fee... I know lots of smaller providers that give you
month-to-month on a per-U basis. (see rippleweb.net - I host my backup servers
there. Something like $80 per U/month, but that includes 1TB data transfer,
and they let you host one of the aforementioned 8 core monsters without
charging extra, and they let you go month to month.)

------
carl_
Lots of great comments so not much add/kinda like others have said:

When dealing direct with vendors (Dell, Sun and IBM from experience):

1) Price you're given is not only based on current purchase, but future
purchase. Telling your sales rep 'I need X servers now, but I expect to need a
further X per quarter growing to X after X time' then they'll fight harder for
your business. Basically, present your business as a growth opportunity to the
vendor.

2) When speaking to your first sales rep, ask at what price/order level they
generally deal in. The reps who handle bigger clients have access to bigger
discounts.

3) Don't under or over-specify your machines without calculating your later
options. Buying an extra processor for one machine now will be a hell of a lot
cheaper/easier than buying the matching processor in 3/6/12 months. In the
same respect, not buying that extra processor now might be more prudent
because a new chassis with the latest and greatest processor in 6/12/24 months
might be a better option. (should be part of your scalability predictions
anyway).

4) Know your stuff.

4.1) We all know reps bs, by knowing what you're talking about you can reduce
this tedium.

4.2) If you ask which processor is better, then it'll always be the more
expensive one (or more importantly, the one with the better margin for them
which is not always the most expensive).

4.3) mmmurf said it, but to re-iterate. Time = Money. If a rep is holding your
hand too much, then you won't get as good deals.

5) Know your vendor and their enemies. 5.1) When is there end of quarter? 5.2)
How have they done compared to comparable vendors over the last quarter, year,
etc? 5.3) Which vendors are pushing X line/type of servers at the moment? 5.4)
Have comparison quotes.

6) Specify a date when you will be looking to purchase. 5.1 has a MASSIVE
impact on this.

~~~
carl_
Also, Dell price-match RAM to crucial for me which might be useful for the
servers you need.

------
merrick33
I used to do IT Consulting and was responsible for procuring hardware from
dell at a few companies including now my own.

What I learned - End of Quarter yields huge savings at Dell. If you are going
to buy 10 servers for sure, create an account, spec out one server and call in
to the small business division. Ask the salesman what he can do if you buy 5
of the one in your cart, he will give you a number let him know you have to
talk to your partner put him on hold or call back. Then ask what he can do if
you buy 10 - watch the price go down again on the quote.

------
iigs
We have a smattering of 100-150 HP DL380s and BL460c blades. We've seen
approximately a 5% infant mortality rate (DOA or issues in the first 90 days).
Depending on the channel HP's custom configurations come as a base and add on
accessories -- if you order a dual 5160 DL380 with 32G of ram and 4x146g SAS
you might get a preassembled 1x5160 4G 1x146 and a CPU, 14 2g DIMMS, and 3
disks, each in a separate shipping carton. You then spend about one person day
per two computers unpacking individual components from cardboard and shipping
foam boxes and installing CPUs, DIMMs, and disks. It's not my favorite thing
to do, and it means that you only get about three computers per pallet. It's
also, of course, not burnt-in beforehand, so you get to rack and memtest them
for a day or two yourself.

In contrast we've had really good luck with a regional whitebox VAR -- they
would assemble Intel reference platforms and Supermicro systems, burn them in,
and ship them, assembled, in the one box the case came in. As far as I know
we've had closer to a 1% failure rate over three years with these (exclusive
of HD failures but not environmental, over-heat, issues).

On balance it's a tough call, because HP (and I assume Dell) have a much
slicker lights-out-management solution than the last IPMI demo I saw. Maybe
the whiteboxes have improved, but I assume the HP machines are still easier to
support.

~~~
yellowbkpk
Another HP story. Not nearly on the scale you're talking about, but the
company I works for uses HP DL360 G5's as the hardware they sell to customers.
Before we ship to customers, we open the box and install software and such. As
a tester/engineer for this software, I've never seen a dead DL360 arrive. Yes,
the shipping containers are huge. HP's iLO2 software (integrated lights-out)
is excellent.

Also, whatever machine you get, make sure you buy some extra rails and
hardware. Store the extras somewhere and label them with the date purchased
and which systems they are for.

------
rmason
I can second the Supermicro solution. Worked for a hosting company in the past
and we never had a Super Micro machine fail. They cost less and are built a
lot beefier than comparable Dell's.

------
volida
You may consider 8anet.com. You can customize the server according to your
needs. They also setup for you the server for 20 or 50 dollars. I don't
remember exactl

~~~
davidu
I've bought over 500 servers from 8anet over the last six years. I'd highly
recommend them. Call them and talk to Mary Z.

------
blender
For whitebox vendor you might want to look at Supermicro

Cheers

------
vaksel
go on eBay, lots of refurbished stuff there, a guy I know buys all his stuff
from one user.

Me personally, I prefer to just lease them

~~~
mrtron
Leasing computer hardware instead of buying results in tax credits in Canada,
so many people take this route.

