Hacker News new | past | comments | ask | show | jobs | submit login

I recently built some data center processing using SuperMicro Twin-Nodes^2 [0] and MicroBlades [1].

We're setup for 38kw [a] possible gross wattage using the dual-node blades using the Xeon D-1541 [2]. The amazing thing about the D-1541 blades is we get around 100W per server, with 8 hyperthreaded cores, and a 3.84T SSD. With the 6U chassis, you have 28 blades with 2 nodes each - 56 medium sized servers for 5.6kw in 6U - under a kw per RU.

For your 70 some server workloads, I'd recommend using the microblades.

For your higher lever workloads, I think the SuperMicro TwinNode makes sense.

Be very very careful about Ceph hype. Ceph is good at redundancy and throughput, but not at iops, and rados iops are poor. We couldn't get over 60k randrw iops across a 120 OSD cluster with 120 SSDs.

For your NFS server, I'd recommend a large FreeNAS system, put a big brain in it and throw spinning platters.

Datacenters can/will do your 30kw

[0] - https://www.supermicro.com/products/nfo/2UTwin2.cfm [1] - https://www.supermicro.com/products/MicroBlade/ [2] - https://www.supermicro.com/products/MicroBlade/module/MBI-62... [a] - Although we have 38kw of possible power there, it's practically well under the 27kw we'll get with 4x208v@60A PDUs at 80%.

If not Ceph, what are you using for storage?

2RU SuperMicro server with 24xSSD with FreeNAS raid10 or raidz2 exporting volumes via iScsi.

Separately I'm using a FreeNAS controller with 4 SAS HBAs supporting 3 JBODs with 45 8TB HGST He8 near line SATA disks each (135 disks or ~1PB) for backups and slow data.

Guidelines | FAQ | Support | API | Security | Lists | Bookmarklet | Legal | Apply to YC | Contact