> So far as I know, nobody else has ever come close to that even with 10x faster CPUs.
With 10-gigabit becoming commonplace in high-end server rooms, dual 10-gigabit cards hitting the market, and 100-gigabit on its way, I've got to ask, what you mean by no one has come close?
(Certainly, 1 GByte/sec is achievable; splice is a large part of that.)
The big print giveth, the fine print taketh away :)
Just because you have a 10 gigabit pipe (which is 1.2Gbyte/sec) that doesn't mean you can fill it. And filling it with a benchmark is different than filling it with NFS traffic.
So far as I know, nobody has come close to what SGI could do years ago, i.e, stream 100's of MB/sec of data off the disk and out the network and keep doing it.
I've tried with Linux to build a disk array that would do gigabit, not 10-gigabit, and couldn't come anywhere close. I think about 60MB/sec was where things started to fail.
I'd love to hear about a pair of boxes that could fill multiple gigabit pipes with the following benchmark:
$ tar cf - big_data | rsh server "tar xf -"
You can set any block size you want in tar, I don't care, the files can be reasonably large, etc.
If you manage to fill one pipe, then try it with 10-gigabit and let's see what that does.
I'd love to be wrong but so far as I can tell, we're nowhere near what SGI could do.
Benchmarks, especially those from card manufactures are suspect. Just because you've got a network ping-pong test running at essentially line speed, doesn't mean I'll see any of that. I fully agree.
Filling any pipe with your 'rsh' benchmark is harder - on all the Linuxes I have on hand, rsh is aliased to ssh, and I've never seen a system deal with the overhead of ssh adequately. (There's high perf ssh if you really want; I've never tried it. http://www.psc.edu/networking/projects/hpn-ssh/) (Would you accept numbers over netcat?)
If I'm not getting 90MBytes/sec over Gigabit, then something, somewhere, is slowing it down.
We sell servers that sustain 300+ MBytes/s per client, via Samba for two clients. Those clients are very fussy about latency (video editing/DI). That's what we sell, so it's empirically that fast; ie not a benchmark. The server itself will actually do 900+ MBytes a second (to three clients, the same as page 14 on your linked pdf), but my small print is the market doesn't want that configuration.
These numbers are what I know, which doesn't include NFS. NFS-RDMA and pNFS are supposedly good for performance, but I don't have any first hand knowledge.
None of what I just stated contradicts you. But if I'm at 900 MBytes/sec with my level of hardware, I find it hard to believe someone with more budget isn't capable of a full Gigabyte/sec over Infiniband, arguably HIPPI's successor.
It's a product for a niche B2B market, but if you're buying, email's in my profile ;)
The network is 10-Gigabit ethernet, from server to a switch, then switch to client, all via CX-4 or SFP+. The data is actually coming off disk at that rate (in ram would be cheating :p ). It's backed by a 16-drive hardware raid using 2TB spinning disks (SATA) in a raid5. The performance we sell is actually 2 streams of read while simultaneously writing a stream to disk, so 600 out, 300 in.
The files are actually the hard part. This supports the dpx 'video' format. The format originated from film scanners, and so each frame is (basically) a 12.5 MByte jpeg. Multiply that by 24-30 fps, add in overhead for reading that many files/sec, and you get 300+ MBytes/sec. (This is also why I'm so sure there's more performance to be had if you don't have the same overhead.)
In my case, it's because it's a pretty good emulator of what we do.
We're the oh-so-loved BitKeeper guys and we're working on tech for very large repositories. A bk clone is essentially the described benchmark. Or should be, if we're slower than that we're doing something wrong.
With 10-gigabit becoming commonplace in high-end server rooms, dual 10-gigabit cards hitting the market, and 100-gigabit on its way, I've got to ask, what you mean by no one has come close?
(Certainly, 1 GByte/sec is achievable; splice is a large part of that.)