HN went down for nearly all of Monday the 6th. I suspected failing hardware.
I configured a new machine that is nearly identical to the old one, but using ZFS instead of UFS. This machine can tolerate the loss of up to two disks. I switched over to it early morning on the 16th, around 1AM PST.
Performance wasn't great. Timeouts were pretty frequent. I looked into it quickly, couldn't see anything obvious, and decided to sleep on it. I switched back to the old server, expecting to call it a night.
Then the old server went down. Again. The filesystem was corrupted. Again. So I switched back to the new server. During this switch some data was lost, but hopefully no more than an hour.
And here we are. I'm sorry that performance is poor, but we're up. I'll work to speed things up as soon as I can, and I'll provide a better write-up once things are over. I'm also really sorry for the data loss, both on the 6th and today.