"Nagios checks to let us know when the filesystem becomes un-writeable"
Any specific reason you don't just panic? I've had RO fs edge cases bite me a few times.
"In the future we may even run fiber to these machines"
Anything against 10g copper? Per port prices are getting crazy cheap.
"DNS servers, LDAP servers, and our Hadoop Namenodes ... need RAID for extra data safety"
Anything you've had issues with? Host level redundancy with the first two has been pretty great IME, can't comment on name odes. The only issue I've run in to is poor LDAP clients not fast failing an unhealthy server when another is available.
RO filesystems can be bad, but usually they're soft failures for us:
* Memcache can still work just fine
* Db servers stop responding (and the app handles that fairly gracefully)
* Web servers serve files from a RAM disk, so they keep working
No reason against 10G copper specifically - we haven't had to address the problem in detail yet. When we do, we might choose copper. Depends what happens when it happens :-)
We had a very nasty incident a few months ago, where the drive in one of the LDAP servers died. Well, it sort-of died. It started to time out a lot but didn't go fully offline.
openldap kept running, but when you connected to it, the TCP connection would open and hang.
This meant that all of our servers saw the server as "OK", but LDAP stopped working and caused all kinds of brilliant havoc :-)
Interesting. I was thinking failure to persist updates, inconsistent health check results, those types of nasty corner cases.
Sure. Premature optimization and all.
Huh. SSD from a major chip vendor perhaps? I had one of those bite me just a few weeks ago. in my case it would have been fine if the drive offlined itself. but the intermittent io pauses killed performance, but not health checks. Actually, same type of problem RO FSs make me scared of.
"In the future we may even run fiber to these machines" Anything against 10g copper? Per port prices are getting crazy cheap.
"DNS servers, LDAP servers, and our Hadoop Namenodes ... need RAID for extra data safety" Anything you've had issues with? Host level redundancy with the first two has been pretty great IME, can't comment on name odes. The only issue I've run in to is poor LDAP clients not fast failing an unhealthy server when another is available.