

Logging the cloud with SimpleDB - pveugen
http://www.peecho.com/blog/logging-the-cloud-with-simpledb.html

======
sehugg
SimpleDB is good for a lot of things, but the unpredictable costs (since
you're charged for box usage which varies) and network latency make it
unsuitable for logging at scale. (like, what if you get 100's of error
messages a second?)

We use a decidedly low-tech scheme. We log everything to files on local
storage. Then we use a command line tool to periodically sync these to S3.
Each instance gets its own folder.

When we want to query, we have several options. We can run good old grep or
tail on one instance -- or on several instances with pssh. We can use the S3
command line tool to process multiple files on an admin machine. Or we could
use Hadoop to mess with files in S3 (haven't done that yet...!)

Not as snazzy as having each log record stored in a structured fashion, but
it's a lot cheaper than SimpleDB.

------
ConstantineXVI
I put together a similar logging setup for our servers recently; except I'm
using CouchDB for my store and NLog (said servers are C#) for the library.
Reasons I'm using Couch instead of SimpleDB:

1) REST+JSON API made integrating it super-simple (had to add a single line to
NLog to set MIME types to make Couch happy)

2) We use SDB in our service; so if SDB blows up, not only would things break,
we couldn't easily get at our logs to see why.

Biggest drawback I've found to Couch in this context is that it loves to
devour disk space; you quickly learn to cron some compaction jobs.

~~~
maratd
Why not use something like rsyslog and have it transfer the logs via udp to a
centralized rsyslog server? Very easy to setup.

~~~
ConstantineXVI
The first motivation for this setup was to make our logs easily parseable to
extract stats from them; Couch seemed like a natural fit to store them. Plus,
Couch's sane API essentially means we're able to grab the logs from any
tool/language with minimal effort. I'm well aware that this isn't necessarily
the best/proper way to handle logs, but we're all fairly pleased here with how
it's worked out.

------
centrical
Thanks for your addition. At Peecho, we figured that if SimpleDB goes down,
logging would be the least of our troubles. Read here how we tried to maintain
our uptime: [http://www.peecho.com/blog/minimizing-downtime-on-amazon-
aws...](http://www.peecho.com/blog/minimizing-downtime-on-amazon-aws.html)

\-------- Please vote for Peecho in the race for the Accenture Innovation
Awards: <http://goo.gl/3xLlC>

~~~
rkalla
Was just going to post this link -- you guys put a lot of time into your AWS
posts and they are excellent. You have a really nice/robust AWS deployment
there.

~~~
centrical
Thanks!

------
mrspandex
Logging seems like the perfect candidate for a relational database. It's a
write-heavy system with need for powerful ad-hoc query capabilities. Am I
missing something? Why put this in a key-value store?

~~~
rbranson
What about a highly-structured ACID datastore designed primarily for short
transactions reading & mutating small sets of random rows makes you think
that? Log data is semi-structured and very heterogenous. I would choose the
filesystem far before an RDBMS.

~~~
mrspandex
Maybe RDBMS isn't the /most/ efficient way to store the data, but the
advantage of being able to write simple and powerful queries against that data
is huge. I seem to lose that with file or key/value based storage. Am I wrong?
(not trying to criticize, I am genuinely curious)

Also, I don't need to worry about files across servers and rolling log files
that file logs usually entail.

~~~
rbranson
I'm not suggesting that a K/V store is better, just that an RDBMS isn't a good
solution to this problem either. How do you effectively utilize the query
capabilities of an RDBMS with semi-structured data? You're better off sticking
the data in a flat file and using text processing tools to work with it.

------
EponymousCoward
How do you get around the 10GB/domain limit?

What do you do when your fleet gets big enough that you start getting
throttled?

What's the query latency?

How do you delete old log data?

------
ww520
What is the cost like? What's the logging volume?

