Stu posted a great description of the logging system he built to make sense of our massive volume of mail logs using Hadoop.  And I gave highscalability.com some follow-up details on how our logging architecture evolved to this point… but the real reason we switched to Hadoop:

> Why didn’t you choose to go to a multiple database
> server architecture instead of relying on one
> database server?

We
knew that was an option, but we felt that moving to a partitioned MySQL
data set would only buy us time, and we’d still need a more scalable
solution at some point down the road.  And to be honest, the coolness
factor of building something with Hadoop really excited us and that
definitely weighed into our decision.

Ya, we’re geeks.

Btw, Hadoop is now a top level Apache project, no longer a sub-project of Lucene.  Congrats to Doug Cutting and crew!