I’ve been asked several times recently what it means for Rackspace now that Webmail.us is using Amazon S3 (and EC2 & SQS) for data backups.  In case you missed it, last month we replaced our tape backups system managed by Rackspace, with a homegrown backups system built on top of Amazon’s web services.

Just yesterday in fact, in a great post on grid computing and Amazon, Joyent asked “So is Webmail.us’s use of Amazon’s web services a success for Amazon or a failure of Rackspace? Or both?”

Well let me answer publicly with what I have been telling everyone who has personally asked me this question…

Yes, our use of Amazon S3 displaced our use of Rackspace’s managed backups.  However, we desperately needed to replace it anyway.  Traditional data backups systems do a horrible job at backing up maildir formatted email data.  This is because with maildir, file names change frequently in order to track meta data such as Flagged, Read and Replied.  Each time the file name changes, the backups system sees a new file and backs the email up again.  This results is several copies of the same email being backed up and wastes backup resources – directly wasting our money.  This would be the case with any general purpose backups system, regardless of if it were a Rackspace hosted solution or not.

What we needed was a smarter backups system.  We needed to build something new; something custom; something designed specifically for the type of data we store.

We are a software and services company, not a hardware company.  Which is why we outsource our data center to Rackspace.  Rackspace owns the hardware, keeps it running, and replaces hardware components that break.  They do a great job at this.  We write and manage the software that runs on the hardware, and we do a great job at that.  A core software development philosophy at Webmail is to maintain a short development cycle; i.e. to release new features early and often.  One of the many ways we accomplish this goal is to build on top of re-usable components, whether that’s software that we write, open source software, or services hosted by other companies.  In this case we built on top of services hosted by another company.

Amazon’s web services allowed us to build something new.  By building on top of their S3 “storage cloud”, we were able to just develop the maildir backup logic and some data cleanup logic.  We were able to skip developing the backup storage system altogether.  We coded the storage client, not the storage server.

Initially we had planned on building both.  But when S3 came out our thoughts quickly shifted to “Screw that, lets just build the client and get this thing released”.

I strongly feel that moving our backups to S3 is a success for Amazon, and not at all a failure of Rackspace.

We’ve announced that this new backups system is saving us 75% monthly.  In the end, our backup data hosting costs would have been about equal had we built the backup storage system and hosted it on servers at Rackspace instead of using S3.  Our 75% cost savings came from building the logic that eliminated backing up the same email multiple times, which we were going to do in either case.

S3 allowed us to build this faster and start saving money earlier.

Will we host other applications on Amazon’s web services in the future?

Yes, if it makes sense to do so.

We have a limited number of programmers.  And as I have said before, when making “build vs buy” decisions it almost always comes down to two things: (1) Where do we feel our internal resources can be best spent?  and (2) Can we find a partner that we can trust with the rest of the stuff at an affordable cost?

Will our use of Amazon web services now or in the future replace our server growth at Rackspace?

Probably not.  We are growing fast and will always need a lot of servers to support our business.  I see these web services as a way to get more done, as opposed to replace existing stuff that we are doing currently.

We are always looking for ways to build new stuff faster.  In some cases this will mean building on top of services hosted by other companies, such as Amazon.  In other cases it will mean building on top of open source software and hosted it on servers at Rackspace.  And still, in other cases it will mean hiring more smart people to build it from scratch and host it on servers at Rackspace.  (speaking of which, if you’re a smart programmer shoot me an email)