Phil Wainewright wrote an article on his Software as Services blog Monday on a vary familiar topic – the ironies of Software as a Service companies who operate their own data centers.  I quote:

"One of the ironies of the on-demand application space is that many of the leading names operate their own data centers. This seems somewhat illogical, if not downright hypocritical. On the one hand, they ask their customers to rely on a third party to provide mission-critical business applications. But instead of doing likewise with their own infrastructure, they host in-house."

This is has been a topic of discussion at Webmail for years… Would it be cheaper buy servers and put them in a collo facility and manage them ourselves?  Or could we build out our own data center here in the VT CRC?  Or should we lease servers and let another company manage the hardware and network?

Our expertise is not infrastructure (even though we have people dedicated to it and that is what I spend 90% of my own time on).  We are a software company with heavy infrastructure usage.  We are also a customer service company.  And our people are great at what they do – programming and pleasing our customers.   Again quoting the article:

"To be very good at [infrastructure] at the same time as writing the software and developing the customer base is very hard. These companies are getting tripped up by things that are mission critical but they’re not really core to the development of the software."

The article goes on to say that many SaaS companies end up bringing their infrastructure back in-house once they have stabilized because:

"They simply don’t trust third party providers to offer the quality, pricing and capabilities they demand."

Believe me, I have done my homework on this topic.  I have looked at collo services from Equinix, Internap; managed hosting services from ServerVault, Rackspace, DigitalNation (now Verio), DialTone, among others, and I have talked with local companies about building our own facility.

When we have these discussions, it always comes down to two things: (1) Where do we feel our internal resources can be best spent?  and (2) Can we find a partner that we can trust with the rest of the stuff at an affordable cost?

We believe that we can be most effective by focusing our technical resources on innovating our software, rather than managing hardware.  We have been outsourcing our infrastructure to Rackspace since 2004, and we were outsourcing to ServerVault before that.  Both are great companies.  Since Rackspace guarantees that our servers and network will always be up, it allows our infrastructure team to work on the system, not the hardware.  We work on things such as:

• scaling our software to take advantage of our growing number of servers
• optimizing performance
• automating common system management tasks
• building monitoring and recovery tools
• securing our systems from spam, viruses and malicious traffic
• intelligent storage systems

I think the comment from the article that sums it up for most companies is that they simply don’t trust third party providers to give them what they need.  There are some horrible hosting companies out there, I know.  And even the bad ones have pretty web sites.  But man… Rackspace has hands down been the best partner I have ever worked with.  They are an amazing company, with great people who have earned my respect and trust; and most importantly they have given us what we need.  We are rapidly approaching 100 servers with Rackspace, and I have yet to find a compelling reason to bring that in-house.

As long as Rackspace continues to scale their business relationship with us as we grow, we will not be one of those companies who deviate from their core competencies by bringing their mission critical infrastructure in-house.

I have a new quote on my wall:

"New hires tend to want to do complex things, but we know complex things break in complex ways. The veterans want simple designs, with simple interfaces and simple constructs that are easy to understand and debug and easy to put back together after they break."

"The best advice is just basically to keep everything as simple as possible—simple processes, simple SKUs, simple engineering. These systems get to be very big very fast. I don’t think there’s really any one particularly hard, gnarly problem, but when you add them all up, there are lots and lots of little problems. As long as you can keep each of those pieces simple, that seems to be the key. It’s more of a philosophy, I think, than anything else."

— Phil Smoot, Product Manager for Hotmail

The full interview with Phil can be found here.

If you tackle every problem with the belief that there is a simple solution, you will usually find one.  Sometimes to solve a complex problem you just need to piece together several individual simple solutions.