Outsource your data center

Phil Wainewright wrote an article on his Software as Services blog Monday on a vary familiar topic – the ironies of Software as a Service companies who operate their own data centers.  I quote:

"One of the ironies of the on-demand application space is that many of the leading names operate their own data centers. This seems somewhat illogical, if not downright hypocritical. On the one hand, they ask their customers to rely on a third party to provide mission-critical business applications. But instead of doing likewise with their own infrastructure, they host in-house."

This is has been a topic of discussion at Webmail for years… Would it be cheaper buy servers and put them in a collo facility and manage them ourselves?  Or could we build out our own data center here in the VT CRC?  Or should we lease servers and let another company manage the hardware and network?

Our expertise is not infrastructure (even though we have people dedicated to it and that is what I spend 90% of my own time on).  We are a software company with heavy infrastructure usage.  We are also a customer service company.  And our people are great at what they do – programming and pleasing our customers.   Again quoting the article:

"To be very good at [infrastructure] at the same time as writing the software and developing the customer base is very hard. These companies are getting tripped up by things that are mission critical but they’re not really core to the development of the software."

The article goes on to say that many SaaS companies end up bringing their infrastructure back in-house once they have stabilized because:

"They simply don’t trust third party providers to offer the quality, pricing and capabilities they demand."

Believe me, I have done my homework on this topic.  I have looked at collo services from Equinix, Internap; managed hosting services from ServerVault, Rackspace, DigitalNation (now Verio), DialTone, among others, and I have talked with local companies about building our own facility.

When we have these discussions, it always comes down to two things: (1) Where do we feel our internal resources can be best spent?  and (2) Can we find a partner that we can trust with the rest of the stuff at an affordable cost?

We believe that we can be most effective by focusing our technical resources on innovating our software, rather than managing hardware.  We have been outsourcing our infrastructure to Rackspace since 2004, and we were outsourcing to ServerVault before that.  Both are great companies.  Since Rackspace guarantees that our servers and network will always be up, it allows our infrastructure team to work on the system, not the hardware.  We work on things such as:

  • scaling our software to take advantage of our growing number of servers
  • optimizing performance
  • automating common system management tasks
  • building monitoring and recovery tools
  • securing our systems from spam, viruses and malicious traffic
  • intelligent storage systems

I think the comment from the article that sums it up for most companies is that they simply don’t trust third party providers to give them what they need.  There are some horrible hosting companies out there, I know.  And even the bad ones have pretty web sites.  But man… Rackspace has hands down been the best partner I have ever worked with.  They are an amazing company, with great people who have earned my respect and trust; and most importantly they have given us what we need.  We are rapidly approaching 100 servers with Rackspace, and I have yet to find a compelling reason to bring that in-house.

As long as Rackspace continues to scale their business relationship with us as we grow, we will not be one of those companies who deviate from their core competencies by bringing their mission critical infrastructure in-house.

Mexican Disney Dollars

If you travel to a resort town outside of the U.S., such as Mazatlan
Mexico, there is a good chance you will meet people who try to get you
to take tours of their resorts.  They will offer you all sorts of free
stuff, such as free airport taxis, a buffet breakfast, cash to use at
their restaurants, etc.  Their goal is to sell you a timeshare.

The minute I stepped outside of the airport terminal in Mexico last
Sunday afternoon, there they were.  They caught me off-guard, and at
first I thought they were the people who organize shuttle service to
the resorts.  They weren’t.  However, they were offering me and Beth a
free $26 taxi ride, so like every other American that stepped off the
plane, I listened to what they had to say.  I said that I was not
interested a few times, but I let her keep talking.  The longer she
talked, the more free stuff she tossed into the offer in exchange for
my 2 hours the following morning.  "You have to eat breakfast any way
right?  So come eat breakfast at our resort while you listen to what we
have to say", she said… What the hell.

I walked away with $200 in cash certificates to be used at their
restaurants, 2 free massages, free taxi service to my resort, and free
taxi service back to the airport the next Saturday.

Beth and I took the tour of Pueblo Bonito.  I’ve been to some nice
resorts before in Jamaica, Cancun, Bermuda and Bonita Springs, but this
blew them all away.  I’m going to have to remember the name of this
company.  I learned a lot about how timeshares work too (Beth says I
ask too many questions sometimes).  I didn’t buy anything, but I asked
the sales guy how many people buy and he said 3 out of 10.  I believe
him – the pitch and the offer was pretty convincing.

With the free $200 we had a lobster dinner, wine, and another free
breakfast later in the week, and we still had $50 left to spend.  But
Beth wouldn’t let me spend it… she wanted to keep the last two
certificates as a souvenir of "how cheap I am".  I kept saying "You
don’t leave Disney World with Disney Dollars do you?  Lets spend the
last $50."

Anyways I’m back, I’m married, and Beth and I had a great time in
Mexico, and we still have two $50 Pueblo Bonito certificates that
expire in 5 days.

I don’t see myself taking another week long trip like this for several
years.  Towards the end of last week I was ready to get back to work.
I guess a 3 day weekend is enough time to unwind.  I’ll post a link to
pictures in a few days.

July 2002

In July 2002 Beth and I took off for 11 days and drove up the entire California coastline.  We started in San Diego, visited a friend in Temecula and then started up the coast on Highway 1.  We skipped over LA and made stops at random hotels in Santa Barbara, Carmel/Monterey, Palo Alto, and spent 2 days in San Francisco & Berkeley.  On day six we ran out of hotel money, so we bought camping gear at Target and continued on.  Before camping we hit a few vineyards between Napa and St. Helena and discovered our new favorite wines: a Cabernet from Silverado Vineyards, and a S. Anderson white that you are supposed to drink warm (it looks like S. Anderson is now owned by Cleff Lede).  Then we made a b-line back to the coast, and spent our first night camping at a random camp site up in the mountains.  The next day we hiked and camped in Sequoia National Park, and the day after that we did the same in the Redwood National Forest (it happened to be bikers week in Redwood National Forest, so all of the small towns were filled with folks wearing leather and bandannas – who knew?).  Our flight back to Virginia was out of San Diego airport, so on day 9 we started back down the coast – this time on the main highways.  We made it all the way from the Redwoods down to Malibu in one day and spent the night camping right off the beach in Malibu… we could hear the ocean; it was awesome.  The next day we hit the beach and ate our first ever Fish Tacos (which has since become one of our favorite home cooked meals).  That night we drove back to San Diego.  We attempted to hit a camp site there but it was full of homeless people, so we got a hotel in Carlsbad.  We hit the beach again that morning in Carlsbad, and then finally caught our flight back home.

It was a great trip.  It was also the one and only time that I have been completely disconnected from Webmail.us since we founded the company in December 1999.  All of my other trips have been short, and I have had full access to email and cell phone, and spent at least 25% of the time working.

Well, after 4 years it is time to once again pull all of the geek stuff out of my head for a week…

Next month Beth and I are getting married (ya, I know… we’ve been together for 6.5 years now; but I’ve been busy!).  After the wedding on March 11th, we are going to Mazatlan, Mexico for 7 days, where I plan to do nothing buy sit in the sun, refresh my scuba skills, and drink a few tequila concoctions.

And tomorrow, me and the crew head to Vegas for a few days for the bachelor party.  I will still be "connected" there, although probably not in a state to do any work.

In 2002, it was a bit rough for the folks at the office while I was away.  But since then we have built a phenomenal engineering and support team that I am more than confident in…  Korey, Kirk and team, the email hosting system is all yours!

Update: proxy software testing

For those testing the new POP3/IMAP proxy, please switch your incoming mail server to beta.webmail.us

Tonight we set up a second Dovecot proxy and slapped a load balancer in front of it, in order to closer simulate how the new proxy system will be running when it goes live.  We also set up a real SSL certificate for beta.webmail.us so that you stop getting those annoying security warnings.

Give it a try.  Post your results as a comment to this blog, or email feedback@beta.webmail.us.  Be sure to mention what mail program you are using, as well as how you are connecting (POP3/IMAP, SSL/TLS/plain-text).

Hackathon 2.0

Look at the cool things that the Webmail.us team built this weekend during Hackathon, as well as photos of this cool bunch.

I missed it unfortunately.  I had to go to DC with Beth this weekend so that we could finalize the details for our wedding next month.  I did get to hit a Wizards game Sunday afternoon while Beth was doing the extreme girly stuff.