Beth’s dad got me a sweet GPS for Christmas. Driving to DC right now and playing with all the features. I just got it to show my geosynchronous satellite calculated speed. And check this out… It also knows the speed limit for the road I’m on and if I exceed 5 miles over, it starts blinking red.
I’m going 65 in a 55 as I type this post one-handed on Route 66. And see that red spot in the lower center of this blury photo?
I was at Starbucks today, and I was ordering a small coffee and a bottle of their Ethos spring water. However, it turned out that they were out of bottled water. So the girl at the counter asked me if I just wanted a cup of ice water instead. I normally would turn that down, because tap water at restaurants sucks ass. But then the girl also said, "Our water is triple filtered and tastes really good." So I gave it a try.
Wow! It was the best water I’ve had since Beth discovered Fiji. Their filtered water was soft, but not excessively soft like Evian. And much better than the popular municipal-source filtered water brands – Aquafina and Dasani.
I know that buying the $2 bottles of Ethos helps children in third world countries get clean drinking water and all. And that’s great. I am glad to see that Starbucks is so dedicated to corporate social responsibility. But (call me a bad person now) I am never going to buy their water again. Starbucks’ free tap water tastes better.
Btw… Did you know that the Coca-Cola Company adds salt to Dasani
water? I don’t know about you, but when I’m thirsty I don’t want to
drink salt-water.
I use webmail in all sorts of weird ways. Pat says I am just weird… but really he knows that there are other people who do some of the same advanced things with webmail that I do. And there are many people who do even crazier things than me. We are power users. And we come in all sorts.
There is no "typical customer" in our business. There is no such thing as an "average user". Even basic users all seem to have their own unique habits and use email in their own unique way. It is very interesting trying to create a product that can meet such diverse demands.
Some of the things that I do are:
I have 18 separate Tasks lists to track just about everything from MP3s that I want to download, to future projects for my team.
I put numbers in front of each of my Tasks, because we can’t sort by priority yet.
I have 54 mail folders, most nested several levels deep. And I have 86 filtering rules to make sure that mail gets delivered directly to one of those folders. (is this really that weird?)
I store notes inside of emails within my Drafts folder, because we don’t have a Notes feature (yet?).
I store files as attachments within my Drafts folder, because we don’t have online file storage (yet?).
And I add notes to emails that I have received and delete attachments from emails that I receive by moving the email to my Drafs folder, opening it, making my changes, saving it, and then moving it back to it’s original folder.
Most of the things I listed are work arounds for features that we do not yet have. Features that I hope we build one day. This is why we created Idea Central, a place where all of us power users can collaborate on our ideas and vote on them in hopes that one day our features get built. The webmail team crancked out a bunch of features from Idea Central with the Concord release and October’s Hackathon. And there are many more features on the way. Make sure you get your votes in for the features that you want – you weirdo.
Yay! I’ve passed the 100,000 unread email milestone…
It’s really time that I clean up the info-alerts that are sent to our engineers from our servers. Do we really need to get an email every time each of our servers downloads a virus signature update? We should just log it, and get an email if too much time goes by without them downloading an update… Lol, which already happens as well.
Here is a gift for you Postfix admins.
See: http://secunia.com/advisories/23483/
/etc/postfix/main.cf:
body_checks =
pcre:/etc/postfix/body_checks.regexp
/etc/postfix/body_checks.regexp:
/(.*)\.pdf\#+/ REPLACE $1.pdf
Whenever I talk with somebody at a company that has a need for dedicated servers, I jump on the opportunity to sell them on Rackspace. No, I don’t get any commission or anything from them. I just feel that when it is so apparently clear to me that Rackspace is exactly what a company needs, I feel compelled share so that they don’t go needlessly down a wrong path.
On Friday, I was having lunch with two guys from one such company in Blacksburg, and they asked me “What is the biggest thing you’ve learned about hosting a system as large as Webmail.us’ at Rackspace?” Man, where do I begin? The biggest thing. Hmmm…
I told them the story of how when we first moved our email hosting system to Rackspace, we were running it on just 5 servers. These were powerful dual-Xeon boxes, lots of RAM, fast expensive SCSI drives, the works. Not cheap boxes. This was 2003, and our business was starting to boom. Soon 5 servers turned into 7 servers. Then 9. Our application started becoming more complex too… adding dns-caching, multiple replicated databases, load balanced spam filtering servers, etc. We had each of our servers running several of these applications so that we could get the most bang-for-the-buck out of the machines. This started to get complex fast, and was about to become a nightmare to manage.
With multiple applications per server, it became increasingly difficult to troubleshoot problems. For example, when a disk starts running slow or a server starts going wacky (technical jargon), how do you determine which of the 4 applications running on that server are the culprit. Lots of stopping and starting services, and watching /proc/* values. But with just 9 servers, you don’t have an excessive amount of redundancy and don’t want to have to do this all that often. Or worse, when an application crashes a box, it takes down all of the apps that were running on that box. If there was a better way to scale, we needed to find it.
We started Webmail.us while still in college, and while we had interned at some pretty neat companies, we didn’t have a whole lot of experience to lean on in order to figure things like this out. In computer engineering / computer science they teach you how to code, but they don’t teach you how to manage clusters of servers. We were learning how to run a technology company by making decisions through gut instinct and trial-and-error – not by doing what has been done in the past at other companies. And even after we had hired a decent number of employees, very smart employees and some with lots of experience, there were still many areas that our team was lacking expertise in. So what did our gut tell us to do in order to learn how to scale things the right way?… Get help from an expert.
Having a company like Rackspace on our side has been a huge asset. With a collection of talented engineers the size of theirs, they seem to always have at least one person who is an expert on just about anything that we have needed help with.
In 2005, by working with people at Rackspace like Paul, Eric, Alex, Antony and others, we decided to re-architect our system to give each of our internal applications and databases their own independent server clusters. The idea was to use smaller servers, and more of them; with smart software to manage it all so that hardware failures can be tolerated (hmm… have you ever heard of a system like this before?). With this approach, each application is completely isolated from the next. When a server starts acting wacky, we can just take it down to replace hardware, re-install the system image, or whatever… and the load balancers and data replication software knows how to deal with that.
We ended up completely ditching the beastly dual-Xeon servers in favor of 54 shiny new single-cpu AMD Athlon boxes, each with a 1 GB RAM and SATA hard drives. Basically equivalent to the hardware you could get at the time in a $1000 Dell desktop. We’ve grown this system over 3x since we first launched it with 54 servers. We still mostly use Athlon cpus, but have some Opteron and even some dual-Opteron boxes now in clusters that require a lot of CPU such as spam filtering.
Today it is just as easy to manage 180 servers as it was with 54 servers, because we’ve built things the right way.
Rackspace’s expertise was invaluable in creating this new system. However, we are not the type of company that likes to be completely dependent on another company, even if that other company is Rackspace. So, we didn’t just let them build this new system for us. We had them show us how to build it. They may have built the first pair of load balancer servers out of basic Linux boxes; but then we ripped them up, and built them again from scratch. Then we did it again. We did this until we understood how each component worked and we didn’t need Eric or Alex’s help anymore. We did this with everything that we built in 2005, and we continue to do this whenever we lean on Rackspace for help.
So my advice for these two guys who had been selling their software for almost 10 years and were about to move it towards a hosted web service model, was this… As much as you think you know about hosting your software, you are going to run into things that nobody at your company will have done before. Things that you guys are not experts at. If you stick your servers in a colo cabinet somewhere, you are going to have to figure those things out on your own. This will be slow and will probably not result in the best solution every time. I highly recommend that you consider hosting your app at a company like Rackspace who can help you when you need it. You are going to pay more to per server going this route if you simply look at the raw hosting cost. However, you will be able to get things online faster, work through problems effectively, and you will learn how to host your system from the best.
My other posts about Rackspace:
Outsource your data center
Amazon vs Rackspace
So I was tagged by Dan Ciruli yesterday for this meme thing. So I’ll do my part to keep this silliness going.
Here are five things that you probably don’t know about me:
(1) My first job was at age 7, stuffing envelopes at my parent’s printing company for 3 cents each.
(2) I hate video games. They’re a waste of time.
(3) I get my best ideas while in the shower or on the toilet.
(4) In 1995, Virginia Tech beat UVA (in football) because of me. It was the last game of the season, and the biggest game of the year. VT was ranked #20 and UVA was ranked #13, and the game was played in Charlottesville. The night before the game my friends and I went to several parties around UVA’s campus. For some reason, I decided to start telling random UVA fans that Tiki Barber, their star running back, was in jail and wouldn’t be playing tomorrow. By the end of the night the rumor amazingly had circulated far and wide, and I had UVA fans telling me that Tiki Barber was in jail and VT was going to kick their ass tomorrow. Well, Tiki Barber wasn’t in jail. It was just a rumor started by some drunk college kid 🙂 But I have to believe, that this rumor convinced many UVA fans that they were going to lose that Saturday and somehow contributed in Tech’s 36-29 victory.
(5) I am on day 3 of Dan’s 2007 pushup challenge (even though he forgot to include me in his post the other day 🙂 Beth is giving it a shot too.