Load balancing DNS with keepalived

Keepalived currently only supports load balancing TCP traffic.  However DNS runs over UDP, so how can you load balance DNS traffic using keepalived?  I had mentioned in my Load Balancing vs Failover post that I’d give you a trick to accomplish this, so here it is…

Keepalived is essentially a wrapper around Linux Virtual Server (LVS) which is built into most modern kernels.  LVS supports load balancing of both TCP and UDP traffic.  (keepalived also includes VRRP and Healthchecks, but that’s off-topic)  Lucky for us, DNS servers happen to listen for TCP connections in addition to UDP.  They do this in order to handle queries that are larger than the old RFC limit of 512-bytes.

The simple trick I use in order to load balance DNS traffic is to configure keepalived to load balance TCP port 53 just like you would load balance any other TCP port, such as http or smtp.  Then use keepalived’s notify_up / notify_down script calling feature to manually configure LVS for load balancing the corresponding UDP port.

For example:

virtual_server 123.456.78.9 53 {
  delay_loop 6
  lb_algo wrr
  lb_kind DR
  persistence_timeout 0
  protocol TCP
  ha_suspend
  real_server 10.0.0.1 53 {
    weight 1
    notify_down "/sbin/ipvsadm -d -u 123.456.78.9:53 -r 10.0.0.1:53"
    notify_up   "/sbin/ipvsadm -a -u 123.456.78.9:53 -r 10.0.0.1:53 -g -w 1"
    TCP_CHECK {
      connect_timeout 6
    }
  }
  real_server 10.0.0.2 53 {
    weight 1
    notify_down "/sbin/ipvsadm -d -u 123.456.78.9:53 -r 10.0.0.2:53"
    notify_up   "/sbin/ipvsadm -a -u 123.456.78.9:53 -r 10.0.0.2:53 -g -w 1"
    TCP_CHECK {
      connect_timeout 6
    }
  }
  real_server 10.0.0.3 53 {
    weight 1
    notify_down "/sbin/ipvsadm -d -u 123.456.78.9:53 -r 10.0.0.3:53"
    notify_up   "/sbin/ipvsadm -a -u 123.456.78.9:53 -r 10.0.0.3:53 -g -w 1"
    TCP_CHECK {
      connect_timeout 6
    }
  }
}

Using the above configuration, whenever healthchecks finds that TCP port 53 is not responding on one of the real servers, it removes the TCP port from the pool of available servers, and then notify_down immediately calls ipvsadm to take the corresponding UDP port also out of it’s pool of available servers.  So keepalived (LVS) stops routing traffic for both TCP and UDP to the dead server whenever the TCP port fails.

Likewise, when healthchecks finds that TCP port 53 is back up, notify_up calls ipvsadm to add it back into the pool of available servers.

There is one additional thing that you must do in order to make this all work.  Keepalived does not call notify_up at startup, so you must also add the ipvsadm command to the keepalived init script, so that the UDP ports get brought up when keepalived launches.

Lot 10

Last year, many of us Webmailers and friends tailgated in Lot 10 for all of the Tech home football games.  We had a great time.  There were kegs, bonfires, loud music, cornhole, wine, cheese, bean dip, tamales, breakfast/lunch/dinner, the occasional co-worker’s bare ass, broken stuff, and a seemingly unlimited flow of burgers and hot dogs.

Needless to say after the fun of last year, David Catalano and David Lawson who own the parking passes, tried very hard to get Lot 10 again.  Even though they both explicitly requested Lot 10, the Hokie Club chose to give them Lot 8.  Lot 8 is a "better" lot according to the Hokie Club, but it is lacking two main things… a large grassy area to party in, and near by parking for 40+ people.  (Lot 8 is a hike, whereas Lot 10 is right next to downtown)

So, after the Lot 8 passes arrived last Monday, I spent the first half of Tuesday on the Tech Sideline message board, attempting to arrange a trade.  I ended up finding a guy in Danville who would trade his Lot 10 for a Lot 5 and I also found another guy in Richmond who would trade his Lot 10 also for a Lot 5.  So then I went hunting for a Lot 5 trade, and I located a guy in Roanoke who would trade his two Lot 5 passes for our two Lot 8 passes.  Without any Lot 5 passes in hand yet I then told the first two guys "I will trade my Lot 5 for your Lot 10"… and the trade was set into motion.  On Wednesday, David Lawson and I met the Lot 5 dude in Roanoke and swapped passes, then headed out to Danville… an hour later we had our first Lot 10 pass!  Later that afternoon back at the office, I dropped the first game’s Lot 5 pass into a FedEx envelope and addressed it to the fellow in Richmond.  And on Friday we got our second Lot 10 pass!

Long story short, we are in Lot 10 again – same location as last year, to the left of the tennis courts on Washington Street.

See you bright and early on the 2nd.

Still reading Blue Ocean Strategy

Have you noticed that the book listed in “I’m Reading” on the right side of my blog hasn’t changed since March?  Its true… I’ve enjoyed Blue Ocean Strategy so far, but I haven’t exactly made time to read lately.

Mini book report:  The book covers how some interesting companies, such as Yellow Tail and Cirque du Soleil, have been able to break away from existing crowded markets and create new demand though product innovation.  I really like the fact that this book illustrates these business concepts using non-technology companies, since most of the innovation that I see every day are technology related.

Well, hopefully I can finally knock this book out on Friday while en route to TechCrunch 6… which looks like a blast!  If you’re in the SF Bay area this weekend and want to meet up, drop me an email.  (But I’m really just going out there to make $200)

Summer of Spam

As the summer winds down, I look at what one of Webmail’s newest stars has been able to do with our anti-spam system in just 3 months.  This summer Mike T:

(1) built new software that integrates customers’ existing sender safelists with Postfix so that mail not only bypasses spam content filtering (as it always has), it also makes sure that the mail doesn’t get rejected during the SMTP session.  (and the reverse for blacklists);

(2) enhanced our safelist and blacklisting system to allow customers to block or accept mail from specific IP addresses; and

(3) added selective greylisting capabilities to our SMTP servers.

#1 and #2 have been released and we began rolling out #3 last week.  As of Friday, we are greylisting mail that arrives to catch-all mailboxes and everything that hits our mx2 servers.  Soon we will also be greylisting mail sent from dynamic IP ranges such as computers connected behind DSL and Cable networks.  And in a few weeks customers will have the option to turn on greylisting for all mail they receive from new sources (this will be disabled by default).  Fyi, much of this was built around a great open-source project called Policyd.

Anyways, I am damn proud of Mike.

When will Gmail reach 5GB?

Gmail advertises:

  • Don’t throw anything away.
    Over 2750.162469 megabytes (and counting) of free storage so you’ll never need to delete another message.

…and their megabyte counter grows continuously.  But when will Gmail reach Webmail.us’s 5GB?

Well, Gmail’s current growth rate is 0.4 MB per day.  That means Gmail will reach the 5GB milestone in 5,925 days — October 21, 2022.

Please test the beta proxy (again)

I think we have finally worked out all of the bugs with the new POP3/IMAP Proxy Software.  The main symptom was when using SSL, saving to the Sent folder would occasionally break.  Here were the fixes.  Thanks Timo!

So, if you’re a Webmail.us customer and you’d like to help us test, please switch your POP3 or IMAP server to:  beta.webmail.us

You should expect everything to work normally.  If you see any differences or encounter any problems, please shoot me an email.  In the email include whether you are using POP3 or IMAP, and connecting with SSL, TLS or plain-text, and also what mail program and version you are running.

If you try it and do not encounter any problems, I am also interested in hearing from you.

Thanks!

Load Balancing vs Failover

This week we added a pair of load balancers to our email hosting system to balance our internal system traffic such as DNS, LDAP MySQL, and a few other services.  Until now we had only used load balancers to balance our external customer traffic (SMTP, POP3, IMAP, Webmail).

Up until now we had our internal services hosted on many independent pairs of servers with master-master failover via heartbeat.  With heartbeat, when there is a problem with one server in a pair, heartbeat tells its partner to take over all traffic for the pair until the problem is resolved.  Master-master pairs can be tricky however, because it is difficult to catch and handle all possible failure conditions.  Plus when a member of a pair dies, fixing it is extremely time sensitive since your service will be down if the other member dies before you restore the redundancy.

It is also messy to scale services that are configured for master-master failover because your application servers need to be configured to each query different database servers, DNS servers etc.  It would be a lot cleaner if the applications could simply be told to "query DNS" and something external to the application server routes the query to the right place.  That is where load balancing comes in.

Now all of our application servers have identical configurations.  They all query the same DNS server, the same LDAP server, and the same MySQL server.  However the "server" that they are configured to query is actually a pair of load balancers which redirect the query to one of the real servers behind it.  When there is a problem with one of the real servers, the load balancers simply stop sending queries to the bad server.  And to scale, we simply add a new real server and tell the load balancer about it.

Our load balancers are custom-built AMD Opteron Dual Core servers with gigabit NICs.  They run keepalived, which uses IPVS for load balancing and VRRP for making the load balancers themselves redundant.

Keepalived out of the box only supports load balancing TCP traffic though, so we had to hack it to do UDP for DNS.  I’ll save that for another post.