Opensourcing Bamboo: Automated Mesos Marathon load balancing

At Qubit, one of the core tenets is to constantly review systems that are working ‘fine’ and to think about how we could do stuff even better. Sometimes these are small incremental changes, sometimes they are large projects.

A while ago, we identified that virtualization/cloud infrastructure (whilst getting better) was substantially less cost efficient than bare metal servers for certain workloads. Over time, these servers have become an important component of Qubit’s infrastructure.

Internally, we use the fantastic Apache Mesos for the management of heterogeneous services in an environment neutral way (i.e. equally happy in both cloud and co-location environments), all scaling dynamically with traffic. As we continued to roll out more services based on Mesos, however, we found that load balancing across various instances of a running service across the Mesos cluster to be a common problem.

Today, we are open sourcing an internal project, Bamboo, that seeks to simplify running a DNS based HAProxy auto configuration and auto service discovery web daemon for Apache Mesos and Marathon.



“Apache Mesos is a cluster manager that simplifies the complexity of running applications on a shared resource and Marathon is a Mesos framework for long-running tasks. ”



In writing Bamboo, we set ourselves some basic goals:

  • Access canary services via a human-friendly DNS name

  • Review products per development branch feature in an automated fashion

  • No service downtime

  • Quickly promote or rollback a service or its dependent web services

  • Reproduce a Test, QA or Third type of environment for different use cases

  • Efficiently allocate infrastructure resources, with support for rapid, dynamic reallocations

  • Simplified, Docker friendly workflow

Just imagine a real bamboo trunk and how it joins all the leaves and keeps them on a straight line.

Bamboo provides a minimal set of features to automate services discovery and web request load balancing to web services running in Mesos cluster. It features:

  • A user interface for configuring DNS mapping to Marathon id

  • A REST API for configuring DNS mapping; during automated application deployment via Marathon with DNS management services an API comes very handy

  • Automatic HAProxy configuration based on templates; you can provision your own template in production to enable SSL and protect HAProxy stats interface, or even configuring different load balance strategy

  • Optional handling of health check endpoint if Marathon application is configured with Healthchecks

  • Stateless daemon; enables horizontal replication and scalability

  • Developed in Golang, deployment on HAProxy instance has no additional dependency

  • Optional integration with StatsD to monitor configuration restart/reload events

 

Automated continuous integration for services requires management of versioning and dependencies. Bamboo can make these services accessible by a human-friendly domain at a linear level. A coordinator process can be implemented on top of third party DNS provider (e.g. AWS rount53) to configure services topologies.

 

 

Bamboo ensures HAProxy understands DNS and directs traffic to the correct web application in Mesos Cluster. When scaling up or down a number of applications running in Marathon, each Bamboo web daemon auto reconfigures HAProxy process and only direct traffic to the application when it passes health check.

 

If you have a small scale web service to manage, the user interface is useful to manually manage and visualize current state of DNS settings. You can find out if a DNS is assigned or missing from the interface. The template is flexible to customize based on your infrastructure requirement. Of course, you can configure HAProxy template to load balance Bamboo web interface too.

 

 

Visit the Bamboo Github page for more information about configuration and deployment guide. We always welcome contributions and love to see what we're missing.