Previously, Explorable Places used Capistrano to deploy to one droplet in DigitalOcean. This setup had a number of downsides and limitations. Among them, we had:
- a single point of failure
- only nearly zero downtime for deployments
- no easy way to scale horizontally
- difficulty in upgrades
When the app started out, it was easy and only necessary to have one box up that hosted everything: nginx, app, db and cache (memcached). Provisioning a new droplet would be a pain in the butt, and I had no way to distribute traffic to each box anyway. Platform upgrades (Ubuntu) proved to be a royal pain in the behind, and required the site to be down for some time, in addition to the risk involved in the update itself.
Dockerizing the app would solve a couple of our issues, and enable us to solve the others. There are multiple articles out there about how to convert your app, no matter the stack, to Docker, so I won’t go into such a large topic here. But for me, the basic steps were:
- Move DB out of our droplet, and into a managed host
- Dockerize our app with docker-compose, and utilize Dockerhub to manage our images
- Create a couple Docker droplets from the DigitalOcean marketplace, managed under a DigitalOcean load balancer
- Write a deploy script that will deploy to each environment
Now that you presumably have your Dockerized app, along with all your separate
docker-compose files, environment variable files, and others that you do not want checked into source countrol, we need a way to get those files to our droplets, and start up our app.
Below, I’ve outlined our deploy script. See the bottom of this post for all the gory details, but what it does is:
- Use DropletKit to get a list of our staging/production droplets
- Loop over each droplet to do the following:
- Take the droplet out of traffic by removing it from the load balancer
- Copy all environment files, docker-compose files, etc. to the droplet.
- Run a start up script within the droplet that:
- Takes down all running containers and removes them, as well as volumes, and images.
- Pulls in latest tagged image from DockerHub
- Brings up containers
- Put the droplet back into traffic by adding back to the load balancer (if everything was successful)
With this in place, you can simply deploy your app with:
The one key here is that we must be certain that our server is up and serving traffic before we add it back to the load balancer to receive traffic. By default, Docker Compose will consider the container to be started and healty as soon as it is up, even if it takes 10-15 seconds for your server within your container to start.
To make sure our app is ready to serve traffic, we can utilize health checks with Docker Compose. You can define this in your compose file like so:
This health check will be run within your container, so you can reference localhost here. If you have other containers that depend on this one, you can define the conditions that will dictate when that container is brought up. Above, we have an nginx service that depends on our app. We tell it to wait until our app is healthy, which will be based on the health check we defined.
Health checks will continue to run, so be sure to exlude it from any:
- SSL validation
- Performance monitoring (will severely skew your ‘typical’ and ‘problem’ response times)
- IP throttling (don’t want to throttle yourself)