Docker Tutorial Part 3: Kamal with Alternative Proxy

In this document we discuss using Kamal orchestration with an external reverse proxy, such as nginx or haproxy. We also discuss opening additional ports on docker containers orchestrated with Kamal.

This document assumes you have a working knowledge of using Kamal for orchestration of Docker containers, as described in Docker Tutorial Part 2: Orchestration with Kamal.

Using an Alternative Proxy

By default, Kamal uses kamal-proxy during deployments to perform health checks on startup, open ports 80 and 443 on your server(s), and allow for a seamless transition from old containers to new.

But sometimes we need to run a different reverse proxy + load balancer like nginx or haproxy, which already do the same thing, so we have no need for kamal-proxy.

We can disable kamal-proxy using a proxy: directive in our deploy.yml file, like so:

servers:
  web:
    hosts:
      - 10.10.10.1
      - 10.10.10.2
    proxy: false

Caution

When doing above, be sure to comment out or remove any other proxy: directives in your deploy.yml.

Opening Ports

Since kamal-proxy is disabled, we will need to use an options: stanza to publish: the ports we want to open. The following would open port 80 on the public interface proxying to port 80 on the container, same for ports 443, 8080 and 4430:

servers:
  web:
    hosts:
      - 10.10.10.1
      - 10.10.10.2
    proxy: false
    options:
      publish:
        - 80:80
        - 443:443
        - 8080:8080
        - 4430:4430

Resolving port conflicts

With the above config we see that the initial kamal setup is fine, but subsequent deploys will fail due to “port already allocated” errors. This is because Kamal will try to start the new container before shutting down the old one, which kamal-proxy used to handle for “seamless deployments”, but is no longer running.

The solution is to run a hook script which will stop containers before Kamal starts the new ones: https://kamal-deploy.org/docs/hooks/overview/

The following pre-app-boot hook script issues docker container stop on the containers first.

.kamal/hooks/pre-app-boot:

#!/bin/python3
#
# Kamal hook script 'pre-app-boot' to try to stop all containers prior to
# kamal starting the new ones.  This is for when you need to listen on multiple
# ports using options: publish: and start getting "port already allocated" errors.
#

import os, subprocess, datetime

# Open a logfile.
f = open('/var/log/pre-app-boot.log', "a+")

# Try to stop containers before kamal boots the app.
hosts = os.environ['KAMAL_HOSTS'].split(',')
for host in hosts:
    x = str(datetime.datetime.now())
    f.write(x + ': stopping containers on ' + host + ' ... ')
    result = subprocess.run(['ssh', host, 'docker container stop $(docker container ls -q)'],
                            text=True, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
    if result.returncode != 0:
        f.write('error: docker container stop failed\n')

    else:
        f.write('success\n')

f.close()

Applying Group Sizes

An issue with above is deploys will by default block until all containers have stopped, causing a downtime which grows exponentially with the number of hosts. The solution for this is to use the boot: option to set group sizes.

For example, say you have 4 hosts and want to deply them in “sets” of 2 at a time, with a 10 second pause between. You can add a boot: stanza to your config, like so:

servers:
  web:
    hosts:
      - 10.10.10.1
      - 10.10.10.2
    proxy: false
    options:
      publish:
        - 80:80
        - 443:443
        - 8080:8080
        - 4430:4430
boot:
 limit: 2
 wait: 10

This tells Kamal to deploy 2 at a time, so the script above also runs/stops 2 at a time. This way you always have (the other) 2 hosts in production to pick up traffic and deploy seamlessly.

More details in the official docs here: https://kamal-deploy.org/docs/configuration/booting/

Setting Environment Variables per-server

You might also have to pass custom --env variables to the docker run command. This can be done with tags. For example say you need to pass the IP address as an environment variable. You add tags by adding them to the server IP addresses separated by a colon. To modify our existing deploy.yml it looks like this:

servers:
  web:
    hosts:
      - 10.10.10.1: 10.10.10.1
      - 10.10.10.2: 10.10.10.2

Note

Although we are using the IP addresses in this example, tags can have any value.

We then add an env: stanza which uses the tags to fill in the variables for those specific hosts. Note too, we also add a clear: tag for a custom environment variable that will be applied globally to all hosts.

env:
  clear:
    MY_GLOBAL_HOST_VAR: true
  tags:
    10.10.10.1:
      MY_HOST_IP: 10.10.10.1
    10.10.10.2:
      MY_HOST_IP: 10.10.10.2

Putting it All Together

The full config.yml:

# Deploy to these servers.
servers:
  web:
    hosts:
    # Tags are done in the form of IP: <tag>, we use the tag here as also the IP address,
    # so we can pass custom ENV vars containing specific IPs to each docker run command
    # (see below)
    - 10.10.10.1: 10.10.10.1
    - 10.10.10.2: 10.10.10.2
  proxy: false
  options:
    publish:
      - 80:80
      - 443:443
      - 8080:8080
      - 4430:4430

boot:
  limit: 2
  wait: 10

env:
  clear:
    MY_GLOBAL_HOST_VAR: true
  tags:
    10.10.10.1:
      MY_HOST_IP: 10.10.10.1
    10.10.10.2:
      MY_HOST_IP: 10.10.10.2