Working with Elixir Releases and performing CI/CD in containers

Elixir has seen fast adoption in Wego with multiple services already in production after the weekend project UltronEx became a daily driver for monitoring real time data it has spurred a movement to explore new options and build robust resilient systems.

As We run almost all our work loads in containers we take advantage of the same build pipelines for building container images to run CI/CD as well. Elixir runs on BEAM and has some nice features such as hot redeploys that is something anti to the concept of containers where they are immutable images none the less it does not mean Elixir is not container friendly. Mix, the elixir build tool similar to what Rake is for Ruby and has great support to generate a bundle with ERTS called a Release that you can take and run anywhere as long as it is the same platform it was built on. This is what works really well with containers keeping the image size significantly small without any external dependencies. Even though this makes the Elixir Release easy to work with it also strips it of its tool chain that you require to perform actions such as running migration or executing tasks as there is no Mix available in a Release. The below chart shows the different approaches and their features

One might ask why use a Release then for that there is a great thread on Elixir forum on why to Always use Releases. Let's say If you were to package your app with Mix and run it using mix run --no-halt to overcome issues with missing Mix what difference does it make on the container size ? The elixir image for 1.11.2-erlang-23.1.3 is about 214MB which is used as the base to either build a release in a multi stage Dockerfile or use it to run the app using Mix. Compared to that Debian image is only 69.2MB. When we use an Elixir Release for a Phoenix app and copy that to a stage based on Debian the final image size comes to 101MB whereas if we were to use Mix and not generate a release the final image size is 627MB a whopping 6X the release image size.

That is some significant advantage releases have over other approaches and also goes on to show the impact multi stage build files can have over image sizes.

So that sorts that we should always use Releases but how to then get the lost functionality of Mix ? For this Elixir and Docker both do their part to seamlessly make things similar to what you could do with a Ruby CI/CD containerised process with rake. For running database migration with releases Elixir has a work around with Ecto Migrator where in you create a separate file that executes the migrations, it is a standardised code and should actually IMO be part of Ecto

defmodule MyApp.Release do
  @app :my_app

  def migrate do
    load_app()

    for repo <- repos() do
      {:ok, _, _} = Ecto.Migrator.with_repo(repo, &Ecto.Migrator.run(&1, :up, all: true))
    end
  end

  def rollback(repo, version) do
    load_app()
    {:ok, _, _} = Ecto.Migrator.with_repo(repo, &Ecto.Migrator.run(&1, :down, to: version))
  end

  defp repos do
    Application.fetch_env!(@app, :ecto_repos)
  end

  defp load_app do
    Application.load(@app)
  end
end

After creating your release you can run you migrations in production with eval for e.g

$ _build/prod/rel/my_app/bin/my_app eval "MyApp.Release.migrate"

Migration  is a special case and Ecto has some nice features to access the Database to make this all possible but for running some tasks in production environment requires a separate approach and some understanding of an Elixir application or for a web application built in Phoenix.  Every Elixir application is built of multiple applications where a Phoenix app is a list of applications where Web and Database are separate applications which also gives Elixir applications its robustness where any single application can not bring down your entire Service. For this you can take lead from the elixir release docs and base understanding of elixir application to formulate your application container image to be used to execute tasks and run the application service as well. Let us take a Phoenix app GhostRider as an example. Here we split the application children into two separate lists base and web . Based on what is required we can handle what children load from an environment variable which can conveniently be passed to a container at runtime.

defmodule GhostRider.Application do
  # See https://hexdocs.pm/elixir/Application.html
  # for more information on OTP Applications
  @moduledoc false

  use Application

  def start(_type, _args) do
    base = [
      # Start the Ecto repository
      GhostRider.Repo,
      {Finch, name: GhostRiderFinch}
    ]

    web = [
      # Start the Telemetry supervisor
      GhostRiderWeb.Telemetry,
      # Start the PubSub system
      {Phoenix.PubSub, name: GhostRider.PubSub},
      # Start the Endpoint (http/https)
      GhostRiderWeb.Endpoint
      # Start a worker by calling: GhostRider.Worker.start_link(arg)
      # {GhostRider.Worker, arg},
    ]

    children =
      case System.get_env("APP_LEVEL") do
        "TASK" -> base
        _ -> base ++ web
      end

    # See https://hexdocs.pm/elixir/Supervisor.html
    # for other strategies and supported options
    opts = [strategy: :one_for_one, name: GhostRider.Supervisor]
    Supervisor.start_link(children, opts)
  end

  # Tell Phoenix to update the endpoint configuration
  # whenever the application is updated.
  def config_change(changed, _new, removed) do
    GhostRiderWeb.Endpoint.config_change(changed, removed)
    :ok
  end
end

Though this approach would work but applications only start when a release is started. To overcome this we need to use the eval to execute a wrapper function bound to perform operation but in order to execute that operation we need to start our application using Application.ensure_all_started(@app) along with the environment variable controlling which children should be loaded.

With Migrations and Task sorted and capable of working with containers the last piece of the CI/CD puzzle is how do you run your test when there is no Mix in your release and the release is byte code and no test folder is available ? This is where docker shines with its multi stage builds and we leverage docker-compose to target a stage to be able to execute tests in the build to ensure they are consistent across different technology stacks we have.

Now some docker and docker-compose recipes to bring this all together to work in a CI/CD pipeline. One thing I have to admit is lack of resources around containerising Elixir/Phoenix applications, something the community should work towards improving. Below is a file for building a Phoenix application and you can see there is a stage called tester this is what we will target using docker-compose

# https://elixirforum.com/t/could-use-some-feedback-on-this-multistage-dockerfile-1st-elixir-phoenix-deployment/30862/10?

########################
### Dependency stage ###
########################
FROM hexpm/elixir:1.11.2-erlang-23.1.3-debian-buster-20201012 AS base

# install build dependencies
RUN apt-get -qq update && \
    apt-get -qq -y install build-essential npm git python --fix-missing --no-install-recommends

# prepare build dir
WORKDIR /app

ARG MIX_ENV
ARG RELEASE_ENV

ENV LANG=C.UTF-8 LC_ALL=C.UTF-8

# Update timezone
ENV TZ=Asia/Singapore

# install hex + rebar
RUN mix local.hex --force && \
    mix local.rebar --force

# set build ENV
ENV MIX_ENV=${MIX_ENV}
ENV RELEASE_ENV=${RELEASE_ENV}

COPY mix.exs mix.lock ./
COPY config config

# install mix dependencies
RUN mix deps.get --only ${MIX_ENV}
RUN mix deps.compile

COPY lib ./lib
COPY priv ./priv
#########################
####### Test Stage ######
#########################

FROM base as tester
WORKDIR /app
COPY test test

########################
# Build Phoenix assets #
########################
# Using stretch for now because it includes Python
# Otherwise you get errors, could use a smaller image though
FROM node:14.15.3-stretch AS assets
WORKDIR /app

COPY --from=base /app/deps /app/deps/
COPY assets/package.json assets/package-lock.json ./assets/
RUN npm --prefix ./assets ci --progress=false --no-audit --loglevel=error

COPY lib ./lib
COPY priv ./priv
COPY assets/ ./assets/

RUN npm run --prefix ./assets deploy


#########################
# Create Phoenix digest #
#########################
FROM base AS digest
WORKDIR /app

# set build ENV
ENV MIX_ENV=${MIX_ENV}
ENV RELEASE_ENV=${RELEASE_ENV}

COPY --from=assets /app/priv ./priv
RUN mix phx.digest


#######################
#### Create release ###
#######################
FROM digest AS release
WORKDIR /app

ARG MIX_ENV
ARG RELEASE_ENV
ENV MIX_ENV=${MIX_ENV}
ENV RELEASE_ENV=${RELEASE_ENV}

COPY --from=digest /app/priv/static ./priv/static

RUN mix do compile, release



#################################################
# Create the actual image that will be deployed #
#################################################
FROM debian:buster-slim AS deploy

# Install stable dependencies that don't change often
RUN apt-get update && \
  apt-get install -y --no-install-recommends \
  apt-utils \
  openssl \
  curl \
  wget && \
  rm -rf /var/lib/apt/lists/*


# Set WORKDIR after setting user to nobody so it automatically gets the right permissions
# When the app starts it will need to be able to create a tmp directory in /app
WORKDIR /app

ARG MIX_ENV
ARG RELEASE_ENV

ENV MIX_ENV=${MIX_ENV}
ENV RELEASE_ENV=${RELEASE_ENV}

ENV LANG=C.UTF-8 LC_ALL=C.UTF-8

# Update timezone
ENV TZ=Asia/Singapore

HEALTHCHECK --start-period=10s \
  --interval=15s \
  --timeout=5s \
  --retries=3 \
  CMD curl -sSf http://localhost:8080/heartbeat || exit 1

COPY --from=release /app/_build/${MIX_ENV}/rel/ghost_rider ./

ENV HOME=/app
# Exposes port to the host machine
EXPOSE 8080

CMD ["bin/ghost_rider", "start"]

A standard docker-compose file to run the application would be similar to this which would spin up a Postgres database and start the application for you.

version: "3.7"
services:
  ghost-rider:
    build:
      context: .
      args:
        MIX_ENV: "${MIX_ENV}"
        RELEASE_ENV: "${RELEASE_ENV}"
    image: wego/ghost-rider
    depends_on:
      - ghost-rider-db
    restart: on-failure
    ports:
      - "8080:8080"
    environment:
      MIX_ENV: "${MIX_ENV}"
      RELEASE_ENV: "${RELEASE_ENV}"
      PORT: "8080"
      SECRET_KEY_BASE: "${SECRET_KEY_BASE}"
      DB_NAME: ghost-rider
      DB_PASSWORD: ghost-rider
      DB_HOST: ghost-rider-db
      APP_LEVEL: WEB

  ghost-rider-db:
    image: postgres
    restart: always
    ports:
      - 5432:5432
    environment:
      POSTGRES_PASSWORD: ghost-rider
      POSTGRES_DB: "${POSTGRES_DB}"
      PGDATA: /var/lib/postgresql/data/pgdata
    volumes:
      - "postgresql-data:/var/lib/postgresql/data"

volumes:
  postgresql-data:

To build and run your container image you would execute

docker-compose -f docker-compose.yml up -d

To run your migration

docker-compose run --rm ghost-rider bin/ghost_rider eval "Release.migrate"

To run the test you would need to create another docker-compose file to override the default values and define the target it should execute up to. If you want you could create a completely new file as well

version: "3.7"
services:
  ghost-rider:
    build:
      target: tester
      args:
        - MIX_ENV=test
        - RELEASE_ENV=test
    image: wego/ghost-rider-test
    ports:
      - "8080:8080"
    environment:
      - MIX_ENV=test
      - RELEASE_ENV=test
      - DB_NAME=ghost-rider-test
      - DB_PASSWORD=ghost-rider
      - DB_HOST=ghost-rider-db

With this you can setup your test database and run test simply by doing

docker-compose -f docker-compose.yml -f docker-compose.test.yml run --rm ghost-rider mix ecto.setup

docker-compose -f docker-compose.yml -f docker-compose.test.yml run --rm ghost-rider mix test

All it takes is some understanding of Elixir and it can be used with Docker tools as any other language to build images and perform CI/CD without any friction.

I hope this helps with containerising Elixir and Phoenix applications as they are amazing technologies along with Docker and containers that make things really simple build pipelines for CI/CD.