Deploy a Distributed Elixir Cluster


This is a guide to deploying a distributed Elixir cluster on Render using libcluster, Phoenix and Distillery. The cluster will be set up to discover nodes automatically and keep the node list current as they join or leave the cluster.

We’ll start off with a bare Phoenix project, modify it to use Distillery and libcluster and deploy it on Render. The full source code for this example is available at https://github.com/render-examples/elixir_cluster.

Create a Phoenix App

  1. Create a new Phoenix app in the terminal. We don’t need a database in this example, so we’re passing the --no-ecto flag to mix.

    $ mix phx.new elixir_cluster_demo --no-ecto # also fetch and install dependencies
    $ cd elixir_cluster_demo
  2. Update mix.exs to add Distillery and libcluster to deps.

    defp deps do
      [ ...,
        {:distillery, "~> 2.0"},
        {:libcluster, "~> 3.0"}
      ]

    Then run mix deps.get in your terminal to update dependencies.

Configure Distillery

  1. We need to configure Distillery to serve a Phoenix app in production. Update the Endpoint config in config/prod.exs so it looks like this:

    config :elixir_cluster_demo, ElixirClusterDemoWeb.Endpoint,
      cache_static_manifest: "priv/static/cache_manifest.json",
      server: true, # critical for Phoenix to run
      root: ".",
      version: Application.spec(:elixir_cluster_demo, :vsn)

    Read more about Distillery configuration.

    Next, we’re going to make sure we don’t add secrets to our repo, because we will manage them in production with Render environment variables. Let’s delete the line at the end of config/prod.exs referring to prod.secret.exs.

    import_config "prod.secret.exs" # delete me

    Feel free to also delete the file config/prod.secret.exs.

  2. You’re now ready to initialize your Distillery release.

    $ mix release.init

    This will create rel/config.exs, rel/vm.args, and the empty directory rel/plugins.

Configure Distributed Erlang

  1. Running Erlang in distributed mode needs a name for each node in the cluster, and a magic cookie that should be the same for each node.

    Both these values can be specified in a vm.args file through Distillery. Render populates the values in this file using environment variables, and sets REPLACE_OS_VARS to true. This allows Distillery to generate the contents of this file at runtime using values set by Render.

    Create rel/prod.vm.args with the following content:

    -name ${RENDER_SERVICE_NAME}@${RENDER_INTERNAL_IP}
    -setcookie <%= release.profile.cookie %>
  2. Configure Distillery to use this file in production by updating rel/config.exs. Change the following line in the environment:prod section from this:

    environment :prod do
     ...
     set vm_args: "rel/vm.args"

    to this:

    environment :prod do
     ...
     set vm_args: "rel/prod.vm.args"

    Distillery will now use your new file in production.

Create a Mix Config Provider

We’re going to use Distillery’s Mix Config Provider for runtime configuration of the web app as well as libcluster.

  1. Create a Mix configuration file at rel/config/config.exs:

    $ mkdir -p rel/config

    Contents of rel/config/config.exs :

    use Mix.Config
    
    port = String.to_integer(System.get_env("PORT") || "4000")
    default_secret_key_base = :crypto.strong_rand_bytes(43) |> Base.encode64
    
    config :elixir_cluster_demo, ElixirClusterDemoWeb.Endpoint,
      http: [port: port],
      url: [host: "localhost", port: port],
      secret_key_base: System.get_env("SECRET_KEY_BASE") || default_secret_key_base

    This sets up the Mix configuration provider to get values from runtime environment variables.

  2. Update rel/config.exs to use your new provider. Change the environment :prod section in the file to this:

    environment :prod do
     set include_erts: true
     set include_src: false
     set cookie: :"GZUAPxTBG1]F%gaBG6.|Fxqpi^]dVX>:AFn^YxR/RY%KE1ys/l6$cd3}8r4h$B4E"
     set vm_args: "rel/prod.vm.args"
     set config_providers: [   {Mix.Releases.Config.Providers.Elixir, ["${RELEASE_ROOT_DIR}/etc/config.exs"]} ] set overlays: [   {:copy, "rel/config/config.exs", "etc/config.exs"} ]end

Configure libcluster

Our setup will create nodes with names like elixir-cluster@10.200.30.4, where the IP addresses are dynamic. Render assigns IPs to nodes when they first start, and every deploy results in a new node IP. This is where libcluster comes in: it enables automatic cluster formation through multiple configurable cluster management strategies.

Given dynamic node IPs, DNS is the best way to reliably form a cluster and keep it up to date. Consequently, we will use libcluster’s DNS strategy for cluster formation.

Let’s add libcluster to the Mix config provider. Add the lines highlighted below to rel/config/config.exs.

config :elixir_cluster_demo, ElixirClusterDemoWeb.Endpoint,
  http: [port: port],
  url: [host: "localhost", port: port],
  secret_key_base: System.get_env("SECRET_KEY_BASE") || default_secret_key_base

dns_name = System.get_env("RENDER_DISCOVERY_SERVICE")app_name = System.get_env("RENDER_SERVICE_NAME")config :libcluster, topologies: [  distillery: [    strategy: Cluster.Strategy.Kubernetes.DNS,    config: [      service: dns_name,      application_name: app_name    ]  ]]

Render automatically populates the RENDER_DISCOVERY_SERVICE and RENDER_SERVICE_NAME environment variables based on the name of your service.

Finally, add libcluster to the application supervisor by adding the lines highlighted below to application.ex:

  def start(_type, _args) do
    # List all child processes to be supervised
    topologies = Application.get_env(:libcluster, :topologies) || []
    children = [
      # start libcluster      {Cluster.Supervisor, [topologies, [name: ElixirClusterDemo.ClusterSupervisor]]},      # Start the endpoint when the application starts
      ElixirClusterDemoWeb.Endpoint
      # Starts a worker by calling: ElixirClusterDemo.Worker.start_link(arg)
      # {ElixirClusterDemo.Worker, arg},
    ]

    # See https://hexdocs.pm/elixir/Supervisor.html
    # for other strategies and supported options
    opts = [strategy: :one_for_one, name: ElixirClusterDemo.Supervisor]
    Supervisor.start_link(children, opts)
  end

Display Connected Nodes

Once everything is wired up, you can access the current node using node() and other nodes in the cluster using Node.list().

Our sample app displays all connected nodes on the homepage. Edit the index view and template in your own app as shown in this commit.

Build a Distillery Release

You’re now ready to build your first release!

$ npm run deploy --prefix assets && MIX_ENV=prod mix do phx.digest, release --env=prod

The output should look like this:

==> Assembling release..
==> Building release elixir_cluster_demo:0.1.0 using environment prod
==> Including ERTS 10.4 from /usr/local/Cellar/erlang/22.0.1/lib/erlang/erts-10.4
==> Packaging release..
Release successfully built!
To start the release you have built, you can use one of the following tasks:

    # start a shell, like 'iex -S mix'
    > _build/prod/rel/elixir_cluster_demo/bin/elixir_cluster_demo console

    # start in the foreground, like 'mix run --no-halt'
    > _build/prod/rel/elixir_cluster_demo/bin/elixir_cluster_demo foreground

    # start in the background, must be stopped with the 'stop' command
    > _build/prod/rel/elixir_cluster_demo/bin/elixir_cluster_demo start

If you started a release elsewhere, and wish to connect to it:

    # connects a local shell to the running node
    > _build/prod/rel/elixir_cluster_demo/bin/elixir_cluster_demo remote_console

    # connects directly to the running node's console
    > _build/prod/rel/elixir_cluster_demo/bin/elixir_cluster_demo attach

For a complete listing of commands and their use:

    > _build/prod/rel/elixir_cluster_demo/bin/elixir_cluster_demo help

You can now deploy your app in production!

Deploying to Render

  1. Create a build script called build.sh at the root of your repo:

    #!/usr/bin/env bash
    export MIX_ENV=prod
    
    # get app name and version from mix.exs
    export APP_NAME="$(grep 'app:' mix.exs | sed -e 's/\[//g' -e 's/ //g' -e 's/app://' -e 's/[:,]//g')"
    export APP_VSN="$(grep 'version:' mix.exs | cut -d '"' -f2)"
    
    # remove existing builds
    rm -rf "_build"
    
    # Compile app and assets
    mix deps.get --only prod
    mix compile
    cd assets && npm install && npm run deploy && cd ..
    
    # create release
    # we don't need to create a tarball because the app will be
    # served directly from the build directory
    mix do phx.digest, release --env=prod --no-tar
    
    echo "Linking release $APP_NAME:$APP_VSN to _render/"
    
    ln -sf "_build/$MIX_ENV/rel/$APP_NAME" _render

    Make the script executable before checking it into Git:

    $ chmod a+x build.sh

    Push the changes to your repo.

  2. Create a new Web Service on Render, and give Render’s GitHub app permission to access your new repo.

  3. Use the following values during creation:

    EnvironmentElixir
    Build Command./build.sh
    Start Command./_render/bin/elixir_cluster_demo foreground

    Under the Advanced section, add the following environment variables:

    KeyValue
    SECRET_KEY_BASEA sufficiently strong secret. You can generate a secret locally by running mix phx.gen.secret

That’s it! Your distributed Elixir web service will be live on your Render URL as soon as the build finishes.


You can add nodes to your cluster by increasing the number of instances for your service under Settings.

Update Server Instance Count


You should see the second node on the homepage as soon as the instance update is live.

Screenshot of Elixir Cluster Nodes


Congratulations! You’ve successfully set up distributed Elixir in production, and your cluster will automatically update as nodes are added or removed from it. 🎉