Prometheus, PostGIS and Phoenix Part 1-Alex Koutmos

Intro

In this two part series, we’ll talk about what exactly monitoring is and why you need it in your production application. We’ll also cover how monitoring differs from logging and some considerations that need to be made when selecting a monitoring solution. Finally we’ll go through setting up a monitoring stack alongside a sample Phoenix application. Without further ado, let’s dive right into things!

What is monitoring?

Monitoring is the process of collecting metrics from your applications, infrastructure, supporting services, and databases and then storing this data over time. Using these metrics, it is possible to gain insight into how exactly your system is behaving. This can be useful if you are inspecting your system in real-time, or if you want to get a historical view of your system. This is particularly useful, if you want to compare performance across releases and determine if service latency has regressed. Once performance baselines can be ascertained from your historical data, your monitoring solution can also include alerting thresholds to ensure that production errors are caught as they occur versus customers reporting them. For example, known error conditions can be caught and alerted upon immediately (such as an API returning 500s).

For those looking for a textbook definition of monitoring, I think Mike Julian’s book Practical Monitoring summarizes it best: “Monitoring is the action of observing and checking the behavior and outputs of a system and its components over time”.

Why use monitoring?

I truly believe that if you don’t have some sort of monitoring tool within your technology stack, your operation is flying blind. As previously mentioned, without monitoring in place, you are hoping that QA will find production issues or you are hoping that customers will report issues to you (in most cases they won’t and will just leave your service). Having a monitoring solution in place (whether on-prem or SaaS) will allow you to proactively address any issues that customers are facing.

Another important outcome of monitoring is that it allows you to forecast system load. Most applications will have some sort of rhythm to their usage (also known as trends and seasonality) which allows you to determine whether your current infrastructure/architecture is able to handle the load that it is receiving. This then allows you to have the discussion with you team as to whether the current architecture needs to be reworked or if more resources need to be allocated to the servers during peak hours. The important takeaway is that you are able to quantify your hypothesis versus chasing ghosts.

Enjoying the blog post so far? Follow me on Twitter for more content Follow @akoutmos

Show me the code!

In order to showcase the capabilities of a monitoring solution, we’ll go through the process of putting together a Docker Compose stack comprised of a Phoenix application, Postgres+PostGIS, Prometheus and Grafana. Our simple Phoenix application will expose a single endpoint that we can use for basic geographical search functionality. For those unfamiliar with geographical search, it is the process of searching for things within certain geographical boundaries. For example, you leverage geographical search any time you use Yelp or Google and want to know where the nearest restaurant is. PostGIS is what allows us to perform geographical searches in SQL and that’s why we’ll be leveraging it in this tutorial.

Our contrived sample application will be used to find the nearest breweries in Washington state. In our make believe application, every zip code in the state of Washington will have several breweries (who knew that Washington state was the mecca of breweries :)). You may be wondering…where does monitoring fit into all of this? That is indeed a good question! We will be using our Prometheus and Grafana setup to monitor what zip codes seem to have the highest traffic of searches, the health/performance of our app, the DB and the BEAM. The full code can be found at https://github.com/akoutmos/elixir_monitoring_prom where each commit lines up with the below steps. This tutorial assumes you have Docker and Docker Compose up and running locally.

Step 1: Create a new Phoenix project - commit

Install the Phoenix project generator (if you don’t already have it installed).

$ mix archive.install hex phx_new 1.4.9

Generate a new project. You can replace elixir_monitoring_prom with what ever your project is called and can also re-enable webpack if your app requires it.

$ mix phx.new elixir_monitoring_prom --no-webpack

In order to ensure that everything works, switch into the project directory and start the server. You’ll probably see a ton of DBConnection.ConnectionError in the terminal when you run the server and that is to be expected given we do not have a database up and running for our app to connect to.

$ cd elixir_monitoring_prom
$ mix phx.server

If all goes well you should see the default Phoenix homepage when you navigate to http://localhost:4000

Step 2: Set up basic Docker Compose stack - commit

Before writing any Elixir code or collecting any metrics, we need to set up a development Docker Compose stack that will launch all of our necessary services. For this step, we’ll be creating two containers. One container will be our Postgres database and the other container will be our Elixir application (we’ll be adding more monitoring related services in part 2). Below is what your docker-compose.yml file should look like. We will be using the mdillon/postgis:11-alpine image for Postgres as we need the PostGIS plugin to perform geospacial queries, and our Elixir container will run our API. The reason that we put our application in a container versus running it on the host is that it will make metrics collection a lot simpler later on in the tutorial.

version: '3.7'

services:
  elixir_app:
    image: elixir:1.9.1
    command: >
      /bin/sh -c 'apt-get update && apt-get install -y inotify-tools &&
      mix local.hex --force &&
      mix local.rebar --force &&
      mix deps.get &&
      mix ecto.setup &&
      mix phx.server'      
    ports:
      - '4000:4000'
    depends_on:
      - postgres
    working_dir: /app
    volumes:
      - ./config:/app/config:ro
      - ./lib:/app/lib:ro
      - ./priv:/app/priv:ro
      - ./.formatter.exs:/app/formatter.exs:ro
      - ./mix.exs:/app/mix.exs:ro
      - ./mix.lock:/app/mix.lock:ro
      - elixir-deps:/app/deps/
      - elixir-build:/app/_build/

  postgres:
    image: mdillon/postgis:11-alpine
    ports:
      - '5432:5432'
    volumes:
      - postgres-data:/var/lib/postgresql/data
    environment:
      POSTGRES_PASSWORD: postgres
      POSTGRES_USER: postgres

volumes:
  elixir-deps: {}
  elixir-build: {}
  postgres-data: {}

After your docker-compose.yml file is complete, you will also need to change your config/dev.exs configuration file so that your application can communicate to the Postgres container. Specifically, you’ll need to change the hostname for Postgres in the Repo configuration (the reason for this being that the name of the Postgres service in the Docker Compose stack is postgres):

config :elixir_monitoring_prom, ElixirMonitoringProm.Repo,
  username: "postgres",
  password: "postgres",
  database: "elixir_monitoring_prom_dev",
  hostname: "postgres",
  show_sensitive_data_on_connection_error: true,
  pool_size: 10

Now that everything is up to date on the Elixir side, feel free to start the Docker Compose stack by running:

$ docker-compose up

If you navigate to localhost:4000, you will once again see the default Phoenix homepage. Now that you have validated that the Docker Compose stack is running correctly, kill it (on Linux/OSX machines this is done with Ctrl + c).

Step 3: Create DB migrations and load data - commit

Before creating any tables and seeding the database, we first need to add a couple of dependencies to our project. In your mix.exs file add the following geo-related dependencies:

defp deps do
  [
    ...
    {:geo, "~> 3.1.0"},
    {:geo_postgis, "~> 3.1.0"}
  ]
end

With that done, run mix deps.get on the host machine to fetch the new dependencies and update the mix.lock file. Now we are going to create the necessary Ecto migration for our geospacial data which will also populate the database with the Washington state zip codes. To create the Ecto migration, run mix ecto.gen.migration zip_code_table in the terminal and input the following into the generated migration file (make sure to change the defmodule line if you named your project something different):

defmodule ElixirMonitoringProm.Repo.Migrations.ZipCodeTable do
  use Ecto.Migration

  alias ElixirMonitoringProm.ZipCodes.ZipCode

  def up do
    execute("CREATE EXTENSION IF NOT EXISTS postgis")

    create table(:zip_codes) do
      add :zip_code, :string, size: 5, null: false
      add :city, :string, null: false
      add :state, :string, size: 2, null: false
      add :timezone, :integer, null: false
      add :dst, :boolean, null: false
    end

    execute("SELECT AddGeometryColumn('zip_codes', 'point', 4326, 'POINT', 2)")
    execute("CREATE INDEX zip_code_point_index on zip_codes USING gist (point)")

    create unique_index(:zip_codes, [:zip_code])

    flush()

    "#{__DIR__}/../wa_zip_codes.csv"
    |> File.read!()
    |> String.split("\n")
    |> Enum.filter(fn line -> String.trim(line) != "" end)
    |> Enum.map(fn csv_line ->
      [zip, city, state, lat, long, tz, dst] =
        csv_line
        |> String.replace("\"", "")
        |> String.replace("\n", "")
        |> String.split(",")

      city = String.downcase(city)
      state = String.downcase(state)

      attrs = %{
        zip_code: zip,
        city: city,
        state: state,
        point: %Geo.Point{coordinates: {long, lat}, srid: 4326},
        timezone: String.to_integer(tz),
        dst: (dst == "1" && true) || false
      }

      ZipCode.changeset(%ZipCode{}, attrs)
    end)
    |> Enum.each(fn zip_code_changeset ->
      ElixirMonitoringProm.Repo.insert(zip_code_changeset, on_conflict: :nothing)
    end)
  end

  def down do
    drop(table(:zip_codes))
    execute("DROP EXTENSION IF EXISTS postgis")
  end
end

Without getting too deep into PostGIS, this migration effectively enables the PostGIS plugin in our application’s database and creates an index for the point representing the zip code. Be sure not to yet run the migration as we are referencing a module (ElixirMonitoringProm.ZipCodes.ZipCode) that does not yet exist. Let’s go ahead and create the Ecto schema to reflect the new zip_codes table. Create the following new directory lib/elixir_monitoring_prom/zip_codes/ to house all of our zip code context modules. After that directory is created, create a new file lib/elixir_monitoring_prom/zip_codes/zip_code.ex and add the following:

defmodule ElixirMonitoringProm.ZipCodes.ZipCode do
  @moduledoc """
  Schema to define zip code entries
  """

  use Ecto.Schema

  import Ecto.Changeset

  alias __MODULE__

  schema "zip_codes" do
    field :zip_code, :string
    field :city, :string
    field :state, :string
    field :point, Geo.PostGIS.Geometry
    field :timezone, :integer
    field :dst, :boolean
  end

  def changeset(%ZipCode{} = zip_code, attrs \\ %{}) do
    all_fields = [:zip_code, :city, :state, :point, :timezone, :dst]

    zip_code
    |> cast(attrs, all_fields)
    |> validate_required(all_fields)
  end
end

We’re so close to wrapping up this step! All we need to do now is configure Postgrex to handle the geospacial types that we leverage from the geo_postgis package (in this case when we try and persist %Geo.Point{...} to the DB in the migration). To configure Postgrex correctly, open up config/dev.exs and add an additional configuration entry to the database section. It should now look like this:

config :elixir_monitoring_prom, ElixirMonitoringProm.Repo,
  username: "postgres",
  password: "postgres",
  database: "elixir_monitoring_prom_dev",
  hostname: "postgres",
  show_sensitive_data_on_connection_error: true,
  pool_size: 10,
  types: ElixirMonitoringProm.PostgresTypes

Next, we need to create the module that we are referencing in the previous code snippet. Create the file lib/elixir_monitoring_prom/postgres_types.ex with the following contents:

Postgrex.Types.define(
  ElixirMonitoringProm.PostgresTypes,
  [Geo.PostGIS.Extension] ++ Ecto.Adapters.Postgres.extensions(),
  json: Jason
)

With all that in place, all that is left is to grab a copy of the zip codes CSV and place it here priv/repo/wa_zip_codes.csv (you can find the CSV HERE). Now we can now run docker-compose up to bring up the development stack. You should see something along these lines:

$ docker-compose up
...
elixir_app_1  | 02:20:27.240 [debug] QUERY OK db=0.1ms
elixir_app_1  | INSERT INTO "zip_codes" ("city","dst","point","state","timezone","zip_code") VALUES ($1,$2,$3,$4,$5,$6) ON CONFLICT DO NOTHING RETURNING "id" ["asotin", true, %Geo.Point{coordinates: {"-117.12916", "46.230508"}, properties: %{}, srid: 4326}, "wa", -8, "99402"]
elixir_app_1  |
elixir_app_1  | 02:20:27.241 [debug] QUERY OK db=0.1ms
elixir_app_1  | INSERT INTO "zip_codes" ("city","dst","point","state","timezone","zip_code") VALUES ($1,$2,$3,$4,$5,$6) ON CONFLICT DO NOTHING RETURNING "id" ["clarkston", true, %Geo.Point{coordinates: {"-117.08313", "46.400365"}, properties: %{}, srid: 4326}, "wa", -8, "99403"]
elixir_app_1  |
elixir_app_1  | 02:20:27.242 [debug] QUERY OK db=0.1ms
elixir_app_1  | INSERT INTO "zip_codes" ("city","dst","point","state","timezone","zip_code") VALUES ($1,$2,$3,$4,$5,$6) ON CONFLICT DO NOTHING RETURNING "id" ["kennewick", true, %Geo.Point{coordinates: {"-119.160173", "46.216706"}, properties: %{}, srid: 4326}, "wa", -8, "99536"]
elixir_app_1  |
elixir_app_1  | 02:20:27.242 [info]  == Migrated 20190821020059 in 1.3s
elixir_app_1  | [info] Running ElixirMonitoringPromWeb.Endpoint with cowboy 2.6.3 at 0.0.0.0:4000 (http)
elixir_app_1  | [info] Access ElixirMonitoringPromWeb.Endpoint at http://localhost:4000

If you see all the insert statements for the zip codes, that means you are on the right track and everything is working!

Step 4: Create API endpoints - commit

With the foundation established for our application, it is time to start implementing some of the important business logic. We’ll start by creating the entry point into our zip codes context. This module will provide some basic geographical search functionality which we will then leverage in our breweries context. We’ll need to create a file at lib/elixir_monitoring_prom/zip_codes/zip_codes.ex with the following contents:

defmodule ElixirMonitoringProm.ZipCodes do
  alias ElixirMonitoringProm.{Repo, ZipCodes.ZipCode}

  def get_zip_codes_in_radius(zip_code, radius_in_miles) do
    # Our raw Postgres query to get all the zip codes within a radius
    query =
      [
        "WITH target AS (SELECT point AS p FROM zip_codes WHERE zip_code = $1::varchar)",
        "SELECT id, zip_code, city, state, timezone, dst, point FROM zip_codes JOIN target ON true",
        "WHERE ST_DWithin(p::geography, zip_codes.point::geography, $2::double precision)"
      ]
      |> Enum.join(" ")

    # The arguments we are passing to the query
    args = [zip_code, miles_to_meters(radius_in_miles)]

    # Since we used a raw SQL query, we'll need to manually (for more information
    # see https://hexdocs.pm/ecto_sql/Ecto.Adapters.SQL.html#query/4)
    case Repo.query(query, args, log: true) do
      {:ok, %Postgrex.Result{columns: cols, rows: rows}} ->
        results =
          Enum.map(rows, fn row ->
            Repo.load(ZipCode, {cols, row})
          end)

        {:ok, results}

      _ ->
        {:error, :not_found}
    end
  end

  defp miles_to_meters(miles), do: miles * 1609.344
end

The meat of the module is the get_zip_codes_in_radius/2 function which makes a raw SQL query using the PostGIS provided function ST_DWithin and fetches all the zip codes whose points fall within a given radius + zip code. With that in place, we can run the following to verify that our search is indeed working (the following commands start the docker compose stack in the background and then spawns an IEx REPL from within the running container). You may have to give the docker-compose up -d command a few moments while everything is brought up:

$ docker-compose up -d
$ docker exec -it $(docker ps --filter name=elixir_app -q) iex -S mix
Erlang/OTP 22 [erts-10.4.4] [source] [64-bit] [smp:8:8] [ds:8:8:10] [async-threads:1] [hipe]

Interactive Elixir (1.9.1) - press Ctrl+C to exit (type h() ENTER for help)
iex(1)> ElixirMonitoringProm.ZipCodes.get_zip_codes_in_radius("98052", 4)
[debug] QUERY OK db=11.2ms decode=0.9ms queue=1.3ms
WITH target AS (SELECT point AS p FROM zip_codes WHERE zip_code = $1::varchar) SELECT id, zip_code, city, state, timezone, dst, point FROM zip_codes JOIN target ON true WHERE ST_DWithin(p::geography, zip_codes.point::geography, $2::double precision) ["98052", 6437.376]
{:ok,
 [
   %ElixirMonitoringProm.ZipCodes.ZipCode{
     __meta__: #Ecto.Schema.Metadata<:loaded, "zip_codes">,
     city: "kirkland",
     dst: true,
     id: 30,
     point: %Geo.Point{
       coordinates: {-122.1934, 47.67903},
       properties: %{},
       srid: 4326
     },
     state: "wa",
     timezone: -8,
     zip_code: "98033"
   },
   %ElixirMonitoringProm.ZipCodes.ZipCode{
     __meta__: #Ecto.Schema.Metadata<:loaded, "zip_codes">,
     city: "redmond",
     dst: true,
     id: 46,
     point: %Geo.Point{
       coordinates: {-122.12138, 47.677471},
       properties: %{},
       srid: 4326
     },
     state: "wa",
     timezone: -8,
     zip_code: "98052"
   }
 ]}
iex(2)>

If you see the output above after running ElixirMonitoringProm.ZipCodes.get_zip_codes_in_radius("98052", 4) then everything is working! Before moving on, run docker-compose up to attach to the background running containers and then kill the containers with Ctrl + c. We’ll want to restart the stack once we have our migration in place. Now to move on to the breweries context. For our breweries context, we’ll start with the module which will provide the schema definition for the Postgres table. Create a file at lib/elixir_monitoring_prom/breweries/brewery.ex with the following contents:

defmodule ElixirMonitoringProm.Breweries.Brewery do
  use Ecto.Schema

  import Ecto.Changeset

  alias __MODULE__

  @derive {Jason.Encoder, only: ~w(brand beers zip_code)a}
  schema "breweries" do
    field :brand, :string
    field :beers, {:array, :string}
    field :zip_code, :string
  end

  def changeset(%Brewery{} = brewery, attrs \\ %{}) do
    all_fields = [:brand, :beers, :zip_code]

    brewery
    |> cast(attrs, all_fields)
    |> validate_required(all_fields)
  end
end

Now to add the breweries context entry point which will allow us to query for nearby breweries. Create a file at lib/elixir_monitoring_prom/breweries/breweries.ex with the following contents:

defmodule ElixirMonitoringProm.Breweries do
  alias ElixirMonitoringProm.{Breweries.Brewery, Repo, ZipCodes}
  import Ecto.Query

  def get_breweries_in_radius(zip_code_to_search, radius_in_miles) do
    zip_codes_in_radius =
      zip_code_to_search
      |> ZipCodes.get_zip_codes_in_radius(radius_in_miles)
      |> case do
        {:ok, zip_codes} -> Enum.map(zip_codes, & &1.zip_code)
        error -> error
      end

    query =
      from brewery in Brewery,
        where: brewery.zip_code in ^zip_codes_in_radius

    Repo.all(query)
  end
end

Finally, in order for our brewery context to work, we need to add an Ecto migration to create the breweries table. We’ll also take leverage the migration to seed the DB with some breweries so that we can query. Before generating the migration, open up mix.exs and add one additional dependency: {:faker, "~> 0.12.0"}. With that done, run mix deps.get && mix ecto.gen.migration brewery_table and populate the file with the following contents:

defmodule ElixirMonitoringProm.Repo.Migrations.BreweryTable do
  use Ecto.Migration

  alias ElixirMonitoringProm.{Breweries.Brewery, Repo, ZipCodes.ZipCode}
  alias Faker.{Beer, Company}

  import Ecto.Query

  def up do
    create table(:breweries) do
      add :brand, :string
      add :beers, {:array, :string}
      add :zip_code, :string
    end

    create index(:breweries, [:zip_code])

    # Flush the database changes so that we can populate the tables with dummy data
    flush()

    Faker.start()

    # Go through all of the zip codes in WA state and create between 1-3 brewers
    ZipCode
    |> Repo.all()
    |> Enum.each(fn %ZipCode{zip_code: zip_code} ->
      num_breweries = Enum.random(1..3)
      generate_breweries(zip_code, num_breweries)
    end)
  end

  defp generate_breweries(_zip_code, 0), do: :ok

  defp generate_breweries(zip_code, count) do
    attrs = %{
      brand: Company.name() <> " Brewers",
      beers: [Beer.name(), Beer.name()],
      zip_code: zip_code
    }

    %Brewery{}
    |> Brewery.changeset(attrs)
    |> Repo.insert()

    generate_breweries(zip_code, count - 1)
  end

  def down do
    drop table(:breweries)
  end
end

With the database migration complete and your contexts all setup, all that is left is to hook up Phoenix with the new functionality. We’ll need to add the route first in lib/elixir_monitoring_prom_web/router.ex. Add the following router scope block to the file:

scope "/api", ElixirMonitoringPromWeb do
  pipe_through :api

  get "/breweries", BreweryController, :index
end

Once that is done, the final step will be to create the controller. Under normal circumstances, you should do a far more involved job of validating the parameters coming into your API…but for the purposes of quickly learning we’ll skip that. Create the controller file at lib/elixir_monitoring_prom_web/controllers/brewery_controller.ex with the following code:

defmodule ElixirMonitoringPromWeb.BreweryController do
  use ElixirMonitoringPromWeb, :controller

  alias ElixirMonitoringProm.Breweries

  def index(conn, %{"zip_code" => zip_code, "mile_radius" => radius}) do
    results = Breweries.get_breweries_in_radius(zip_code, String.to_integer(radius))

    json(conn, results)
  end

  def index(conn, _) do
    conn
    |> json(%{
      error: "\"zip_code\" and \"mile_radius\" are both required fields"
    })
  end
end

With all that in place you can bring up your Docker Compose stack again with docker-compose up and if all goes well, you should be able to make a GET request to your API and retrieve the list of breweries that are within the provided zip code + radius. You can either use you browser or curl to fetch data. Your output may differ from what I have below given that the breweries are randomly generated (for those curious I have the json command aliased to jsome -s 2, so if you have something else that pretty prints JSON in the terminal feel free to use that):

$ curl -s -X GET "http://localhost:4000/api/breweries?zip_code=98052&mile_radius=4" | json
[
  {
    brand: "Leffler, Boyle and Orn Brewers",
    beers: ["Sierra Nevada Bigfoot Barleywine Style Ale", "Westmalle Trappist Tripel"],
    zip_code: "98033"
  },
  {
    brand: "Predovic LLC Brewers",
    beers: ["Arrogant Bastard Ale", "Trappistes Rochefort 8"],
    zip_code: "98033"
  },
  {
    brand: "Klein LLC Brewers",
    beers: ["Alpha King Pale Ale", "Dreadnaught IPA"],
    zip_code: "98033"
  },
  {
    brand: "Cruickshank Group Brewers",
    beers: ["Nugget Nectar", "Racer 5 India Pale Ale, Bear Republic Bre"],
    zip_code: "98052"
  }
]

Closing thoughts

Well done and thanks for sticking with me to the end! We covered quite a lot of ground and hopefully you picked up a couple of cool tips and tricks along the way. In Part 1 of this tutorial we covered a bit of the theory surrounding application monitoring and why it is important to have monitoring as part of your production system. While we didn’t touch anything necessarily monitoring related this time around, we set up a Docker Compose stack to run our Phoenix application and Postgres+PostGIS. In Part 2 we’ll be deep diving into Prometheus and Grafana and adding them to our application stack.

Feel free to leave comments or feedback or even what you would like to see in the next tutorial. Till next time!

Enjoying the blog post so far? Follow me on Twitter for more content Follow @akoutmos

Additional Resources

Below are some additional resources if you would like to deep dive into any of the topics covered in the post.