6 Dec 2014

Elixir - Setting up Poolboy and Redis

Since we’ll be using redis, we need to limit the number of connections at any one time. There’s a nice library called Poolboy for doing this. We’ll essentially be making a pool of redis connections that will be re-used. Note that pools can be registered globally across all the erlang nodes or locally. We’ll be going with a local registration as we plan to create and destroy nodes.

Code is at https://github.com/tjheeta/elixir_web_crawler. Checkout step-2.

git clone https://github.com/tjheeta/elixir_web_crawler.git
git checkout step-2

We have to do the following:

Download and compile the dependencies
Setup the configuration file that states the master_ip
Create supervision for the pool in case something fails.

Dependencies - mix.exs

There are a few dependencies we need to add. Eredis is the standard redis library for erlang, poolboy for setting up the pool itself, and confort to setup simple configuration files. First, we have to add the dependency to mix.exs and run mix deps.get and mix deps.compile.

defmodule ElixirWebCrawler.Mixfile do
  use Mix.Project

  def project do
    [app: :elixir_web_crawler,
     version: "0.0.1",
     elixir: "~> 1.0-dev",
     conf_path: "config/main.conf",
     deps: deps]
  end

  def application do
    [applications: [:logger, :eredis, :mix, :confort]]
  end

  defp deps do
    [
      {:eredis,  github: "wooga/eredis" },
      {:poolboy,  github: "devinus/poolboy" },
      {:confort, github: "zambal/confort" }
    ]
  end
end

Configuration file - config/main.conf

We have set this up using ansible already. It should contain the ip address of the master node.

erlang@worker1:/vagrant$ cat config/main.conf 
[ master_ip: "10.0.3.235",
  redis: [ key1: 42, key2: :hello ],
  my_app2: [ key1: 10, key2: :world ] ]

Pool setup - an explanation

Now comes the fun part, setting up the redis pool and placing it under a supervisor. For poolboy, we’ll setup a local pool ( :local instead of :global ) with a size of 5 and an overflow of 10.

:poolboy.child_spec takes three arguments:

name of the pool
arguments for the pool itself - worker_module, size, max_overflow, etc.

pool_options = [
   {:name, {:local, :redis_pool}},
   {:worker_module, :eredis},
   {:size, 5},
   {:max_overflow, 10}
]

arguments to pass to the module - in this case for eredis we want to pass the host and the port

eredis_args = [
  {:host, String.to_char_list(Confort.get(:master_ip))},
  {:port, 6379}
  ]

Finally, we need to setup a supervisor with a strategy of one_for_one to restart any failing children.

children = [
  :poolboy.child_spec(:redis_pool, pool_options, eredis_args )
]
supervise(children, strategy: :one_for_one)

We don’t want the headache of checking workers in and out, so we’ll define a query function that uses transactions. They will check out a worker, run the query, and then check it back in.

def q(args) do
  {:ok, item} = :poolboy.transaction(:redis_pool, fn(worker) -> :eredis.q(worker, args, 5000) end)
end

redis.ex - putting it all together

We can put all the pieces together into a single file and then try running our new pool.


defmodule DL.RedisSupervisor do
  use Supervisor

  # this will create a linked process and call init
  def start_link do
    Supervisor.start_link(__MODULE__, [])
  end

  def init([]) do
    pool_options = [
      {:name, {:local, :redis_pool}},
      {:worker_module, :eredis}, 
      {:size, 5},
      {:max_overflow, 10}
      ]
    eredis_args = [
      {:host, String.to_char_list(Confort.get(:master_ip))},
      {:port, 6379}
      ]
    children = [
      :poolboy.child_spec( :redis_pool, pool_options, eredis_args )
      ]
    supervise(children, strategy: :one_for_one)
  end

  # a redis query transaction function
  def q(args) do
    {:ok, item} = :poolboy.transaction(:redis_pool, fn(worker) -> :eredis.q(worker, args, 5000) end)
  end
end

Let’s try using it.

erlang@worker1:~$ ~/startup.sh                                  
All dependencies up to date
==> eredis (compile)
==> poolboy (compile)
Erlang/OTP 17 [erts-6.2] [source-aaaefb3] [64-bit] [smp:4:4] [async-threads:10] [hipe] [kernel-poll:false]

Compiled lib/elixir_web_crawler.ex
Compiled lib/elixir_web_crawler/redis.ex
Generated elixir_web_crawler.app
Interactive Elixir (1.0.2) - press Ctrl+C to exit (type h() ENTER for help)
iex(node@10.0.3.179)1> ElixirWebCrawler.RedisSupervisor.start_link
{:ok, #PID<0.120.0>}

Checkout a worker and run queries with it:

iex(node@10.0.3.179)2> pid = :poolboy.checkout(:redis_pool)
#PID<0.129.0>
iex(node@10.0.3.179)3> :eredis.q(pid, ["GET", "foo"], 5000)
{:ok, :undefined}
iex(node@10.0.3.179)4> :eredis.q(pid, ["SET", "foo", "bar"], 5000)
{:ok, "OK"}
iex(node@10.0.3.179)5> :eredis.q(pid, ["GET", "foo"], 5000)       
{:ok, "bar"}
iex(node@10.0.3.179)6> :poolboy.checkin(:redis_pool, pid)
:ok

Or just use the transaction function:

iex(node@10.0.3.179)7> ElixirWebCrawler.RedisSupervisor.q(["GET", "foo"])
{:ok, "bar"}

Let’s use up all the workers and then try to run a transaction (there should be a timeout):

iex(node@10.0.3.179)8> pids = Enum.map(1..15, fn(i) -> :poolboy.checkout(:redis_pool) end)
[#PID<0.129.0>, #PID<0.128.0>, #PID<0.127.0>, #PID<0.126.0>,
#PID<0.123.0>, #PID<0.132.0>, #PID<0.133.0>, #PID<0.134.0>,
#PID<0.135.0>, #PID<0.136.0>, #PID<0.137.0>, #PID<0.138.0>,
#PID<0.139.0>, #PID<0.140.0>, #PID<0.141.0>]
iex(node@10.0.3.179)9> ElixirWebCrawler.RedisSupervisor.q(["GET", "foo"])                 
** (exit) exited in: :gen_server.call(:redis_pool, {:checkout, true}, 5000)
    ** (EXIT) time out
                    (stdlib) gen_server.erl:190: :gen_server.call/3
                             src/poolboy.erl:53: :poolboy.checkout/3
                             src/poolboy.erl:72: :poolboy.transaction/3
        (elixir_web_crawler) lib/elixir_web_crawler/redis.ex:28: ElixirWebCrawler.RedisSupervisor.q/1

Let’s check a worker back in and try again:

iex(node@10.0.3.179)10> :poolboy.checkin(:redis_pool, Enum.at(pids,0))
:ok
iex(node@10.0.3.179)11> ElixirWebCrawler.RedisSupervisor.q(["GET", "foo"]) 
{:ok, "bar"}

Let’s verify that the supervision is working correctly by killing a pid:

iex(node@10.0.3.179)12> pid = :poolboy.checkout(:redis_pool)                                
#PID<0.152.0>
iex(node@10.0.3.179)13> ElixirWebCrawler.RedisSupervisor.q(["GET", "foo"])
** (exit) exited in: :gen_server.call(:redis_pool, {:checkout, true}, 5000)
    ** (EXIT) time out
                (stdlib) gen_server.erl:190: :gen_server.call/3
                         src/poolboy.erl:53: :poolboy.checkout/3
                         src/poolboy.erl:72: :poolboy.transaction/3
    (elixir_web_crawler) lib/elixir_web_crawler/redis.ex:28: ElixirWebCrawler.RedisSupervisor.q/1
iex(node@10.0.3.179)14> Process.exit(Enum.at(pids,1), "because")          
true
iex(node@10.0.3.179)15> ElixirWebCrawler.RedisSupervisor.q(["GET", "foo"])
{:ok, "bar"}

Everything is working as expected and we’ve got a pool of workers to use. We could have also setup a global pool across all of our erlang instances, but that would cause us some issues because we plan to delete vm’s on the fly.

tl;dr - setting up a set of redis workers with poolboy is trivial < 30 lines of code

Nothing interesting...

About

Elixir - Setting up Poolboy and Redis

Dependencies - mix.exs

Configuration file - config/main.conf

Pool setup - an explanation

redis.ex - putting it all together