Elixir - Setting up Poolboy and Redis
Since we’ll be using redis, we need to limit the number of connections at any one time. There’s a nice library called Poolboy for doing this. We’ll essentially be making a pool of redis connections that will be re-used. Note that pools can be registered globally across all the erlang nodes or locally. We’ll be going with a local registration as we plan to create and destroy nodes.
Code is at https://github.com/tjheeta/elixir_web_crawler. Checkout step-2.
git clone https://github.com/tjheeta/elixir_web_crawler.git
git checkout step-2
We have to do the following:
- Download and compile the dependencies
- Setup the configuration file that states the master_ip
- Create supervision for the pool in case something fails.
Dependencies - mix.exs
There are a few dependencies we need to add. Eredis is the standard redis library for erlang, poolboy for setting up the pool itself, and confort to setup simple configuration files. First, we have to add the dependency to mix.exs and run mix deps.get and mix deps.compile.
defmodule ElixirWebCrawler.Mixfile do
use Mix.Project
def project do
[app: :elixir_web_crawler,
version: "0.0.1",
elixir: "~> 1.0-dev",
conf_path: "config/main.conf",
deps: deps]
end
def application do
[applications: [:logger, :eredis, :mix, :confort]]
end
defp deps do
[
{:eredis, github: "wooga/eredis" },
{:poolboy, github: "devinus/poolboy" },
{:confort, github: "zambal/confort" }
]
end
end
Configuration file - config/main.conf
We have set this up using ansible already. It should contain the ip address of the master node.
erlang@worker1:/vagrant$ cat config/main.conf
[ master_ip: "10.0.3.235",
redis: [ key1: 42, key2: :hello ],
my_app2: [ key1: 10, key2: :world ] ]
Pool setup - an explanation
- Now comes the fun part, setting up the redis pool and placing it under a supervisor. For poolboy, we’ll setup a local pool ( :local instead of :global ) with a size of 5 and an overflow of 10.
:poolboy.child_spec takes three arguments:
- name of the pool
- arguments for the pool itself - worker_module, size, max_overflow, etc.
pool_options = [
{:name, {:local, :redis_pool}},
{:worker_module, :eredis},
{:size, 5},
{:max_overflow, 10}
]
- arguments to pass to the module - in this case for eredis we want to pass the host and the port
eredis_args = [
{:host, String.to_char_list(Confort.get(:master_ip))},
{:port, 6379}
]
Finally, we need to setup a supervisor with a strategy of one_for_one to restart any failing children.
children = [
:poolboy.child_spec(:redis_pool, pool_options, eredis_args )
]
supervise(children, strategy: :one_for_one)
We don’t want the headache of checking workers in and out, so we’ll define a query function that uses transactions. They will check out a worker, run the query, and then check it back in.
def q(args) do
{:ok, item} = :poolboy.transaction(:redis_pool, fn(worker) -> :eredis.q(worker, args, 5000) end)
end
redis.ex - putting it all together
We can put all the pieces together into a single file and then try running our new pool.
defmodule DL.RedisSupervisor do
use Supervisor
# this will create a linked process and call init
def start_link do
Supervisor.start_link(__MODULE__, [])
end
def init([]) do
pool_options = [
{:name, {:local, :redis_pool}},
{:worker_module, :eredis},
{:size, 5},
{:max_overflow, 10}
]
eredis_args = [
{:host, String.to_char_list(Confort.get(:master_ip))},
{:port, 6379}
]
children = [
:poolboy.child_spec( :redis_pool, pool_options, eredis_args )
]
supervise(children, strategy: :one_for_one)
end
# a redis query transaction function
def q(args) do
{:ok, item} = :poolboy.transaction(:redis_pool, fn(worker) -> :eredis.q(worker, args, 5000) end)
end
end
Let’s try using it.
erlang@worker1:~$ ~/startup.sh
All dependencies up to date
==> eredis (compile)
==> poolboy (compile)
Erlang/OTP 17 [erts-6.2] [source-aaaefb3] [64-bit] [smp:4:4] [async-threads:10] [hipe] [kernel-poll:false]
Compiled lib/elixir_web_crawler.ex
Compiled lib/elixir_web_crawler/redis.ex
Generated elixir_web_crawler.app
Interactive Elixir (1.0.2) - press Ctrl+C to exit (type h() ENTER for help)
iex(node@10.0.3.179)1> ElixirWebCrawler.RedisSupervisor.start_link
{:ok, #PID<0.120.0>}
Checkout a worker and run queries with it:
iex(node@10.0.3.179)2> pid = :poolboy.checkout(:redis_pool)
#PID<0.129.0>
iex(node@10.0.3.179)3> :eredis.q(pid, ["GET", "foo"], 5000)
{:ok, :undefined}
iex(node@10.0.3.179)4> :eredis.q(pid, ["SET", "foo", "bar"], 5000)
{:ok, "OK"}
iex(node@10.0.3.179)5> :eredis.q(pid, ["GET", "foo"], 5000)
{:ok, "bar"}
iex(node@10.0.3.179)6> :poolboy.checkin(:redis_pool, pid)
:ok
Or just use the transaction function:
iex(node@10.0.3.179)7> ElixirWebCrawler.RedisSupervisor.q(["GET", "foo"])
{:ok, "bar"}
Let’s use up all the workers and then try to run a transaction (there should be a timeout):
iex(node@10.0.3.179)8> pids = Enum.map(1..15, fn(i) -> :poolboy.checkout(:redis_pool) end)
[#PID<0.129.0>, #PID<0.128.0>, #PID<0.127.0>, #PID<0.126.0>,
#PID<0.123.0>, #PID<0.132.0>, #PID<0.133.0>, #PID<0.134.0>,
#PID<0.135.0>, #PID<0.136.0>, #PID<0.137.0>, #PID<0.138.0>,
#PID<0.139.0>, #PID<0.140.0>, #PID<0.141.0>]
iex(node@10.0.3.179)9> ElixirWebCrawler.RedisSupervisor.q(["GET", "foo"])
** (exit) exited in: :gen_server.call(:redis_pool, {:checkout, true}, 5000)
** (EXIT) time out
(stdlib) gen_server.erl:190: :gen_server.call/3
src/poolboy.erl:53: :poolboy.checkout/3
src/poolboy.erl:72: :poolboy.transaction/3
(elixir_web_crawler) lib/elixir_web_crawler/redis.ex:28: ElixirWebCrawler.RedisSupervisor.q/1
Let’s check a worker back in and try again:
iex(node@10.0.3.179)10> :poolboy.checkin(:redis_pool, Enum.at(pids,0))
:ok
iex(node@10.0.3.179)11> ElixirWebCrawler.RedisSupervisor.q(["GET", "foo"])
{:ok, "bar"}
Let’s verify that the supervision is working correctly by killing a pid:
iex(node@10.0.3.179)12> pid = :poolboy.checkout(:redis_pool)
#PID<0.152.0>
iex(node@10.0.3.179)13> ElixirWebCrawler.RedisSupervisor.q(["GET", "foo"])
** (exit) exited in: :gen_server.call(:redis_pool, {:checkout, true}, 5000)
** (EXIT) time out
(stdlib) gen_server.erl:190: :gen_server.call/3
src/poolboy.erl:53: :poolboy.checkout/3
src/poolboy.erl:72: :poolboy.transaction/3
(elixir_web_crawler) lib/elixir_web_crawler/redis.ex:28: ElixirWebCrawler.RedisSupervisor.q/1
iex(node@10.0.3.179)14> Process.exit(Enum.at(pids,1), "because")
true
iex(node@10.0.3.179)15> ElixirWebCrawler.RedisSupervisor.q(["GET", "foo"])
{:ok, "bar"}
Everything is working as expected and we’ve got a pool of workers to use. We could have also setup a global pool across all of our erlang instances, but that would cause us some issues because we plan to delete vm’s on the fly.
tl;dr - setting up a set of redis workers with poolboy is trivial < 30 lines of code