Elixir - supervision trees

We’ll be creating a supervision tree that will monitor the download and parse workers. This puts together the earlier posts and uses the remote file saving and the redis pool for the downloading, but we won’t go over any of the other code for downloading and parsing pages because that just isn’t that interesting. We could use poolboy again to limit the number of concurrent downloads, but that doesn’t really buy us much at this point.

Code is at https://github.com/tjheeta/elixir_web_crawler. Checkout step-4.

git clone https://github.com/tjheeta/elixir_web_crawler.git
git checkout step-4

We created a pool of redis workers earlier while setting up a redis pool with poolboy . Now we will create an additional supervisor to supervise that, hence, the tree.

First, let’s setup an anonymous function to setup our download workers. Our download workers essentially pop something off a queue, download it, and then save it on the remote node, rinse and repeat. We’ll be using the “worker” helper to setup our worker from Supervisor.Spec and the default values for this are:

worker(SomeModule, [])

# the return values of the worker function
[id: SomeModule,
  function: :start_link,
  restart: :permanent,
  shutdown: 5000,
  modules: [SomeModule]]

The start_link function should return an {:ok, pid} for the supervisor to be able to monitor it, so the functions we are calling should also. We’ll use a loop to download and then spawn_link to setup the loop and return the pid as we can’t monitor the loop directly. Here’s a quick sample:

def loop_test() do
  pid = :erlang.pid_to_list(self())|> to_string
  IO.puts "test_loop_#{pid}"
  :timer.sleep(3000)
  loop_test()
end

def spawn_test() do
  pid=spawn_link(__MODULE__, :loop_test, [])
  {:ok, pid}
end

worker(ElixirWebCrawler.Worker, [], restart: :permanent, id: "worker_test", function: :spawn_test)

And here is the real init code where we create the tree. An anonymous function sets up 5 different workers for download, the redis pool, and a parse worker.

def init([]) do
  spec_fun = fn(x) ->
    worker_name = "worker_download" <> to_string(x)
    worker(ElixirWebCrawler.Worker, [], restart: :permanent, id: worker_name, function: :spawn_download)
  end
  # setup 5 download workers
  download_workers = Enum.map(1..5, spec_fun)

  children = [
    worker(ElixirWebCrawler.RedisSupervisor, []),
    worker(ElixirWebCrawler.Worker, [], restart: :permanent, id: "worker_parse", function: :spawn_parse)
    worker(ElixirWebCrawler.Worker, [], restart: :permanent, id: "worker_requeue", function: :spawn_requeue)
  ] ++ download_workers

  supervise(children, strategy: :one_for_one)
end

Let’s try running it. First, on the redis node, let’s inject a url.

erlang@storage1:~$ redis-cli flushdb
OK
erlang@storage1:~$ redis-cli sadd download_queue https://en.wikipedia.org/wiki/Main_Page
(integer) 1
erlang@storage1:~$ ~/startup.sh 
Generated elixir_web_crawler.app
Interactive Elixir (1.0.2) - press Ctrl+C to exit (type h() ENTER for help)
iex(node@10.0.3.235)1>

And on the worker node, let’s fire up the crawler:

iex(node@10.0.3.179)1> ElixirWebCrawler.Worker.start_link
parse_loop_<0.173.0>
download_loop_<0.175.0>
download_loop_<0.176.0>
download_loop_<0.177.0>
download_loop_<0.178.0>
download_loop_<0.179.0>
{:ok, #PID<0.162.0>}
https://en.wikipedia.org/wiki/ - START process_item
count = en.wikipedia.org - 1
processing https://en.wikipedia.org/wiki/
Sleeping for 12588

So we have now got 5 concurrent workers fired up on the worker, downloading from wikipedia. If one of the worker loops crashes for some reason, it will be restarted by the supervisor. That is the point of the supervisor after all. The only problem is now that we’re firing up the application manually and that also needs to be automated.