Benchmarking Elixir and Erlang database drivers
I’ve never used an ORM before, mainly because I’ve had some previous experience watching people use Hibernate and it scared the bejeebus out of me. It seems easier to just write the SQL and there are some performance advantages to do so. Was planning to try Ecto, so at this point, it’s a good time to do some very simple benchmarks for select. So we’ll be benchmarking ecto, postgrex, epgsql, mariaex, and emysql, and mysql-otp.
** Updated benchmarks at this link **
We’ll be using a connection pool of 10 with an overflow of 10 and selecting a single row out of a table. All results are being compiled between two lxc guests on a laptop.
Results
- eredis (1.0.7) - 12000 qps
- mysql-otp - 10500 qps
- epgsql (git:epgsql/epgsql) - 8350 qps
- emysql (git:Eonblast/Emysql) - 7200 qps
- postgrex (0.8.1) - 5800 qps
- ecto (0.12.0-rc) with postgrex (0.8.1) - 5200 qps
- ecto (0.12.0-rc) with mariaex (0.1.6) - 500 qps
Ecto seems to be giving about 85%-90% of the performance of the raw Postgrex. However, the erlang epgsql driver is a bit more mature and about 30% faster. That’s a lot. Interestingly new mysql-otp driver which is still in beta is leading the way over the older emysql.
Regarding Ecto, I wouldn’t mind using it as it seems to come with a lot of free functionality. Migrations, allows setting up tests easily, cleans up queries against injection, at 90% of the speed of the database driver. The problem is that the Elixir driver seems to be less mature than the Erlang version for now. Competition is good, and the more options the better.
The raw results are below and the source code for this is on github.
./wrk -c 100 -d 10 -t 20 http://localhost:8800/redis
Running 10s test @ http://localhost:8800/redis
20 threads and 100 connections
Thread Stats Avg Stdev Max +/- Stdev
Latency 8.29ms 1.70ms 32.23ms 88.41%
Req/Sec 607.19 54.48 750.00 73.20%
121048 requests in 10.02s, 25.74MB read
Requests/sec: 12078.89
Transfer/sec: 2.57MB
./wrk -c 100 -d 10 -t 20 http://localhost:8800/mysqlotp
Running 10s test @ http://localhost:8800/mysqlotp
20 threads and 100 connections
Thread Stats Avg Stdev Max +/- Stdev
Latency 9.71ms 3.62ms 82.28ms 95.15%
Req/Sec 527.80 61.73 1.01k 81.10%
105228 requests in 10.04s, 22.78MB read
Requests/sec: 10483.41
Transfer/sec: 2.27MB
./wrk -c 100 -d 10 -t 20 http://localhost:8800/epgsql
Running 10s test @ http://localhost:8800/epgsql
20 threads and 100 connections
Thread Stats Avg Stdev Max +/- Stdev
Latency 12.30ms 5.73ms 131.06ms 97.26%
Req/Sec 420.66 49.37 505.00 86.87%
83852 requests in 10.05s, 17.91MB read
Requests/sec: 8347.20
Transfer/sec: 1.78MB
./wrk -c 100 -d 10 -t 20 http://localhost:8800/emysql
Running 10s test @ http://localhost:8800/emysql
20 threads and 100 connections
Thread Stats Avg Stdev Max +/- Stdev
Latency 14.17ms 5.65ms 113.59ms 96.04%
Req/Sec 362.98 46.54 636.00 86.58%
72702 requests in 10.10s, 15.46MB read
Requests/sec: 7198.07
Transfer/sec: 1.53MB
./wrk -c 100 -d 10 -t 20 http://localhost:8800/postgrex
Running 10s test @ http://localhost:8800/postgrex
20 threads and 100 connections
Thread Stats Avg Stdev Max +/- Stdev
Latency 21.07ms 29.50ms 377.27ms 97.36%
Req/Sec 297.64 40.85 370.00 77.03%
58262 requests in 10.04s, 12.61MB read
Requests/sec: 5803.25
Transfer/sec: 1.26MB
./wrk -c 100 -d 10 -t 20 http://localhost:8800/ecto
Running 10s test @ http://localhost:8800/ecto
20 threads and 100 connections
Thread Stats Avg Stdev Max +/- Stdev
Latency 21.20ms 20.01ms 372.78ms 97.93%
Req/Sec 264.36 38.43 323.00 86.00%
52236 requests in 10.05s, 11.06MB read
Requests/sec: 5200.08
Transfer/sec: 1.10MB