Subscribe to access all episodes. View plans →
Published August 10, 2017
Elixir 1.4
Source code on GitHub
Elixir is fast, but how can we distinguish between the performance of two pieces of code that have the same output?
Benchmarking is the answer and Benchee is a great framework for benchmarking in Elixir.
In this episode we’ll use Benchee to benchmark some code and see what’s faster.
Let’s get started.
We’ll open our mixfile and add ‘benchee’ as a dependency. And we’ll set the only: :dev option since we only want it in our development environment.
mix.exs
defp deps do
[{:benchee, "~> 0.9", only: :dev}]
end
Then we’ll go to the command line and download our dependencies.
$ mix deps.get
Now we need something to benchmark.
Let’s write a recursive function that takes a list of numbers, adds one to it, and then returns the updated list. We’ll compare it against Enum.map/2.
First let’s define a new function we’ll call ‘incrementer’.
It will take a list as its first parameter. We’ll match against it to get the ‘head’, which will be the first number in the list, and the ‘tail’, which will be the remainder of the list.
The second parameter will be an accumulator, which will contain our updated list.
Since this is a recursive function our ‘incremental’ function will then call itself. We’ll pass in the tail of our list and then build our updated list by adding one to the ‘head’ of our list and then adding it to the accumulator.
And since a recursive call is the last expression, our function is tail call optimized.
Now we’ll need to create another function that pattern matches against an empty list. This will happen when we’ve processed all the items in our list.
And in this function we’ll then call Enum.reverse/1 since we want the order of the list to match.
And finally let’s add a function head that will let us set a default value for the accumulator, which we’ll set as an empty list.
lib/teacher.ex
defmodule Teacher do
def incrementer(list, acc \\ [])
def incrementer([], acc), do: Enum.reverse(acc)
def incrementer([head|tail], acc) do
incrementer(tail, [head + 1 | acc])
end
end
Great with that let’s head to the command line and test it out.
We’ll create a list of 1 through 5.
Then we’ll feed that into our ‘Teacher.incrementer’ function.
And since we set a default value for our list accumulator, we only need to pass in our list that we want to be processed.
$ iex -S mix
> list = [1, 2, 3, 4, 5]
[1, 2, 3, 4, 5]
> Teacher.incrementer(list)
[2, 3, 4, 5, 6]And perfect our function is working just like we want it to.
Let’s go back to our code and define our benchmark.
We’ll create a directory named ‘benchmarks’ and in it a file named ‘increment_example.exs’
First let’s create our list of numbers. Let’s create one with numbers ‘1’ through ‘100,000’.
Now we’ll call Benchee.run/2 .
We’ll pass in a map of the functions we want to benchmark. With the key being a readable name and the value a function with the code we want to benchmark.
Let’s first include our ‘tail call optimized’ function and then we’ll define a function that calls our Teacher.incrementer/2 function with the list defined above.
The second function we want to benchmark is Enum.map/2 so let’s give it a name and then we’ll define another function that calls Enum.map/2 and again uses the list we defined above, we’ll then give it a function that simply takes the number from the list and increments it.
benchmarks/increment_example.exs
list = Enum.to_list(1..100_000)
Benchee.run(%{
"tail_call_opt" => fn -> Teacher.incrementer(list) end,
"Enum.map" => fn -> Enum.map(list, &(&1 + 1)) end
})
With that, let’s run our Benchmark. We’ll go to the command line and call:
$ mix run benchmarks/increment_example.exs
We can see that it tells us some info about our benchmark, like what version of Elixir and Erlang we’re running.
And great it ran the benchmark and we can see the Enum.map/2 function was 1.6 times slower than our ‘incrementer’ fucntion.
Benchee has a lot of great configuration options we can use to customize our benchmarks.
Let’s update our benchmark to see how they work.
Back in our benchmark we’ll add the option ‘time’, which sets how long each job should be run and measured for. Let’s set this to 10 seconds.
Next we’ll we’ll add the ‘parallel’ option, which allows each job to be executed in a parallel number of processes. By default there is no parallel execution. Let’s update ours to ‘2’.
benchmarks/increment_example.exs
list = Enum.to_list(1..100_000)
Benchee.run(%{
"tail_call_opt" => fn -> Teacher.incrementer(list) end,
"Enum.map" => fn -> Enum.map(list, &(&1 + 1)) end
}, time: 10, parallel: 2)
Then we’ll go back to the command line and re-run our benchmark.
$ mix run benchmarks/increment_example.exs
And we can see our benchmark ran with our updated options - parallel set to 2 and a time of 10 seconds.
Now that we know how to customize our benchmark, let’s remove our customizations and just use the defaults for now.
A nice bonus of using Benchee is there are a few different plugins we can use to provide nice visualizations of the benchmarks we run. One of them is ‘benchee_html’, which will take our benchmarks and display them in graphs saved in a standalone HTML files.
Let’s add benchee_html to our project.
We’ll open the ‘mixfile’ and add benchee_html as a dependency, with the only: :dev option since we only want it in our development environment.
mix.exs
defp deps do
[{:benchee, "~> 0.9", only: :dev},
{:benchee_html, "~> 0.3", only: :dev}]
end
Then we’ll download it with $ mix deps.get
Now we can got back to our benchmark script and tell it how we want to format our benchmarks.
We’ll add the formatters option and in it we’ll include the formatter we want to include.
Then we’ll also include the ‘formatter_option’ and tell it where we want our results written. Let’s save ours to our ‘benchmarks’ directory.
benchmarks/increment_example.exs
list = Enum.to_list(1..100_000)
Benchee.run(%{
"tail_call_opt" => fn -> Teacher.incrementer(list) end,
"Enum.map" => fn -> Enum.map(list, &(&1 + 1)) end
},
formatters: [
&Benchee.Formatters.HTML.output/1
],
formatter_options: [html: [file: "benchmarks/results.html"]]
)
With that let’s go back to the terminal and re-run our script.
$ mix run benchmarks/increment_example.exs
It prints that our results were generated, but the console results are missing. That’s because they’re included by default and now that we’re explicitly defining our formatters we’ll have to include it as well.
Let’s add it.
benchmarks/increment_example.exs
list = Enum.to_list(1..100_000)
Benchee.run(%{
"tail_call_opt" => fn -> Teacher.incrementer(list) end,
"Enum.map" => fn -> Enum.map(list, &(&1 + 1)) end
},
formatters: [
&Benchee.Formatters.HTML.output/1,
&Benchee.Formatters.Console.output/1
],
formatter_options: [html: [file: "benchmarks/results.html"]]
)
Back in our terminal we’ll run our script one last time.
$ mix run benchmarks/increment_example.exs
And great the results are printed in the console and our benchmark files were generated.