Our Test automation journey

Tue, 26 Feb 2019 13:00:00 +0000

We have been doing test automation for a year and a half, and we are really proud of where we are today. This is the story of how we got here and where we are right now.

Keep in mind that this is a description of our journey and it should not be used as a template of how to do it. We hope that our story can help or inspire others on their journey; if it does, then our purpose for writing this will have been fulfilled.

Selecting the framework

Choosing a framework was a big challenge. You always need to keep many things in mind: what kind of tests will be written, what language you want to use, will developers be involved, who will write tests, who will support them, how you want to get the results, how much you want to cover and so on.

Cross platform

Since our application is cross-platform (iOS and Android) the most obvious choice was to choose a cross-platform framework. The biggest advantage is that there will be a common part and that you, as a Test automation (TA) engineer, do not need to learn several languages and support several systems. Also, there will only be one reporting system—which is the most important part of testing. Of course, it is not that simple as you are working with two completely different applications despite having similar designs, and even if you have one framework you will, in most cases, need to write different tests. So what is the point? The most complicated part is not writing tests but working with data and generating the right reports. In the case of a cross-platform framework, the latter will remain the same.

Make sure there is an active community working with the framework you choose!

With that in mind, the task was not that hard anymore. There are not that many frameworks that support both platforms. Frankly, if you want to choose something that will last for several years and will be easy to work with then you need something that is widely used and documented well. The last part is important. When you use the framework there will be problems you will not be able to solve yourself. You will have to google, A LOT, and ask the developers of the framework, and google again. This is unavoidable. We had to be sure the framework community regularly answers questions and that there are a lot of answers on websites such as StackOverflow, or Google groups. Also, the language… it will be much easier if you know it.

Test the framework

So the search began. The final candidates were Appium and Calabash. For backup, we had Espresso for Android tests and XCUITest for iOS. Appium looked great, it was quite easy to install everything and write the first test. Also, they support a couple of languages to choose from. Java suited really well, however after a successful first test on Android there was a huge disappointment: iOS was slow. Not the kind of slow you can deal with, but REALLY slow. Whatever solution we thought there was—nothing worked, days were wasted and Appium was still too slow. Time to move on.

#justworks

The second on the list was Calabash. Calabash only supports Ruby, which is fine, because part of our legacy backend is written in Ruby. This means that developers will be able to help out and educate. Calabash did not have a fancy UI compared to Appium and it seemed to be less used. Nonetheless, it was considered one of the best options out there. Surprisingly after an hour reading docs and preparing the environment everything just worked. It worked evenly on both platforms. After years of fighting with different kinds of software, I understood the golden rule: if it works without much effort—stick with it. That was it.

Architecture

Whenever you read about test automation, you always run into warnings that it is not worth the effort, because it always becomes so hard to maintain. So, how did we tackle this?

Given When Then

One great thing you can do is to write scenarios using Cucumber. Using this pattern when writing tests really helps in making the tests easy to read, not just for those of us working with test automation but for everyone within the company. It gives you a great error report format while at the same time describing the behaviour of the system, and it actually gives you a living documentation of the system.

It is also very easy for the quality assurance engineers to see what is covered by test automation and what they should verify manually. We have even used these scenarios to teach new engineers the behaviour of the app. When we rebuilt parts of the app in a project, we often got stuck in discussions on how a particular feature was supposed to work but then we could simply look at the test cases to sort out the behaviour.

Page object pattern

Page object patterns are widely used in test engineering, and they give you a nice structure of your code, making it reusable and maintainable. Using a pattern turned out to be a great idea. Now, it was easier to navigate between tests, to see common parts and differences between Android and iOS. We have implemented it in a way that we have a page object class for each view (or sometimes part of a view), and all of the IDs and methods for interacting with the different controls of the view are gathered in one place. Some useful links: Martin Fowler - Page Object, Page Object Model (POM) | Design Pattern

Even more maintainable

We also tried to keep our structure in modules. The reason for that is simple: when you decide about test architecture you should think about what happens if the framework you have chosen dies? Should you just throw away years of work and start all over again? That is not the best way of doing it. All of our tests have separate layers which, in case of emergency, can be moved to a different framework or easily rewritten.

This recently happened to us with a module that reads all of the data for ads since we are switching to a different system. Now we have to rewrite only that part and nothing else.

We also have a helper class with common methods, our constants are separated in another class (which helps a lot to store it in only one place if you want to change timeouts or something similar), several data classes for working getting data, filtering it, and a couple more divided according to their purpose.

Continuous Integration (CI)

After the framework was set in place and we covered some parts of our app with tests, we needed to make sure everything worked as expected, so it was now time to set up the way we run everything on a regular basis. So we needed CI.

Start with hosting the CI locally

Throughout the company there was no common way to run builds and unit tests for clients (except for backend, they were doing great), so we decided that we needed something for that as well, not only for end-to-end (E2E) tests. The idea was to find something that can be run locally, at least from the start. We had to use emulators or real devices for E2E tests and it is easier to maintain yourself, you get less flaky tests, and it is more stable. Should that work well and we need to grow bigger—there will always be an option to move everything to the cloud. We had our Mac mini as a server where we could start testing and trying different tools.

Team City vs. Jenkins

After googling and trying different ones the best two candidates were Team City and Jenkins. With Team City it did not really work as we had hoped, we were struggling to set everything up. Nothing worked, so we decided to try to do the same thing with Jenkins, and if that did not work—come back to Team City. However, we never needed to. Jenkins just worked. It has thousands of plugins for anything you need, which is nice. The downside is that it is quite complex to deal with plug-ins because they are written by different people and there is not always an obvious way of using them. I would say it is quite hard and time-consuming to set everything up, but after it was done—you barely need to do anything (only in cases when they change something in the plug-ins).

This is how it looks now:

Choose what to automate with E2E tests

Another big question within test automation is “What should we automate?”. The following approach we took to answering that question.

Based on analytics—our most used views

We started by gathering data about the usage of the different views and the more views a page had, the higher priority it got. We already had analytics in place, so it was quite easy to get this data. With some help from our colleagues, we also learned some BigQuery, which is very useful to know as a test engineer. Backing up an argument with data, as to why we should spend time fixing a bug, will definitely increase your credibility.

Learn along the way

We also made sure to cover critical bugs that we found either during release testing or that got out to our users. We held a post-mortem for critical bugs where we discussed how we can prevent this from happening again. Not all bugs were suited for getting covered by E2E tests, it could lead to a unit test, or in some cases even a manual test.

Trust the tests

Sometimes it is just not worth the effort of creating an automated test. A test that is very hard to build, or if you are building it with the knowledge that it might get flaky, it is probably better to not build it. Although, if you still think it is worth the effort, make sure to run it in a separate build where you can check it manually. Having a test suite that is constantly red will only make you ignore it, and you will miss important bugs.

Test data is tricky

Test data is always an issue, and we tried to avoid building tests that were dependant on data, but eventually, we couldn’t avoid it anymore. We are working on a solution where we can start the client with a predefined state, but until that is in place, we are using our API to get IDs to entities that match our needs. This actually turned out to be a really good solution but we’ve decided to move on with the more complex solution since that will give even more stability to the tests, and fulfil some needs that we don’t cover today.

Reporting

Test runs without good reporting do not make any sense. This part is really worth putting some extra effort into. If you have readable reports, people will get engaged. It also makes your daily work with the tests so much easier, if you can get a good overview of their state.

Good default reports

For E2E test reports we used a standard Cucumber plugin which generates a nice web page integrated in Jenkins.

Customized test result reports

After a while we changed the concept of how we treat our results. The idea was to write all scenarios we want to cover on all features of the app, but they will be in a pending state. That is how we are measuring the coverage of system. The issue was that standard report considered pending tests as a failure and we did not want that. This forced us to write our own report. Actually, that is what I really like about Jenkins—it is amazingly customizable. After our report generator was ready it was really easy to integrate it with Jenkins.

So now it looks like this (much more modern and stylish, we like it much better):

We also use JUnit plugin for unit tests JUnit plugin and Slack notification plugin for Slack reports.

Slack

We have our custom Slack reports for E2E tests as well. A simple Ruby app that generates exactly what we need with the right data and logo (luckily Slack has very nice open API that everyone can use). It was very easy to call it from a Jenkins’s job as a post-build task.

We have reports from most of our builds reported to different Slack channels. There is one test-automation channel where most E2E test reports are displayed, some builds send reports to iOS, Android, or even squad channels. There is a link in each report where you can find a more detailed test report to see exactly what step is failing. You even see screenshots of which state the client was in when the test failed.

These reports are used by people in all different roles. It can be an iOS developer checking what failed in the latest nightly build or a QA engineer checking the state of the client before releasing.

Feature coverage report

This function is very useful when investigating if a failing test failed for the first time in a longer period, or if there is a pattern to the failures. For example, we found that one of our tests was failing every Sunday, due to a bug in the test code, only happening on every Sunday. This would have been very hard to investigate without this report. Given you get the date it fails, it is really easy to go back to Jenkins and see if it fails on the same thing. If you have flaky tests (which you, of course, should not have, but let’s face it, sometimes we just do), this is very valuable to make sure you are fixing the flakiness.

We also have the feature coverage percentage displaying here, letting us know how much of the features we intend to cover with E2E tests, that are actually covered.

Nanoleaf

Finally, on our most high-level report, we have a triangle made of Nanoleaf panels, where each panel represents a selected build. This is the result after a hack week we had in one of the squads.

Here, the bottom row represents client builds and unit test runs and the middle layer shows if E2E tests on iOS and Android have passed or failed.

It also keeps track of other essential things, like if it is Friday and Beer O’clock :)

It is all about visualization

We have put them in a place where almost everyone passes by, so when tests are red people ask us why they are failing. It has been a good and fun way to visualize test results.

Summary

Our path was long and complicated but also fun and exciting! And it is way far from its end. We are hoping that our journey will bring some value and inspiration. Good luck and remember: more challenges you face—the stronger you get.

How I sped up my XML parser by 15 times

Thu, 19 Apr 2018 13:00:00 +0000

In the previous blog post we have showcased how possible optimizations can be made to binary parsing implementation in Elixir. This article aims to delineate how I have applied these techniques, in order to drastically speed up XML parsing.

XML is one of the most widely used formats for data interchange on the web today. At Forza Football, in addition to JSON and MessagePack, we use XML in several facets of our work: receiving football data from providers, obtaining headlines from RSS feeds. Ensuring that we have an efficent method to parse the data we receive will boost the performance of the application as a whole and thus improve user experience. With speed and usability as my two ultimate goals, I started writing the first implementation of the Saxy project.

Not so “saxy”

After two months of work and two aborted proofs of concept, The first version was published to the world. Since speed was, from the beginning, the desired goal of this project, I decided to do some benchmarks on Saxy against other XML parsing libraries.

Benchmarking with XML is hard because the result highly depends on the complexity of the document, so a relatively simple XML file was picked. The chosen contestants were xmerl—the standard XML parser in Erlang OTP, Erlsom, and fast_xml. But in the end I could not get fast_xml to work as intended, so only xmerl and Erlsom went into the benchmark script.

	Microseconds per run
Saxy 0.3.0	24.4232
xmerl 1.3.16	299.242
Erlsom 1.4.1	65.8912

“Eureka! Eureka! Saxy is 2.7 times faster than erlsom and 12 times faster than xmerl 🎉🎉🎉” I excitedly screamed out.

But after taking some time to reflect and cool down, it turned out that I had a fatal mistake right in the benchmark script. When the script was fixed, reality slapped me really hard in the face.

	Microseconds per run
Saxy 0.3.0	943.401
xmerl 1.3.16	285.5068
Erlsom 1.4.1	66.8736

“Focus on speed … 😏, good that now you know how ignorant you are!”, I told my dispointed self.

Shortcomings

I decided to give it another try, but with a different strategy this time. Instead of throwing away yet another proof of concept and writing the new implementation, I spent a few days pondering why the previous parser was so unacceptably slow. What I did was: 1) Study how Erlsom does it (thus learning from the winner) 2) I read the Erlang Efficiency Guide 3) And used our previous blog post as the guideline.

It became clear that there were some shortcomings in my original approach:

Sub-binary creation

Let’s look at a common rule matching function in Saxy.

def match(buffer, position, :element, state) do
    with {:ok, {:<, _token_value}, {new_buffer, new_pos}, new_state} <-
           match(buffer, position, :<, state),
         {:ok, {:Name, tag_name}, {new_buffer, new_pos}, new_state} <-
           match(new_buffer, new_pos, :Name, new_state),
         {:ok, {:SAttribute, attributes}, {new_buffer, new_pos}, new_state} <-
           zero_or_more(new_buffer, new_pos, :SAttribute, new_state, []),
         {:ok, {:S, _s_char}, {new_buffer, new_pos}, new_state} <-
           zero_or_one(new_buffer, new_pos, :S, new_state) do
      {:ok, {:element, tag_name}, {new_buffer, new_pos}, new_state}
    end
end

A quick summary of what the code does: it tries matching < token, then :Name rule, :SAttribute, and finally :S. For each token/rule that is matched, it returns a four-element tuple representing the matched token/rule, the buffer and its current position, as well as the parsing state.

But why does it make the whole parsing flow slow? The answer is because it completely avoids the compiler from keeping the initial match context, because every match function returns its sub-binary. This topic has been extensively covered in the binary parsing optimization post.

Pattern Matching Mania

In Erlang, pattern-matching in functions, as well as in case and receive are usually optimized.

defmodule Foo do
  def bar([item]), do: item
  def bar({item}), do: item
  def bar([item | rest]), do: item
end

The generated Erlang VM instructions below will show us how the compiler rearranges the clauses and produces a more optimized code. See here for more information of how to generate these assembler codes.

{:function, :bar, 1, 10,
 [
   {:label, 9},
   {:line, [{:location, 'lib/foo.ex', 2}]},
   {:func_info, {:atom, Foo}, {:atom, :bar}, 1},
   {:label, 10},
   {:test, :is_nonempty_list, {:f, 12}, [x: 0]}, % <—————————————
   {:get_list, {:x, 0}, {:x, 1}, {:x, 2}},
   {:test, :is_nil, {:f, 11}, [x: 2]}, % <—————————————
   {:move, {:x, 1}, {:x, 0}},
   :return,
   {:label, 11},
   {:move, {:x, 1}, {:x, 0}},
   :return,
   {:label, 12},
   {:test, :is_tuple, {:f, 9}, [x: 0]}, % <—————————————
   {:test, :test_arity, {:f, 9}, [{:x, 0}, 1]},
   {:get_tuple_element, {:x, 0}, 0, {:x, 0}},
   :return
 ]}

But unfortunately, binary matching is one of the exceptions that the Erlang compiler does not rearrange clauses yet.

defp match_token(<<"-->", _rest::bits>>, :"-->") do
  {:ok, {"-->", 3}}
end

defp match_token(<<_::bits>>, :"-->") do
  :error
end

defp match_token(<<"-->", _rest::bits>>, :CommentChar) do
  :error
end

defp match_token(<<char::utf8, _rest::bits>>, :CommentChar) do
  {:ok, {<<char::utf8>>, byte_size(<<char::utf8>>)}}
end

# 50 more token matching.

defp match_token(<<charcode::utf8, _rest::bits>>, :HexChar) do
  if hex_char?(charcode) do
    char = <<charcode::utf8>>
    {:ok, {char, byte_size(char)}}
  else
    :error
  end
end

Pattern matching is absolutely one of the coolest features in Elixir/Erlang and makes writing conditional statements easier than ever, especially for someone who has background from other languages. However, like everything else in the Universe, using it excessively would eventually come back to bite us.

In the code above, every token type is passed as the second argument in the match_token’s clause. In order to match a :HexChar token, the Erlang VM has to try matching every clause of the function (because the first argument is matching binary) in a top-to-toe manner before it can reach the function genuinely doing the matching work. Working in this manner means costing a great deal of overheads and gaining nothing in return.

Binary construction

For better usability, Saxy was designed to emit SAX event data in binary, instead of a characters list, as in other libraries. This makes all the operations afterwards easier for the library users.

I am going to demonstrate how Saxy previously operated:

defp zero_or_more(buffer, position, rule, state, acc) do
  case match(buffer, position, rule, state) do
    {:ok, {^rule, value}, {new_buffer, current_pos}, new_state} ->
      zero_or_more(new_buffer, current_pos, rule, new_state, acc(acc, value))

    {:error, ^rule, {new_buffer, mismatch_pos}, new_state} ->
      {:ok, {rule, acc}, {new_buffer, mismatch_pos}, new_state}
  end
end

defp acc(acc, value) when is_binary(acc), do: acc <> value

defp match_token(<<charcode::utf8, _rest::bits>>, :NameChar) do
  if name_char?(charcode) do
    char = <<charcode::utf8>>
    {:ok, {char, byte_size(char)}}
  else
    :error
  end
end

At first, I suspected that binary appending operations were creating a bottleneck caused by allocations, but it turned out that the Erlang VM is way smarter than I expected. Unlike binary matching which is optimized by the compiler, the optimization work, primarily to avoid copying, of binary appending is done by the runtime system, therefore the optimization only fails in very few circumstances.

The good thing about open source is that you can learn from any random person on the Internet by looking at their code. I peaked at how other libraries implemented this and learned that Jason took a fairly smart approach. As mentioned, binaries in Erlang are designed in a way that they can be referenced internally, a sub binary is only a reference into a part of another binary. So instead of constructing the result binary, we can slice it out from the original binary with Kernel.binary_part/3.

Make Saxy great again

Taking into account all of these learnings: continuous binary matching, reasonable pattern matching, sub binaries instead of new binary construction, I implemented the next version of Saxy.

def parse_open_tag_name(<<charcode, rest::bits>>, cont, original, pos, state, len)
    when is_name_char(charcode) do
  parse_open_tag_name(rest, cont, original, pos, state, len + 1)
end

def parse_open_tag_name(<<charcode::utf8, rest::bits>>, cont, original, pos, state, len)
    when is_name_char(charcode) do
  parse_open_tag_name(rest, cont, original, pos, state, len + compute_unicode_len(charcode))
end

def parse_open_tag_name(<<buffer::bits>>, cont, original, pos, state, len) do
  name = binary_part(original, pos, len)
  state = %{state | stack: [name | state.stack]}
  parse_sattribute(buffer, cont, original, pos + len, state, [])
end

This implementation has many improvements in comparison to the previous one:

Sub-binaries in each parsing function are now continuously passed to the next without being returned or used. This prevents new ones from being created and keeps the original match context.
Every rule now gets its own function, instead of the rule name being passed as the second argument and the VM does crazy clause evaluations.
Data can be sliced out from the original binary using binary_part/3 without having to be constructed. Some times binary construction cannot be completely avoided, for example mixing character references in XML content "Tom & Jerry", in that case we can avoid binary appending by building iodata (["Tom ", 0x26, " Jerry"]).
It also improves error handling. For example if the look-ahead code point is not an expected token, we can immediately return a parsing error, instead of having to do a throw/catch block in the previous approach.

And a happy ending

After applying all the optimizations above, I ran the benchmark again (with the correct script 🙈). And the result started to look very promising:

	Microseconds per run
Saxy 0.4.0	64.401
xmerl 1.3.16	285.5068
Erlsom 1.4.1	66.8736

THAT IS 14.7x SPEED UP (yeah I rounded it up a bit in the blog title).

Results with other samples and a proper benchmark library.

Benchmarking against Hackernoon RSS, Saxy is 1.59 times faster than Erlsom.

	IPS	99th %
Saxy 0.4.0	437.79	3.21 ms
Erlsom 1.4.1	275.23	5.03 ms

This speed is particularly noticeable with deeply nested XML—4.35 times.

	IPS	99th %
Saxy 0.4.0	15.37	76.18 ms
Erlsom 1.4.1	3.53	294.30 ms

More detailed benchmark results can be found on Github.

Summary

I had many different ups and downs, and learned a bunch of things while building the library. I hope that this blog is a fun read and provides a real-world example on optimizing binary parsing.

Binary parsing optimizations in Elixir

Thu, 25 Jan 2018 14:00:00 +0000

Binary parsing is a significant part of almost every application: interactions with databases, data-interchange formats, integers and datetimes in string representation, etc. Making optimizations to those domains might result in substantial speed-up across the whole application.

In the previous blog post I mentioned that we intensively use the MessagePack format where possible (soon our mobile clients start using it too). Comparing to JSON for example, it is much faster, has smaller footprint, and it shows better compression speed and ratio.

In this article I will be using the MessagePack library of our choice—Msgpax—to showcase optimizations you could apply to your implementations of binary parsing.

Measuring the impact

The actual outcome of any optimizations to binary parsing really depends on the data being parsed. For our showcase we will be using the following, somewhat close to reality, data:

inner_map =
  {0.0, [true, -129]}
  |> Stream.iterate(fn {key, val} -> {key + 1, val} end)
  |> Stream.map(fn {key, val} -> {Float.to_string(key), [key | val]} end)
  |> Enum.take(20)
  |> Map.new()

content =
  %{
    "foobar" => nil,
    "baz" => List.duplicate(9001, 20),
    "plugh" => inner_map
  }
  |> List.duplicate(20)

Also we are going to use a rather simple and comprehensible benchmarking script:

payload = Msgpax.pack!(content, iodata: false)

run_count = 10_000

for _ <- 1..1000 do
  Msgpax.unpack(payload)
end

total_spent = Enum.reduce(1..run_count, 0, fn _, result ->
  {run_spent, _result} = :timer.tc(Msgpax, :unpack, [payload])
  result + run_spent
end)

IO.puts(total_spent / run_count) # microseconds per run

Before getting to the applied optimizations it is worth checking the pre-optimized version of Msgpax. For completeness, let’s include our current go-to JSON library—Jason, and yet the most popular JSON library on Hex—Poison.

Elixir 1.5.2 and OTP 20.1.5 yield:

Msgpax 1.1.0	417.8994
Jason 1.0.0	564.4407
Poison 3.1.0	671.3923

And the optimized Msgpax gives the powerful 228.1887 μs. It is a 45% speed-up.

How did we get there?

The rocky road of the single match context optimization

In general, every binary matching (<<...>>) generates a match context. Also for each part that accessed out of a binary a sub binary is created, that is a reference to that accessed part of the binary. This makes binary matching relatively cheap because the actual binary data is never copied.

Nevertheless, the Erlang compiler avoids generating code that creates a sub binary if the compiler sees that shortly afterwards a new match context is created and the sub binary is discarded. Instead of creating a sub binary, the initial match context is kept.

The compiler can point out the places where such optimiation could be applied (or have already been applied) by using the bin_opt_info option:

ERL_COMPILER_OPTIONS=bin_opt_info mix run lib/msgpax/unpacker.ex

It will produce dozens of NOT OPTIMIZED warnings, that could be split into two types:

sub binary is used or returned
called function … does not begin with a suitable binary matching instruction

The first type literally means what the warning says, thus creating a sub binary cannot be avoided. This is the code that generates such warning:

def unpack(<<0xC3, value, rest::bytes>>) do
  {value, rest}
end

In the code above, the rest variable represents that returned sub binary. In some cases it is indeed a desired behaviour, but not when such function is called in a recursive manner:

defp unpack_list(<<>>, 0, result) do
  result
end

defp unpack_list(rest, length, result) do
  {value, rest} = unpack(rest)
  unpack_list(rest, length - 1, [value | result])
end

For example, when we unpack a 5-element list the unpack_list/4 function will be called 5 times, each time creating a new match context and a new sub binary, even if the only use of the sub binary is the next call binary matching (hence its creation can be avoided and the match context can be reused).

Without our help the compiler could not understand that intention. Continuous parsing is the way to get the code optimized:

def unpack(<<buffer::bytes>>) do
  unpack(buffer, [], 1)
end

defp unpack(<<>>, [value], 0) do
  value
end

defp unpack(<<0xC3, value, rest::bytes>>, result, count) do
  unpack(rest, [value | result], count - 1)
end

With this code the optimization works, however, only if the unpack/2 clauses do not produce the second type of NOT OPTIMIZED warnings.

called function … does not begin with a suitable binary matching instruction

To resolve the warning each clause of the unpack/2 function must start with a binary matching, that is, the first argument is <<...>>. Note that with continuous parsing we also were able to avoid 2-element tuple creation.

Starting with OTP 21, the compiler might reorder function arguments to fix the warning and make the optimization happen (see the pull request).

Diving deeper for more optimizations

Keeping a single match context when parsing certainly increases its speed. Yet there are more things to consider. Unnecessary check elimination is one of such.

For example, in the previous code snippet the ::bytes part of the binary matching is not strictly necessary. It adds an additional check to ensure that rest is divisible by 8. Changing ::bytes to ::bits will eliminates the check (28947f9).

To observe the result of this change let’s inspect the produced assembler code. The following command should output it to the Elixir.Msgpax.Unpacker.S file:

mix run --eval "beam = :code.which(Msgpax.Unpacker); {:ok, {_, [abstract_code: {:raw_abstract_v1, abstract_code}]}} = :beam_lib.chunks(beam, [:abstract_code]); :compile.forms(abstract_code, [:S])"

The ::bits usage will result in the eliminated bs_test_unit bytecode instructions in the assembly.

Keeping the argument order intact also helps with the instruction elimination. The 69cd8df commit ensures that the argument order in unpack_continue/6 and unpack_continue/5 is the same. This makes the following difference to the produced assembly:

Another type of optimizations is function clauses reordering. In fact, the compiler does function clauses reordering when possible. But many times it requires manual work to receive faster code.

Putting higher-probability clauses to the top (or the opposite—lower-probability clauses to the bottom) can give better speed. Though changes like that, I believe, warrant regular benchmarking and the produced assembler code inspection.

For Msgpax there were 2 attempts of function clauses reordering (be19258, and 1955c5e)—both giving noticeable boost. But ultimately the code inlining gave much better outcome. The top function clause was integrated into several places where it actually belongs to (81330c3). After that the :inline directive has helped to get rid of the code duplication (6eacf81).

Wrapping up

All the aforementioned optimization were bundled into a single pull request to Msgpax.

I think many kinds of applications would benefit from binary parsing optimizations. And the compiler has all the necessary tools to assist you.

There are also more examples of these optimizations in the wild:

Integer.parse/2 in Elixir
Decoding in Xandra—our Cassandra driver for Elixir

I’m going to close the article with the step by step parsing time progress:

4ac16cf (pre-optimized)	417.8994
3ad47e2…3d3f5b5	321.7947
171faab	311.3362
4b2dba7	304.8067
5e3a213	313.7435
08549bf	334.0085
1955c5e (function clauses reordering)	266.7469
f22686b	262.2508
81330c3 (function clause inlining)	239.2266
2eb1a9f (unnecessary check elimination)	229.2112
69cd8df (keeping the argument order)	228.1887

Until the next one. 🚀

Maximizing HTTP/2 performance with GenStage

Tue, 07 Nov 2017 13:00:00 +0000

A core feature of our Forza Football app is push notifications about live match events. With Apple moving their push notifications services to HTTP/2, we wanted to take advantage of the functionalities that their new API provides and at the same maximize performance and improve resource usage with the new platform.

At the end of 2015, Apple announced an update to APNs (Apple Push Notification service) API. Part of the update was to move communication over HTTP/2 in order to improve performance and be able to deliver instant feedback to clients sending push notifications. A few months ago, since we were rewriting our push notifications system (we wrote about it in the past), we decided to use the new HTTP/2 API so that we would be prepared for the future and the eventual deprecation of the original API.

A (very) quick HTTP/2 primer

HTTP/2 is the new version of HTTP. It brings many improvements while keeping the same semantics of HTTP/1.1. One of these improvements is multiplexing: all communication that happens between client and server through TCP happens on one HTTP/2 connection. However, on one connection there can be multiple streams. A stream is similar to an HTTP/1.1 request but can be initiated - opened, in HTTP/2 lingo - by both server or client. For example, if a browser (client) wants to load a website from a server using HTTP/2, it will first open a connection (which is a direct mapping to a TCP connection) with the server. After that, it will initiate one stream for each resource it needs to load, so for example one stream for the HTML content, one stream per image, one stream for the CSS documents, and one for the JavaScript code. What makes HTTP/2 great is that different streams on a single connection are concurrent: at any given time, many streams can be open on one connection. The exact number of streams that can be open at the same time is negotiated between the client and server during the life of the connection.

HTTP/2 has many more features and improvements over HTTP/1.1, but concurrent streams on a single connection are the one you need to know for the rest of this article. If you want more information about HTTP/2, the HTTP/2 website and the HTTP/2 FAQs are great resources to start with.

HTTP/2 and APNs

The new HTTP/2 API that Apple released uses a single HTTP/2 connection to send multiple push notifications. First, a client (for example, a server where your application lives) opens the HTTP/2 connection with an APN server. This connection can be used to send a (theoretically) unlimited amount of notifications, so it’s advised that you keep this connection open, unless you only send notifications in bursts for only a few times each day. When the client needs to send a notification, it opens a stream on the connection and send a POST request on that stream. The APN server replies on that same stream right away and it closes the stream. This interaction is similar to having one request per notification in HTTP/1.1, but performance is much better because of the single shared underlying connection. When one connection is not enough, Apple advises to simply open many connections and distribute the load of notifications on those.

Maximizing performance

We send millions of notifications each day but they mostly happen in big bursts, for example when a goal is scored in a match between two big teams. Since the notifications we send are time-sensitive, we need to send as many as we can in the smallest possible amount of time.

The first quick-and-dirty approach we took was to simply use one APNs connection to send all of our notifications. We kept this connection open at all times. Problems appeared soon in this prototype implementation: putting the terrible performance aside (caused by having a single sink in the single HTTP/2 connection), we were not handling a fundamental characteristic of HTTP/2, that is, the limit on streams that can be open at the same time (we’ll call this limit max concurrent streams). The current max concurrent streams limit Apple negotiates is 1000 concurrent streams. When sending more than 1000 notifications, say 1001, we would open 1001 streams and if no streams would have freed up by the time we were to open the 1001st stream, we would get an HTTP/2 error from Apple’s server telling us we were opening too many streams.

So, one requirement was to never go over the max concurrent streams limit. At the same time, however, we wanted to maximize the performance of each connection by using it as much as possible. In this case, it means that we should strive to have as many streams open at the same time as we can. Ideally, the number of concurrent open streams should be always close to 1000. We tried to achieve this (and improve the single-connection approach) by using many connections alive at the same time and distributing notifications over these connections. The problem with this approach was that we were distributing notifications to connections in a “fair” way, but connections were processing notifications at different speeds (which can be caused by many reasons, like scheduling, connection usage, and so on). We ended up in a situation where some connections striving to keep up with the notifications to send while other connections would be idle. We traced this down to a design problem: we were using a push pattern where we were pushing notifications to connections without considering the state of each connection. The first solution to this that came to mind was starting to use a pull approach where each connection would express its availability so that all connections would get a fair amount of notifications to send, based on their state and usage.

The more we looked at the problem, the more we recognized a pattern:

many events come into our system (the notifications to send to each user)
we need to “forward” these events to another system (push the notifications to Apple’s platform)
we need to rate-limit these events (keeping the number of open streams under the limit)
we need “consumers” (APNS connections) to pull data (pending notifications) based on their availability

These requirements naturally drew us to GenStage, which is an Elixir library that provides abstractions for “stages” (producers and consumers) that exchange data between each other in a rate-limited, pulling-oriented way. We decided we wanted to rewrite the handling of notifications on top of GenStage to be able to take advantage of its features.

Before moving our implementation to GenStage, we felt that none of the existing HTTP/2 clients for Erlang or Elixir would satisfy our performance and reliability requirements. So before doing anything else, we wrote an custom optimized HTTP/2 client to use just in this project.

The current implementation

In our current implementation, we have a separate service (an HTTP API) which receives notifications to send to different providers (like APNs for Apple or GCM for Google). This HTTP API keeps several connections to APNs open (around 10) and is responsible for all the rate-limiting of open streams and the spreading of events (notifications) over all the open connections. This service is where GenStage lives. Its architecture looks like this:

The notifications “lake”

Requests that contain multiple notifications to send come into the HTTP API. The first step that we take is to temporarily store these notifications in what we call a “lake” (play on words around a big “pool”). APNS.Lake is a GenStage producer that holds a bounded queue of notifications to send. We use a bounded queue so that we have a back-pressure mechanism if more notifications come in than we can handle. All processes that handle HTTP requests queue notifications in a single APNS.Lake process: while this was thought as a first step before having multiple lakes as well, we found performance was great this way so we are holding off before adding more lake processes.

A simplified version of the APNS.Lake code looks something like this:

defmodule APNS.Lake do
  use GenStage

  def init(bounded_queue_size) do
    state = %{
      pending_demand: 0,
      queue: BoundedQueue.new(bounded_queue_size),
    }
    {:producer, state}
  end

  # This is triggered manually from the HTTP API.
  def handle_call({:send_notifs, notifs}, _from, state) do
    case BoundedQueue.in_many(state.queue, notifs) do
      {:ok, queue} ->
        {events, state} = dispatch_demand(queue, state.pending_demand)
        {:reply, :ok, events, state}

      :full ->
        {:reply, {:error, :full}, [], state}
    end
  end

  defp dispatch_demand(state, pending_demand) do
    # Dispatches events until either the pending demand
    # is satisfied (goes to 0) or we run out of events.
  end
end

In the code above, we could let GenStage handle the buffering of events and demand using GenStage’s internal buffer instead of an explicit bounded queue. However, GenStage’s internal buffer drops events when it gets filled up, and we can’t afford to lose events (that is, not send notifications). With a manual implementation through a bounded queue, we have tighter control over the behaviour when the queue becomes full. In our case, we reply {:error, :full} to the HTTP API which can the act accordingly: for example, it can retry later or it can reply with an error to the HTTP client so that that client can retry later (pushing the responsibility of retrying further and further).

The APNS connections

APNS.Lake, the GenStage producer, pushes events to APNs connections (APNS.Connection) processes, which are GenStage consumers. Each connection has a maximum demand of events that is exactly the same as the maximum number of concurrent streams. Conceptually, the logical thing to do would be to initially ask for 1000 events (one notification for each stream) and then to ask for one event each time a stream is closed. In practice, this didn’t work optimally because asking for one event at a time would mean that producer and consumer would exchange many messages, ending up degrading performance. Instead, we buffer demand and make the connection wait until a small number of streams (50 in our case) frees up and ask the producer for this number of events. A consumer never asks for more than max_concurrent_streams - currently_open_streams events, so that the total number of concurrently open streams can’t go over the max concurrent streams limit.

Buffering events also helped us improve the performance of our HTTP/2 client. Each sending of a notification corresponds to a new HTTP/2 request, which boils down to a write on the HTTP/2’s connection TCP socket. This means that by not buffering events we would do many TCP writes close to each other. Now that we buffer events, our HTTP/2 client is designed to send requests issued at the same time as a single blob of bytes down the TCP socket. If we buffer 50 events at a time and send them all at once, we only do one TCP write for all those corresponding HTTP/2 requests.

The work each connection does to process events – that is, send notifications – is asynchronous because it sends a request to Apple and then asynchronously waits for the response. For this reason, we can’t use GenStage’s internal demand mechanism but we have to use the :manual demand mode and send demand to the producer ourselves.

Simplified code for the connections looks like this:

defmodule APNS.Connection do
  use GenStage

  def init(host_info) do
    {:ok, client, mcs} = HTTP2Client.connect(host_info)
    state = %{
      client: client,
      events_to_ask: mcs,
      max_concurrent_streams: mcs
    }
    {:consumer, state, subscribe_to: [APNS.Lake]}
  end

  def handle_subscribe(:producer, _options, from, state) do
    state = maybe_ask_demand(%{state | subscription: from})
    {:manual, state} # we set the demand to :manual
  end

  def handle_events(notifications, _from, state) do
    # Work is async so we send notifications without
    # sending any demand upstream.
    state = send_notifications(notifications, state)
    {:noreply, [], state}
  end

  def handle_info(stream_response, state) do
    process_stream_response(stream_response)
    events_to_ask = state.events_to_ask + 1
    state = maybe_ask_demand(%{state | events_to_ask: events_to_ask})
    {:noreply, [], state}
  end

  def maybe_ask_demand(state) do
    if state.events_to_ask >= 50 do
      GenStage.ask(state.subscription, state.events_to_ask)
      %{state | events_to_ask: 0}
    else
      state
    end
  end
end

The number of connections we have today (around 10) was the result of experimenting with the optimal number of connections for a single machine.

Performance and robustness

We’re very happy with the performance we’re getting out of this system. The graph below shows how a day with many big matches looks like in our metrics dashboards:

In our system we have two servers handling Apple notifications, and the graph above shows the performance for the two of them combined. At peak times, we are able to send more than 1.3 million notifications in around 10 seconds which is a number we’re satisfied with for the time being.

While we aimed mostly for performance, rearchitecting this system also improved the robustness of our system. Thanks to the work being very distributed over many HTTP/2 connections, the bounded queue that ensures we don’t get overflows, and the isolation that Elixir and the Erlang VM provide, we are confident that the problems that may happen will only affect small parts of the system without significantly degrading the overall performance.

Conclusions and next steps

In this post, we explored how we took advantage of GenStage to build a system around the new HTTP/2 Apple API that is able to reliably deliver millions of push notifications to our users. While we’re happy with the current numbers, we have still improvements that we can make to the system in case we’ll want to squeeze out even more performance. One of them is possible thanks to how we distribute the workload in parallel and over multiple servers (albeit only two for now): this means that the system is scalable, which in turn means that if we throw more servers at it, we’ll be able to increase the throughput of notifications we’re able to send.

Internationalization of Elixir applications with Gettext and Transifex

Mon, 19 Dec 2016 13:00:00 +0000

Our Forza Football app is translated into many languages, and therefore our push notifications have to be translated as well. This is an interesting problem to solve when the languages to translate to become more than you can handle by yourself.

One of the most important features of our Forza Football app is sending notifications to subscribed users when events such as goals happen in football matches (we wrote about our notifications before). Since our users are based in many different countries, we need to translate such notifications in many different languages. In this post, I will talk about how we tackle this problem in Elixir (that’s what we wrote our push system in) and about the tools we use to do so.

Gettext for Elixir

The main tool we’re using for translation is Gettext for Elixir. This library is an implementation of GNU Gettext for Elixir applications; in short, Gettext is a system for internationalization of software based on having source strings in the source code that are extracted to translation files where the translations for different languages live. The Gettext for Elixir README explains this in more depth if you’re interested.

When we’re building a notification payload to send to a user, we have code that looks like this:

import Pushboy.Gettext, only: [gettext: 2]

gettext "%{minute}′ Red Card - %{player_name} (%{team_name})",
        minute: minute,
        player_name: player_name,
        team_name: team_name

Don’t worry about the name Pushboy, that’s just how our “pusher” app is called. gettext/2 in the code above is a macro automatically defined in our Pushboy.Gettext module by Gettext:

defmodule Pushboy.Gettext do
  use Gettext, otp_app: :pushboy
end

The source string ("%{minute}′ Red Card ...") is how we will identify this translation in the future, and Gettext will find the translation for the right language at runtime based on such source string.

Extracting translations

Once we’re ready for internationalizing our code, the first thing we do is extract the strings that need translation out of the source code. Gettext provides a Mix task for this:

$ mix gettext.extract
Extracted priv/gettext/default.pot

This task reads all the calls to Gettext macros (such as gettext/2 above) at compile-time and extracts them to POT files (with a .pot extension) that look like this:

# This would go in priv/gettext/default.pot

#: lib/pushboy/event/red_card.ex:86
msgid "%{minute}′ Red Card - %{player_name} (%{team_name})"
msgstr ""

POT files are template files that are only meant to hold a list of all the strings to translate. In the file shown above, msgid is the identifier of the string to translate (which is the string itself) and msgstr is where the translation goes. However, translations are not stored in POT files, because this POT file is not specific to any language. Instead, translated strings are stored into PO (.po) files; each PO file is stored in a directory specific to a language. For example, if we wanted to translate our string to Italian, we would have a file that looks like this:

# This would go in priv/gettext/it/LC_MESSAGES/default.po

#: lib/pushboy/event/red_card.ex:86
msgid "%{minute}′ Red Card - %{player_name} (%{team_name})"
msgstr "%{minute}′ Cartellino rosso - %{player_name} (%{team_name})"

Gettext reads such PO files at compile time to make the lookup of translations as fast as possible. As you can see, the things between %{ and } are interpolation variables: they’re not meant to be translated, and they will be replaced at runtime with some dynamic value.

With the POT file and the PO file above, our translation would look like this:

iex> import Pushboy.Gettext, only: [gettext: 2]
iex> Gettext.put_locale(Pushboy.Gettext, "it")
iex> gettext "%{minute}′ Red Card - %{player_name} (%{team_name})",
...>         minute: 38,
...>         player_name: "Cristiano Ronaldo",
...>         team_name: "Real Madrid"
"38′ Cartellino rosso - Cristiano Ronaldo (Real Madrid)"

Translating into different languages

While we solved the problem of how we internationalize our application, we still haven’t solved the problem of how we translate our strings. Our app supports many more languages that our team members speak, so translating in-house is not an option.

To fix this problem, we use Transifex. Transifex is a website that provides both an easy-to-use interface for organizing and editing translations as well as support for outsourcing translations. Basically, they partnered up with a few translation services so that you can “order” translations from such services and get strings translated to different languages of your choice by users that speak those languages.

While we use external translation services for most languages, we also have native speakers for around six languages in the company; with Transifex, we can easily let these native speakers contribute to the translations. This is possible since Transifex’s interface is straightforward and less tech-savvy users can use it intuitively.

Gettext integration

Transifex integrates with many translation platforms, and Gettext is one of such platforms. You can upload PO/POT files to Transifex and download translations as PO files. Transifex has the concept of “resources”, which are different “domains” of translations (for example, one resource could be notifications, while another one could be error messages, and so on). Luckily, Transifex resources map exactly to Gettext domains: in Gettext, you can use the dgettext/3 macro to extract a translation to a different domain (gettext/2 uses the "default" domain), and each domain ends up in a different PO(T) file (we had default.pot in the examples above).

The workflow is roughly this: first, we extract strings to translate from our source code into a POT file with the mix gettext.extract task, as shown above. Then, we upload this POT file to Transifex in order to update, add, or remove the strings to translate.

After that, we wait for our coworkers to translate what they can and for the translation services to take care of translating. Once all the translations are ready, we download the PO files for all the languages we need from Transifex.

Command-line tools

The workflow described above works but is pretty tedious, involves a lot of manual interactions, and becomes slower the more languages we need to translate to. Lucky for us, Transifex provides a nifty command-line tool called tx: this tool allows us to “push” source strings to translate when we run mix gettext.extract and to “pull” translated strings into PO files once they’re available.

tx can be configured via the .tx/config file in the root of your project. Ours looks like this:

[main]
host = https://www.transifex.com

[pushboy.default]
type = PO
source_file = priv/gettext/default.pot
source_lang = en
file_filter = priv/gettext/<lang>/LC_MESSAGES/default.po

In this file, we configure the pushboy.default resource (which maps to our "default" Gettext domain) and instruct the tx tool that:

we want to use the Gettext format (PO and POT)
our source strings to translate live in priv/gettext/default.pot
our source language is English
the translations that we’ll download should end up inside priv/gettext, under the language they’re translated to (<lang> is replaced by tx)

With the configuration file above, we can now push new, edited, or removed source strings to translate with $ tx push --source (--source ensures we only push the updated POT) and pull updated translations with tx pull.

Cleaning up Transifex PO files

We’re pretty strict when it comes to style in our codebase, and we found that the PO files that Transifex generated for us were not perfect for our taste. Luckily, Gettext provides tools to easily parse and modify PO/POT files. What we ended up having is a new Mix task, mix translations.pull, that abstracts the interaction with tx away.

In such task, we first call tx pull, then we iterate over all pulled PO files and we “reformat” them how we want:

defp reformat_po_file(path) do
  reformatted_po =
    path
    |> Gettext.PO.parse_file!()
    |> reformat_headers()
    |> remove_top_of_the_file_comments()

  File.write!(path, Gettext.PO.dump(reformatted_po))
end

# We get rid of the Last-Translator header
defp reformat_headers(%Gettext.PO{headers: headers} = po) do
  new_headers = Enum.reject(headers, &String.starts_with?(&1, "Last-Translator"))
  %Gettext.PO{po | headers: new_headers}
end

# We get rid of comments that Transifex leaves at the top of the PO file
defp remove_top_of_the_file_comments(%Gettext.PO{} = po) do
  %Gettext.PO{po | top_of_the_file_comments: []}
end

The complete workflow

Our complete workflow is currently the following:

after modifying our source code, we run mix gettext.extract and get a bunch of updated POT files
we run tx push --source to update the strings to translate on Transifex
we wait for our coworkers and for the translation services to translate the updated strings
we run mix translations.pull and get updated translations

We really enjoy this workflow as it allows us to programmatically update, push, and pull translations, and at the same time it allows us to scale really easily when adding new languages (as nothing changes in the workflow, we just need to order translations for those languages as well on Transifex).

Conclusion

We took a look at how we take care of translating push notifications for our Forza Football app in several languages in a way that is easy to use, fast, and scales well with the number of translations and languages we have.

Gathering metrics in Elixir applications

Fri, 26 Aug 2016 14:00:00 +0000

Metrics are a fundamental part of most pieces of software. They’re a great insight on how a system behaves and performs, and can be used for different purposes, such as performance monitoring or alerting in case of anomalies.

Elixir is often praised as a fast language, especially for distributed, concurrent applications (like the ones we build at Forza Football). However, “fast” can be tricky to define without enough measurements available; without metrics, refactoring (especially aimed at improving speed) also becomes a risky thing to do, as it can be hard to determine if the performance is improving or not.

Let’s have a look at in what way and with what tools we gather metrics in our Elixir applications.

Overview of the stack

Our metrics architecture is roughly shaped like this:

Let’s explore this architecture and the components that form it in more depth.

Storing metrics

Let’s start from the core of the architecture: the database we use to store metrics. We’re using InfluxDB, an open-source time-series database which suits storing metrics perfectly; it performs really well (it may be interesting to know it’s written in Go) and provides a bunch of very useful functionalities.

InfluxDB databases are formed by a set of measurements. Each measurement is a collection of data points; each data point (that’s the InfluxDB official term) is formed by a timestamp, a set of tags, and a set of values. Tags are for “tagging” the measurements with metadata about it: for example, a common tag attached to measurements is host, which holds the value of the host reporting the measurement. Tags are shared by all the points in a measurement. Values, instead, are the subject of the measurement - what is being measured. For example, a value could represent the time it takes to complete an HTTP request, and be called request_time. A data point can have multiple values; when a point has a single value, we often call that value just value, since the measurement name will be enough to determine what that value represents. In the rest of the post, we will often refer to “measurements” as both measurements and points, to make the reading experience more fluid.

InfluxDB also provides a rich SQL-like language to query the data it stores, and ways to resample data for optimal storing. We won’t look at either of those in this post; if you’re interested, check out their documentation.

InfluxDB supports a couple of ways to store data in it: measurements can be reported via an HTTP API or via an UDP interface, and can be encoded either as JSON or using InfluxDB’s own “line protocol”. The UDP interface is naturally faster since it lacks the overhead of something like the HTTP protocol and the TCP transport; however, as always, it provides close to no guarantees of successful delivery of data. Choosing between the HTTP API and the UDP interface can be hard, but we will see how it’s possible to avoid this choice completely and get the best of both worlds.

Last but not least: to improve the availability of InfluxDB and be more conservative with our metrics, we use influxdb-relay, which sets up a simple load-balancer-like architecture to add some replication to the metrics storage. This only works for writing metrics though, so we point directly to an InfluxDB instance when we need to read metrics.

Reporting metrics

The most straightforward way to report metrics to InfluxDB would be to send them directly from our Elixir applications to InfluxDB, using either the HTTP API or the UDP interface mentioned in the previous section.

This choice, however, is a tough one for us: we have a high load of metrics, and we often need to report hundreds of thousands of metrics in a matter of seconds. This brings out disadvantages in both reporting methods: using the HTTP API would mean slowing down our applications, which is not something you want metrics reporting to do, but using the UDP interface would mean risking to drop metrics, especially because of the substantial number of metrics being reported.

Telegraf

Luckily, the same folks behind InfluxDB provide another nifty open-source tool called Telegraf. Telegraf is basically a daemon that can collect inputs from different sources and delivers it to different outputs every n seconds; both inputs and outputs are managed via “input plugins” and “output plugins”. We’re using primarily two main input sources, and one output.

Inputs

The inputs that Telegraf is gathering for us are two. The first one is “system metrics”, meaning system data such as CPU usage, memory usage, disk usage, network usage, and so on. Telegraf takes care of reading this measurements out of the box. The other input we’re using is the UDP listener built into Telefgraf: this plugin provides a “mirror” of InfluxDB’s UDP interface, exposed locally through Telegraf. This is useful: for example, it means InfluxDB bindings in different languages can be built to work with the UDP interface, and automatically work with running instances of both InfluxDB as well as Telegraf. Fun fact: me and @lexmag actually contributed this plugin to Telegraf because it was just perfect for our needs!

Outputs

We’re using only one output plugin: the InfluxDB plugin. This plugin is straightforward: it reports measurements gathered via input plugins to InfluxDB. It also provides some niceties such as global tags that can be applied to all measurements from every input plugin (we use this to set the host tag for both application and system metrics) and measurement filtering. But for us the most useful feature is that this plugin talks to InfluxDB’s HTTP API instead of UDP interface: this means that we can rely on the guarantees that HTTP (and the underlying TCP transport) provide to ensure metrics arrive to the InfluxDB server.

Aggregation

For us, Telegraf works as a middlemann between our application reporting metrics and InfluxDB, and this has multiple advantages:

We can use the UDP interface to send metrics from our Elixir applications to Telegraf; this greatly benefits the speed at which we can report these metrics, and, since Telegraf is running locally, the risk of losing packets because of UDP is substantially reduced; since Telegraf reports these metrics to InfluxDB via HTTP, the chance of losing measurements is very low.
We can throttle the number of measurements that are reported to InfluxDB: Telegraf aggregates metrics for n seconds (usually 5 or 10 seconds is a good number, depending on the application), before sending them to InfluxDB. This means that Telegraf will only send bulks of metrics to InfluxDB, once every n seconds, reducing the number of hits to InfluxDB and the network traffic.

Telegraf/InfluxDB driver

When we started using InfluxDB and Telegraf, there were no Elixir drivers for InfluxDB’s UDP interface (we could only find one for InfluxDB’s HTTP API). But fear not: we built one! Fluxter is a straightforward Elixir library that provides a pool of UDP connections to InfluxDB/Telegraf (it doesn’t know the distinction, since the UDP protocol is the same) and a simple API to report metrics.

Each Elixir application has its own Fluxter module:

defmodule MyApp.Fluxter do
  use Fluxter
end

The code above turns MyApp.Fluxter into a supervised pool of UDP connections. To make things fault-tolerant, we make sure to start this pool under the application’s supervision tree:

def start(_type, _args) do
  children = [
    MyApp.Fluxter,
    # ...
  ]

  Supervisor.start_link(children, strategy: :one_for_one)
end

Finally, we configure the pool via the application’s configuration:

# Assuming Telegraf is running on localhost:8086, and has the UDP input running:
config :fluxter,
  host: "localhost",
  port: 8086

Then, once the Fluxter pool is started, we can easily report metrics through it:

def my_operation() do
  # Perform my operation
  MyApp.Fluxter.write("something_done", [my_tag: "foo"], 1)
end

For more information on Fluxter, you can consult its documentation.

Erlang VM metrics

Given we report metrics for everything, we couldn’t skip metrics from the Erlang VM: we gather detailed information about number of running processes, memory usage, process mailboxes, garbage collection, and more. To do that, we use vmstats, a tiny Erlang application that collects measurements from the Erlang VM and reports them to a “sink”: a sink is just an Erlang/Elixir module implementing the :vmstats_sink behaviour. We use the Fluxter pool as the sink, and we have an identical setup for it in each of our Elixir applications:

defmodule MyApp.Fluxter do
  use Fluxter

  @behaviour :vmstats_sink

  def collect(_type, name, value) do
    write(name, value: value)
  end
end

Have a look at vmstats’ README for more information on what it can do.

Counters

Since we report a very high number of metrics in a very short span of time, we experienced trouble with this setup, since each reported measurement would mean a UDP packet sent to Telegraf (which makes things prone to overflowing and packets get easily lost). For this reason we introduced counters in Fluxter: a counter is simply another level of aggregation of numeric measurements, at the Elixir level this time. Counters will add every measurement reported to them and only send the final sum to Telegraf/InfluxDB when “flushed”.

counter = MyApp.Fluxter.start_counter("my_operation_success", host: "eu-west")

Enum.each(1..1_000_000, fn _ ->
  my_operation()
  MyApp.Fluxter.increment_counter(counter, 1)
end)

MyApp.Fluxter.flush_counter(counter)

Visualizing metrics

All the metrics we report and store would be close to useless without a way of visualizing them. As our visualization tool, we use Grafana, which has native support for InfluxDB (meaning it can query InfluxDB out of the box) and which packs a ton of useful features.

Our dashboards look roughly like this one:

The dashboard above is the one that we use to monitor the health of all our applications: it shows metrics from the system and from the Erlang VM. The select box on the top left corner (where it says “pushboy”) lets us choose which application to visualize metrics of, so that we can have an overview of the system and Erlang VM metrics for each application with just a couple of clicks.

Grafana provides several useful features: for example, it can group measurements by tag (for example, useful for showing each host in a cluster as a different line on a graph), it can perform all sorts of aggregations on the data, and is very customizable in its appearance.

Grafana is probably the simplest part of the whole architecture, but a crucial one nonetheless.

Alerting

We mainly use metrics to have an insight on how our applications perform, but we also take advantage of them for another thing: alerting. Metrics can be often bent to communicate the health of an application: for example, if the number of Erlang processes in one of our applications goes way up the average value, we know it’s something we should look into.

The lovely people behind InfluxDB and Telegraf have this covered as well with another open-source tool called Kapacitor. Kapacitor is a daemon that can repeatedly perform queries on InfluxDB and act on the results of such queries. Kapacitor works by interpreting “Kapacitor scripts”, which are scripts written in a Kapacitor-specific language. A lot can be achieved with these scripts, but let’s keep it simple for this post: the script below is taken directly out of our system and shows how we use the Erlang VM metrics gathered by vmstats to determine the health of the application.

stream
  |from()
    .measurement('pushboy_vm_modules')
  |groupBy('host')
  |deadman(15.0, 1m) //
    .id('Pushboy [{{ index .Tags "host" }}]')
    .message('{{ .ID }} is {{ if eq .Level "OK" }}up{{ else }}down{{ end }}')
    .stateChangesOnly()
    .slack()

This small script checks for the vm_modules measurement and triggers an alert if the throughput drops below 15 points per minute (a healthy application with our configuration reports around 60 points per minute). As you may guess by the last line, this script will notify us by posting on a Slack channel (which is specified in Kapacitor’s configuration). Kapacitor supports several ways of notifying about alerts other than Slack, such as PagerDuty or email.

Conclusion

This post showed how we gather as many metrics as we can from our Elixir applications (and the servers they run on) and how we use them to monitor the health and performance of such applications and to get alerts when problems arise. We’re quite happy with this system, as it works almost flawlessly and has made us much more scientific in our approach to software development: we now tend to experiment, gather data, and draw conclusions based on data instead of relying a bit too much on our gut.

The Pursuit of Instant Pushes

Tue, 02 Aug 2016 13:00:00 +0000

At Forza Football we strive to provide our apps users with the best possible experience, and lightning-fast push notifications is one of its aspects.

This photo (by Argonne National Laboratory) shows Cherenkov radiation glowing in the core of the Advanced Test Reactor. It’s emitted when a charged particle passes through a dielectric medium at a speed greater than the phase velocity of light in that medium.

My name is Aleksei and I spent most of last year building a new push notification system.

This blog post tells an introductory story of why it was needed, how it operates today, the migration path to the new system, and what the plans for the future are.

From an extremely high view, each push system consists of two logical parts: subscription tracking and notification sending.

Overview of our legacy system

The legacy push system is an integral part of a much bigger Ruby on Rails application, which is responsible for many more things besides push notifications handling.

Let’s take a closer look at it.

Subscription tracking

There is an API for handling subscribe/unsubscribe HTTP requests. Each request goes through a validation step and then subscriptions get inserted/deleted.

Before going further, I should explain how subscriptions look like. Every subscription contains:

device_token – an identifier in the specific vendor service
subject – match, team, or league
topic – the matter of interest: "match_reminder", "goal", etc.
language
country

For instance, having the following subscription:

device_token	xfL3k6QxHagFNf8Y01a
subject	team, 42
topic	goal
language	en
country	SE

We know that someone from Sweden, who speaks English, wants to receive a notification every time their favourite team scores a goal.

Notification sending

The process of sending notifications starts when a match reminder is generated, an event has happened during the football match, or a transfer rumour is received. Notifier loads all the relevant subscriptions, filters out token duplicates, removes muted devices, and as the last step builds translated messages (sometimes limiting selection to specific countries). Yes, we have to deal with token duplicates, since an individual user could be subscribed to both teams in a single match (also to a league).

After all these steps, the generated payload is sent to Dispatcher, which does the actual sending to the particular vendor service (Apple Push Notification, Google Cloud Messaging, or Windows Push Notification), and that’s the moment when users start to receive their notifications.

Slow pushes are slow

After having successfully used this setup for a long time we have reached a point when notification sending time has increased unacceptably. For instance, it could take a few minutes, depending on application load, to send a push notification to 1 million subscribers. Quick investigation has showed that it is all due to Notifier being extremely slow in loading the relevant subscriptions. This was caused by the nature of the query (which included many JOIN expressions) and by the big size of the whole data set.

An attempt to fix the problem had already been done: subscribers sharding was introduced and subscribers querying was moved to a slave machine with a fancy index. There was a strong need to replace MySQL with something more appropriate (and change the data model). Ideally it should have allowed us to parallelize subscriptions handling (translation, country filtering, etc.) in some way, instead of doing it sequentially.

There was only one problem: too many data stores to choose from. Of course, we had to do research to pick the right one, so I limited the choice to a handful of databases (some I’ve worked with previously, some I selected based on their storage model and plenty of articles). I usually say “let’s think about future in the future”, but this wasn’t that case, thus I went with a 200 million subscriptions sample data set measuring 4 million subscriptions extraction. The process was conceptually simple but rather time consuming, it took about two months in total to pick a suitable database: Cassandra.

Cassandra data modeling

There are many articles written on this topic, however one rule is worth to repeat on every occasion: model around your queries. This leads to data duplication and a heavier write load, which is a preferred tradeoff in Cassandra.

The new push system operates with two tables in Cassandra: one to handle subscription tracking and another one for notification sending.

Both tables leverage compound primary key to store subscriptions which belongs to the same topic and subject sequentially on disk on a single node (plus replicas). It makes reads extremely efficient.

The role of the second table is to store normalized subscriptions. I mentioned previously, somebody can freely have intersected subscriptions, therefore this table effectively removes the burden of token duplicates handling during notification sending.

Moving to the new system

Now our new push system has got three specialized components built around Cassandra database. These components are written in Elixir.

For the inter-system data exchange we use MessagePack format (with Msgpax library) throughout the system. It is much faster than JSON, and has smaller footprint.

Subscription tracking

Besides the old API we have the Pushkeeper application, which is responsible for managing subscriptions, and it also took responsibility of feedback handling, which isn’t depicted. API operates as previously to support the legacy system, and also duplicates incoming requests to Pushkeeper to allow gradual migration to the new system.

Notification sending

The Pushboy application is replacing Notifier: it does the same tasks (loading subscriptions, performing messages translation, etc.). However, there is one important difference: subscriptions are already normalized, and Pushboy effectively streams them out of the Cassandra table. It means that users start to receive their notificatins right away.

The Pushpass application is a middleware between the former two, which is responsible for data normalization, that is, filtering out token duplicates and populating the specialized table.

Measure all the things

Migration to new systems is hard. To verify that the new system works as expected, we run it in parallel with the legacy system for about a half year, performing all operations except the very last step - sending data to vendor services. Only a limited set of test devices has been whitelisted. This helped us to refine deployment process, failure recovery mechanism, and of course, to measure all the things.

Our metrics system (which deserves its own blog post) was actually built from scratch during the migration to the new push system. I’ll just briefly mention some tools, libraries, and technologies we use for metrics: applications report metrics via Fluxter library to locally installed Telegraf agent, which sends them further to InfluxDB store.

According to a vast range of metrics the new system performs great, let’s see how long it takes to send a notification to 1.3 million subscribers.

The legacy system requires 40 seconds to prepare payload, and an additional 5 seconds to dispatch it.

The new system finishes sending in 16 seconds, and this is in a streaming way: this means that a half of subscribers will receive their notification in 8 seconds.

Bottleneck shifts to other place

Components of the new system reside in one datacenter while the legacy system is in another one (yes, one more migration, we’re moving to Amazon Web Services cloud). This quite badly affects the request time for sending generated payload to Dispatcher. In addition to that, Pushboy produces hundreds of such requests per second.

In fact, Pushboy finishes processing as fast as the slowest subscriptions chunk being processed in Dispatcher.

Long live the new system!

Since last month, we are running new push system in production for some particular notification topics.

Besides the pure performance improvement, there is a bunch of things improved:

No more “MySQL as a queue” anti-pattern, which brings locking issues and data growth control problem.
Notification sending executes in parallel, in a streaming manner.
Main Ruby on Rails application no longer affects push system operation (for example, CPU stealing was a noticeable problem before), and its complexity has decreased.
Tiny targeted components make maintenance easier.
Started to practice canary deployments.
We simply need less resources than was required previously.

Elixir matters a lot for the system

Starting with the programmer happiness, there are many, much more pragmatic reasons to choose Elixir.

Elixir’s compiler is wonderful. It is smart, fast, and incredibly helpful. It actively helps in the elimination of bugs early on in the software cycle.

Erlang Runtime System enabled us to build highly concurrent system with strong fault tolerant characteristics. It provides better introspection capabilities and live system debugging.

Furthermore, we utilize distributed setup for several things, for example: to reduce the probability of duplicated delivery, or to coordinate subscriptions normalization.

Beyond the speed of light

Things indeed improved but there are still many left:

Complete switch to the new system in coming weeks.
Replace Dispatcher with the Pushgate application, which is located in the same datacenter.
Introduce API v2, revised and enhanced.

In addition, we have quite ambitious plan to release several Elixir libraries. Stay tuned.