A/B testing and HTTP caching with `plug_http_cache`

In the previous article, we discussed how to setup plug_http_cache to cache Phoenix’s response.

In this article, we’ll take a look at the specific case of caching when doing A/B testing.

A/B testing consists in presenting 2 different versions of a page (page A and page B) to users and comparing the users’ behaviour to determine which version performs better.

Usually, a user is randomly assigned a group (A or B) when first visiting the site implementing A/B testing and sticks to this group. The value of the group is stored in a persistent cookie.

We’ll try to implement caching A/B testing in plug_http_cache_demo. By default, the result of the Fibonacci calculation is shown in Arabic numerals:

How about Roman numerals? Time to check if it’s more convenient to users:

In both case, we have the same URL: http://localhost:4000/fibo?number=19.

We will not implement the actual measurements of the users’ behaviour. We are only interested in caching the page. Or pages?

Caching different versions of a page

To find a cached response from a request, HTTP caches build a cache key from the request. http_cache, the library used by plug_http_cache, takes into account the URL, the method, the body (in case, for instance, of caching POST search requests) and a bucket (to differentiate private caches - not used by plug_http_cache):

request_key({Method, Url, _Headers, Body}, Opts) ->
    MethodDigest = crypto:hash(sha256, Method),
    UrlDigest = url_digest(Url, Opts),
    BodyDigest = crypto:hash(sha256, iolist_to_binary(Body)),
    BucketDigest = crypto:hash(sha256, erlang:term_to_binary(map_get(bucket, Opts))),
    crypto:hash(sha256,
                <<MethodDigest/binary, UrlDigest/binary, BodyDigest/binary, BucketDigest/binary>>).

Other caches use slightly different parameters. Varnish only takes into account the URL and the host (or IP):

sub vcl_hash {
    hash_data(req.url);
    if (req.http.host) {
        hash_data(req.http.host);
    } else {
        hash_data(server.ip);
    }
    return (lookup);
}

No trace of cookie here, but the A/B test group is stored in the cookie! If we enable HTTP caching here, only one of the two versions will be cached, and served, to everyone. The cached page will simply be the page returned upon the first visit, and will be served until the cached version expires.

Common solutions to this problem include manually altering the cache key, changing the URL to reflect the A/B testing group as a query parameter, or somehow redirecting to different URLs depending on it.

Changing the cache key is not possible with plug_http_cache, and other usually proposed solutions are cumbersome and not elegant.

So we’re back to the question: how to store different versions of a same page?

The response is in the documentation, to put it politely.

This mechanism has existed for decades and is used for the accept-* headers. HTTP caches actually can and will store different versions of a page as long as the vary header is set. Follow the link for a refresh about this header if needed.

Using it looks like a good option for our use-case. Let’s see how to implement it.

Implementation with plug_http_cache

Let’s sum up what needs to be done to implement it in Phoenix with plug_http_cache:

1. first check an A/B test value exists in the session, and create it otherwise
2. then we need to set an ab-test header with the A/B testing group
3. somewhere in the backend, we need to check the A/B testing group to return different pages
4. finally, we need to set the vary header accordingly, otherwise caches will not know the page changes depending on this header’s value

We’ll use the "a" and "b" string values for the 2 groups, but it could be another value (it’s often 0 and 1).

Plug is an ideal mechanism to set the cookie and header. Let’s start with the cookie:

defmodule PlugHTTPCacheDemoWeb.Plug.SetABTestCookie do
  @behaviour Plug

  @impl true
  def init(opts), do: opts

  @impl true
  def call(conn, _opts) do
    case Plug.Conn.get_session(conn, "ab-test") do
      "a" ->
        conn

      "b" ->
        conn

      nil ->
        Plug.Conn.put_session(conn, "ab-test", rand_a_or_b())
    end
  end

  defp rand_a_or_b() do
    if :rand.uniform() < 0.5, do: "a", else: "b"
  end
end

If the cookie doesn’t exist, we set “a” or “b” with a 50/50 probability.

The A/B testing group being set in the session, we can then set the header:

defmodule PlugHTTPCacheDemoWeb.Plug.SetABTestHeader do
  @behaviour Plug

  @impl true
  def init(opts), do: opts

  @impl true
  def call(conn, _opts) do
    ab_test = Plug.Conn.get_session(conn, "ab-test") || raise "Missing ab test cookie value"

    Plug.Conn.put_req_header(conn, "ab-test", ab_test)
  end
end

We can now add them to our pipeline in the router in the right order:

  pipeline :browser do
    # ...
    plug Plug.SetABTestCookie
    plug Plug.SetABTestHeader
    plug PlugHTTPCache, Application.compile_env(:plug_http_cache_demo, :plug_http_cache_opts)
  end

Since plug_http_cache will read the ab-test request header to pick the right version, it’s important to set this header before the PlugHTTPCache plug.

It’s time to return different versions of the page. We’ll not discuss what is the best way to handle A/B tests in Phoenix, as the author has no clue what the best practices are :)

Let’s just read the session and return the numbers in Roman numerals if the A/B testing group is “b”:

defmodule PlugHTTPCacheDemoWeb.FiboController do
  use PlugHTTPCacheDemoWeb, :controller

  alias PlugHTTPCacheDemo.RomanNumeral

  #...

  def index(conn, %{"number" => number_str}) do
    {number, _} = Integer.parse(number_str)
    result = PlugHTTPCacheDemo.fib(number)

    ab_test = Plug.Conn.get_session(conn, "ab-test")

    conn
    |> set_alternate_keys(result)
    |> Plug.Conn.prepend_resp_headers([{"vary", "ab-test"}])
    |> render("index.html",
      number: format_abtest(number, ab_test),
      result: format_abtest(result, ab_test)
    )
  end

  #...

  defp format_abtest(number, "a"), do: number
  defp format_abtest(number, "b"), do: RomanNumeral.convert(number)
end

If we open 2 private windows, and with a bit of luck, we can now see the 2 versions of the page:

Which version do you prefer?

Let’s check in the browser they’re correctly cached:

• The age header shows a cached response was returned
• The cache-control header is correctly set
• The vary headers shows that the response varies on ab-test and accept-encoding (plug_http_cache automatically compresses text responses, which is why the responses also vary on accept-encoding)

All good!

We can actually go deeper and take a look at the http_cache_store_memory_table_object table containing the cached responses (courtesy of the http_cache_store_memory store used here) in the observer:

As you can see, the value of the request header (“a” or “b” - 97 or 98 in the second column) has to be saved in the cached response metadata so that we can pick the right answer.

It also looks like the primary keys (1st column) are the same, but it’s not the case and can’t be in an ordered_set ETS table:

• for “a” it’s {<<87,34,228,152,15,9,33,144,234,100,243,187,249,73,166,247,238,97,112,73,156, 155,45,194,141,104,145,252,131,185,12,18>>, <<144,142,233,211,151,219,19,41,13,41,90,248,164,40,49,123,56,20,206,50,57, 14,52,175,213,21,79,234,235,5,244,184>>}
• for “b” it’s {<<87,34,228,152,15,9,33,144,234,100,243,187,249,73,166,247,238,97,112,73,156, 155,45,194,141,104,145,252,131,185,12,18>>, <<85,91,180,57,41,57,7,165,10,95,231,160,8,168,82,141,240,47,72,161,12,20, 103,123,41,231,183,116,97,228,59,127>>}

The first element of the tuple is the request key, and are equal. The second element is the part that actually takes the varying headers into account.

Storing the cached responses in this manner allows finding all the variants of a page very quickly thanks to an ordered_set trick. Indeed, ordered_set tables are implemented as a tree, and matching on the first element of tuple is thus extremely efficient as data is collocated.

Looks like we’re done, right?

Something might be very wrong

If your Phoenix server is serving users directly, then there’s no problem. However, what happens if you have intermediate shared caches (CDN…) in-between?

In this case, the shared cache will see that the response varies on the ab-test header, will look at the request to fetch the value of this header, which is missing. Then it will store the response, one of the two so either with Arab numerals or Roman numerals, and return it to all subsequent requests that don’t have an ab-test header, that is to all requests.

Congratulations, you have accidentaly cached one version of the page for all you users, until the cached page expires!

This happens because the ab-test header is set by your Phoenix instance, and not at the shared cache level. To fix it, either:

• prevent caching by your CDN, but setting the cache-control header back to private, no-cache, no-store, but you’ll no longer benefit from your CDN; or
• set the cookie and the header at the CDN

The code of the implementation is available on the ab-test branch of the plug_http_cache_demo.

We’ll soon continue this series of articles with a very exciting thing: publicly caching private pages. Stay tuned!