Avoid test setup pollution (and 4 problems it creates)

One problem with shared setup in Elixir is that it can pollute our tests.

Take, for example, the default UserLiveTest generated by running mix phx.gen.live Accounts User users. The “Index” describe block creates a user before each test:

describe "Index" do
  setup [:create_user]

  # tests...
end

defp create_user(_) do
  user = user_fixture()
  %{user: user}
end

Some people like that because it makes the user record available through our test context:

test "lists all users", %{conn: conn, user: user} do
  {:ok, _index_live, html} = live(conn, Routes.user_index_path(conn, :index))

  assert html =~ "Listing Users"
  assert html =~ user.name
end

But that brings a few problems:

• it creates unused data,
• it introduces mystery guests,
• it requires subsequent updates, and
• it obscures critical information.

Let’s look at each in turn and at one beneficial use case at the end.

Shared setups create unused data

It’s common for one or two tests to be sufficiently different from the rest that they don’t need the shared setup.

For example, within the “Index” describe block, the “saves new user” test doesn’t need the user created. Since the test is trying to create a user, it only takes the conn struct from the test context:

test "saves new user", %{conn: conn} do
  {:ok, index_live, _html} = live(conn, Routes.user_index_path(conn, :index))

  # ...

  {:ok, _, html} =
    index_live
    |> form("#user-form", user: @create_attrs)
    |> render_submit()
    |> follow_redirect(conn, Routes.user_index_path(conn, :index))

  assert html =~ "User created successfully"
  assert html =~ "some name"
end

Nevertheless, the shared setup creates the user before we run the test. So, we slow down our test suite with a database operation even though we don’t need the record.

The additional delay of unnecessary database operations might not be perceptible when running a single test. But, we certainly feel the impact when we have many tests that use the shared setup.

Shared setups introduce mystery guests

Because the setup is separate from the test body, the data created can act as an unexpected side effect.

Imagine adding a new test that makes an assertion about the empty state of the page:

test "displays empty state when there are no users", %{conn: conn} do
  {:ok, _live, html} = live(conn, Routes.user_index_path(conn, :index))

  # this unexpectedly fails because it turns out we have a user
  assert html =~ "You don't have users yet! Invite someone here."
end

Unexpectedly, our test fails because our list of users isn’t empty, so we never get the empty state! Where did the user come from? (You and I both know 😉)

Shared setups require subsequent updates

Since shared setups try to create data that is applicable to all tests, we often end up with the lowest common denominator – a basic record. Then, we’re forced to tweak it within each test to make it useful.

For example, what if some of our tests need more than just a regular user? What if one test needs an archived user, another needs an admin, and a third needs an archived admin? We either need to separate each test under a different describe block and call different versions of setup:

describe "Index for archived admin user" do
  setup [:create_user, :make_admin, :archived]

  test "archived admin cannot see admin panel", %{conn: conn, user: user} do
    # ...
  end
end

Or we have to update the user passed in through the context to fit our test’s needs:

test "archived admin cannot see admin panel", %{conn: conn, user: user} do
  archived_admin = Accounts.update_user(user, %{archived: true, admin: true})
  # ...
end

In either case, we’re no longer only creating a user for each test, but we’re now also updating that database record in each test just to fit our setup needs.

Compare that to the alternative of creating the user for each test within that test body. We can use the same user_fixture function with some additional options:

test "archived admin cannot see admin panel", %{conn: conn} do
  archived_admin = user_fixture(admin: true, archived: true)
  # ...
end

We avoid extra database operations and keep all the logic required to set up the test within the test itself!

Shared setups obscure critical information

Like other code, tests benefit from being self-contained and easily understood.

But shared setups separate the context in which a test runs from the test’s exercise and verification steps – making the test much harder to understand. That is all the more apparent as a test file grows:

Imagine writing the tenth test that uses the same setup.

We look up to see what setup is already done. But the setup is so far above that we don’t know at a glance what is and isn’t created. Either we scroll up and load the context into memory and go back to our test, or we ignore the previously created data, potentially create duplicate data (slowing down tests even further), and risk running into mystery guests.

My point here is not that we shouldn’t abstract the user setup at all – I think functions like user_fixture/1 do that well enough. But, in using setup we separate the context of our test from our test in unhelpful ways.

So, my advice is to set up the necessary data inside each test, using functions to abstract irrelevant details, but keeping the test a single, self-contained, understandable block of code.

Is there a good use of setup?

I find one use of shared setup beneficial – to set up the machinery required to run tests.

Let’s once again consider our autogenerated LiveView tests. They all use the conn struct that comes from ConnCase’s shared setup:

  setup tags do
    # ... sandbox setup

    {:ok, conn: Phoenix.ConnTest.build_conn()}
  end

conn is different from the user we’ve seen because the connection struct is required to run the test, but it is not part of the domain we’re trying to test.

Therefore, putting the conn in shared setup clarifies rather than obscures our tests by removing irrelevant setup information.