Behavior-Driven Test Data

23rd of August, 2020
rails, ruby, testing

Don't discard the database state at the end of your test. Use it!
by Tom Rothe


When running behavior tests, we seed the database with a defined snapshot called fixpoint. We do run the behavior test and save the resulting database state as another fixpoint. This method allows testing complex business processes in legacy applications without having to implement fixtures/factories upfront. By building one fixpoint on top of another, we can ensure that the process chain works without any gaps. Comparing each resulting database state at the end of a test with a previously recorded state ensures that refactoring did not have unintended side effects.

The Problem

I recently worked on a large system which had grown since 2014. Getting to know the system took a while. Grasping the business domain and the implemented features was tedious and slow. There were no tests and no documentation. So I could choose choose between annoying my peers with tons of questions or playing detective for hours on end.

When I started a new feature, I would writing a few browser tests to get to know the relevant areas of the system. This strategy worked well for features early in the user flow/business process chain. So for example, writing tests for creating a user account was easy because it did not rely on prior actions. The latest ticket however involved invoicing. Invoicing required the service to be carried out and adding the respective records in the database. To carry out the service, there are million other processes to be carried out before. Long story short, I needed a lot of database setup before I could even begin to explore & write tests.

classical: ignore test outcome and implement test setup from scratch
classical: ignore test outcome and implement test setup from scratch

I diligently wrote factories and identified more constraints, more dependencies and more quirks of the data model – frustrating work. Anyhow, since I could not get the data quite right for the feature I wanted to test, started writing a test for a feature that came earlier in the user journey. I found myself backtracking further and further to understand the necessary data constraints for invoicing.

The Idea

produce the database setup for a test by the test before it
produce the database setup for a test by the test before it

After each test completed, I had gained enough knowledge to refine the database setup for a subsequent test later in the process chain. But, implementing test fixtures/factories seemed redundant. Idea! I wrote a test that produced a state in the database. This exact state could serve as the basis of the next test, but I was not using it. Instead, I resorted to mimicking the state as closely as necessary in my test setup. That seemed inefficient and superfluous. The idea for fixpoints was born. Instead of neglecting the data at the end of a test, we save it and use it for the subsequent tests.

The Advantages

After doing a quick spike, I had a working implementation that worked with Rails system tests and RSpec:

it 'registers a user' do
  visit new_user_path
  fill_in 'Name', with: 'Hans'
  click_on 'Save'

  store_fixpoint :registred_user
  # creates YAML files containing all records (/spec/fixpoints/[table_name].yml)

it 'posts an item' do
  restore_fixpoint :registered_user
  user = User.find_by(name: 'Hans')
  visit new_item_path(user)
  fill_in 'Item', with: '...'
  click_on 'Post'

  compare_fixpoint(:posted_item, ignore_columns: [:release_date], store_fixpoint_and_fail: true)
  # compares the database state with the previously saved fixpoint and
  # raises if there is a difference. when there is no previous fixpoint,
  # it writes it and fails the test (so it can be re-run)  

I was delighted because:

compare the database state after a test with a prior run
compare the database state after a test with a prior run


Behavior-driven test data can be a helpful tool for large legacy applications without much test coverage2. Instead of spending time on implementing data setup for a test, we can invest time in producing the setup via a behavior test (which has value in its own). The full comparison of database state at the end of a test alerts the developer about problems when changing complex applications. There are lots more advantages (and disadvantages) which are not covered in this post, so there is more to come.

I have not yet released the fixpoint implementation as a Ruby gem. Please drop a comment on hackernews if you are interested.

  1. I later implemented incremental fixpoints, so only the difference between two database states is persisted. This made understanding & investigating changes to data records during tests even easier.

  2. Fixpoints can also be used in a green-field scenarios to avoid data differences/gaps between tests. But later more on this.