JSON dynamic attributes with Rails

12th of April, 2019
rails, json

We had a tricky problem. We knew that the columns for one particular model would need to change over time. That was easily solved with a JSON field, but we also wanted to support validations on this field – validations that are assigned to the model at the time of creation.
Heitor Silva

This was a joint effort between me, Tom Rothe and Dmitry Parshenko. After we did some brainstorming and white board sessions, we decided to start with a Model called Vehicle. It held a schema and a description keeping its characteristics.

The schema

The schema defines attributes, telling us their type and default value. This enables us to have a different set of attributes for each record of the table. Required fields are also being declared in the schema for later validations.

We want every new instance of vehicle to have its schema filled by a default value declared in the database table. Let’s create our Vehicle migration:

class CreateVehicles < ActiveRecord::Migration[5.2]
  def change
    create_table :vehicles do |t|
      t.jsonb :schema, null: false, default: { "required": ["name", "brand"], "attributes": { "name": { "type": "string", "default": "" }, "brand": { "type": "string", "default": "" }, "color": { "type": "string", "default": "" }, "wheels": { "type": "integer", "default": 2 } } }
      t.jsonb :description
      t.timestamp
    end
  end
end

We are following the structures of JSON Schema, but only the ones necessary for our objectives.

Manipulating data

Now, to work with both new instances and existing records of our Vehicle model, we must create our Vehicle class:

class Vehicle < ApplicationRecord
  store_accessor :schema, :required, :attributes
  serialize :description

  after_initialize :define_default_values

  private

  def define_default_values
    return unless self.description.blank?
    self.description = self.schema['attributes'].keys.reduce({}) do |acc, field|
      acc[field] = self.schema['attributes'][field]['default']
      acc
    end
  end
end

The store_accessor defines the getters and setters of the schema. The serialize is required because we are using JSONB (PostgreSQL). For every new instance, we loop over the schema attributes and set the description default values.

Dealing with validations

When we try to save or update an instance of our Vehicle, we loop over the required fields that are listed in the schema:

class Vehicle < ApplicationRecord
  # ...
  before_validation :validate_from_schema

  private

  def validate_from_schema
    self.schema['required'].each do |field|
      errors.add(field, 'is required') if self.description[field].blank?
    end
  end
end

Here we only implement a presence validation, but one can add involved validations.

Presenting our records

To create our forms in the HTML, we use the schema definition and loop over it:

# views/vehicles/new.html.haml
%h1 Vehicle creation
= form_for @vehicle, url: { action: "create" } do |f|
  = f.fields_for :description, OpenStruct.new(vehicle.description) do |df|
    - vehicle.schema['attributes'].each do |field, info|
      %p
        = field
        - if info['type'] == 'boolean'
          = df.check_box field
        - else
          = df.text_field field
        = vehicle.errors[field]
  = f.submit

Notice that we use the OpenStruct to make Rails form able to fill the values of existent records. Without it, we get empty fields always.

Handling queries

Because we are using PostgreSQL and JSONB, we gain the power of queries like this:

# description->sound = 'Pioneer'
Vehicle.where('description @> ?', {sound: 'Pioneer'}.to_json)

# description->fuels = ['Gasoline', 'Etanol']
Vehicle.where("description -> 'fuels' ? :fuel", fuel: 'Gasoline')

# description->alarm AND description->sound
Vehicle.where('description ?& array[:keys]', keys: ['alarm', 'sound'])

# description->automatic OR description->gps
Vehicle.where('description ?| array[:keys]', keys: ['automatic', 'gps'])

# description->color = 'Red' AND description->wheels = 4
Vehicle.where('description @> ?', {color: 'Red', wheels: 4}.to_json)

Remember that with JSONB, we have indexes called GIN / GiST to help with performance.

Conclusion

We are now able to have a relational database delivering multiple records of the same table, each one with different set of attributes. With the schema definition and a little Rails magic, we also enable validations. Old records stay valid and usable if the default value of the schema changes, and querying inside the description is ok due to PostgreSQL power with JSONB fields.

A complete version of this idea can be seen in this GitHub repo. Also note the heitor, tom and dmitry branches, which have different approaches to the same problem. Yet, the master branch combines them into more complete solution.

If you have any questions, feel free to reach out.

P.S.: This code is not production ready. We refrained from refactoring it to a gem/DSL, so the code examples read a little nicer.