How we built formula attributes

We just launched one of the most requested features in Attio’s history - formula attributes.

Building formulas into Attio was an exciting engineering journey, with many careful decisions and complex infrastructure implications along the way.

In this post, we’d like to take a moment to reflect on some of the challenges we tackled on the way - why we decided to build our own formula language from scratch, how its type system works, how we execute formulas at Attio's scale, and how we taught AI to write formulas for you.

Building from scratch

There are many 3rd party formula libraries out there, implementing a broad range of functions, tabular interfaces, and reactivity. You might ask, why not just reach for one of these and call it a day?

As with everything for Attio, we try to look beyond the quick solution and think about how a feature will scale with the product and the company.

Bringing a 3rd party formula library would initially save us some time (perhaps on implementing the plethora of math functions!), but would mean we’d lose the ability to tune the formula language into perfect sync with our Particle data store, define custom language constructs that fit our user’s needs, or execute the formulas in a way that scales well with our backend.

Defining a formula language

Ultimately, we decided to build our own Formula grammar based on Lezer. Lezer is an incremental, error-insensitive, LR parser generator. It’s used and maintained by CodeMirror, which is one of the most ubiquitous code editors on the web.

It allowed us to define our own language, with its own syntax highlighting that seamlessly fits into CodeMirror. In turn, CodeMirror gave us total extensibility and we were able to build a fully fledged code editor with error highlighting, auto-completion, and rich Attio elements like attribute and actor references.

We also built a declarative type system that neatly wraps the formula language and provides first-class support for Particle concepts such as record attributes, historical references or complex types like actor references or phone numbers.

It also gives us useful internal machinery like generics or runtime type inference and coercion. These make it easy to implement new formula functions and ensure that the data flows through without type errors.

[FormulaScriptFunction.AND]: {
    generics: {
        T: [checkbox()],
    },
    parameters: {
        left: "T",
        right: "T",
    } as const,
    returnType: checkbox(),
}

Example of and() function declaration in the Formula type system.

Executing formulas

Compared to other tools, which might preload all necessary data into the browser or server memory for the execution, Attio’s scale requires a much more nuanced approach.

Our datasets contain millions of records, including entire change histories of each attribute and relationship attributes that tie records together.

When recomputing against the entire dataset, formula execution pipeline fans out the executions into batches and utilizes Particle, our foundational data model, to efficiently load all the required data and handle the intricacies of relationship resolution.

For compute-intensive, synchronous operations, we allow formula functions that act on potentially unbounded datasets (for example hasBeenIn()) to be expressed in SQL, which allows us to offload the execution onto Google's Spanner, our primary database, and avoid blocking the execution workers.

Formula execution pipeline is a message-driven, multi-tenant system, where event delivery delays and noisy neighbor issues can easily occur. Moreover, when executing formulas against a dataset, it is desired that each individual execution runs against the state of the data as it was when the workload was triggered. Spanner lets us elegantly maintain consistency by executing every batch against a common transaction read timestamp (ie. execution trigger time).

We orchestrated the execution using Google’s PubSub messaging service. PubSub supports ordered event delivery, which has an interesting property - it guarantees that a single ordering key can be only consumed by a single consumer (execution worker) at a time.

This property can be used to maintain fairness across tenants and prevents the noisy neighbor problem. We use an ordering key based on workspace’s ID to ensure that each tenant is served by a limited slice of the worker pool and capacity is left for others.

Keeping computations up to date

Formula language and execution are the most visible parts of the puzzle, but the real value lies in the reactive nature of formula attributes. As in a spreadsheet, a formula needs to recompute whenever the underlying data changes, to keep its result fresh.

To do this, we completely rebuilt the write side of our Particle data model as a highly performant Rust service, and gave it the ability to emit change events through PubSub as the data gets written. Through a subscription system, formulas listen to these events and trigger immediately as their dependencies change.

Letting AI write formulas for you

Despite the capabilities of the formula editor, writing a complex formula is not an easy task, especially for non-technical users. Here’s where AI comes into play.

We leveraged the infrastructure built for Universal Context to create a formula builder agent aware of your workspace’s context, including the schema of your objects, attribute types, and select & stage options.

Naturally, AIs can make mistakes, especially when working with a custom DSL and a large number of UUIDs, slugs, and other identifiers and tokens found in the object schema. We leveraged our type system to enforce correctness of the output and all the references the formula builder makes, feeding the issues back into the agent and letting it correct itself.

Similarly, this also powers the auto fix feature, making the formula builder agent a helpful assistant when dealing with errors.

What’s next

In this article, we looked at some of the core parts of building formula attributes into Attio. However, the work has just begun.

Formula attributes provide a powerful way to encode predictable logic into Attio and we are going to extend their capabilities by expanding our library of useful functions and bringing formulas to other parts of the product, like workflows. Stay tuned!