History at scale

When a deploy adds a view or a projection, or bumps a version, the new consumer receives the history recorded before it. On Cloudflare the deploy does that work and waits for it. This page explains how, how long it takes, measured on Cloudflare, and what to keep in mind as a Domain grows.

The deploy waits for the history

Alongside your Worker, the Domain declares a backfill run for every consumer the model has. A deploy that adds or rebuilds one starts the run and waits until every new consumer has its history:

[DomainBackfillRun] creating
[DomainBackfillRun] created — Backfilling view:Deposits, view:Timeline (8s) ... (9.7s)

When the deploy finishes, history is already complete. A deploy that changes no consumer waits for nothing. alchemy dev has nothing to wait for: the local Domain backfills itself on its first call.

How the work spreads out

Every instance lists itself in a directory on its first commit: a tree of Durable Objects, one subtree per aggregate and view, where each node lists up to 256 instances and splits into 16 children when it fills. The directory is only an index. A backfill reads it and does its work elsewhere.

The work belongs to the run, and every part of the run is an object of a class the deploy adds. A new view’s class exists only in the code that adds it, and a rebuilt view gets a new class too (DepositsViewV1), so none of these objects can start on the code from before the deploy while Cloudflare rolls it out.

The run’s root lists the directory into a few hundred subtrees and hands them to relays. Each relay splits its share six ways, the most calls one Durable Object makes at once, so a million instances are reached in about eight levels of calls. At the bottom, each instance’s history goes to its keys:

• A view keyed by its aggregate, like a summary per account, has one source per key. The key reads its account’s stored events, routes them with its own code and applies them.
• Any other consumer, like a timeline every account feeds, gets a scanner per instance. The scanner reads the instance’s events, routes them with the consumer’s code and pushes each key its part. Once every scanner has pushed, the run tells the keys to apply what they have, in the order it was recorded.

An instance hands over its stored events whatever code it runs, so the deploy never waits for Cloudflare to move existing instances onto the new code. Switching an instance to live delivery does need its new code; until then a commit holds back the new consumer’s events, and the reader picks them up from the log when it checks back.

Deploy safely

A backfill is a run on Cloudflare, not a step on your machine. Each consumer has one run per generation, held by one Durable Object, and the first deploy to reach it starts it; a deploy that reaches it later finds it started. The deploy starts the runs its model needs, waits for them, and reports how far they have got. It holds none of their state, so you can repeat a deploy, let it time out or stop it, and nothing is lost or done twice.

While it waits, the deploy reports each run’s phase and how far through the directory it is:

[DomainBackfillRun] Backfilling view:Deposits reading history 41% (1,690/4,096 subtrees, 410,211 instances) (48s) ...
[DomainBackfillRun] Backfilled view:Deposits (112.4s)

A consumer keyed by its source reads its history in one pass. Any other reads it, then applies it once every source’s history is in. refused counts calls Cloudflare turned down for the account’s load; the run sends them again, more slowly.

The deploy waits 30 minutes at most. Set backfillTimeout to change that:

const BankLive = Bank.layer(BankPolicies).pipe(
  Layer.provide(DomainCloudflare.layerDurableObjects({ backfillTimeout: "2 hours" })),
);

Past it, the deploy fails with BackfillStillRunning, naming how far each run got, and the runs keep going on Cloudflare. Deploy again and it waits on the same runs from where they are, or returns at once if they finished meanwhile. Stopping a deploy with Ctrl-C works the same way.

A run reads the sources its consumer read when the run started. If you change what a view reads, say it now reads a journal as well as accounts, its history would miss the new source unless you bump its version, so the deploy fails and says so:

view:Timeline now reads Account, Journal, but its backfill was started reading Account. Bump its version to rebuild it from every source's history.

To run a backfill again, bump the consumer’s version. A view keyed by the one aggregate it reads is rebuilt in place instead, with no backfill. For any other consumer the new generation is a new class with its own run, and the same Worker upload deletes the previous generation, its data and its run. Until the deploy returns, reads of the view see the new generation fill in. To stop a backfill, remove the consumer and deploy: its classes, and the run with them, are deleted. Either way it is one upload, so a consumer never has two runs at once.

Width

We measured it on Cloudflare: accounts with 10 deposits each, then a deploy that rebuilds a view per account.

Accounts	Events	The deploy waited
10,000	100,000	13 seconds
100,000	1,000,000	58 seconds
1,000,000	10,000,000	8 minutes

Past about a hundred thousand accounts, the time grows with the number of accounts. That isn’t the tree: a backfill creates one object per key, and Cloudflare caps how fast an account can create and call Durable Objects. Past the cap, calls fail with Your account is generating too much load on Durable Objects. The run paces itself to the cap: it sends more while the account keeps up, sends half as much when Cloudflare refuses calls, and sends the refused work again later. The million-account run held about 2,000 keys a second. If your Domain needs more, Cloudflare can raise an account’s limits.

A long history behind one key

Each key applies its history in order, so a key’s time grows with its own history, whatever the width of the rest of the Domain. A key reads its source’s events a thousand at a time.

A bank-wide total has the longest history of all. It’s one value, like the total deposited across every account, that every account’s events add to, so its one key applies every event in the Domain. It works, but in a large Domain give it a key per day or per region and add the parts up when you read it.

What it costs

A backfill is billed like any Durable Object work: requests, rows written and read, and duration. Each key costs a few requests, plus a row written for each event it applies, and recording the events in the first place cost about as much. Each consumer you add or rebuild in one deploy is its own backfill: adding two views costs twice as much.

A rebuild from the history costs a full backfill, and so does every view and projection that reads the rebuilt view. A view keyed by the one aggregate it reads costs less: it’s rebuilt in place, each key reading only its own account’s events, and it can fill lazily when it is added.

Run a large Domain

Every aggregate instance, view key and projection key is its own Durable Object, and Cloudflare scales them out without limit. What limits a large Domain is how much lands on one of them, how fast the account as a whole goes, and a few platform rules worth knowing before you hit them.

One object handles about 1,000 requests a second. A key that every instance feeds, like a global feed or a counter, tops out there, live as well as during a backfill. Partition it: key it by day, by region or by a hash of the source, and read the parts together.

One object stores up to 10 GB. An aggregate instance keeps its whole event history, so give long-lived streams of events their own instances (an account per period, a ledger per month) rather than one instance that grows forever.

One request runs at most 6 calls at a time. A request that fans out to many instances, say a command sent to every member of a group, runs 6 at a time however many there are. Spread wide fan-out over many requests, or hand it to a saga or a workflow. The backfill fans out through its relays for the same reason.

An account creates and calls Durable Objects at a capped rate. A backfill or a bulk import of millions of instances runs at that rate, and backs off when Cloudflare says so.

An object lives where it was first created. Cloudflare places each Durable Object near the request that created it, and it never moves. A Domain that grows from your users’ traffic spreads across the locations they’re in; a bulk import from one machine puts everything it creates in that one location, so import from a Worker near your users.

Deploys roll out over minutes. During a deploy, some objects run the new code while others still run the old. The backfill doesn’t wait for them, but your own code should stay compatible across one deploy.