Part 7: Observe it

The bank already records a span for every command, event, view and policy. Nothing receives them yet. In this part you send them to Axiom, then add the browser’s spans, so one trace runs from a click to every Durable Object it reaches.

the bank's Worker and Durable Objects ──────────────────────────▶ Axiom
the browser app ──────────────────▶ the bank's Worker (relay) ──▶ Axiom

You need a configured Axiom profile.

Install the Axiom provider

bun add @alchemy.run/axiom@latest

Let the stack create Axiom resources

alchemy.run.ts

import * as Alchemy from "alchemy";
import * as Axiom from "@alchemy.run/axiom";
import * as Cloudflare from "@alchemy.run/cloudflare";
import * as Effect from "effect/Effect";
import * as Layer from "effect/Layer";
import BankWorker from "./src/BankWorker.ts";

export default Alchemy.Stack(
  "Bank",
  { providers: Cloudflare.providers(), state: Cloudflare.state() },
  {
    providers: Layer.mergeAll(Cloudflare.providers(), Axiom.providers()),
    state: Cloudflare.state(),
  },
  Effect.gen(function* () {
    const bank = yield* BankWorker;
    const app = yield* Cloudflare.Website.Foldkit("BankApp", {
      main: "src/web/worker.ts",
      env: { BANK: bank },
      assets: { runWorkerFirst: ["/api", "/api/*"] },
    });
    return { url: bank.url.as<string>(), app: app.url.as<string>() };
  }),
);

Axiom.providers() sits next to Cloudflare.providers(), so the stack can create Axiom datasets and tokens alongside the bank.

Create the datasets and an ingest token

src/Observability.ts

import * as Axiom from "@alchemy.run/axiom";

// Where spans go.
export const Traces = Axiom.Dataset("Traces", {
  name: "bank-traces",
  kind: "otel:traces:v1",
});

// Where logs go.
export const Logs = Axiom.Dataset("Logs", {
  name: "bank-logs",
  kind: "otel:logs:v1",
});

// A token that may write to those two datasets, and nothing else.
export const Ingest = Axiom.ApiToken("Ingest", {
  name: "bank-ingest",
  datasetCapabilities: {
    "bank-traces": { ingest: ["create"] },
    "bank-logs": { ingest: ["create"] },
  },
});

These are three Axiom resources. The stack creates them the first time a Worker uses them.

Note

Dataset names are global to your Axiom org. If you deploy more than one stage, give each its own datasets: see per-stage datasets.

Send the bank’s telemetry to Axiom

src/BankWorker.ts

import * as Axiom from "@alchemy.run/axiom";
import * as Cloudflare from "@alchemy.run/cloudflare";
import * as DomainCloudflare from "@alchemy.run/cloudflare/Domain";
import * as Effect from "effect/Effect";
import { HttpServerRequest } from "effect/http/HttpServerRequest";
import * as Layer from "effect/Layer";
import { Bank, BankHandlers } from "./Bank.model.ts";
import { BankApi } from "./BankApi.api.ts";
import { Ingest, Logs, Traces } from "./Observability.ts";
import { SessionLive, sessionRoute } from "./Session.middleware.ts";

const BankLive = SessionLive.pipe(
// Export every span and log to Axiom.
const TelemetryLive = Axiom.Telemetry({
  token: Ingest,
  traces: Traces,
  logs: Logs,
  serviceName: "bank",
});

const BankLive = Layer.mergeAll(SessionLive, TelemetryLive).pipe(
  Layer.provideMerge(Bank.layer(BankHandlers)),
  Layer.provide(DomainCloudflare.DurableObjects),
);

export default class BankWorker extends Cloudflare.Worker<BankWorker>()(
  "BankWorker",
  { main: import.meta.url },
  Effect.gen(function* () {
    const api = yield* BankApi;
    return {
      fetch: Effect.gen(function* () {
        const request = yield* HttpServerRequest;
        const path = new URL(request.url, "http://localhost").pathname;
        return path === "/api/session" ? yield* sessionRoute : yield* api.fetch;
      }),
    };
  }).pipe(Effect.provide(BankLive)),
) {}

This is the only change the bank needs. Axiom.Telemetry gives the Worker the datasets’ addresses and the token, and the Worker and every Durable Object it hosts export their spans and logs after each response.

Follow a deposit

bun alchemy deploy

Open the app, deposit, and open bank-traces in Axiom. The deposit is one trace:

Domain.send Account.DepositMoney         the bank's Worker received the command
└─ Domain.decide Account.DepositMoney    the account's Durable Object decided and committed
   │   event MoneyDeposited              one span event per committed event
   ├─ Domain.view AccountSummary         the summary view applied the event
   └─ Domain.view Statement              the statement view applied the event

The trace crosses the Worker and three Durable Objects. Each event carries the span that committed it, so the views that react to it continue the same trace.

See how a command ended

Every Domain.decide span records how its command ended in domain.outcome. Your deposit reads accepted. A withdrawal larger than the balance reads InsufficientFunds, the tag of its rejection. A rejection is an answer, not a failure, so the span still ends cleanly; only a defect marks it as failed.

Follow a flow across traces

Registering a customer opens their account through the MainAccountProvisioning policy. The policy runs after the registration commits, so it starts its own trace, linked to the one that caused it.

Every span carries domain.correlation_id, the id of the first command in the flow. Filter on it to see the registration, the policy and the account opening together.

Record the amount on every withdrawal

src/WithdrawMoney.command.ts

import * as Domain from "@alchemy.run/domain";
import * as Schema from "effect/Schema";
import { AccountId } from "./Account.id.ts";
import { NotOpen } from "./NotOpen.rejection.ts";
import { NotOwner } from "./NotOwner.rejection.ts";
import { Session } from "./Session.middleware.ts";

export class MoneyWithdrawn extends Domain.Event("MoneyWithdrawn", {
  amount: Schema.Int,
  balanceAfter: Schema.Int,
}) {}

export class InsufficientFunds extends Domain.Rejection("InsufficientFunds", {
  balance: Schema.Int,
  requested: Schema.Int,
}) {}

export class WithdrawMoney extends Domain.Command("WithdrawMoney", {
  target: AccountId,
  middleware: [Session],
  input: { amount: Schema.Int },
  events: [MoneyWithdrawn],
  rejects: [NotOpen, NotOwner, InsufficientFunds],
  reply: { balance: Schema.Int },
  attributes: (cmd) => ({ "bank.amount": cmd.amount }),
}) {}

Spans record nothing from a command’s input unless you ask. attributes adds bank.amount to every span the withdrawal makes, so you can find large withdrawals in Axiom. Events and rejections accept attributes too, and a Redacted value is always recorded as <redacted>.

Trace the browser

The bank is traced. The trace still starts at the Worker, though, not at the click that caused it. The rest of this part adds the browser’s spans.

The browser can’t send to Axiom itself: that would put the ingest token in every visitor’s hands. Instead it sends its spans to the bank’s Worker, and the Worker forwards them to Axiom with its own token.

Export the sign-in check

src/Session.middleware.ts

const signedIn = (headers: Headers.Headers): Option.Option<string> =>
export const signedIn = (headers: Headers.Headers): Option.Option<string> =>

The Worker will forward spans only from signed-in customers, so a stranger can’t fill your dataset. Export the signedIn check from Part 6 so it can use it.

Forward the browser’s spans to Axiom

src/BankWorker.ts

import * as Axiom from "@alchemy.run/axiom";
import * as Cloudflare from "@alchemy.run/cloudflare";
import * as DomainCloudflare from "@alchemy.run/cloudflare/Domain";
import * as Telemetry from "alchemy/Telemetry";
import * as Effect from "effect/Effect";
import { HttpServerRequest } from "effect/http/HttpServerRequest";
import * as Layer from "effect/Layer";
import * as Option from "effect/Option";
import { Bank, BankHandlers } from "./Bank.model.ts";
import { BankApi } from "./BankApi.api.ts";
import { Ingest, Logs, Traces } from "./Observability.ts";
import { SessionLive, sessionRoute } from "./Session.middleware.ts";
import { SessionLive, sessionRoute, signedIn } from "./Session.middleware.ts";

// Export every span and log to Axiom.
const TelemetryLive = Axiom.Telemetry({
  token: Ingest,
  traces: Traces,
  logs: Logs,
  serviceName: "bank",
});

const BankLive = Layer.mergeAll(SessionLive, TelemetryLive).pipe(
  Layer.provideMerge(Bank.layer(BankHandlers)),
  Layer.provide(DomainCloudflare.DurableObjects),
);

// Accepts spans from the browser and forwards them to Axiom, for signed-in customers only.
const relayBrowserSpans = Telemetry.relay({
  authorize: Effect.map(HttpServerRequest, (request) =>
    Option.isSome(signedIn(request.headers)),
  ),
});

export default class BankWorker extends Cloudflare.Worker<BankWorker>()(
  "BankWorker",
  { main: import.meta.url },
  Effect.gen(function* () {
    const api = yield* BankApi;
    return {
      fetch: Effect.gen(function* () {
        const request = yield* HttpServerRequest;
        const path = new URL(request.url, "http://localhost").pathname;
        return path === "/api/session" ? yield* sessionRoute : yield* api.fetch;
        if (path === "/api/session") return yield* sessionRoute;
        if (path.startsWith("/api/telemetry/")) return yield* relayBrowserSpans;
        return yield* api.fetch;
      }),
    };
  }).pipe(Effect.provide(BankLive)),
) {}

Telemetry.relay is an HTTP handler that receives spans and sends them wherever the Worker’s own telemetry goes, which is Axiom. It lives under /api, so the app’s Worker from Part 6 already forwards it to the bank.

Give the app a tracer

src/web/entry.ts

import * as Domain from "@alchemy.run/domain";
import * as Effect from "effect/Effect";
import * as FetchHttpClient from "effect/http/FetchHttpClient";
import * as HttpClient from "effect/http/HttpClient";
import * as HttpClientRequest from "effect/http/HttpClientRequest";
import * as Layer from "effect/Layer";
import * as OtlpSerialization from "effect/observability/OtlpSerialization";
import * as OtlpTracer from "effect/observability/OtlpTracer";
import * as Socket from "effect/socket/Socket";
import { Runtime } from "foldkit";
import { BankApi } from "../BankApi.api.ts";
import { CustomerId } from "../Customer.id.ts";
import { Flags, init, Model, subscriptions, update, view } from "./Bank.ts";

// A stand-in for a login page: `?customer=c-2` picks who you are.
const customerId = CustomerId.make(new URLSearchParams(location.search).get("customer") ?? "c-1");

// Sign in, so the browser holds the session cookie for every call and watch.
const signIn = Effect.gen(function* () {
  const client = HttpClient.filterStatusOk(yield* HttpClient.HttpClient);
  yield* client.execute(
    HttpClientRequest.post("/api/session").pipe(HttpClientRequest.bodyJsonUnsafe({ customerId })),
  );
});

const BankConnection = Layer.unwrap(
  signIn.pipe(
    Effect.orDie,
    Effect.as(
      BankApi.layerClient.pipe(
        Layer.provide(Domain.protocolWebSocket(`${location.origin}/api`)),
        Layer.provide(Socket.layerWebSocketConstructorGlobal),
      ),
    ),
  ),
).pipe(Layer.provide(FetchHttpClient.layer));
);

// Send the app's spans to the bank's relay, on this origin.
const Tracing = OtlpTracer.layer({
  url: `${location.origin}/api/telemetry/v1/traces`,
  resource: { serviceName: "bank-web" },
}).pipe(Layer.provide(OtlpSerialization.layerJson));

const Resources = Layer.mergeAll(BankConnection, Tracing).pipe(
  Layer.provide(FetchHttpClient.layer),
);

Runtime.run(
  Runtime.makeApplication({
    Model,
    Flags,
    init,
    update,
    view,
    subscriptions,
    resources: BankConnection,
    resources: Resources,
    container: document.getElementById("root"),
  }),
  { flags: Effect.succeed({ customerId }) },
);

Tracing is a standard OpenTelemetry exporter pointed at the relay. foldkit records a span for every command it runs, and every Api call carries the current trace to the bank.

Follow a click

bun alchemy deploy

Deposit again. The trace now starts in the browser:

Deposit                                     the app ran its Deposit command
└─ Domain.send Account.DepositMoney         the bank's Worker received the command
   └─ Domain.decide Account.DepositMoney    the account's Durable Object decided and committed
      │   event MoneyDeposited
      ├─ Domain.view AccountSummary
      └─ Domain.view Statement

One trace shows the click, the command it sent, and everything it changed.

Note

Metrics, logs from handlers, and turning off view spans are in Observability. To get alerted on what you now record, see Monitors & Alerting.