Observability

A model records what it does as it does it. Every command, every event it commits, and every policy, view, projection and saga that reacts to it gets a span, named after the building block. The spans connect into traces that follow a flow from a click to everything it caused.

Export the traces

A hosted model records into whatever tracer the runtime provides. Send it to an OpenTelemetry backend, such as Axiom, Honeycomb or your own collector, to see each flow as one connected trace. The bank declares its Axiom datasets and an ingest token next to its Worker:

return Axiom.Telemetry({ token: ingest, traces, logs, serviceName: "bank" });

and provides that Layer to the Worker, with the rest of what it hosts. Alchemy.Telemetry.layerOtlp does the same for any OTLP endpoint.

Cloudflare.Telemetry() shows a Worker’s spans in Cloudflare’s dashboard instead, with no backend to run. Cloudflare starts a new trace for every Worker and Durable Object event, so a flow that crosses objects shows up as separate traces, joined only by domain.correlation_id, and spans from a browser can’t be added.

Follow a command

A withdrawal sent from the app shows up as one trace, crossing the Worker and two Durable Objects:

Domain.send Account.WithdrawMoney           the Worker, where the Api received it
└─ Domain.decide Account.WithdrawMoney      the account's Durable Object: decide and commit
   │   event MoneyWithdrawn                 one span event per committed event
   ├─ Domain.view AccountSummary
   ├─ Domain.view DailyOutflow
   └─ Domain.policy FraudReview
      ├─ Domain.external FraudCheck.score
      └─ Domain.send Account.FreezeAccount
         └─ Domain.decide Account.FreezeAccount

Each event carries the span that committed it, so the policies and views that react to it continue the same trace, however long the delivery took.

Trace the browser

foldkit traces every Command it runs, and every Api call carries the caller’s trace on, so the browser can start the trace: a click, the commands it sends, and everything they cause on the server. Give the app a tracer that exports to its own origin:

const Tracing = OtlpTracer.layer({
  url: `${location.origin}/api/telemetry/v1/traces`,
  resource: { serviceName: "bank-web" },
}).pipe(Layer.provide(OtlpSerialization.layerJson));

Collect it through your Worker

Telemetry.relay receives the browser’s spans in your own Worker and forwards them to wherever the Worker’s telemetry goes, with the Worker’s credentials. No ingest token reaches a browser, and only signed-in users may send:

const telemetry = Telemetry.relay({
  authorize: Effect.map(HttpServerRequest, (request) =>
    Option.isSome(actorOf(request.headers)),
  ),
});

Mount it under the Api’s path, so the app’s Worker forwards it like any other call:

if (path.startsWith("/api/telemetry/")) return yield* telemetry;

route decides where each batch goes: the Worker’s own destinations, a separate dataset for browser traces, or anywhere else. Cloudflare Workers Observability only records a Worker’s own spans, so send browser spans to an OTLP destination such as Axiom.

See every outcome

A decide span records how the command ended in domain.outcome: accepted, or the tag of its rejection. A rejection is an answer, not a failure, so its span ends cleanly; only a defect marks a span as failed.

Follow a flow across traces

Some work starts later, or has many causes. A saga can run for days, a projection batch handles events from many commits, and a scheduled command fires long after it was scheduled. Each starts its own trace and links to the span that caused it.

Every span carries domain.correlation_id, the id of the first command in the flow. One query on it finds a whole business flow, across every trace it took.

Record what matters to you

Nothing from a command or event is recorded unless you ask for it. Return the attributes you want from attributes, on the declaration:

attributes: (cmd) => ({ "bank.amount": cmd.amount }),

Commands, events and rejections all accept it. Events and rejections take it as a third argument: Domain.Event("MoneyWithdrawn", { amount: Cents }, { attributes: (e) => ({ ... }) }).

Keep secrets out

A Redacted value is recorded as <redacted>, even when you return it from attributes. Make a secret field Schema.Redacted(...) and it never reaches a trace, or a log.

Annotate from a handler

An effectful decide, a policy, a saga step and a projection all run inside their span, so plain Effect annotates it:

WithdrawMoney: Effect.fn(function* (s, cmd) {
  yield* Effect.annotateCurrentSpan("bank.tier", s.tier);
  yield* Effect.log("large withdrawal");
  // ...
}),

Logs from a handler carry the span’s trace and its domain.* attributes, so they line up with the trace that caused them.

Watch it keep up

The model counts and times its work as metrics: domain.commands and domain.command.duration by command and outcome, domain.events by event, domain.lag from commit to handling for each policy and projection, domain.runs by outcome, domain.projection.batch_size, and domain.external.calls. Lag is the number that tells you whether reactions keep up with commits.

Trace less

Views apply every event they fold, so they’re the busiest spans. Turn them off on the Layer that hosts the model:

Bank.layer(BankHandlers, { telemetry: { views: false } })