Operations

Health endpoints

Route	Checks	Use for
`/healthz`	Process is up	Liveness probe
`/readyz`	DB reachable + engine reachable + max engine `seq`	Readiness probe, on-call triage
`/metrics`	Prometheus	Dashboards, alerting

curl -s http://localhost:4000/readyz | jq

{
  "ok": true,
  "db": "ok",
  "engine": "ok",
  "engineMaxSeq": 918342
}

/readyz is the endpoint that distinguishes "no positions" from "no database". Without DATABASE_URL, reads return empty lists happily — /readyz is where the degradation is visible.

The metric that matters

bff_engine_max_seq — the highest engine WAL sequence the BFF has observed.

Compare it against the indexer's own high-water mark. The gap is indexer lag:

Gap	Meaning
~0	Healthy
Growing steadily	Indexer falling behind — reads are stale, live frames are fine
Frozen while trades happen	Indexer stalled, or the SSE bridge dropped

Live WebSocket frames come straight from SSE, so live data can be correct while REST reads are stale. A user seeing a fresh order book but an empty position list is this, not a UI bug.

Config reference

Var	Effect
`DATABASE_URL`	Postgres (read-only). Unset → empty reads
`ENGINE_BASE_URL`	Default `http://localhost:8080`
`ENGINE_ADMIN_TOKEN`	Bearer for the SSE firehose. Unset → no live frames
`WS_ORDERBOOK_POLL_MS`	Orderbook snapshot poll interval
`FAUCET_ENABLED`	Enables `POST /v1/users/:party/faucet`
`AUTH_DISABLED`	`true` makes `authenticate` / `requireOwnParty` no-ops

ENGINE_ADMIN_TOKEN must never reach a browser. It grants the full engine admin firehose. The whole reason the BFF fans out over WebSocket instead of letting clients hit SSE directly is to keep this token server-side.

Staging monitoring

Staging is watched by a Cloudflare Worker uptime monitor plus a status agent on the server, alerting into Discord.

Worker secrets vs plain text vars. The uptime monitor's credentials must be configured as secrets, not as plain-text environment variables. Plain-text vars are readable from the dashboard and get captured in deploy logs. This has been a real footgun on this project — check it after any Worker redeploy.

Local Postgres

The server database user is titanadmin on port 5433 (not the default 5432).

The api's DATABASE_URL must match that user and port. A mismatch does not error loudly — it produces empty candle and market data, which reads as "the market is quiet" rather than "the connection is wrong".

DATABASE_URL='postgres://titanadmin:<password>@localhost:5433/titan'

Deploy checklist

Apply indexer migrations first. Confirm 0014_tpsl_v2 and 0015_fired_limit are present.

Confirm the engine is at or above c879f7f (the market=all floor).

Deploy the api. Hit /readyz and confirm db: ok, engine: ok.

Grep the logs for ordersForUser: live open-orders fetch failed — its absence is the check that the engine version floor is actually satisfied.

Deploy the interface last.