Skip to content

Monitoring and routine operations

Monitoring should distinguish service availability from cryptographic or operational trust. A healthy HTTP response does not prove that a gateway is honest, that a threshold was independently checked, or that a log is append-only.

Health and metrics

Use /health for a load-balancer health check:

curl -f https://gateway.example.com/health

The gateway exposes /metrics. Configure WITNESS_METRICS_TOKEN and send Authorization: Bearer <token>, or block the endpoint at the reverse proxy. Without either control, metrics are public.

Useful signals include:

  • total attestations and attestations in the last 24 hours;
  • signatures collected per witness;
  • batch and external-anchor counts;
  • witness health;
  • request duration and process uptime.

The currently documented metric names include witness_attestations_total, witness_signatures_collected, witness_batches_total, witness_external_anchors_total, witness_attestations_24h, witness_witness_health, witness_uptime_seconds, and witness_request_duration_seconds.

Watch for threshold failures, witness availability changes, unusual admission volume, anchor failures, database errors, and unexpected authentication failures. Per-IP limits are only defense in depth and should not be treated as an abuse-control perimeter.

Monitor WebSocket connections and route exposure as well. Without WITNESS_WS_AUTH_TOKEN, each new reservation is broadcast immediately with its lowercase digest and timestamp, so an internet-reachable /ws/events can expose submission activity even before an attestation is confirmed.

Common operational symptoms

If startup reports a missing witness auth_token, every witness entry in network.json needs a non-empty token. Do not bypass this check by placing a public signing endpoint in front of an unauthenticated witness.

If SQLite reports that the database is locked, confirm that only one gateway process uses the database file. The gateway is not a multi-writer SQLite cluster.

If timestamp requests have high tail latency, inspect witness health and latency. An unreachable or slow witness can delay threshold collection even though collection stops once the configured threshold is reached.

After a prolonged witness outage, also inspect the age of pending/retryable jobs. Their reservation timestamps do not move, and witnesses reject an old timestamp after max_clock_skew (300 seconds by default). Such a job can no longer succeed merely because witnesses recover; treat it as a terminal operational failure rather than expecting a refreshed timestamp.

If Freebird verification fails, confirm that the configured verifier URL is reachable from the gateway, that the request shape is { "token_b64": "..." }, and that the selected /v1/verify or /v1/check path matches the configured consuming mode. The hardened client rejects private address ranges, so the verifier must be reachable within the deployment's permitted network design.

For external anchors, do not infer the same protection from a successful configuration review: Trillian and DNS use unrestricted reqwest, and Ethereum validates the RPC URL before initialization but does not use the hardened client for subsequent RPC traffic. Review and monitor these paths as an operational risk; provider-specific SSRF handling remains a TODO.

Logs

Set RUST_LOG to control tracing verbosity. Use normal verbosity for routine operation and increase it temporarily for troubleshooting. Review log and reverse-proxy retention because source IPs, timing, User-Agent strings, paths, and application identifiers can be identifying metadata. Never log private keys, bearer tokens, Freebird tokens, or full secret-bearing configuration.

Operational logs and the SQLite database are not automatically tamper-evident. For independent evidence, clients should verify attestations and proof bundles, and auditors should verify signed tree-head and consistency-proof chains.

Incident first actions

If a key, bearer token, API key, database, or deployment configuration may have been exposed:

  1. Restrict the affected service and preserve relevant logs and configuration safely.
  2. Determine which network, witness, route, or external provider was affected.
  3. Check attestations, signed tree heads, consistency proofs, and independent verification for anomalous behavior.
  4. Restore only from a known-good database/configuration backup when required.
  5. Update the reviewed configuration and restart services deliberately.

Witness does not document a universal revoke mechanism. Operators must use the credential and deployment controls available in their own environment and review the resulting network configuration with clients.