Operations

Health

• GET /health/live: the node is up.
• GET /health/ready: the database answers and the node is not draining. Balancers and orchestrators use this one; it turns 503 the moment a drain starts.

Metrics

GET /metrics serves Prometheus text (set METRICS_TOKEN for the scraper; without it only an admin session may read it). A Grafana dashboard is in deploy/grafana/conveyor.json. The series that matter:

Series	Alert when
conveyor_ingest_ack_latency_us (histogram)	p99 above 250 ms for 5 minutes: Bazel's end-of-build wait grows
conveyor_ingest_writer_flush_duration (histogram, ms)	p99 above 500 ms: PostgreSQL slow or contended
conveyor_repo_query_queue_time (histogram, ms)	p99 above 50 ms: pool too small or database saturated
conveyor_ingest_batch_committed_events (counter)	rate drops to zero while streams are open
conveyor_ingest_streams_count, conveyor_ingest_workers_count (gauges)	near MAX_STREAMS_PER_KEY times keys
conveyor_ingest_fenced_count (counter)	any increase that repeats: two nodes wrote one invocation (balancer or lifecycle problem)
conveyor_oban_jobs_count{queue,state}, conveyor_oban_oldest_available_seconds{queue} (gauges)	oldest waiting job older than 10 minutes: the queue is not draining
conveyor_blobs_errors_count{op} (counter)	any increase: blob store credentials, bucket policy or disk
vm_memory_total, vm_system_counts_process_count	memory growth without stream growth

Ready-made rules for these are in deploy/prometheus/alerts.yml.

Stuck jobs

Execution-log parsing, profile fetches and maintenance run as Oban jobs (oban_jobs). A node that dies mid-job leaves it executing; the Lifeline plugin moves it back to available after 15 minutes (or discards it when its attempts are used up). Two rules learned on the AWS trial:

• Rescue with Oban's API, never by editing rows: Oban.retry_all_jobs(query) or Oban.retry_job/1 raise max_attempts as needed. A row set to available by hand with attempt = max_attempts is never fetched again, and looks exactly like a stalled queue.
• Lifeline is naive: a job that legitimately runs longer than 15 minutes is "rescued" while still running, runs twice and burns an attempt each time. Long jobs must finish or fail inside their timeout/1 (the parser's is 10 minutes).

Execution-log parses run two at a time on the parse queue (default carries profile fetches and summaries, maintenance the nightly jobs); conveyor_oban_jobs_count and conveyor_oban_oldest_available_seconds carry a label per queue.

Logs

Structured, one line per event, with the invocation id on every ingest message. Ingest errors are rare by design: a stream failure is UNAVAILABLE (draining), RESOURCE_EXHAUSTED (key limit), UNAUTHENTICATED (bad key) or FAILED_PRECONDITION (out-of-order sequence), all of which Bazel handles by retrying or failing the upload, never the build.

Retention and storage

Nightly at 02:00 UTC builds older than RETENTION_DAYS (or the project's own retention, set in Settings) are deleted in batches per project, execution-log spawns older than RETENTION_SPAWN_DAYS (per project too) are deleted and their builds marked expired on the Actions tab, and the input lists nothing references any more are pruned; hourly the raw segment partitions older than RETENTION_RAW_DAYS are dropped and upcoming partitions created; at 03:30 unpinned blobs older than CAS_TTL_DAYS and blobs no remaining build references (their builds were retained away) are pruned. All three are Oban jobs; failures retry and show in the logs.

Blobs are stored per project under the project's key prefix (<S3_PREFIX>/<prefix>/ in S3, <BLOB_DIR>/<prefix>/ on disk; the prefix is the slug unless changed in Settings), so one project's data can be listed, given a bucket lifecycle rule or removed on its own. Projects never share blobs. Blobs written before 0.2 sit in the old flat layout and stay readable; new writes go under the prefix.

Watch pg_stat_user_tables for dead tuples on invocations (autovacuum runs at 2 %) and the size of the TOAST relation of invocations (options is the large column).

After upgrading to 0.2.4 (input lists moved out of spawns): the migration copies the lists into spawn_inputs and drops the column, but PostgreSQL keeps a dropped column's data until the table is rewritten. Run VACUUM FULL spawns once, at a quiet moment (it locks the table for the rewrite: 6 s for the trial's 1.2 GB table on its smallest instance) to get the space back; until then the table stays its old size.

Every ten minutes SweepStale marks builds disconnected that are still in_progress but whose row has not changed for two idle windows (INGEST_IDLE_TIMEOUT_MS, 10 minutes by default): a node died under them and Bazel gave up. A client that comes back later resumes the build as after an idle timeout.

Splitting a project

Builds land in the project of the API key that sent them. To split an existing project (everything went to default before teams had keys), move its finished builds by search query, then hand each team a key of its new project:

bin/conveyor rpc 'Conveyor.Release.move_builds("default", "grpc", "repo:grpc", "gRPC")'

The target is created when missing. Every blob a moved build references is copied under the target's prefix first (the source copies are pruned as orphans), rollups of the affected hours and both projects' tag facets are recomputed, and the move is recorded in the audit log. Builds still streaming are skipped.

Raw archive

With RAW_ARCHIVE_ENABLED=true, every 15 minutes the ArchiveRaw job enqueues one job per build that finished more than RAW_ARCHIVE_AFTER_HOURS ago (and whose row has not changed for two idle windows) on the archive queue. Each copies the build's event segments into one blob (a zstd-compressed --build_event_binary_file) and its log into another, checks the counts against the row, and points the build at them (invocations.raw_status = 'archived', raw_blob, log_blob). From then on the Events tab, the log viewer and the downloads read from the blob store (S3 in production): raw data lives there after a day, PostgreSQL keeps only the build's rows. The segment rows stay until their daily partition is dropped; the hourly partition job keeps a day that still holds a finished build the archive has not reached, so an outage of the store delays the drop instead of losing events.

• Storage: one object per build for events and one for the log, under the project's prefix, pinned; build retention makes them orphans and the nightly prune removes them. Requests: two PUTs per finished build (about $0.50/day of S3 PUTs at 100k builds/day) plus a GET per Events-tab or log view of an archived build.
• Metrics: conveyor_raw_archived_count, conveyor_raw_archived_bytes, conveyor_raw_archive_failures_count{reason} and conveyor_raw_archive_overdue_seconds (how long the oldest due build has waited past the delay). Alerts: ConveyorRawArchiveBehind (over 6 hours for 30 minutes: store credentials, bucket policy or a failing queue; nothing is lost, the database grows meanwhile) and ConveyorRawArchiveMismatch (a build's segments did not match its row; it is marked skipped and keeps its segments until RETENTION_RAW_DAYS — run Conveyor.Ingest.Verify.check/2 on it).
• Rollout: enable, watch the gauge fall to zero and the failures stay at zero for a day, then lower RETENTION_RAW_DAYS (at least RAW_ARCHIVE_AFTER_HOURS/24 + 1, so 2 with the default delay). Turning it off again is safe: archived builds keep reading from the store, new ones stay in the segments.

Runbooks

Each alert in deploy/prometheus/alerts.yml maps to one of these.

Ack latency high (ConveyorAckLatencyHigh). Bazel waits for acks at the end of every build, so this is what engineers feel. Check conveyor_ingest_writer_flush_duration first: slow flushes mean PostgreSQL (CPU, IOPS, credits on burstable instances, dead tuples on invocations); fast flushes with high ack latency mean the nodes are saturated (vm_total_run_queue_lengths_total, add nodes). Never raise POOL_SIZE past what max_connections allows for all nodes.

Fenced commits (ConveyorFencedCommits). Two nodes wrote the same invocation. A handful after a client reconnected through a balancer is normal: the resend lands on another node, which resumes from the row, and the first node's late batch is fenced. A steady rate without client drops means a balancer sends one build's connections to several nodes, or lifecycle events reach a node that starts work it does not own; check the balancer's stickiness and the version of Conveyor (lifecycle events must never start a worker).

Job backlog (ConveyorJobBacklog). conveyor_oban_jobs_count{state="available"} grows and the oldest waiting job ages: no node is fetching (all nodes down or the queue paused), jobs with attempt = max_attempts left in available by a manual edit (rescue with Oban.retry_all_jobs), or parses slower than their timeout. See "Stuck jobs" above.

Blob store errors (ConveyorBlobErrors). Reads or writes failed against disk or S3: credentials expired, bucket policy changed, disk full. Profiles and test outputs of new builds show as unavailable until fixed; nothing about the build itself is lost.

Node not draining / not ready. /health/ready stays 503 after a deploy: the database is unreachable (connection string, TLS bundle, security group) or migrations failed at boot (docker logs / pod logs show "Could not create schema migrations table" when the database itself is the problem). A node that never finishes draining holds open streams longer than SHUTDOWN_DRAIN_SECONDS; the balancer's deregistration delay must exceed it.

Database full or slow. Retention is the lever. With execution logs uploaded, spawn rows are the bulk (SELECT pg_total_relation_size('spawns')): lower RETENTION_SPAWN_DAYS. Otherwise enable the raw archive and lower RETENTION_RAW_DAYS (raw events and logs), then RETENTION_DAYS per project in Settings; run VACUUM on invocations if dead tuples exceed a few percent. Growth per build is about 27 KB compressed for small builds and scales with targets, actions and log size.

Backups

PostgreSQL is the only state that matters; blobs are derivable (Bazel can re-upload profiles) but cheap to keep. Rehearsed on the AWS trial (2026-09-22): an RDS snapshot of the live db.t3.micro took about 3 minutes, restoring it into a new instance about 9, and aws s3 sync of the blob bucket (825 objects) 16 seconds; a Conveyor release pointed at the restored instance (bin/conveyor eval with Ecto.Migrator.with_repo) reported the same row counts (invocations, spawns, blobs, artifacts, projects, keys) and all 11 migrations applied. A fresh deploy/trial stack takes -var db_snapshot_identifier=…. Standard base backups plus WAL archiving, or the managed service's snapshots, are enough; Conveyor holds no local state beyond the blob directory in disk mode.

Rollouts

SIGTERM starts a drain: no new streams, readiness 503, up to SHUTDOWN_DRAIN_SECONDS for open streams, then exit. Migrations run on boot and take the advisory lock Ecto uses, so several nodes can boot at once. Balancer deregistration delay must exceed the drain.

Runbook

• Ack latency high, database fine: check conveyor_ingest_writer_flush_duration and pool queue time. If flush is slow and Postgres CPU is high, the database is the bottleneck: bigger instance or more nodes will not help; fewer statements per commit would (see scale).
• Builds stuck in progress: the client went away without finishing; the worker marks the build disconnected after INGEST_IDLE_TIMEOUT_MS.
• RESOURCE_EXHAUSTED in Bazel: the key's stream limit on that node; raise MAX_STREAMS_PER_KEY or add nodes.
• Profiles missing: the cache endpoint for the project is not reachable, or Bazel ran without a remote cache; see the build's Timeline tab message and Cache endpoints.
• Node will not join the cluster: RELEASE_COOKIE differs, the distribution port is blocked, or POD_IP/NODE_IP is not routable; bin/conveyor remote then Node.list() to inspect.