Operations
Queues, pools, readiness, and the env knobs that matter in v0.1
A short operator guide for deploys that survive more than one video and one replica.
Valkey / Redis
BullMQ stores job state in Redis. Set maxmemory-policy noeviction. LRU eviction can silently drop queue keys and look like random job loss.
- Self-contained Valkey is configured correctly out of the box.
- Managed Redis often defaults to an eviction policy; change it in the provider console.
- If the provider blocks
CONFIG GET, setVALKEY_SKIP_EVICTION_CHECK=trueonly after you have confirmednoevictionyourself. Phase 1/readystill fails when the policy is readable and wrong.
Worker concurrency
| Variable | Default | Meaning |
|---|---|---|
PREP_WORKER_CONCURRENCY | 4 | ffmpeg / S3 prep jobs |
API_WORKER_CONCURRENCY | 2 | embed + Whisper jobs |
ENQUEUE_CONCURRENCY | 16 | Max concurrent BullMQ adds + DB writes when fan-out enqueueing one file |
Raise prep without raising API concurrency one-for-one. If you see gateway 429s, lower API_WORKER_CONCURRENCY.
Job retries
Prep and API jobs use exponential backoff and a max age from enqueue time. After max attempts or max age, the job fails permanently.
| Prefix | Typical knobs |
|---|---|
PREP_JOB_* | ATTEMPTS, BACKOFF_MS, MAX_AGE_MS |
API_JOB_* | same |
Defaults live in the deploy .env.example files.
Database pool
DB_POOL_MAX (default 10) is the max connections per API / worker / search process. Keep:
replicas × DB_POOL_MAX < Postgres max_connectionsManaged plans are often tight; size pools before scaling workers.
Readiness
| Service | /ready roughly checks |
|---|---|
| indexer-api | AI_GATEWAY_API_KEY, schema, S3, Valkey noeviction, both queues |
| search-api | AI_GATEWAY_API_KEY, schema |
indexer-worker has no /ready; it exits on startup if AI_GATEWAY_API_KEY is missing.
/health is liveness. Application containers do not run migrations on boot; use db-push / docker-compose.init.yml.
Pipeline safety
- Only one active parent task per file per modality. A concurrent second start returns 409.
- Workers shut down on
SIGTERM/SIGINTby closing prep and API workers (in-flight jobs finish, then exit). - Re-process is a normal action; pipelines are written to tolerate re-runs without unique-index blowups looking like product bugs.
Costs
Each path records usage when the gateway returns cost metadata. Missing metadata fails clearly rather than inventing numbers. Use the per-modality and hybrid costs screens in the admin UI for A/B on the same fixture video.
Admin UI playback
Presigned playback links use S3_PUBLIC_ENDPOINT. That URL must be reachable from the operator's browser.