perf(api): bound worker pools and add telemetry

Share one configured PostgreSQL pool across worker repositories and make API and worker connection budgets explicit.

Add privacy-safe aggregate request, query, pool, sync, and outbox telemetry, a loopback-only representative load fixture, bounded worker shutdown, and focused regression and integration coverage.

Document the database-wide budget formula and defer pg_stat_statements activation to a reviewed migration and planned restart.

infra/backend/.env.production.example

@@ -4,6 +4,13 @@ POSTGRES_USER=sauci
 POSTGRES_PASSWORD=generate-a-long-random-password
 DATABASE_URL=postgresql://sauci:URL_ENCODED_PASSWORD@postgres:5432/sauci
 
+# Keep this total below PostgreSQL max_connections:
+# (API replicas * API_DATABASE_POOL_MAX) + (worker replicas * WORKER_DATABASE_POOL_MAX) + migration/admin headroom.
+API_DATABASE_POOL_MAX=10
+WORKER_DATABASE_POOL_MAX=4
+DATABASE_POOL_IDLE_TIMEOUT_MS=30000
+DATABASE_POOL_CONNECTION_TIMEOUT_MS=10000
+
 # Hosted Supabase Auth only. The service-role key is used solely by the
 # server-side Auth Admin endpoint for account deletion and admin identity
 # enrichment. It belongs on the API only, never the worker, MCP, or a client.

infra/backend/README.md

@@ -111,6 +111,53 @@ docker compose --env-file infra/backend/.env.production.local \
   -f infra/backend/compose.production.yaml config
 ```
 
+## PostgreSQL connection and telemetry contract
+
+The API and worker each create exactly one process-owned PostgreSQL pool and
+hand it to every repository. They have separate deployment variables:
+`API_DATABASE_POOL_MAX` (default 10) and `WORKER_DATABASE_POOL_MAX` (default
+4). Set PostgreSQL `max_connections` above:
+
+```text
+(API replicas * API_DATABASE_POOL_MAX) + (worker replicas * WORKER_DATABASE_POOL_MAX) + migration/admin headroom
+```
+
+The headroom covers the one-shot migrator, an operator session, and any
+backup/monitoring connection; allocating the full server limit to application
+pools is unsafe. `DATABASE_POOL_IDLE_TIMEOUT_MS` and
+`DATABASE_POOL_CONNECTION_TIMEOUT_MS` apply to both process pools.
+
+API and worker write privacy-safe JSON telemetry to normal logs. Fixed event
+types are request, query, pool, sync, and outbox; fields are route templates,
+method/status/outcome, durations, counts, and pool/outbox sizes. They never
+include SQL, query values, raw paths, identities, tokens, content, dedupe keys,
+or request bodies. There is deliberately no public `/metrics` endpoint.
+Request, query, and sync events are aggregated into one fixed-interval JSON
+record per label set (and worker flushes after its tick); pool state is sampled
+once per minute rather than per request. This prevents telemetry itself from
+becoming a connection or log-volume amplifier. Outbox observation runs after
+work with a 250ms best-effort deadline and uses the same due/unlocked predicate
+as claims, so a slow observation cannot delay the next lease.
+
+`pg_stat_statements` is useful for aggregate query investigation but is not
+enabled by this repository change. A database operator must first confirm the
+PostgreSQL image/provider supports it, add `shared_preload_libraries =
+'pg_stat_statements'`, and restart PostgreSQL in a planned window. After that,
+the extension itself must be introduced by a reviewed, committed migration in
+`apps/api/drizzle/`; do not run ad hoc provider/operator DDL. This changes
+server startup and remains deferred from ordinary API deployments.
+
+For a controlled local observation, `npm run load:representative -w @sauci/api`
+performs five authenticated sync reads, five authenticated chat reads, and a
+read-only outbox-state query. It requires `DATABASE_URL`,
+`SAUCI_LOAD_AUTHORIZATION`, `SAUCI_LOAD_USER_ID`, and `SAUCI_LOAD_MATCH_ID`.
+It accepts only `http` loopback API port 3003 (or the explicit local
+`SAUCI_LOAD_API_PORT`) and loopback PostgreSQL, verifies `/health/live` and the
+authenticated fixture identity before load, and emits counts plus p50/p95/p99
+summaries without any pass/fail latency threshold. Use disposable local or
+designated non-production Auth credentials only; never pass a staging or
+production token to this fixture.
+
 Run the repeatable disposable container check before changing the Dokploy
 application. It uses hard-coded test-only credentials, creates a unique Compose
 project, verifies migration/startup/security/shutdown behaviour, and removes its

infra/backend/compose.production.yaml

@@ -8,6 +8,9 @@ x-common-environment: &common-environment
 x-api-environment: &api-environment
   <<: *common-environment
   PORT: 3003
+  DATABASE_POOL_MAX: ${API_DATABASE_POOL_MAX:-10}
+  DATABASE_POOL_IDLE_TIMEOUT_MS: ${DATABASE_POOL_IDLE_TIMEOUT_MS:-30000}
+  DATABASE_POOL_CONNECTION_TIMEOUT_MS: ${DATABASE_POOL_CONNECTION_TIMEOUT_MS:-10000}
   SUPABASE_AUTH_URL: ${SUPABASE_AUTH_URL:?set SUPABASE_AUTH_URL}
   SUPABASE_AUTH_AUDIENCE: ${SUPABASE_AUTH_AUDIENCE:-authenticated}
   SUPABASE_AUTH_SERVICE_ROLE_KEY: ${SUPABASE_AUTH_SERVICE_ROLE_KEY:-}
@@ -20,6 +23,9 @@ x-api-environment: &api-environment
 
 x-worker-environment: &worker-environment
   <<: *common-environment
+  DATABASE_POOL_MAX: ${WORKER_DATABASE_POOL_MAX:-4}
+  DATABASE_POOL_IDLE_TIMEOUT_MS: ${DATABASE_POOL_IDLE_TIMEOUT_MS:-30000}
+  DATABASE_POOL_CONNECTION_TIMEOUT_MS: ${DATABASE_POOL_CONNECTION_TIMEOUT_MS:-10000}
   OPENROUTER_API_KEY: ${OPENROUTER_API_KEY:-}
   CLASSIFIER_ENABLED: ${CLASSIFIER_ENABLED:-true}
   CLASSIFIER_MODEL: ${CLASSIFIER_MODEL:-openai/gpt-4o}