feat(backend): add standalone data plane

Replace the Supabase application data plane with the Node/PostgreSQL API, migrate mobile, admin, web, and MCP callers, add controlled cutover tooling, and define hardened Dokploy infrastructure.
This commit is contained in:
jjbrunton committed 2026-08-28 14:42:11 +01:00
1 parent 01bb8225ae
commit 006761d3b0
294 files changed
+16259 -6138

No files matched your search

+8
View File
@@ -0,0 +1,8 @@
# Local development only. Docker Compose already supplies these defaults.
# Do not reuse these credentials for a deployed environment.
# Database name/user/password are deliberately fixed by compose.yaml so shell
# variables from another environment cannot accidentally change this stack.
SAUCI_POSTGRES_PORT=54320
# The API runs on the host during development, outside Docker Compose.
SAUCI_API_PORT=3003
+32
View File
@@ -0,0 +1,32 @@
# Copy these names into Dokploy's environment configuration. Do not commit values.
POSTGRES_DB=sauci
POSTGRES_USER=sauci
POSTGRES_PASSWORD=generate-a-long-random-password
DATABASE_URL=postgresql://sauci:URL_ENCODED_PASSWORD@postgres:5432/sauci
# Hosted Supabase Auth only. The service-role key is used solely by the
# server-side Auth Admin endpoint for account deletion and admin identity
# enrichment. It belongs on the API only, never the worker, MCP, or a client.
SUPABASE_AUTH_URL=https://your-project.supabase.co
SUPABASE_AUTH_AUDIENCE=authenticated
SUPABASE_AUTH_SERVICE_ROLE_KEY=set-a-server-only-auth-admin-key
# Opaque standalone admin API credential. Configure both values together.
# The UUID must identify an active least-privilege admin_users row. MCP receives
# the same token as SAUCI_ADMIN_API_TOKEN, not this user ID and no Supabase key.
ADMIN_API_SERVICE_TOKEN=generate-at-least-32-random-characters
ADMIN_API_SERVICE_USER_ID=set-to-an-active-migrated-admin-profile-uuid
REVENUECAT_API_KEY=set-the-secret-revenuecat-api-key
REVENUECAT_ENTITLEMENT_ID=Sauci Pro
REVENUECAT_WEBHOOK_SECRET=generate-a-long-random-webhook-secret
MEDIA_SIGNING_SECRET=generate-at-least-32-random-characters
MEDIA_PUBLIC_BASE_URL=https://api.example.com
# Operations worker. Discord is optional.
CLASSIFIER_ENABLED=true
OPENROUTER_API_KEY=set-the-secret-openrouter-key
CLASSIFIER_MODEL=openai/gpt-4o
ADMIN_PRIVATE_KEY_JWK=set-the-private-jwk-json-used-for-message-moderation
DISCORD_WEBHOOK_URL=
WORKER_POLL_INTERVAL_MS=30000
WORKER_BATCH_SIZE=25
+115
View File
@@ -0,0 +1,115 @@
# Standalone backend infrastructure
This Compose project is the local data plane for the standalone Sauci API. It
does not run Supabase services. Supabase remains the hosted identity provider;
the API verifies its access tokens and owns all application data access.
Use `scripts/dev-local.sh` rather than invoking Compose directly. PostgreSQL is
bound to loopback and persists in the `sauci-local_sauci-postgres-data` volume.
The committed credentials are deliberately local-only and must not be reused by
a deployed environment.
Private media is stored on the API's `media-data` volume. Mobile clients upload
with bearer authentication, persist opaque `media:<uuid>` references, and ask
the API for short-lived signed read URLs. The public content route accepts only
a valid unexpired HMAC capability; it never exposes directory paths.
## Dokploy deployment contract
`compose.production.yaml` is a deployment-ready baseline containing the Node
API and worker, PostgreSQL, and a one-shot migration service. Configure Dokploy to build the
Compose file from the repository root and route the public API hostname to the
`api` service on port `3003`. Do not publish PostgreSQL.
Copy the variable names from `.env.production.example` into Dokploy's secret
configuration. `POSTGRES_PASSWORD` and the password embedded (URL encoded) in
`DATABASE_URL` must match. Generate them for the deployment; never reuse the
committed local password. The API needs only the hosted Supabase Auth URL to
discover its issuer and JWKS endpoint. The server-only service-role key is used
solely by the hosted Auth Admin deletion endpoint; it must never reach a client
or be used for application data or Storage access. The backend must not receive
a Supabase anon key, database password, or direct Supabase Storage connection.
Administrative automation uses a separate opaque credential pair. Configure
`ADMIN_API_SERVICE_TOKEN` and `ADMIN_API_SERVICE_USER_ID` on the API together;
the UUID must resolve to an active, least-privilege `admin_users` row imported or
created in the standalone database. Configure MCP with the same token under the
different name `SAUCI_ADMIN_API_TOKEN` plus its independent caller-facing
`SAUCI_MCP_API_KEY`. The token does not confer Supabase authority: the API maps
it to the configured administrator and applies normal permissions and auditing.
Do not give either admin token to mobile, admin browser, or worker containers.
Secret ownership is intentionally split:
| Secret | API | Worker | MCP |
|---|:---:|:---:|:---:|
| Hosted Auth service-role key | yes, Auth Admin only | no | no |
| RevenueCat API/webhook secrets | yes | no | no |
| Admin API service token | yes | no | yes, as `SAUCI_ADMIN_API_TOKEN` |
| OpenRouter/Discord worker credentials | no | yes | no |
| PostgreSQL/media signing configuration | yes | yes | no |
On each deployment, `migrate` waits for PostgreSQL, takes a PostgreSQL advisory
lock, applies the committed `apps/api/drizzle` migrations, and exits. The `api`
service starts only after migration succeeds. A failed migration therefore
fails closed instead of starting new application code against an old schema.
The image also exposes the same operation as:
```sh
node dist/db/migrate.js
```
The container runs as the unprivileged `node` user with a read-only filesystem,
dropped Linux capabilities, an init process, and bounded graceful shutdown.
Docker checks `/health/live`; Dokploy should use `/health/ready` for routing so
traffic is admitted only when PostgreSQL is reachable.
Before the first deployment, provide:
1. The API hostname and TLS route.
2. Persistent Dokploy volumes for `postgres-data` and `media-data`, with one
coordinated backup/restore policy covering both.
3. The hosted Supabase project URL and expected JWT audience.
4. Fresh PostgreSQL credentials stored only in Dokploy secrets.
5. A rollback and restore point before any migrated production data is imported.
6. A separately generated `MEDIA_SIGNING_SECRET` (at least 32 characters) and
the HTTPS API origin as `MEDIA_PUBLIC_BASE_URL`.
7. A non-production hosted Auth project and disposable, onboarding-complete test
identities mapped into the staging database for authenticated mobile acceptance.
Blob deletion is retryable. Database cascades and avatar replacement enqueue
the server-generated storage key in `media_deletion_queue`; the operations
worker removes the file and acknowledges the queue row only after a
successful deletion (missing files are treated as already removed). Failed
metadata transactions remove the newly written file immediately. Operators
should alert on an old/growing deletion queue and include it in database
backups; never delete files by constructing paths outside this lifecycle.
The worker also leases notification, moderation, and optional Discord jobs from
`operations_outbox`. Jobs are retried with bounded exponential backoff and are
not selected again after five attempts. Alert on rows that remain unsent after
five attempts; investigate and replay them explicitly rather than deleting the
idempotency record or blindly retrying provider calls.
Validate configuration without starting anything:
```sh
docker compose --env-file infra/backend/.env.production.local \
-f infra/backend/compose.production.yaml config
```
Run the repeatable disposable container check before changing the Dokploy
application. It uses hard-coded test-only credentials, creates a unique Compose
project, verifies migration/startup/security/shutdown behaviour, and removes its
volume on exit:
```sh
infra/backend/verify-production.sh
```
Do not point this Compose project at an existing production database during
local validation. Validate against a disposable local volume first. Production
cutover additionally requires a stopped-source final sync, row/count checks,
legacy-media-reference checks, and authenticated mobile behaviour checks against
the non-production fixture; container health alone is not proof of a successful
migration.
+125
View File
@@ -0,0 +1,125 @@
name: sauci-backend
x-common-environment: &common-environment
NODE_ENV: production
DATABASE_URL: ${DATABASE_URL:?set DATABASE_URL}
MEDIA_ROOT: /data/media
MEDIA_SIGNING_SECRET: ${MEDIA_SIGNING_SECRET:?set MEDIA_SIGNING_SECRET}
MEDIA_PUBLIC_BASE_URL: ${MEDIA_PUBLIC_BASE_URL:?set MEDIA_PUBLIC_BASE_URL}
x-api-environment: &api-environment
<<: *common-environment
PORT: 3003
SUPABASE_AUTH_URL: ${SUPABASE_AUTH_URL:?set SUPABASE_AUTH_URL}
SUPABASE_AUTH_AUDIENCE: ${SUPABASE_AUTH_AUDIENCE:-authenticated}
SUPABASE_AUTH_SERVICE_ROLE_KEY: ${SUPABASE_AUTH_SERVICE_ROLE_KEY:?set SUPABASE_AUTH_SERVICE_ROLE_KEY}
REVENUECAT_API_KEY: ${REVENUECAT_API_KEY:?set REVENUECAT_API_KEY}
REVENUECAT_ENTITLEMENT_ID: ${REVENUECAT_ENTITLEMENT_ID:-Sauci Pro}
REVENUECAT_WEBHOOK_SECRET: ${REVENUECAT_WEBHOOK_SECRET:?set REVENUECAT_WEBHOOK_SECRET}
ADMIN_API_SERVICE_TOKEN: ${ADMIN_API_SERVICE_TOKEN:?set ADMIN_API_SERVICE_TOKEN}
ADMIN_API_SERVICE_USER_ID: ${ADMIN_API_SERVICE_USER_ID:?set ADMIN_API_SERVICE_USER_ID}
ADMIN_PRIVATE_KEY_JWK: ${ADMIN_PRIVATE_KEY_JWK:-}
x-worker-environment: &worker-environment
<<: *common-environment
OPENROUTER_API_KEY: ${OPENROUTER_API_KEY:-}
CLASSIFIER_ENABLED: ${CLASSIFIER_ENABLED:-true}
CLASSIFIER_MODEL: ${CLASSIFIER_MODEL:-openai/gpt-4o}
ADMIN_PRIVATE_KEY_JWK: ${ADMIN_PRIVATE_KEY_JWK:-}
DISCORD_WEBHOOK_URL: ${DISCORD_WEBHOOK_URL:-}
WORKER_POLL_INTERVAL_MS: ${WORKER_POLL_INTERVAL_MS:-30000}
WORKER_BATCH_SIZE: ${WORKER_BATCH_SIZE:-25}
services:
postgres:
image: postgres:17-alpine
environment:
POSTGRES_DB: ${POSTGRES_DB:-sauci}
POSTGRES_USER: ${POSTGRES_USER:-sauci}
POSTGRES_PASSWORD: ${POSTGRES_PASSWORD:?set POSTGRES_PASSWORD}
healthcheck:
test: ["CMD-SHELL", "pg_isready -U $${POSTGRES_USER} -d $${POSTGRES_DB}"]
interval: 5s
timeout: 3s
retries: 20
start_period: 10s
volumes:
- postgres-data:/var/lib/postgresql/data
restart: unless-stopped
stop_grace_period: 60s
security_opt:
- no-new-privileges:true
migrate:
build:
context: ../..
dockerfile: apps/api/Dockerfile
command: ["node", "dist/db/migrate.js"]
environment:
NODE_ENV: production
DATABASE_URL: ${DATABASE_URL:?set DATABASE_URL}
depends_on:
postgres:
condition: service_healthy
restart: "no"
init: true
read_only: true
tmpfs:
- /tmp
cap_drop:
- ALL
security_opt:
- no-new-privileges:true
api:
build:
context: ../..
dockerfile: apps/api/Dockerfile
environment: *api-environment
expose:
- "3003"
depends_on:
migrate:
condition: service_completed_successfully
restart: unless-stopped
init: true
read_only: true
volumes:
- media-data:/data/media
tmpfs:
- /tmp
cap_drop:
- ALL
security_opt:
- no-new-privileges:true
stop_grace_period: 15s
worker:
build:
context: ../..
dockerfile: apps/api/Dockerfile
command: ["node", "dist/workers/index.js"]
environment: *worker-environment
depends_on:
migrate:
condition: service_completed_successfully
restart: unless-stopped
# The image health check probes the HTTP API. This process deliberately has
# no listener; restart-on-exit is its liveness contract.
healthcheck:
disable: true
init: true
read_only: true
volumes:
- media-data:/data/media
tmpfs:
- /tmp
cap_drop:
- ALL
security_opt:
- no-new-privileges:true
stop_grace_period: 30s
volumes:
postgres-data:
media-data:
+23
View File
@@ -0,0 +1,23 @@
name: sauci-local
services:
postgres:
image: postgres:17-alpine
environment:
POSTGRES_DB: sauci
POSTGRES_USER: sauci
POSTGRES_PASSWORD: sauci_local_only
ports:
- "127.0.0.1:${SAUCI_POSTGRES_PORT:-54320}:5432"
healthcheck:
test: ["CMD-SHELL", "pg_isready -U $${POSTGRES_USER} -d $${POSTGRES_DB}"]
interval: 2s
timeout: 3s
retries: 20
start_period: 5s
volumes:
- sauci-postgres-data:/var/lib/postgresql/data
restart: unless-stopped
volumes:
sauci-postgres-data:
+49
View File
@@ -0,0 +1,49 @@
#!/usr/bin/env bash
set -euo pipefail
repo_root="$(cd "$(dirname "${BASH_SOURCE[0]}")/../.." && pwd)"
compose_file="$repo_root/infra/backend/compose.production.yaml"
project_name="sauci-deploy-verify-$$"
# Deliberately fixed disposable values: this check must never consume deployment
# secrets or connect to an externally configured database.
export POSTGRES_PASSWORD='sauci-container-test-only'
export DATABASE_URL='postgresql://sauci:sauci-container-test-only@postgres:5432/sauci'
export SUPABASE_AUTH_URL='https://example.supabase.co'
export SUPABASE_AUTH_AUDIENCE='authenticated'
export SUPABASE_AUTH_SERVICE_ROLE_KEY='container-test-auth-admin-only'
export ADMIN_API_SERVICE_TOKEN='container-test-admin-service-token-123456789'
export ADMIN_API_SERVICE_USER_ID='00000000-0000-4000-8000-000000000002'
export REVENUECAT_API_KEY='container-test-revenuecat-only'
export REVENUECAT_WEBHOOK_SECRET='container-test-webhook-only'
export MEDIA_SIGNING_SECRET='sauci-media-container-test-only-secret'
export MEDIA_PUBLIC_BASE_URL='https://api.sauci.test'
export CLASSIFIER_ENABLED='false'
cleanup() {
docker compose -p "$project_name" -f "$compose_file" down --volumes --remove-orphans >/dev/null 2>&1 || true
}
trap cleanup EXIT
docker compose -p "$project_name" -f "$compose_file" config >/dev/null
docker compose -p "$project_name" -f "$compose_file" up -d --build --wait
docker compose -p "$project_name" -f "$compose_file" exec -T api node -e \
"Promise.all(['/health/live','/health/ready','/v1/me'].map(async p=>[p,(await fetch('http://127.0.0.1:3003'+p)).status])).then(results=>{const expected=[200,200,401];if(results.some((result,index)=>result[1]!==expected[index])){console.error(results);process.exit(1)};console.log(results)})"
docker compose -p "$project_name" -f "$compose_file" run --rm migrate
docker compose -p "$project_name" -f "$compose_file" exec -T api sh -c 'test "$(id -u)" != 0'
docker compose -p "$project_name" -f "$compose_file" exec -T worker sh -c 'test "$(id -u)" != 0'
docker compose -p "$project_name" -f "$compose_file" exec -T api sh -c 'touch /data/media/write-check && rm /data/media/write-check'
if docker compose -p "$project_name" -f "$compose_file" exec -T api sh -c 'touch /app/should-not-write'; then
echo 'API filesystem unexpectedly allowed writes' >&2
exit 1
fi
docker compose -p "$project_name" -f "$compose_file" stop api
docker compose -p "$project_name" -f "$compose_file" logs --no-color api | grep -q 'Received SIGTERM; shutting down'
docker compose -p "$project_name" -f "$compose_file" stop worker
docker compose -p "$project_name" -f "$compose_file" logs --no-color worker | grep -q 'stopping operations worker'
echo 'Production container verification passed'