RedditBot/docs/database.md

Data model and persistence

Storage model

The application uses one better-sqlite3 database, normally data/reddit-bot.db, in WAL mode with foreign keys enabled. The persistent Docker volume also holds browser profiles and downloaded media, so a database-only backup is not a complete account/content backup.

src/database/client.ts is the schema authority. Migrations are ordered functions guarded by SQLite PRAGMA user_version and run inside transactions. The repository does not use an ORM.

Ownership categories

Account registry and account content

• reddit_accounts: verified Reddit/Matrix identity, encrypted-at-rest only to the degree provided by the host/volume, cookie JSON, Matrix credentials, active/status timestamps.
• reddit_conversations: account-scoped cached Matrix conversation payloads.
• reddit_account_sync_state: account-scoped sync cursor, full/delta timestamps, and invitation cooldown.
• reddit_media: account-scoped private-media metadata and relative file path.

The special legacy-unassigned account exists only to preserve rows created before account ownership was introduced. New runtime data must use a verified account ID.

Outreach and outcomes

• processed_posts: account/post key, source metadata, analysis, generated opener, send marker, and terminal error/rejection reason.
• sent_messages: account-owned opener/reply attribution, strategy/experiment/model metadata, Matrix room, response/image flags, and source type.
• successful_messages: examples derived from positive outcomes for prompt context.
• conversation_outcomes: normalized outcome for a sent message (engaged, declined, boundary, upsell, or unknown).
• activity_log: operator-facing events with optional recipient/post context.
• reddit_users: global mapping between Reddit fullname/username and Matrix identity.
• recipient_suppressions: process-global recipient reservation/contact record used to prevent cross-account duplicates.

Durable work and idempotency

• candidate_queue: global candidate snapshot, source/priority, state, prepared payload, worker lease, retry timing, selected account, and rejection/error details.
• delivery_outbox: account-scoped prepared delivery and idempotency status retained for legacy/recovery paths.
• conversation_states: account/conversation policy and derived state JSON.
• conversation_automation_claims: account/conversation/inbound-event claim with intended reply and pending/sent state.
• runtime_state: small process controls such as whether bot work is running.

Configuration and telemetry

• app_config: one secret-free validated application configuration record.
• ai_usage: requested/actual model, token/cost, latency, success/error, purpose, and account attribution.
• legacy_content_imports: records one-time imports from legacy content stores.

Critical invariants

1. Account-produced rows carry account_id; cross-account views must opt into global queries explicitly.
2. Recipient identity comparisons are case-insensitive and global for outreach suppression.
3. A sent message is persisted only after remote confirmation.
4. Candidate and automation claims are lease/claim based; a worker must own the expected lease before it changes state.
5. Conversation policy is snapshotted in state JSON when automation starts for that conversation.
6. Secrets and the database path are not stored in app_config.
7. Private media paths in SQLite are relative to the persistent data root; HTTP access remains authenticated.

Repository map

src/database/repositories.ts contains the domain repositories for posts/messages/outcomes/activity/outbox/state/claims/users/recipient history. account-repositories.ts owns account records and cached account content. Candidate and AI telemetry have dedicated repositories.

Prefer narrow repository methods such as claimReady, markSent, or getByChatRoomId to exposing SQL or generic CRUD. This keeps transaction and ownership rules near the schema.

Candidate leases

Discovery inserts candidates without overwriting terminal work. Screening and dispatch use a worker ID plus lease expiry. On startup all interrupted leases can be recovered; during operation expired leases are recovered and due retries promoted. A transient account failure belongs to the selected account, so delivery can be made visible quickly for another eligible runtime without losing the prepared candidate.

Any new candidate state needs:

• an allowed predecessor and terminal/retry semantics;
• a lease ownership rule;
• recovery behavior after process death;
• counts/reasons for the operator funnel;
• tests for stale lease, wrong owner, restart, and duplicate post/author cases.

Conversation reply claims

The unique identity is account + conversation + inbound message key. The claim persists the intended reply before the remote send. A pending claim is not blindly retried: Matrix history is checked first. This closes the process-crash window between remote acceptance and local completion.

Changes to this flow must prove at-most-once observable replies across retry and restart tests.

Configuration persistence

On an empty database, config.yaml is parsed, environment-owned values are injected, the result is validated, and the application-owned subset is saved to app_config. Later starts load SQLite first. Dashboard changes update that record.

The persisted record excludes:

• OPENROUTER_API_KEY;
• dashboard/agent tokens;
• DATABASE_PATH.

Those values are supplied by the environment on every load.

Adding a migration

1. Inspect the final schema and all repository assumptions.
2. Append one migration function; never reorder or modify an already-released migration.
3. Make the operation transactional and compatible with existing data.
4. Use addColumnIfNotExists only where repeatability is appropriate; table rebuilds need explicit foreign-key handling and foreign_key_check.
5. Update repository types and operations.
6. Add tests for a fresh in-memory database and, when relevant, an older schema upgraded in place.
7. Update this document and affected API/contracts.

Do not edit a live SQLite file by hand. Back it up, stop writers when required, apply code-owned migrations, and verify integrity_check, foreign_key_check, row counts, and application reads.

Backup and recovery

For a running WAL database, use SQLite's online backup mechanism or stop the container before copying the database plus -wal/-shm state. A full production backup should include the entire persistent volume because sessions and media live beside the database. Validate that the backup can be opened and contains expected tables before any destructive reset.