jjbrunton/RedditBot
1
0 Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

docs: add agent codebase guide

Browse Source
This commit is contained in:
jjbrunton committed 2026-08-09 16:21:57 +01:00
1 parent 742c06d0c4
commit c630303b47
12 files changed
+881 -4483
Download Patch File Download Diff File Expand all files Collapse all files

No files matched your search

docs/database.md
+69 -515
View File
@@ -1,550 +1,104 @@
# Database Schema
# Data model and persistence
Complete reference for the SQLite database schema, migrations, and query patterns.
## Storage model
---
The application uses one `better-sqlite3` database, normally `data/reddit-bot.db`, in WAL mode with foreign keys enabled. The persistent Docker volume also holds browser profiles and downloaded media, so a database-only backup is not a complete account/content backup.
## Table of Contents
`src/database/client.ts` is the schema authority. Migrations are ordered functions guarded by SQLite `PRAGMA user_version` and run inside transactions. The repository does not use an ORM.
1. [Overview](#overview)
2. [Database File](#database-file)
3. [Schema](#schema)
4. [Indexes](#indexes)
5. [Migrations](#migrations)
6. [Query Patterns](#query-patterns)
7. [Repository Methods](#repository-methods)
8. [Data Retention](#data-retention)
---
## Overview
RedditBot uses SQLite (via `better-sqlite3`) for persistent storage. The database tracks processed posts, sent messages, and successful message/response pairs.
**Features**:
- WAL mode for better concurrent read performance
- Foreign key constraints enabled
- Progressive migrations for schema evolution
- Repository pattern for data access
---
## Database File
**Default path**: `./data/reddit-bot.db`
Configured in `config.yaml`:
```yaml
database:
path: "./data/reddit-bot.db"
```
The database file is auto-created on first run.
---
## Schema
### Table: `processed_posts`
Tracks all posts that have been discovered and processed.
```sql
CREATE TABLE processed_posts (
id TEXT PRIMARY KEY, -- Reddit post ID
subreddit TEXT NOT NULL, -- Source subreddit
author TEXT NOT NULL, -- Post author username
title TEXT, -- Post title
image_url TEXT, -- Original image URL
image_analysis TEXT, -- AI-generated JSON description
generated_message TEXT, -- AI-generated DM text
processed_at INTEGER NOT NULL, -- Unix timestamp
message_sent INTEGER DEFAULT 0, -- 0 = not sent, 1 = sent
error TEXT -- Error message if processing failed
);
```
**Column Details**:
| Column | Type | Constraints | Description |
|---|---|---|---|
| `id` | TEXT | PRIMARY KEY | Reddit post ID (e.g., "abc123") |
| `subreddit` | TEXT | NOT NULL | Source subreddit name |
| `author` | TEXT | NOT NULL | Post author username |
| `title` | TEXT | | Post title (nullable if unavailable) |
| `image_url` | TEXT | | Original image URL |
| `image_analysis` | TEXT | | JSON string from AI analysis |
| `generated_message` | TEXT | | The DM text that was generated |
| `processed_at` | INTEGER | NOT NULL | Unix timestamp when processed |
| `message_sent` | INTEGER | DEFAULT 0 | 0 = not sent, 1 = successfully sent |
| `error` | TEXT | | Error message if processing failed |
**Example Row**:
```json
{
"id": "abc123",
"subreddit": "gonewild",
"author": "user123",
"title": "Summer vibes",
"image_url": "https://i.redd.it/abc123.jpg",
"image_analysis": "{\"gender\":\"F\",\"description\":\"Young woman...\"}",
"generated_message": "Hey, you look amazing...",
"processed_at": 1716234567,
"message_sent": 1,
"error": null
}
```
---
### Table: `sent_messages`
Tracks all DMs sent by the bot.
```sql
CREATE TABLE sent_messages (
id INTEGER PRIMARY KEY AUTOINCREMENT,
post_id TEXT NOT NULL,
recipient TEXT NOT NULL,
subject TEXT DEFAULT 'Stunning',
message TEXT NOT NULL,
sent_at INTEGER NOT NULL,
strategy_id TEXT,
got_response INTEGER DEFAULT 0,
response_checked_at INTEGER,
FOREIGN KEY (post_id) REFERENCES processed_posts(id)
);
```
**Column Details**:
| Column | Type | Constraints | Description |
|---|---|---|---|
| `id` | INTEGER | PRIMARY KEY AUTOINCREMENT | Internal ID |
| `post_id` | TEXT | NOT NULL, FK | Source post ID |
| `recipient` | TEXT | NOT NULL | Recipient username |
| `subject` | TEXT | DEFAULT 'Stunning' | DM subject line |
| `message` | TEXT | NOT NULL | DM body text |
| `sent_at` | INTEGER | NOT NULL | Unix timestamp when sent |
| `strategy_id` | TEXT | | A/B test variant name |
| `got_response` | INTEGER | DEFAULT 0 | 0 = no response, 1 = responded |
| `response_checked_at` | INTEGER | | Timestamp of last response check |
**Example Row**:
```json
{
"id": 1,
"post_id": "abc123",
"recipient": "user123",
"subject": "Stunning",
"message": "Hey, you look amazing...",
"sent_at": 1716234567,
"strategy_id": "compliment",
"got_response": 1,
"response_checked_at": 1716238167
}
```
---
### Table: `successful_messages`
Stores successful message/response pairs for few-shot learning.
```sql
CREATE TABLE successful_messages (
id INTEGER PRIMARY KEY AUTOINCREMENT,
sent_message_id INTEGER,
original_message TEXT NOT NULL,
image_analysis TEXT,
post_title TEXT,
response_snippet TEXT,
subreddit TEXT,
created_at INTEGER NOT NULL,
FOREIGN KEY (sent_message_id) REFERENCES sent_messages(id)
);
```
**Column Details**:
| Column | Type | Constraints | Description |
|---|---|---|---|
| `id` | INTEGER | PRIMARY KEY AUTOINCREMENT | Internal ID |
| `sent_message_id` | INTEGER | FK, nullable | Related sent message ID |
| `original_message` | TEXT | NOT NULL | The DM that got a response |
| `image_analysis` | TEXT | | Context from image analysis |
| `post_title` | TEXT | | Original post title |
| `response_snippet` | TEXT | | First reply text |
| `subreddit` | TEXT | | For weighted example selection |
| `created_at` | INTEGER | NOT NULL | Unix timestamp |
**Example Row**:
```json
{
"id": 1,
"sent_message_id": 1,
"original_message": "Hey, you look amazing...",
"image_analysis": "{\"gender\":\"F\",\"description\":\"...\"}",
"post_title": "Summer vibes",
"response_snippet": "Thank you so much! 😊",
"subreddit": "gonewild",
"created_at": 1716238167
}
```
---
## Indexes
### `processed_posts` Indexes
```sql
CREATE INDEX idx_posts_subreddit ON processed_posts(subreddit);
CREATE INDEX idx_posts_author ON processed_posts(author);
```
**Purpose**: Fast lookup by subreddit (for processing stats) and by author (for skip-checking).
---
### `sent_messages` Indexes
```sql
CREATE INDEX idx_messages_recipient ON sent_messages(recipient);
CREATE INDEX idx_messages_strategy ON sent_messages(strategy_id);
CREATE INDEX idx_messages_response ON sent_messages(got_response);
```
**Purpose**:
- `idx_messages_recipient`: Fast lookup when matching responses
- `idx_messages_strategy`: Analytics by strategy variant
- `idx_messages_response`: Filter for responded/not-responded
---
## Ownership categories
### `successful_messages` Indexes
### Account registry and account content
```sql
CREATE INDEX idx_success_subreddit ON successful_messages(subreddit);
CREATE INDEX idx_success_message ON successful_messages(sent_message_id);
```
- `reddit_accounts`: verified Reddit/Matrix identity, encrypted-at-rest only to the degree provided by the host/volume, cookie JSON, Matrix credentials, active/status timestamps.
- `reddit_conversations`: account-scoped cached Matrix conversation payloads.
- `reddit_account_sync_state`: account-scoped sync cursor, full/delta timestamps, and invitation cooldown.
- `reddit_media`: account-scoped private-media metadata and relative file path.
**Purpose**:
- `idx_success_subreddit`: Weighted example selection by subreddit
- `idx_success_message`: Link back to sent message
The special legacy-unassigned account exists only to preserve rows created before account ownership was introduced. New runtime data must use a verified account ID.
---
### Outreach and outcomes
## Migrations
- `processed_posts`: account/post key, source metadata, analysis, generated opener, send marker, and terminal error/rejection reason.
- `sent_messages`: account-owned opener/reply attribution, strategy/experiment/model metadata, Matrix room, response/image flags, and source type.
- `successful_messages`: examples derived from positive outcomes for prompt context.
- `conversation_outcomes`: normalized outcome for a sent message (`engaged`, `declined`, `boundary`, `upsell`, or `unknown`).
- `activity_log`: operator-facing events with optional recipient/post context.
- `reddit_users`: global mapping between Reddit fullname/username and Matrix identity.
- `recipient_suppressions`: process-global recipient reservation/contact record used to prevent cross-account duplicates.
Migrations are defined in `src/database/client.ts` and run automatically on every startup.
### Durable work and idempotency
### Migration Logic
- `candidate_queue`: global candidate snapshot, source/priority, state, prepared payload, worker lease, retry timing, selected account, and rejection/error details.
- `delivery_outbox`: account-scoped prepared delivery and idempotency status retained for legacy/recovery paths.
- `conversation_states`: account/conversation policy and derived state JSON.
- `conversation_automation_claims`: account/conversation/inbound-event claim with intended reply and pending/sent state.
- `runtime_state`: small process controls such as whether bot work is running.
```typescript
// Pseudo-code for migration process
function runMigrations(db: Database) {
// Enable features
db.exec("PRAGMA journal_mode=WAL;");
db.exec("PRAGMA foreign_keys=ON;");
### Configuration and telemetry
// Create tables if they don't exist
db.exec(`CREATE TABLE IF NOT EXISTS processed_posts (...)`);
db.exec(`CREATE TABLE IF NOT EXISTS sent_messages (...)`);
db.exec(`CREATE TABLE IF NOT EXISTS successful_messages (...)`);
- `app_config`: one secret-free validated application configuration record.
- `ai_usage`: requested/actual model, token/cost, latency, success/error, purpose, and account attribution.
- `legacy_content_imports`: records one-time imports from legacy content stores.
// Create indexes if they don't exist
db.exec(`CREATE INDEX IF NOT EXISTS idx_posts_subreddit ON processed_posts(subreddit)`);
// ... more indexes
## Critical invariants
// Safe column additions (for schema evolution)
db.exec(`ALTER TABLE processed_posts ADD COLUMN error TEXT`);
}
```
1. Account-produced rows carry `account_id`; cross-account views must opt into global queries explicitly.
2. Recipient identity comparisons are case-insensitive and global for outreach suppression.
3. A sent message is persisted only after remote confirmation.
4. Candidate and automation claims are lease/claim based; a worker must own the expected lease before it changes state.
5. Conversation policy is snapshotted in state JSON when automation starts for that conversation.
6. Secrets and the database path are not stored in `app_config`.
7. Private media paths in SQLite are relative to the persistent data root; HTTP access remains authenticated.
### Migration Safety
## Repository map
- All `CREATE TABLE` statements use `IF NOT EXISTS`
- All `CREATE INDEX` statements use `IF NOT EXISTS`
- Column additions use `ALTER TABLE ADD COLUMN IF NOT EXISTS` pattern
- Migrations are idempotent (safe to run multiple times)
`src/database/repositories.ts` contains the domain repositories for posts/messages/outcomes/activity/outbox/state/claims/users/recipient history. `account-repositories.ts` owns account records and cached account content. Candidate and AI telemetry have dedicated repositories.
### Adding New Columns
Prefer narrow repository methods such as `claimReady`, `markSent`, or `getByChatRoomId` to exposing SQL or generic CRUD. This keeps transaction and ownership rules near the schema.
When adding new columns to existing tables:
## Candidate leases
```typescript
// In src/database/client.ts
const tableInfo = db.prepare(`PRAGMA table_info(processed_posts)`).all();
const hasErrorColumn = tableInfo.some((col: any) => col.name === 'error');
Discovery inserts candidates without overwriting terminal work. Screening and dispatch use a worker ID plus lease expiry. On startup all interrupted leases can be recovered; during operation expired leases are recovered and due retries promoted. A transient account failure belongs to the selected account, so delivery can be made visible quickly for another eligible runtime without losing the prepared candidate.
if (!hasErrorColumn) {
db.exec("ALTER TABLE processed_posts ADD COLUMN error TEXT");
}
```
Any new candidate state needs:
---
- an allowed predecessor and terminal/retry semantics;
- a lease ownership rule;
- recovery behavior after process death;
- counts/reasons for the operator funnel;
- tests for stale lease, wrong owner, restart, and duplicate post/author cases.
## Query Patterns
## Conversation reply claims
### Post Processing
The unique identity is account + conversation + inbound message key. The claim persists the intended reply before the remote send. A pending claim is not blindly retried: Matrix history is checked first. This closes the process-crash window between remote acceptance and local completion.
**Check if post was already processed**:
Changes to this flow must prove at-most-once observable replies across retry and restart tests.
```sql
SELECT id FROM processed_posts WHERE id = ?;
```
## Configuration persistence
**Mark post as processed**:
On an empty database, `config.yaml` is parsed, environment-owned values are injected, the result is validated, and the application-owned subset is saved to `app_config`. Later starts load SQLite first. Dashboard changes update that record.
```sql
INSERT OR REPLACE INTO processed_posts
(id, subreddit, author, title, image_url, image_analysis,
generated_message, processed_at, message_sent, error)
VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?);
```
The persisted record excludes:
**Mark post as sent**:
- `OPENROUTER_API_KEY`;
- dashboard/agent tokens;
- `DATABASE_PATH`.
```sql
UPDATE processed_posts
SET message_sent = 1
WHERE id = ?;
```
Those values are supplied by the environment on every load.
**Update post with analysis**:
## Adding a migration
```sql
UPDATE processed_posts
SET image_analysis = ?, generated_message = ?
WHERE id = ?;
```
1. Inspect the final schema and all repository assumptions.
2. Append one migration function; never reorder or modify an already-released migration.
3. Make the operation transactional and compatible with existing data.
4. Use `addColumnIfNotExists` only where repeatability is appropriate; table rebuilds need explicit foreign-key handling and `foreign_key_check`.
5. Update repository types and operations.
6. Add tests for a fresh in-memory database and, when relevant, an older schema upgraded in place.
7. Update this document and affected API/contracts.
---
Do not edit a live SQLite file by hand. Back it up, stop writers when required, apply code-owned migrations, and verify `integrity_check`, `foreign_key_check`, row counts, and application reads.
### Message Tracking
## Backup and recovery
**Record a sent message**:
```sql
INSERT INTO sent_messages
(post_id, recipient, subject, message, sent_at, strategy_id)
VALUES (?, ?, 'Stunning', ?, ?, ?);
```
**Get recent messages**:
```sql
SELECT * FROM sent_messages
ORDER BY sent_at DESC
LIMIT ?;
```
**Update response status**:
```sql
UPDATE sent_messages
SET got_response = 1, response_checked_at = ?
WHERE id = ?;
```
**Check if user was previously messaged**:
```sql
SELECT COUNT(*) FROM sent_messages WHERE recipient = ?;
```
---
### Strategy Analytics
**Get strategy statistics**:
```sql
SELECT
strategy_id,
COUNT(*) as sent,
SUM(got_response) as responded
FROM sent_messages
WHERE strategy_id IS NOT NULL
GROUP BY strategy_id;
```
**Get response rate by subreddit**:
```sql
SELECT
p.subreddit,
COUNT(*) as sent,
SUM(s.got_response) as responded
FROM sent_messages s
JOIN processed_posts p ON s.post_id = p.id
GROUP BY p.subreddit;
```
**Get daily response trend**:
```sql
SELECT
DATE(sent_at, 'unixepoch') as date,
COUNT(*) as sent,
SUM(got_response) as responded
FROM sent_messages
GROUP BY DATE(sent_at, 'unixepoch')
ORDER BY date DESC
LIMIT 30;
```
---
### Few-Shot Learning
**Get successful examples for a subreddit**:
```sql
SELECT
original_message,
image_analysis,
post_title,
response_snippet
FROM successful_messages
WHERE subreddit = ?
ORDER BY created_at DESC
LIMIT 3;
```
**Store successful pair**:
```sql
INSERT INTO successful_messages
(sent_message_id, original_message, image_analysis,
post_title, response_snippet, subreddit, created_at)
VALUES (?, ?, ?, ?, ?, ?, ?);
```
---
## Repository Methods
### PostsRepository
```typescript
class PostsRepository {
/** Check if a post was already processed */
getById(id: string): ProcessedPost | undefined
/** Mark a post as processed */
insert(post: ProcessedPostInsert): void
/** Mark a post as having a message sent */
markSent(id: string): void
/** Update post with AI analysis results */
updateAnalysis(id: string, analysis: string, message: string): void
/** Get total count of processed posts */
count(): number
/** Get count by subreddit */
countBySubreddit(subreddit: string): number
}
```
### MessagesRepository
```typescript
class MessagesRepository {
/** Record a sent message */
insert(message: MessageInsert): number
/** Get recent sent messages */
getRecent(limit: number): SentMessage[]
/** Get messages by recipient */
getByRecipient(recipient: string): SentMessage[]
/** Update response status */
markResponded(id: number, checkedAt: number): void
/** Get strategy statistics */
getStrategyStats(): StrategyStat[]
/** Get response rate by subreddit */
getSubredditStats(): SubredditStat[]
/** Get daily response trend */
getDailyTrend(days: number): DailyTrend[]
/** Get total count of sent messages */
count(): number
/** Get count of responded messages */
countResponded(): number
}
```
### SuccessfulMessagesRepository
```typescript
class SuccessfulMessagesRepository {
/** Store a successful message/response pair */
insert(pair: SuccessfulMessageInsert): void
/** Get examples for few-shot learning */
getExamples(subreddit: string, count: number): Example[]
/** Get total count */
count(): number
/** Get recent examples */
getRecent(limit: number): SuccessfulMessage[]
}
```
---
## Data Retention
### Current Behavior
All data is retained indefinitely. There is no automatic cleanup or retention policy.
| Table | Retention | Notes |
|---|---|---|
| `processed_posts` | Indefinite | Grows with each post processed |
| `sent_messages` | Indefinite | Grows with each message sent |
| `successful_messages` | Indefinite | Grows with each response received |
### Estimated Growth
With default settings (20 messages/hour, 24/7 operation):
| Timeframe | Posts | Messages | Responses (est. 30%) |
|---|---|---|---|
| 1 day | ~480 | ~288 | ~86 |
| 1 week | ~3,360 | ~2,016 | ~605 |
| 1 month | ~14,400 | ~8,640 | ~2,592 |
| 1 year | ~175,200 | ~105,120 | ~31,536 |
### Recommended Cleanup (Future)
Consider implementing automated cleanup:
```sql
-- Archive old processed posts
CREATE TABLE processed_posts_archive AS
SELECT * FROM processed_posts
WHERE processed_at < ?;
-- Delete archived posts
DELETE FROM processed_posts
WHERE processed_at < ?;
-- Keep last N months of data
```
> **Note**: Successful messages should NEVER be deleted as they are used for few-shot learning.
For a running WAL database, use SQLite's online backup mechanism or stop the container before copying the database plus `-wal`/`-shm` state. A full production backup should include the entire persistent volume because sessions and media live beside the database. Validate that the backup can be opened and contains expected tables before any destructive reset.