Restructure chat service, web routes, and add test suite

Split monolithic chat service into modular structure with Matrix client support, extract web server into per-resource route modules, introduce bootstrap/main-loop separation, and add comprehensive vitest coverage across services, repositories, and utilities. Adds project docs and CLAUDE.md agent instructions.
2026-05-21 01:32:10 +01:00
1 parent dfa0580859
commit 633bb0ec04
@@ -0,0 +1,550 @@
+# Database Schema
+
+Complete reference for the SQLite database schema, migrations, and query patterns.
+
+---
+
+## Table of Contents
+
+1. [Overview](#overview)
+2. [Database File](#database-file)
+3. [Schema](#schema)
+4. [Indexes](#indexes)
+5. [Migrations](#migrations)
+6. [Query Patterns](#query-patterns)
+7. [Repository Methods](#repository-methods)
+8. [Data Retention](#data-retention)
+
+---
+
+## Overview
+
+RedditBot uses SQLite (via `better-sqlite3`) for persistent storage. The database tracks processed posts, sent messages, and successful message/response pairs.
+
+**Features**:
+- WAL mode for better concurrent read performance
+- Foreign key constraints enabled
+- Progressive migrations for schema evolution
+- Repository pattern for data access
+
+---
+
+## Database File
+
+**Default path**: `./data/reddit-bot.db`
+
+Configured in `config.yaml`:
+
+```yaml
+database:
+  path: "./data/reddit-bot.db"
+```
+
+The database file is auto-created on first run.
+
+---
+
+## Schema
+
+### Table: `processed_posts`
+
+Tracks all posts that have been discovered and processed.
+
+```sql
+CREATE TABLE processed_posts (
+  id TEXT PRIMARY KEY,           -- Reddit post ID
+  subreddit TEXT NOT NULL,       -- Source subreddit
+  author TEXT NOT NULL,          -- Post author username
+  title TEXT,                    -- Post title
+  image_url TEXT,                -- Original image URL
+  image_analysis TEXT,           -- AI-generated JSON description
+  generated_message TEXT,        -- AI-generated DM text
+  processed_at INTEGER NOT NULL, -- Unix timestamp
+  message_sent INTEGER DEFAULT 0, -- 0 = not sent, 1 = sent
+  error TEXT                     -- Error message if processing failed
+);
+```
+
+**Column Details**:
+
+| Column | Type | Constraints | Description |
+|---|---|---|---|
+| `id` | TEXT | PRIMARY KEY | Reddit post ID (e.g., "abc123") |
+| `subreddit` | TEXT | NOT NULL | Source subreddit name |
+| `author` | TEXT | NOT NULL | Post author username |
+| `title` | TEXT | | Post title (nullable if unavailable) |
+| `image_url` | TEXT | | Original image URL |
+| `image_analysis` | TEXT | | JSON string from AI analysis |
+| `generated_message` | TEXT | | The DM text that was generated |
+| `processed_at` | INTEGER | NOT NULL | Unix timestamp when processed |
+| `message_sent` | INTEGER | DEFAULT 0 | 0 = not sent, 1 = successfully sent |
+| `error` | TEXT | | Error message if processing failed |
+
+**Example Row**:
+
+```json
+{
+  "id": "abc123",
+  "subreddit": "gonewild",
+  "author": "user123",
+  "title": "Summer vibes",
+  "image_url": "https://i.redd.it/abc123.jpg",
+  "image_analysis": "{\"gender\":\"F\",\"description\":\"Young woman...\"}",
+  "generated_message": "Hey, you look amazing...",
+  "processed_at": 1716234567,
+  "message_sent": 1,
+  "error": null
+}
+```
+
+---
+
+### Table: `sent_messages`
+
+Tracks all DMs sent by the bot.
+
+```sql
+CREATE TABLE sent_messages (
+  id INTEGER PRIMARY KEY AUTOINCREMENT,
+  post_id TEXT NOT NULL,
+  recipient TEXT NOT NULL,
+  subject TEXT DEFAULT 'Stunning',
+  message TEXT NOT NULL,
+  sent_at INTEGER NOT NULL,
+  strategy_id TEXT,
+  got_response INTEGER DEFAULT 0,
+  response_checked_at INTEGER,
+  FOREIGN KEY (post_id) REFERENCES processed_posts(id)
+);
+```
+
+**Column Details**:
+
+| Column | Type | Constraints | Description |
+|---|---|---|---|
+| `id` | INTEGER | PRIMARY KEY AUTOINCREMENT | Internal ID |
+| `post_id` | TEXT | NOT NULL, FK | Source post ID |
+| `recipient` | TEXT | NOT NULL | Recipient username |
+| `subject` | TEXT | DEFAULT 'Stunning' | DM subject line |
+| `message` | TEXT | NOT NULL | DM body text |
+| `sent_at` | INTEGER | NOT NULL | Unix timestamp when sent |
+| `strategy_id` | TEXT | | A/B test variant name |
+| `got_response` | INTEGER | DEFAULT 0 | 0 = no response, 1 = responded |
+| `response_checked_at` | INTEGER | | Timestamp of last response check |
+
+**Example Row**:
+
+```json
+{
+  "id": 1,
+  "post_id": "abc123",
+  "recipient": "user123",
+  "subject": "Stunning",
+  "message": "Hey, you look amazing...",
+  "sent_at": 1716234567,
+  "strategy_id": "compliment",
+  "got_response": 1,
+  "response_checked_at": 1716238167
+}
+```
+
+---
+
+### Table: `successful_messages`
+
+Stores successful message/response pairs for few-shot learning.
+
+```sql
+CREATE TABLE successful_messages (
+  id INTEGER PRIMARY KEY AUTOINCREMENT,
+  sent_message_id INTEGER,
+  original_message TEXT NOT NULL,
+  image_analysis TEXT,
+  post_title TEXT,
+  response_snippet TEXT,
+  subreddit TEXT,
+  created_at INTEGER NOT NULL,
+  FOREIGN KEY (sent_message_id) REFERENCES sent_messages(id)
+);
+```
+
+**Column Details**:
+
+| Column | Type | Constraints | Description |
+|---|---|---|---|
+| `id` | INTEGER | PRIMARY KEY AUTOINCREMENT | Internal ID |
+| `sent_message_id` | INTEGER | FK, nullable | Related sent message ID |
+| `original_message` | TEXT | NOT NULL | The DM that got a response |
+| `image_analysis` | TEXT | | Context from image analysis |
+| `post_title` | TEXT | | Original post title |
+| `response_snippet` | TEXT | | First reply text |
+| `subreddit` | TEXT | | For weighted example selection |
+| `created_at` | INTEGER | NOT NULL | Unix timestamp |
+
+**Example Row**:
+
+```json
+{
+  "id": 1,
+  "sent_message_id": 1,
+  "original_message": "Hey, you look amazing...",
+  "image_analysis": "{\"gender\":\"F\",\"description\":\"...\"}",
+  "post_title": "Summer vibes",
+  "response_snippet": "Thank you so much! 😊",
+  "subreddit": "gonewild",
+  "created_at": 1716238167
+}
+```
+
+---
+
+## Indexes
+
+### `processed_posts` Indexes
+
+```sql
+CREATE INDEX idx_posts_subreddit ON processed_posts(subreddit);
+CREATE INDEX idx_posts_author ON processed_posts(author);
+```
+
+**Purpose**: Fast lookup by subreddit (for processing stats) and by author (for skip-checking).
+
+---
+
+### `sent_messages` Indexes
+
+```sql
+CREATE INDEX idx_messages_recipient ON sent_messages(recipient);
+CREATE INDEX idx_messages_strategy ON sent_messages(strategy_id);
+CREATE INDEX idx_messages_response ON sent_messages(got_response);
+```
+
+**Purpose**:
+- `idx_messages_recipient`: Fast lookup when matching responses
+- `idx_messages_strategy`: Analytics by strategy variant
+- `idx_messages_response`: Filter for responded/not-responded
+
+---
+
+### `successful_messages` Indexes
+
+```sql
+CREATE INDEX idx_success_subreddit ON successful_messages(subreddit);
+CREATE INDEX idx_success_message ON successful_messages(sent_message_id);
+```
+
+**Purpose**:
+- `idx_success_subreddit`: Weighted example selection by subreddit
+- `idx_success_message`: Link back to sent message
+
+---
+
+## Migrations
+
+Migrations are defined in `src/database/client.ts` and run automatically on every startup.
+
+### Migration Logic
+
+```typescript
+// Pseudo-code for migration process
+function runMigrations(db: Database) {
+  // Enable features
+  db.exec("PRAGMA journal_mode=WAL;");
+  db.exec("PRAGMA foreign_keys=ON;");
+
+  // Create tables if they don't exist
+  db.exec(`CREATE TABLE IF NOT EXISTS processed_posts (...)`);
+  db.exec(`CREATE TABLE IF NOT EXISTS sent_messages (...)`);
+  db.exec(`CREATE TABLE IF NOT EXISTS successful_messages (...)`);
+
+  // Create indexes if they don't exist
+  db.exec(`CREATE INDEX IF NOT EXISTS idx_posts_subreddit ON processed_posts(subreddit)`);
+  // ... more indexes
+
+  // Safe column additions (for schema evolution)
+  db.exec(`ALTER TABLE processed_posts ADD COLUMN error TEXT`);
+}
+```
+
+### Migration Safety
+
+- All `CREATE TABLE` statements use `IF NOT EXISTS`
+- All `CREATE INDEX` statements use `IF NOT EXISTS`
+- Column additions use `ALTER TABLE ADD COLUMN IF NOT EXISTS` pattern
+- Migrations are idempotent (safe to run multiple times)
+
+### Adding New Columns
+
+When adding new columns to existing tables:
+
+```typescript
+// In src/database/client.ts
+const tableInfo = db.prepare(`PRAGMA table_info(processed_posts)`).all();
+const hasErrorColumn = tableInfo.some((col: any) => col.name === 'error');
+
+if (!hasErrorColumn) {
+  db.exec("ALTER TABLE processed_posts ADD COLUMN error TEXT");
+}
+```
+
+---
+
+## Query Patterns
+
+### Post Processing
+
+**Check if post was already processed**:
+
+```sql
+SELECT id FROM processed_posts WHERE id = ?;
+```
+
+**Mark post as processed**:
+
+```sql
+INSERT OR REPLACE INTO processed_posts 
+  (id, subreddit, author, title, image_url, image_analysis, 
+   generated_message, processed_at, message_sent, error)
+VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?);
+```
+
+**Mark post as sent**:
+
+```sql
+UPDATE processed_posts 
+SET message_sent = 1 
+WHERE id = ?;
+```
+
+**Update post with analysis**:
+
+```sql
+UPDATE processed_posts 
+SET image_analysis = ?, generated_message = ? 
+WHERE id = ?;
+```
+
+---
+
+### Message Tracking
+
+**Record a sent message**:
+
+```sql
+INSERT INTO sent_messages 
+  (post_id, recipient, subject, message, sent_at, strategy_id)
+VALUES (?, ?, 'Stunning', ?, ?, ?);
+```
+
+**Get recent messages**:
+
+```sql
+SELECT * FROM sent_messages 
+ORDER BY sent_at DESC 
+LIMIT ?;
+```
+
+**Update response status**:
+
+```sql
+UPDATE sent_messages 
+SET got_response = 1, response_checked_at = ? 
+WHERE id = ?;
+```
+
+**Check if user was previously messaged**:
+
+```sql
+SELECT COUNT(*) FROM sent_messages WHERE recipient = ?;
+```
+
+---
+
+### Strategy Analytics
+
+**Get strategy statistics**:
+
+```sql
+SELECT 
+  strategy_id,
+  COUNT(*) as sent,
+  SUM(got_response) as responded
+FROM sent_messages
+WHERE strategy_id IS NOT NULL
+GROUP BY strategy_id;
+```
+
+**Get response rate by subreddit**:
+
+```sql
+SELECT 
+  p.subreddit,
+  COUNT(*) as sent,
+  SUM(s.got_response) as responded
+FROM sent_messages s
+JOIN processed_posts p ON s.post_id = p.id
+GROUP BY p.subreddit;
+```
+
+**Get daily response trend**:
+
+```sql
+SELECT 
+  DATE(sent_at, 'unixepoch') as date,
+  COUNT(*) as sent,
+  SUM(got_response) as responded
+FROM sent_messages
+GROUP BY DATE(sent_at, 'unixepoch')
+ORDER BY date DESC
+LIMIT 30;
+```
+
+---
+
+### Few-Shot Learning
+
+**Get successful examples for a subreddit**:
+
+```sql
+SELECT 
+  original_message,
+  image_analysis,
+  post_title,
+  response_snippet
+FROM successful_messages
+WHERE subreddit = ?
+ORDER BY created_at DESC
+LIMIT 3;
+```
+
+**Store successful pair**:
+
+```sql
+INSERT INTO successful_messages 
+  (sent_message_id, original_message, image_analysis, 
+   post_title, response_snippet, subreddit, created_at)
+VALUES (?, ?, ?, ?, ?, ?, ?);
+```
+
+---
+
+## Repository Methods
+
+### PostsRepository
+
+```typescript
+class PostsRepository {
+  /** Check if a post was already processed */
+  getById(id: string): ProcessedPost | undefined
+
+  /** Mark a post as processed */
+  insert(post: ProcessedPostInsert): void
+
+  /** Mark a post as having a message sent */
+  markSent(id: string): void
+
+  /** Update post with AI analysis results */
+  updateAnalysis(id: string, analysis: string, message: string): void
+
+  /** Get total count of processed posts */
+  count(): number
+
+  /** Get count by subreddit */
+  countBySubreddit(subreddit: string): number
+}
+```
+
+### MessagesRepository
+
+```typescript
+class MessagesRepository {
+  /** Record a sent message */
+  insert(message: MessageInsert): number
+
+  /** Get recent sent messages */
+  getRecent(limit: number): SentMessage[]
+
+  /** Get messages by recipient */
+  getByRecipient(recipient: string): SentMessage[]
+
+  /** Update response status */
+  markResponded(id: number, checkedAt: number): void
+
+  /** Get strategy statistics */
+  getStrategyStats(): StrategyStat[]
+
+  /** Get response rate by subreddit */
+  getSubredditStats(): SubredditStat[]
+
+  /** Get daily response trend */
+  getDailyTrend(days: number): DailyTrend[]
+
+  /** Get total count of sent messages */
+  count(): number
+
+  /** Get count of responded messages */
+  countResponded(): number
+}
+```
+
+### SuccessfulMessagesRepository
+
+```typescript
+class SuccessfulMessagesRepository {
+  /** Store a successful message/response pair */
+  insert(pair: SuccessfulMessageInsert): void
+
+  /** Get examples for few-shot learning */
+  getExamples(subreddit: string, count: number): Example[]
+
+  /** Get total count */
+  count(): number
+
+  /** Get recent examples */
+  getRecent(limit: number): SuccessfulMessage[]
+}
+```
+
+---
+
+## Data Retention
+
+### Current Behavior
+
+All data is retained indefinitely. There is no automatic cleanup or retention policy.
+
+| Table | Retention | Notes |
+|---|---|---|
+| `processed_posts` | Indefinite | Grows with each post processed |
+| `sent_messages` | Indefinite | Grows with each message sent |
+| `successful_messages` | Indefinite | Grows with each response received |
+
+### Estimated Growth
+
+With default settings (20 messages/hour, 24/7 operation):
+
+| Timeframe | Posts | Messages | Responses (est. 30%) |
+|---|---|---|---|
+| 1 day | ~480 | ~288 | ~86 |
+| 1 week | ~3,360 | ~2,016 | ~605 |
+| 1 month | ~14,400 | ~8,640 | ~2,592 |
+| 1 year | ~175,200 | ~105,120 | ~31,536 |
+
+### Recommended Cleanup (Future)
+
+Consider implementing automated cleanup:
+
+```sql
+-- Archive old processed posts
+CREATE TABLE processed_posts_archive AS
+SELECT * FROM processed_posts
+WHERE processed_at < ?;
+
+-- Delete archived posts
+DELETE FROM processed_posts
+WHERE processed_at < ?;
+
+-- Keep last N months of data
+```
+
+> **Note**: Successful messages should NEVER be deleted as they are used for few-shot learning.