RedditBot/docs/database.md

# Database Schema

Complete reference for the SQLite database schema, migrations, and query patterns.

---

## Table of Contents

1. [Overview](#overview)
2. [Database File](#database-file)
3. [Schema](#schema)
4. [Indexes](#indexes)
5. [Migrations](#migrations)
6. [Query Patterns](#query-patterns)
7. [Repository Methods](#repository-methods)
8. [Data Retention](#data-retention)

---

## Overview

RedditBot uses SQLite (via `better-sqlite3`) for persistent storage. The database tracks processed posts, sent messages, and successful message/response pairs.

**Features**:
- WAL mode for better concurrent read performance
- Foreign key constraints enabled
- Progressive migrations for schema evolution
- Repository pattern for data access

---

## Database File

**Default path**: `./data/reddit-bot.db`

Configured in `config.yaml`:

```yaml
database:
  path: "./data/reddit-bot.db"
```

The database file is auto-created on first run.

---

## Schema

### Table: `processed_posts`

Tracks all posts that have been discovered and processed.

```sql
CREATE TABLE processed_posts (
  id TEXT PRIMARY KEY,           -- Reddit post ID
  subreddit TEXT NOT NULL,       -- Source subreddit
  author TEXT NOT NULL,          -- Post author username
  title TEXT,                    -- Post title
  image_url TEXT,                -- Original image URL
  image_analysis TEXT,           -- AI-generated JSON description
  generated_message TEXT,        -- AI-generated DM text
  processed_at INTEGER NOT NULL, -- Unix timestamp
  message_sent INTEGER DEFAULT 0, -- 0 = not sent, 1 = sent
  error TEXT                     -- Error message if processing failed
);
```

**Column Details**:

| Column | Type | Constraints | Description |
|---|---|---|---|
| `id` | TEXT | PRIMARY KEY | Reddit post ID (e.g., "abc123") |
| `subreddit` | TEXT | NOT NULL | Source subreddit name |
| `author` | TEXT | NOT NULL | Post author username |
| `title` | TEXT | | Post title (nullable if unavailable) |
| `image_url` | TEXT | | Original image URL |
| `image_analysis` | TEXT | | JSON string from AI analysis |
| `generated_message` | TEXT | | The DM text that was generated |
| `processed_at` | INTEGER | NOT NULL | Unix timestamp when processed |
| `message_sent` | INTEGER | DEFAULT 0 | 0 = not sent, 1 = successfully sent |
| `error` | TEXT | | Error message if processing failed |

**Example Row**:

```json
{
  "id": "abc123",
  "subreddit": "gonewild",
  "author": "user123",
  "title": "Summer vibes",
  "image_url": "https://i.redd.it/abc123.jpg",
  "image_analysis": "{\"gender\":\"F\",\"description\":\"Young woman...\"}",
  "generated_message": "Hey, you look amazing...",
  "processed_at": 1716234567,
  "message_sent": 1,
  "error": null
}
```

---

### Table: `sent_messages`

Tracks all DMs sent by the bot.

```sql
CREATE TABLE sent_messages (
  id INTEGER PRIMARY KEY AUTOINCREMENT,
  post_id TEXT NOT NULL,
  recipient TEXT NOT NULL,
  subject TEXT DEFAULT 'Stunning',
  message TEXT NOT NULL,
  sent_at INTEGER NOT NULL,
  strategy_id TEXT,
  got_response INTEGER DEFAULT 0,
  response_checked_at INTEGER,
  FOREIGN KEY (post_id) REFERENCES processed_posts(id)
);
```

**Column Details**:

| Column | Type | Constraints | Description |
|---|---|---|---|
| `id` | INTEGER | PRIMARY KEY AUTOINCREMENT | Internal ID |
| `post_id` | TEXT | NOT NULL, FK | Source post ID |
| `recipient` | TEXT | NOT NULL | Recipient username |
| `subject` | TEXT | DEFAULT 'Stunning' | DM subject line |
| `message` | TEXT | NOT NULL | DM body text |
| `sent_at` | INTEGER | NOT NULL | Unix timestamp when sent |
| `strategy_id` | TEXT | | A/B test variant name |
| `got_response` | INTEGER | DEFAULT 0 | 0 = no response, 1 = responded |
| `response_checked_at` | INTEGER | | Timestamp of last response check |

**Example Row**:

```json
{
  "id": 1,
  "post_id": "abc123",
  "recipient": "user123",
  "subject": "Stunning",
  "message": "Hey, you look amazing...",
  "sent_at": 1716234567,
  "strategy_id": "compliment",
  "got_response": 1,
  "response_checked_at": 1716238167
}
```

---

### Table: `successful_messages`

Stores successful message/response pairs for few-shot learning.

```sql
CREATE TABLE successful_messages (
  id INTEGER PRIMARY KEY AUTOINCREMENT,
  sent_message_id INTEGER,
  original_message TEXT NOT NULL,
  image_analysis TEXT,
  post_title TEXT,
  response_snippet TEXT,
  subreddit TEXT,
  created_at INTEGER NOT NULL,
  FOREIGN KEY (sent_message_id) REFERENCES sent_messages(id)
);
```

**Column Details**:

| Column | Type | Constraints | Description |
|---|---|---|---|
| `id` | INTEGER | PRIMARY KEY AUTOINCREMENT | Internal ID |
| `sent_message_id` | INTEGER | FK, nullable | Related sent message ID |
| `original_message` | TEXT | NOT NULL | The DM that got a response |
| `image_analysis` | TEXT | | Context from image analysis |
| `post_title` | TEXT | | Original post title |
| `response_snippet` | TEXT | | First reply text |
| `subreddit` | TEXT | | For weighted example selection |
| `created_at` | INTEGER | NOT NULL | Unix timestamp |

**Example Row**:

```json
{
  "id": 1,
  "sent_message_id": 1,
  "original_message": "Hey, you look amazing...",
  "image_analysis": "{\"gender\":\"F\",\"description\":\"...\"}",
  "post_title": "Summer vibes",
  "response_snippet": "Thank you so much! 😊",
  "subreddit": "gonewild",
  "created_at": 1716238167
}
```

---

## Indexes

### `processed_posts` Indexes

```sql
CREATE INDEX idx_posts_subreddit ON processed_posts(subreddit);
CREATE INDEX idx_posts_author ON processed_posts(author);
```

**Purpose**: Fast lookup by subreddit (for processing stats) and by author (for skip-checking).

---

### `sent_messages` Indexes

```sql
CREATE INDEX idx_messages_recipient ON sent_messages(recipient);
CREATE INDEX idx_messages_strategy ON sent_messages(strategy_id);
CREATE INDEX idx_messages_response ON sent_messages(got_response);
```

**Purpose**:
- `idx_messages_recipient`: Fast lookup when matching responses
- `idx_messages_strategy`: Analytics by strategy variant
- `idx_messages_response`: Filter for responded/not-responded

---

### `successful_messages` Indexes

```sql
CREATE INDEX idx_success_subreddit ON successful_messages(subreddit);
CREATE INDEX idx_success_message ON successful_messages(sent_message_id);
```

**Purpose**:
- `idx_success_subreddit`: Weighted example selection by subreddit
- `idx_success_message`: Link back to sent message

---

## Migrations

Migrations are defined in `src/database/client.ts` and run automatically on every startup.

### Migration Logic

```typescript
// Pseudo-code for migration process
function runMigrations(db: Database) {
  // Enable features
  db.exec("PRAGMA journal_mode=WAL;");
  db.exec("PRAGMA foreign_keys=ON;");

  // Create tables if they don't exist
  db.exec(`CREATE TABLE IF NOT EXISTS processed_posts (...)`);
  db.exec(`CREATE TABLE IF NOT EXISTS sent_messages (...)`);
  db.exec(`CREATE TABLE IF NOT EXISTS successful_messages (...)`);

  // Create indexes if they don't exist
  db.exec(`CREATE INDEX IF NOT EXISTS idx_posts_subreddit ON processed_posts(subreddit)`);
  // ... more indexes

  // Safe column additions (for schema evolution)
  db.exec(`ALTER TABLE processed_posts ADD COLUMN error TEXT`);
}
```

### Migration Safety

- All `CREATE TABLE` statements use `IF NOT EXISTS`
- All `CREATE INDEX` statements use `IF NOT EXISTS`
- Column additions use `ALTER TABLE ADD COLUMN IF NOT EXISTS` pattern
- Migrations are idempotent (safe to run multiple times)

### Adding New Columns

When adding new columns to existing tables:

```typescript
// In src/database/client.ts
const tableInfo = db.prepare(`PRAGMA table_info(processed_posts)`).all();
const hasErrorColumn = tableInfo.some((col: any) => col.name === 'error');

if (!hasErrorColumn) {
  db.exec("ALTER TABLE processed_posts ADD COLUMN error TEXT");
}
```

---

## Query Patterns

### Post Processing

**Check if post was already processed**:

```sql
SELECT id FROM processed_posts WHERE id = ?;
```

**Mark post as processed**:

```sql
INSERT OR REPLACE INTO processed_posts 
  (id, subreddit, author, title, image_url, image_analysis, 
   generated_message, processed_at, message_sent, error)
VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?);
```

**Mark post as sent**:

```sql
UPDATE processed_posts 
SET message_sent = 1 
WHERE id = ?;
```

**Update post with analysis**:

```sql
UPDATE processed_posts 
SET image_analysis = ?, generated_message = ? 
WHERE id = ?;
```

---

### Message Tracking

**Record a sent message**:

```sql
INSERT INTO sent_messages 
  (post_id, recipient, subject, message, sent_at, strategy_id)
VALUES (?, ?, 'Stunning', ?, ?, ?);
```

**Get recent messages**:

```sql
SELECT * FROM sent_messages 
ORDER BY sent_at DESC 
LIMIT ?;
```

**Update response status**:

```sql
UPDATE sent_messages 
SET got_response = 1, response_checked_at = ? 
WHERE id = ?;
```

**Check if user was previously messaged**:

```sql
SELECT COUNT(*) FROM sent_messages WHERE recipient = ?;
```

---

### Strategy Analytics

**Get strategy statistics**:

```sql
SELECT 
  strategy_id,
  COUNT(*) as sent,
  SUM(got_response) as responded
FROM sent_messages
WHERE strategy_id IS NOT NULL
GROUP BY strategy_id;
```

**Get response rate by subreddit**:

```sql
SELECT 
  p.subreddit,
  COUNT(*) as sent,
  SUM(s.got_response) as responded
FROM sent_messages s
JOIN processed_posts p ON s.post_id = p.id
GROUP BY p.subreddit;
```

**Get daily response trend**:

```sql
SELECT 
  DATE(sent_at, 'unixepoch') as date,
  COUNT(*) as sent,
  SUM(got_response) as responded
FROM sent_messages
GROUP BY DATE(sent_at, 'unixepoch')
ORDER BY date DESC
LIMIT 30;
```

---

### Few-Shot Learning

**Get successful examples for a subreddit**:

```sql
SELECT 
  original_message,
  image_analysis,
  post_title,
  response_snippet
FROM successful_messages
WHERE subreddit = ?
ORDER BY created_at DESC
LIMIT 3;
```

**Store successful pair**:

```sql
INSERT INTO successful_messages 
  (sent_message_id, original_message, image_analysis, 
   post_title, response_snippet, subreddit, created_at)
VALUES (?, ?, ?, ?, ?, ?, ?);
```

---

## Repository Methods

### PostsRepository

```typescript
class PostsRepository {
  /** Check if a post was already processed */
  getById(id: string): ProcessedPost | undefined

  /** Mark a post as processed */
  insert(post: ProcessedPostInsert): void

  /** Mark a post as having a message sent */
  markSent(id: string): void

  /** Update post with AI analysis results */
  updateAnalysis(id: string, analysis: string, message: string): void

  /** Get total count of processed posts */
  count(): number

  /** Get count by subreddit */
  countBySubreddit(subreddit: string): number
}
```

### MessagesRepository

```typescript
class MessagesRepository {
  /** Record a sent message */
  insert(message: MessageInsert): number

  /** Get recent sent messages */
  getRecent(limit: number): SentMessage[]

  /** Get messages by recipient */
  getByRecipient(recipient: string): SentMessage[]

  /** Update response status */
  markResponded(id: number, checkedAt: number): void

  /** Get strategy statistics */
  getStrategyStats(): StrategyStat[]

  /** Get response rate by subreddit */
  getSubredditStats(): SubredditStat[]

  /** Get daily response trend */
  getDailyTrend(days: number): DailyTrend[]

  /** Get total count of sent messages */
  count(): number

  /** Get count of responded messages */
  countResponded(): number
}
```

### SuccessfulMessagesRepository

```typescript
class SuccessfulMessagesRepository {
  /** Store a successful message/response pair */
  insert(pair: SuccessfulMessageInsert): void

  /** Get examples for few-shot learning */
  getExamples(subreddit: string, count: number): Example[]

  /** Get total count */
  count(): number

  /** Get recent examples */
  getRecent(limit: number): SuccessfulMessage[]
}
```

---

## Data Retention

### Current Behavior

All data is retained indefinitely. There is no automatic cleanup or retention policy.

| Table | Retention | Notes |
|---|---|---|
| `processed_posts` | Indefinite | Grows with each post processed |
| `sent_messages` | Indefinite | Grows with each message sent |
| `successful_messages` | Indefinite | Grows with each response received |

### Estimated Growth

With default settings (20 messages/hour, 24/7 operation):

| Timeframe | Posts | Messages | Responses (est. 30%) |
|---|---|---|---|
| 1 day | ~480 | ~288 | ~86 |
| 1 week | ~3,360 | ~2,016 | ~605 |
| 1 month | ~14,400 | ~8,640 | ~2,592 |
| 1 year | ~175,200 | ~105,120 | ~31,536 |

### Recommended Cleanup (Future)

Consider implementing automated cleanup:

```sql
-- Archive old processed posts
CREATE TABLE processed_posts_archive AS
SELECT * FROM processed_posts
WHERE processed_at < ?;

-- Delete archived posts
DELETE FROM processed_posts
WHERE processed_at < ?;

-- Keep last N months of data
```

> **Note**: Successful messages should NEVER be deleted as they are used for few-shot learning.