RedditBot/docs/database.md

Database Schema

Complete reference for the SQLite database schema, migrations, and query patterns.

---

Table of Contents

1. Overview
2. Database File
3. Schema
4. Indexes
5. Migrations
6. Query Patterns
7. Repository Methods
8. Data Retention

---

Overview

RedditBot uses SQLite (via better-sqlite3) for persistent storage. The database tracks processed posts, sent messages, and successful message/response pairs.

Features:

• WAL mode for better concurrent read performance
• Foreign key constraints enabled
• Progressive migrations for schema evolution
• Repository pattern for data access

---

Database File

Default path: ./data/reddit-bot.db

Configured in config.yaml:

database:
  path: "./data/reddit-bot.db"

The database file is auto-created on first run.

---

Schema

Table: processed_posts

Tracks all posts that have been discovered and processed.

CREATE TABLE processed_posts (
  id TEXT PRIMARY KEY,           -- Reddit post ID
  subreddit TEXT NOT NULL,       -- Source subreddit
  author TEXT NOT NULL,          -- Post author username
  title TEXT,                    -- Post title
  image_url TEXT,                -- Original image URL
  image_analysis TEXT,           -- AI-generated JSON description
  generated_message TEXT,        -- AI-generated DM text
  processed_at INTEGER NOT NULL, -- Unix timestamp
  message_sent INTEGER DEFAULT 0, -- 0 = not sent, 1 = sent
  error TEXT                     -- Error message if processing failed
);

Column Details:

Column	Type	Constraints	Description
id	TEXT	PRIMARY KEY	Reddit post ID (e.g., "abc123")
subreddit	TEXT	NOT NULL	Source subreddit name
author	TEXT	NOT NULL	Post author username
title	TEXT		Post title (nullable if unavailable)
image_url	TEXT		Original image URL
image_analysis	TEXT		JSON string from AI analysis
generated_message	TEXT		The DM text that was generated
processed_at	INTEGER	NOT NULL	Unix timestamp when processed
message_sent	INTEGER	DEFAULT 0	0 = not sent, 1 = successfully sent
error	TEXT		Error message if processing failed

Example Row:

{
  "id": "abc123",
  "subreddit": "gonewild",
  "author": "user123",
  "title": "Summer vibes",
  "image_url": "https://i.redd.it/abc123.jpg",
  "image_analysis": "{\"gender\":\"F\",\"description\":\"Young woman...\"}",
  "generated_message": "Hey, you look amazing...",
  "processed_at": 1716234567,
  "message_sent": 1,
  "error": null
}

---

Table: sent_messages

Tracks all DMs sent by the bot.

CREATE TABLE sent_messages (
  id INTEGER PRIMARY KEY AUTOINCREMENT,
  post_id TEXT NOT NULL,
  recipient TEXT NOT NULL,
  subject TEXT DEFAULT 'Stunning',
  message TEXT NOT NULL,
  sent_at INTEGER NOT NULL,
  strategy_id TEXT,
  got_response INTEGER DEFAULT 0,
  response_checked_at INTEGER,
  FOREIGN KEY (post_id) REFERENCES processed_posts(id)
);

Column Details:

Column	Type	Constraints	Description
id	INTEGER	PRIMARY KEY AUTOINCREMENT	Internal ID
post_id	TEXT	NOT NULL, FK	Source post ID
recipient	TEXT	NOT NULL	Recipient username
subject	TEXT	DEFAULT 'Stunning'	DM subject line
message	TEXT	NOT NULL	DM body text
sent_at	INTEGER	NOT NULL	Unix timestamp when sent
strategy_id	TEXT		A/B test variant name
got_response	INTEGER	DEFAULT 0	0 = no response, 1 = responded
response_checked_at	INTEGER		Timestamp of last response check

Example Row:

{
  "id": 1,
  "post_id": "abc123",
  "recipient": "user123",
  "subject": "Stunning",
  "message": "Hey, you look amazing...",
  "sent_at": 1716234567,
  "strategy_id": "compliment",
  "got_response": 1,
  "response_checked_at": 1716238167
}

---

Table: successful_messages

Stores successful message/response pairs for few-shot learning.

CREATE TABLE successful_messages (
  id INTEGER PRIMARY KEY AUTOINCREMENT,
  sent_message_id INTEGER,
  original_message TEXT NOT NULL,
  image_analysis TEXT,
  post_title TEXT,
  response_snippet TEXT,
  subreddit TEXT,
  created_at INTEGER NOT NULL,
  FOREIGN KEY (sent_message_id) REFERENCES sent_messages(id)
);

Column Details:

Column	Type	Constraints	Description
id	INTEGER	PRIMARY KEY AUTOINCREMENT	Internal ID
sent_message_id	INTEGER	FK, nullable	Related sent message ID
original_message	TEXT	NOT NULL	The DM that got a response
image_analysis	TEXT		Context from image analysis
post_title	TEXT		Original post title
response_snippet	TEXT		First reply text
subreddit	TEXT		For weighted example selection
created_at	INTEGER	NOT NULL	Unix timestamp

Example Row:

{
  "id": 1,
  "sent_message_id": 1,
  "original_message": "Hey, you look amazing...",
  "image_analysis": "{\"gender\":\"F\",\"description\":\"...\"}",
  "post_title": "Summer vibes",
  "response_snippet": "Thank you so much! 😊",
  "subreddit": "gonewild",
  "created_at": 1716238167
}

---

Indexes

processed_posts Indexes

CREATE INDEX idx_posts_subreddit ON processed_posts(subreddit);
CREATE INDEX idx_posts_author ON processed_posts(author);

Purpose: Fast lookup by subreddit (for processing stats) and by author (for skip-checking).

---

sent_messages Indexes

CREATE INDEX idx_messages_recipient ON sent_messages(recipient);
CREATE INDEX idx_messages_strategy ON sent_messages(strategy_id);
CREATE INDEX idx_messages_response ON sent_messages(got_response);

Purpose:

• idx_messages_recipient: Fast lookup when matching responses
• idx_messages_strategy: Analytics by strategy variant
• idx_messages_response: Filter for responded/not-responded

---

successful_messages Indexes

CREATE INDEX idx_success_subreddit ON successful_messages(subreddit);
CREATE INDEX idx_success_message ON successful_messages(sent_message_id);

Purpose:

• idx_success_subreddit: Weighted example selection by subreddit
• idx_success_message: Link back to sent message

---

Migrations

Migrations are defined in src/database/client.ts and run automatically on every startup.

Migration Logic

// Pseudo-code for migration process
function runMigrations(db: Database) {
  // Enable features
  db.exec("PRAGMA journal_mode=WAL;");
  db.exec("PRAGMA foreign_keys=ON;");

  // Create tables if they don't exist
  db.exec(`CREATE TABLE IF NOT EXISTS processed_posts (...)`);
  db.exec(`CREATE TABLE IF NOT EXISTS sent_messages (...)`);
  db.exec(`CREATE TABLE IF NOT EXISTS successful_messages (...)`);

  // Create indexes if they don't exist
  db.exec(`CREATE INDEX IF NOT EXISTS idx_posts_subreddit ON processed_posts(subreddit)`);
  // ... more indexes

  // Safe column additions (for schema evolution)
  db.exec(`ALTER TABLE processed_posts ADD COLUMN error TEXT`);
}

Migration Safety

• All CREATE TABLE statements use IF NOT EXISTS
• All CREATE INDEX statements use IF NOT EXISTS
• Column additions use ALTER TABLE ADD COLUMN IF NOT EXISTS pattern
• Migrations are idempotent (safe to run multiple times)

Adding New Columns

When adding new columns to existing tables:

// In src/database/client.ts
const tableInfo = db.prepare(`PRAGMA table_info(processed_posts)`).all();
const hasErrorColumn = tableInfo.some((col: any) => col.name === 'error');

if (!hasErrorColumn) {
  db.exec("ALTER TABLE processed_posts ADD COLUMN error TEXT");
}

---

Query Patterns

Post Processing

Check if post was already processed:

SELECT id FROM processed_posts WHERE id = ?;

Mark post as processed:

INSERT OR REPLACE INTO processed_posts 
  (id, subreddit, author, title, image_url, image_analysis, 
   generated_message, processed_at, message_sent, error)
VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?);

Mark post as sent:

UPDATE processed_posts 
SET message_sent = 1 
WHERE id = ?;

Update post with analysis:

UPDATE processed_posts 
SET image_analysis = ?, generated_message = ? 
WHERE id = ?;

---

Message Tracking

Record a sent message:

INSERT INTO sent_messages 
  (post_id, recipient, subject, message, sent_at, strategy_id)
VALUES (?, ?, 'Stunning', ?, ?, ?);

Get recent messages:

SELECT * FROM sent_messages 
ORDER BY sent_at DESC 
LIMIT ?;

Update response status:

UPDATE sent_messages 
SET got_response = 1, response_checked_at = ? 
WHERE id = ?;

Check if user was previously messaged:

SELECT COUNT(*) FROM sent_messages WHERE recipient = ?;

---

Strategy Analytics

Get strategy statistics:

SELECT 
  strategy_id,
  COUNT(*) as sent,
  SUM(got_response) as responded
FROM sent_messages
WHERE strategy_id IS NOT NULL
GROUP BY strategy_id;

Get response rate by subreddit:

SELECT 
  p.subreddit,
  COUNT(*) as sent,
  SUM(s.got_response) as responded
FROM sent_messages s
JOIN processed_posts p ON s.post_id = p.id
GROUP BY p.subreddit;

Get daily response trend:

SELECT 
  DATE(sent_at, 'unixepoch') as date,
  COUNT(*) as sent,
  SUM(got_response) as responded
FROM sent_messages
GROUP BY DATE(sent_at, 'unixepoch')
ORDER BY date DESC
LIMIT 30;

---

Few-Shot Learning

Get successful examples for a subreddit:

SELECT 
  original_message,
  image_analysis,
  post_title,
  response_snippet
FROM successful_messages
WHERE subreddit = ?
ORDER BY created_at DESC
LIMIT 3;

Store successful pair:

INSERT INTO successful_messages 
  (sent_message_id, original_message, image_analysis, 
   post_title, response_snippet, subreddit, created_at)
VALUES (?, ?, ?, ?, ?, ?, ?);

---

Repository Methods

PostsRepository

class PostsRepository {
  /** Check if a post was already processed */
  getById(id: string): ProcessedPost | undefined

  /** Mark a post as processed */
  insert(post: ProcessedPostInsert): void

  /** Mark a post as having a message sent */
  markSent(id: string): void

  /** Update post with AI analysis results */
  updateAnalysis(id: string, analysis: string, message: string): void

  /** Get total count of processed posts */
  count(): number

  /** Get count by subreddit */
  countBySubreddit(subreddit: string): number
}

MessagesRepository

class MessagesRepository {
  /** Record a sent message */
  insert(message: MessageInsert): number

  /** Get recent sent messages */
  getRecent(limit: number): SentMessage[]

  /** Get messages by recipient */
  getByRecipient(recipient: string): SentMessage[]

  /** Update response status */
  markResponded(id: number, checkedAt: number): void

  /** Get strategy statistics */
  getStrategyStats(): StrategyStat[]

  /** Get response rate by subreddit */
  getSubredditStats(): SubredditStat[]

  /** Get daily response trend */
  getDailyTrend(days: number): DailyTrend[]

  /** Get total count of sent messages */
  count(): number

  /** Get count of responded messages */
  countResponded(): number
}

SuccessfulMessagesRepository

class SuccessfulMessagesRepository {
  /** Store a successful message/response pair */
  insert(pair: SuccessfulMessageInsert): void

  /** Get examples for few-shot learning */
  getExamples(subreddit: string, count: number): Example[]

  /** Get total count */
  count(): number

  /** Get recent examples */
  getRecent(limit: number): SuccessfulMessage[]
}

---

Data Retention

Current Behavior

All data is retained indefinitely. There is no automatic cleanup or retention policy.

Table	Retention	Notes
processed_posts	Indefinite	Grows with each post processed
sent_messages	Indefinite	Grows with each message sent
successful_messages	Indefinite	Grows with each response received

Estimated Growth

With default settings (20 messages/hour, 24/7 operation):

Timeframe	Posts	Messages	Responses (est. 30%)
1 day	~480	~288	~86
1 week	~3,360	~2,016	~605
1 month	~14,400	~8,640	~2,592
1 year	~175,200	~105,120	~31,536

Recommended Cleanup (Future)

Consider implementing automated cleanup:

-- Archive old processed posts
CREATE TABLE processed_posts_archive AS
SELECT * FROM processed_posts
WHERE processed_at < ?;

-- Delete archived posts
DELETE FROM processed_posts
WHERE processed_at < ?;

-- Keep last N months of data

Note

: Successful messages should NEVER be deleted as they are used for few-shot learning.