docs: add agent codebase guide
This commit is contained in:
jjbrunton committed
1 parent
742c06d0c4
commit
c630303b47
12 files changed
+881
-4483
No files matched your search
@@ -0,0 +1,59 @@
|
||||
# RedditBot documentation
|
||||
|
||||
This directory is the starting point for agents and maintainers. It explains the system at the level needed to make safe changes; source code, tests, database migrations, and the generated API reference remain authoritative.
|
||||
|
||||
## Read this first
|
||||
|
||||
RedditBot is a stateful, multi-account outreach and conversation application. It discovers eligible Reddit posts, prepares AI-generated openers, sends them through Reddit's Matrix chat service, follows replies under bounded policies, records outcomes, and exposes an authenticated operator dashboard and agent API.
|
||||
|
||||
The most important operational fact is that local and production configuration may be live-capable. Installing dependencies, building, and testing are safe. Starting the backend, invoking control endpoints, importing sessions, sending messages, or deploying may affect real Reddit accounts and requires explicit authorization.
|
||||
|
||||
## Documentation map
|
||||
|
||||
| Document | Use it when |
|
||||
| --- | --- |
|
||||
| [Product goals and safety](product-and-safety.md) | You need the purpose, product boundaries, eligibility policy, or definition of success. |
|
||||
| [Architecture](architecture.md) | You need the system shape, process lifecycle, dependency boundaries, or repository map. |
|
||||
| [Runtime flows](data-flow.md) | You are changing discovery, screening, candidate delivery, replies, chat sync, media, or realtime updates. |
|
||||
| [Service map](services.md) | You need to find the module that owns a responsibility or identify extension seams. |
|
||||
| [Data model](database.md) | You are changing persistence, migrations, account scoping, idempotency, or analytics. |
|
||||
| [Configuration](configuration.md) | You are adding a setting, changing precedence, or diagnosing startup validation. |
|
||||
| [API](api.md) | You are using or changing HTTP/MCP operations. This file is generated. |
|
||||
| [Frontend](frontend.md) | You are changing the React dashboard, shared contracts, queries, routes, or SSE behavior. |
|
||||
| [Development and testing](development.md) | You are setting up locally, implementing a change, or choosing verification commands. |
|
||||
| [Deployment and operations](deployment.md) | You are diagnosing or performing Docker/Dokploy work, backups, recovery, or live verification. |
|
||||
|
||||
## Fast orientation by task
|
||||
|
||||
- Candidate selection or outreach: read product-and-safety, data-flow, services, and database.
|
||||
- Conversation automation: read product-and-safety, data-flow, and the conversation state/claim sections in database.
|
||||
- Dashboard/API work: read frontend, api, and architecture.
|
||||
- Authentication or account work: read architecture, configuration, database, and deployment.
|
||||
- Production incident: read deployment first, then the flow and service that own the failing state.
|
||||
- Schema or config change: read database/configuration and update both transport contracts and tests where applicable.
|
||||
|
||||
## Sources of truth
|
||||
|
||||
Use this order when facts conflict:
|
||||
|
||||
1. Observable behavior and focused tests.
|
||||
2. Shared Zod transport contracts in `packages/api-contracts/src/`.
|
||||
3. Runtime Zod configuration in `src/config/index.ts` and migrations in `src/database/client.ts`.
|
||||
4. Owning implementation under `src/` or `apps/dashboard/src/`.
|
||||
5. Generated `docs/api.md` and live `/api/openapi.json`.
|
||||
6. Narrative documentation in this directory.
|
||||
|
||||
Documentation should explain why boundaries exist and where behavior lives. Avoid copying entire schemas, hard-coded production identifiers, current counts, or secrets into prose.
|
||||
|
||||
## Keeping documentation current
|
||||
|
||||
Every code change must include the documentation changes needed to keep this map accurate. In particular:
|
||||
|
||||
- Add or remove links here when documents change.
|
||||
- Update architecture and service ownership when dependencies move.
|
||||
- Update runtime flows when ordering, eligibility, retries, or idempotency changes.
|
||||
- Update configuration and data model documents with schema changes.
|
||||
- Run `npm run docs:api` after modifying `src/web/api-catalog.ts`.
|
||||
- Keep historical plans out of current-state documents; use Git history for retired designs.
|
||||
|
||||
See the repository-level [`agents.md`](../agents.md) for the mandatory agent workflow.
|
||||
+126
-904
File diff suppressed because it is too large.
Load diff
+69
-469
@@ -1,504 +1,104 @@
|
||||
# Configuration Guide
|
||||
# Configuration
|
||||
|
||||
Complete reference for configuring RedditBot.
|
||||
## Precedence and lifecycle
|
||||
|
||||
---
|
||||
Configuration is validated by `ConfigSchema` in `src/config/index.ts`.
|
||||
|
||||
## Table of Contents
|
||||
1. `DATABASE_PATH` identifies the database before application settings can be read.
|
||||
2. If `app_config` exists, the application-owned configuration is loaded from SQLite.
|
||||
3. Otherwise, `config.yaml` is loaded once and the validated application-owned subset is seeded into SQLite.
|
||||
4. Environment-owned secrets and database location are injected on every start.
|
||||
|
||||
1. [Configuration Overview](#configuration-overview)
|
||||
2. [Configuration Storage](#configuration-storage)
|
||||
3. [Environment Variables](#environment-variables)
|
||||
4. [Application Settings](#application-settings)
|
||||
5. [Runtime Configuration](#runtime-configuration)
|
||||
6. [Configuration Validation](#configuration-validation)
|
||||
7. [Troubleshooting](#troubleshooting)
|
||||
After the initial seed, editing `config.yaml` does not replace an existing `app_config` row. Use the authenticated settings/API workflow or a deliberate database migration. The config route validates the complete result before saving.
|
||||
|
||||
---
|
||||
## Environment-owned values
|
||||
|
||||
## Configuration Overview
|
||||
| Variable | Required | Purpose |
|
||||
| --- | --- | --- |
|
||||
| `OPENROUTER_API_KEY` | Yes | OpenRouter authentication. |
|
||||
| `WEB_DASHBOARD_TOKEN` | When web is enabled | Administrator UI/API credential. |
|
||||
| `DATABASE_PATH` | No | SQLite location; defaults to `./data/reddit-bot.db`. |
|
||||
| `REDDIT_BOT_AGENT_READ_TOKEN` | No | Read-only API/SSE bearer credential. |
|
||||
| `REDDIT_BOT_AGENT_CONTROL_TOKEN` | No | Read plus versioned control/job bearer credential. |
|
||||
| `REDDIT_BOT_API_URL` | MCP only | Base URL; defaults to `http://127.0.0.1:3000`. |
|
||||
| `REDDIT_BOT_AGENT_TOKEN` | MCP only | Preferred MCP bearer credential override. |
|
||||
|
||||
RedditBot uses a layered configuration approach:
|
||||
Never document real values. `.env` is ignored and may contain production credentials; do not print or blindly source it.
|
||||
|
||||
```
|
||||
┌──────────────────────────────────────────────────────────────┐
|
||||
│ Priority Order (High → Low) │
|
||||
├──────────────────────────────────────────────────────────────┤
|
||||
│ 1. Environment Variables │
|
||||
│ (OPENROUTER_API_KEY, WEB_DASHBOARD_TOKEN) │
|
||||
│ │
|
||||
│ 2. config.yaml │
|
||||
│ (Primary configuration file) │
|
||||
│ │
|
||||
│ 3. Defaults │
|
||||
│ (Hardcoded defaults in code) │
|
||||
└──────────────────────────────────────────────────────────────┘
|
||||
```
|
||||
## Application sections
|
||||
|
||||
Configuration is loaded at startup by `loadConfig()` in `src/config/index.ts`, which:
|
||||
### `reddit`
|
||||
|
||||
1. Reads the singleton `app_config` record from SQLite
|
||||
2. Imports `config.yaml` only when upgrading an existing database that has no settings record
|
||||
3. Validates the entire configuration tree using Zod
|
||||
4. Returns a strongly-typed `Config` object
|
||||
- `subreddits`: at least one item with `name`, `enabled`, `type`, optional `weight`, `tone`, optional scripted opener, and conversation policy.
|
||||
- `pollIntervalMs`: legacy/scheduled polling cadence.
|
||||
- `userAgent`: Reddit JSON request identifier.
|
||||
- `targetGenders`: allowed parsed values.
|
||||
- `filterSellers`, `sellerKeywords`, `sellerDomains`: seller policy inputs.
|
||||
- `prioritizeRecentPosts`, `recentWindowMinutes`: recency/fallback behavior.
|
||||
- `maxReadyCandidates`: maximum prepared backlog admitted before screening pauses; valid range 1–500.
|
||||
|
||||
Application-owned settings are edited from the dashboard and saved to SQLite. `OPENROUTER_API_KEY`, `WEB_DASHBOARD_TOKEN`, `REDDIT_BOT_AGENT_READ_TOKEN`, `REDDIT_BOT_AGENT_CONTROL_TOKEN`, and `DATABASE_PATH` remain environment-owned bootstrap values and are never stored in `app_config`.
|
||||
Subreddit `type` is `image` or `chat`. Conversation policy contains `mode`, `maxAutomatedTurns`, and `rapportTurnsBeforeMediaAsk`; hybrid rapport turns must be lower than the maximum.
|
||||
|
||||
---
|
||||
### `openrouter`
|
||||
|
||||
## Configuration Storage
|
||||
Contains image/chat primary and fallback models plus the analysis, message, and chat-opener prompt templates. The API key is environment-only. Model fallback and prompt changes affect cost, behavior, and experiment attribution and need focused tests plus operator-visible metadata.
|
||||
|
||||
### Legacy config.yaml
|
||||
### `rateLimit`
|
||||
|
||||
`config.yaml` is retained temporarily as a one-time migration source for existing installations. Once every deployed database contains an `app_config` row, it can be removed in a follow-up release.
|
||||
- `minDelayBetweenMessagesMs`;
|
||||
- `maxMessagesPerHour`;
|
||||
- `jitterMs`.
|
||||
|
||||
```
|
||||
RedditBot/
|
||||
├── config.yaml # ← Main configuration
|
||||
├── package.json
|
||||
├── tsconfig.json
|
||||
└── src/
|
||||
```
|
||||
These are per account. Reddit JSON request pacing is separately controlled by `RedditRequestCoordinator` and is process-global.
|
||||
|
||||
### Data Directory
|
||||
### `browser`
|
||||
|
||||
Runtime data is stored in `data/` (created automatically):
|
||||
- `sessionDir`: root for persistent account browser profiles.
|
||||
- `dryRun`: when true, candidate preparation may run but remote sends must not.
|
||||
|
||||
```
|
||||
data/
|
||||
├── reddit-bot.db # SQLite database
|
||||
├── browser/ # Browser session data (cookies, storage)
|
||||
├── media/ # Downloaded media files
|
||||
│ ├── index.json # Media index (SHA-1 → filename)
|
||||
│ ├── abc123.jpg # Media file 1
|
||||
│ └── def456.png # Media file 2
|
||||
└── chats/ # Cached conversation JSON files
|
||||
├── chat_abc123.json
|
||||
└── chat_def456.json
|
||||
```
|
||||
Treat changing `dryRun` from true to false as enabling a real side effect.
|
||||
|
||||
---
|
||||
### `database`
|
||||
|
||||
## Environment Variables
|
||||
Contains `path`, but runtime ownership is `DATABASE_PATH`, not the persisted application config.
|
||||
|
||||
Environment variables provide secrets and bootstrap settings that are intentionally excluded from SQLite.
|
||||
### `media`
|
||||
|
||||
| Variable | Required | Type | Description | Example |
|
||||
|---|---|---|---|---|
|
||||
| `OPENROUTER_API_KEY` | Yes | string | OpenRouter API key for AI services | `sk-or-v1-abc123...` |
|
||||
| `WEB_DASHBOARD_TOKEN` | Yes when dashboard enabled | string | Dashboard access token | `long-random-secret` |
|
||||
| `REDDIT_BOT_AGENT_READ_TOKEN` | No | string | Least-privilege GET/SSE agent access | `separate-read-secret` |
|
||||
| `REDDIT_BOT_AGENT_CONTROL_TOKEN` | No | string | Read access plus `/api/v1` controls/jobs | `separate-control-secret` |
|
||||
| `REDDIT_BOT_API_URL` | MCP only | URL | HTTP base URL used by `npm run mcp` | `http://127.0.0.1:3000` |
|
||||
| `REDDIT_BOT_AGENT_TOKEN` | MCP only | string | Token selected for the MCP process | `separate-read-secret` |
|
||||
| `DATABASE_PATH` | No | string | SQLite database location | `./data/reddit-bot.db` |
|
||||
Controls periodic sync enablement and interval. Media work remains authentication-bound.
|
||||
|
||||
### Setting Environment Variables
|
||||
### `strategies` and `chatStrategies`
|
||||
|
||||
**Option 1: `.env` file** (recommended for development)
|
||||
Each has exploration rate, minimum mature samples, response window, and weighted variants. Variants include an ID and prompt override. Image and chat sources use separate sections so their results are not compared as one experiment.
|
||||
|
||||
Create a `.env` file in the project root:
|
||||
### `web`
|
||||
|
||||
```env
|
||||
OPENROUTER_API_KEY=sk-or-v1-abc123...
|
||||
WEB_DASHBOARD_TOKEN=long-random-secret
|
||||
REDDIT_BOT_AGENT_READ_TOKEN=separate-read-secret
|
||||
REDDIT_BOT_AGENT_CONTROL_TOKEN=separate-control-secret
|
||||
```
|
||||
|
||||
> **Note**: Add `.env` to `.gitignore` - never commit credentials.
|
||||
|
||||
**Option 2: Shell export**
|
||||
|
||||
```bash
|
||||
export OPENROUTER_API_KEY=sk-or-v1-abc123...
|
||||
export WEB_DASHBOARD_TOKEN=long-random-secret
|
||||
export REDDIT_BOT_AGENT_READ_TOKEN=separate-read-secret
|
||||
export REDDIT_BOT_AGENT_CONTROL_TOKEN=separate-control-secret
|
||||
```
|
||||
|
||||
**Option 3: Docker**
|
||||
|
||||
Pass via `docker-compose.yml` or Docker CLI:
|
||||
|
||||
```yaml
|
||||
environment:
|
||||
- OPENROUTER_API_KEY=sk-or-v1-abc123...
|
||||
- WEB_DASHBOARD_TOKEN=long-random-secret
|
||||
- REDDIT_BOT_AGENT_READ_TOKEN=separate-read-secret
|
||||
- REDDIT_BOT_AGENT_CONTROL_TOKEN=separate-control-secret
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Application Settings
|
||||
|
||||
Use **Settings** in the dashboard for monitoring, rate limits, dry-run behaviour, subreddits, models, prompts, filtering, media sync, and strategy configuration. The advanced JSON editor exposes settings that do not yet have dedicated controls; its payload is validated by the server before it is saved.
|
||||
|
||||
The YAML examples below document the legacy field layout used during one-time migration.
|
||||
|
||||
Complete reference for every configuration option.
|
||||
|
||||
### Reddit Settings
|
||||
|
||||
Each subreddit also has a conversation behaviour in the dashboard:
|
||||
|
||||
- `legacy` preserves the existing three-turn media-request flow.
|
||||
- `r4r` continues general conversation autonomously and never requests media.
|
||||
- `r4r-media` continues general conversation for `rapportTurnsBeforeMediaAsk`, makes one optional non-explicit image request, and never repeats it.
|
||||
- `manual` records the reply for human handling without sending an automated response.
|
||||
|
||||
`maxAutomatedTurns` is configurable for `r4r` and `r4r-media` up to 20. In hybrid mode, `rapportTurnsBeforeMediaAsk` must be lower than `maxAutomatedTurns`. A conversation snapshots these values when its first verified reply arrives, so later settings changes affect new conversations only.
|
||||
|
||||
```yaml
|
||||
reddit:
|
||||
# Subreddits to monitor for new posts
|
||||
subreddits:
|
||||
- gonewild
|
||||
- ratemyboobs
|
||||
- HomemadeNsfw
|
||||
- realgirls
|
||||
- plussize_gonewild
|
||||
# ... add or remove subreddits
|
||||
|
||||
# Minutes between polling cycles
|
||||
pollInterval: 15
|
||||
|
||||
# Target gender for image analysis filtering
|
||||
# Options: "F" (female), "M" (male)
|
||||
targetGender: "F"
|
||||
|
||||
# Keywords used to detect sellers/spam profiles
|
||||
# Posts from users matching these keywords are skipped
|
||||
sellerKeywords:
|
||||
- seller
|
||||
- onlyfans
|
||||
- link in bio
|
||||
- check my profile
|
||||
- promo
|
||||
- discount
|
||||
- affiliate
|
||||
```
|
||||
|
||||
### OpenRouter AI Settings
|
||||
|
||||
```yaml
|
||||
openrouter:
|
||||
# OpenRouter API key (can also be set via OPENROUTER_API_KEY env var)
|
||||
apiKey: "sk-or-v1-abc123..."
|
||||
|
||||
# Primary AI model for image analysis
|
||||
# Models must be available on OpenRouter
|
||||
model: "meta-llama/llama-3.2-90b-vision-preview"
|
||||
|
||||
# Fallback models tried if the primary model refuses or fails
|
||||
fallbackModels:
|
||||
- "google/gemini-2.0-flash-001"
|
||||
- "meta-llama/llama-3.2-11b-vision-instruct"
|
||||
|
||||
# Quick detection model (cheaper/faster, used for gender pre-filter)
|
||||
quickModel: "google/gemini-2.0-flash-lite-001"
|
||||
|
||||
# System prompt for image analysis
|
||||
# The AI will describe the image and detect gender
|
||||
systemPrompt: |
|
||||
Analyze this image and provide:
|
||||
1. Gender detected (F or M)
|
||||
2. Detailed description of the image content
|
||||
3. Notable features (hair color, clothing, setting, etc.)
|
||||
|
||||
Return as JSON: {"gender": "F", "description": "...", "features": [...]}
|
||||
|
||||
# System prompt for message generation
|
||||
# Used when crafting DMs to post authors
|
||||
messagePrompt: |
|
||||
Generate a natural, friendly DM for this Reddit post.
|
||||
|
||||
Image analysis: {description}
|
||||
Post title: {title}
|
||||
Strategy: {strategy}
|
||||
|
||||
Guidelines:
|
||||
- Be complimentary but not creepy
|
||||
- Reference something specific from the image
|
||||
- Keep it under 2 sentences
|
||||
- Sound like a genuine person, not a bot
|
||||
- Do NOT ask for anything in return
|
||||
```
|
||||
|
||||
### Rate Limiting
|
||||
|
||||
```yaml
|
||||
rateLimit:
|
||||
# Minimum delay between messages (milliseconds)
|
||||
# Higher values reduce detection risk
|
||||
minDelayMs: 30000
|
||||
|
||||
# Maximum messages per hour
|
||||
# Bot will pause when this limit is reached
|
||||
maxMessagesPerHour: 20
|
||||
|
||||
# Random jitter added to delay (milliseconds)
|
||||
# Adds unpredictability to timing
|
||||
jitterMs: 15000
|
||||
```
|
||||
|
||||
**Timing example with defaults:**
|
||||
```
|
||||
Minimum delay: 30,000ms (30 seconds)
|
||||
Jitter: 0 to 15,000ms (0-15 seconds)
|
||||
Effective delay: 30-45 seconds between messages
|
||||
Max messages: 20 per hour
|
||||
```
|
||||
|
||||
### Browser Settings
|
||||
|
||||
```yaml
|
||||
browser:
|
||||
# Persistent managed-browser data populated only by verified imports
|
||||
sessionDir: "./data/browser-session"
|
||||
dryRun: false
|
||||
```
|
||||
|
||||
**Browser session persistence:**
|
||||
|
||||
```
|
||||
data/browser/
|
||||
├── Default/
|
||||
│ ├── Cookies # Reddit session cookies
|
||||
│ ├── Local Storage # Local storage data
|
||||
│ └── Preferences # Browser preferences
|
||||
├── Login Data # Saved login info
|
||||
└── Preferences # Browser state
|
||||
```
|
||||
|
||||
> **Important**: Keep this directory intact. Deleting it requires re-login.
|
||||
|
||||
### Database Settings
|
||||
|
||||
```yaml
|
||||
database:
|
||||
# Path to SQLite database file
|
||||
path: "./data/reddit-bot.db"
|
||||
```
|
||||
|
||||
The database is auto-created on first run. Migrations run automatically on startup.
|
||||
|
||||
### Media Sync Settings
|
||||
|
||||
```yaml
|
||||
mediaSync:
|
||||
# Enable periodic media synchronization
|
||||
enabled: true
|
||||
|
||||
# Interval between sync cycles (minutes)
|
||||
intervalMinutes: 30
|
||||
```
|
||||
|
||||
**Sync behavior:**
|
||||
- **Incremental sync** (automatic): Downloads media from top 10 conversations
|
||||
- **Full sync** (manual via dashboard): Downloads media from all conversations
|
||||
|
||||
### Auto-Reply Settings
|
||||
|
||||
```yaml
|
||||
autoReply:
|
||||
# Enable automatic chat replies
|
||||
enabled: true
|
||||
|
||||
# Auto-send replies (true) or mark as pending for approval (false)
|
||||
autoSend: false
|
||||
|
||||
# Minutes between checking for new messages
|
||||
checkInterval: 10
|
||||
```
|
||||
|
||||
**Reply modes:**
|
||||
|
||||
| autoSend | Behavior |
|
||||
|---|---|
|
||||
| `true` | Bot sends AI-generated replies automatically |
|
||||
| `false` | Bot suggests replies, user approves via dashboard |
|
||||
|
||||
### Strategy Variants (A/B Testing)
|
||||
|
||||
```yaml
|
||||
strategyVariants:
|
||||
# Variant 1: Compliment-based
|
||||
- name: "compliment"
|
||||
prompt: |
|
||||
Write a friendly compliment about the person's appearance.
|
||||
Be genuine and specific. Keep it short and natural.
|
||||
enabled: true
|
||||
|
||||
# Variant 2: Question-based
|
||||
- name: "question"
|
||||
prompt: |
|
||||
Write a friendly message that includes a question.
|
||||
Engage them in conversation naturally.
|
||||
enabled: true
|
||||
|
||||
# Variant 3: Casual observation
|
||||
- name: "observation"
|
||||
prompt: |
|
||||
Write a casual observation about the photo or setting.
|
||||
Sound like you're commenting to a friend.
|
||||
enabled: true
|
||||
```
|
||||
|
||||
### Web Dashboard Settings
|
||||
|
||||
```yaml
|
||||
web:
|
||||
# Port for the web dashboard
|
||||
port: 3000
|
||||
```
|
||||
|
||||
### Global Settings
|
||||
|
||||
```yaml
|
||||
# Global dry run mode
|
||||
# When true, all actions are logged but not actually performed
|
||||
dryRun: false
|
||||
|
||||
# Number of successful message examples to include in prompts
|
||||
fewShotExamples: 3
|
||||
|
||||
# Minimum responses needed before trusting strategy stats
|
||||
minStrategySamples: 10
|
||||
|
||||
# Exploration rate for A/B testing (0.0 to 1.0)
|
||||
# 0.15 = 15% of messages use random variants (exploration)
|
||||
explorationRate: 0.15
|
||||
```
|
||||
Controls enabled state, port, optional tokens after environment injection, and dangerous debug endpoint enablement. Production secrets should never live in YAML.
|
||||
|
||||
---
|
||||
|
||||
## Runtime Configuration
|
||||
|
||||
The following settings can be modified via the web dashboard without restarting the bot:
|
||||
|
||||
| Setting | Endpoint | Type | Description |
|
||||
|---|---|---|---|
|
||||
| `rateLimit.minDelayMs` | `PUT /api/config` | number | Message delay |
|
||||
| `rateLimit.maxMessagesPerHour` | `PUT /api/config` | number | Hourly cap |
|
||||
| `rateLimit.jitterMs` | `PUT /api/config` | number | Jitter range |
|
||||
| `reddit.pollInterval` | `PUT /api/config` | number | Poll frequency |
|
||||
| `dryRun` | `PUT /api/config` | boolean | Dry run mode |
|
||||
|
||||
### Updating Runtime Config
|
||||
|
||||
Via dashboard:
|
||||
1. Open the settings modal
|
||||
2. Modify desired values
|
||||
3. Click "Save"
|
||||
|
||||
Via API:
|
||||
```bash
|
||||
curl -X PUT http://localhost:3000/api/config \
|
||||
-H "Content-Type: application/json" \
|
||||
-d '{
|
||||
"rateLimit": {
|
||||
"minDelayMs": 45000,
|
||||
"maxMessagesPerHour": 15,
|
||||
"jitterMs": 20000
|
||||
},
|
||||
"dryRun": true
|
||||
}'
|
||||
```
|
||||
|
||||
> **Note**: Runtime changes are in-memory only. They are lost when the bot restarts. To make changes persistent, update `config.yaml`.
|
||||
## Runtime-editable subset
|
||||
|
||||
---
|
||||
|
||||
## Configuration Validation
|
||||
The dashboard/API exposes a deliberate subset: rate limits, dry-run, recency and ready-funnel limits, subreddit definitions/conversation behavior, and selected advanced configuration. Shared transport schemas live in `packages/api-contracts/src/config.ts`; the backend source schema is still the final validator.
|
||||
|
||||
All configuration is validated using Zod at startup. Invalid configuration will cause the bot to exit with a descriptive error.
|
||||
|
||||
### Validation Rules
|
||||
|
||||
| Field | Validation |
|
||||
|---|---|
|
||||
| `reddit.subreddits` | Must be non-empty array of strings |
|
||||
| `reddit.pollInterval` | Must be positive integer |
|
||||
| `reddit.targetGender` | Must be "F" or "M" |
|
||||
| `openrouter.apiKey` | Must be non-empty string |
|
||||
| `openrouter.model` | Must be non-empty string |
|
||||
| `openrouter.fallbackModels` | Must be non-empty array |
|
||||
| `rateLimit.minDelayMs` | Must be positive integer |
|
||||
| `rateLimit.maxMessagesPerHour` | Must be positive integer |
|
||||
| `rateLimit.jitterMs` | Must be non-negative integer |
|
||||
| `browser.sessionDir` | Must be non-empty string |
|
||||
| `browser.dryRun` | Must be boolean |
|
||||
| `database.path` | Must be non-empty string |
|
||||
| `mediaSync.enabled` | Must be boolean |
|
||||
| `mediaSync.intervalMinutes` | Must be positive integer (if enabled) |
|
||||
| `autoReply.enabled` | Must be boolean |
|
||||
| `autoReply.autoSend` | Must be boolean |
|
||||
| `autoReply.checkInterval` | Must be positive integer (if enabled) |
|
||||
| `strategyVariants` | Must be array with name + prompt |
|
||||
| `web.port` | Must be integer 1-65535 |
|
||||
| `dryRun` | Must be boolean |
|
||||
When adding a runtime-editable setting, update all of:
|
||||
|
||||
### Common Validation Errors
|
||||
1. `ConfigSchema` and defaults.
|
||||
2. `config.yaml` seed/example when appropriate.
|
||||
3. `ConfigStore` persistence rules if ownership changes.
|
||||
4. Shared request/response schemas.
|
||||
5. The config route's merge/write/read-back behavior.
|
||||
6. Dashboard model/control.
|
||||
7. backend, contract, route, and dashboard tests.
|
||||
8. this document.
|
||||
|
||||
**Error**: `Invalid config: openrouter.apiKey must be a string`
|
||||
**Fix**: Set `OPENROUTER_API_KEY` environment variable or add `apiKey` to `config.yaml`
|
||||
## Validation and change safety
|
||||
|
||||
**Error**: `Invalid config: reddit.subreddits must be a non-empty array`
|
||||
**Fix**: Add at least one subreddit to `reddit.subreddits` in `config.yaml`
|
||||
- Use Zod for every external or persisted shape.
|
||||
- Reject unknown transport fields with strict contracts.
|
||||
- Validate relationships (not just types), such as the hybrid turn constraint.
|
||||
- Do not log the full config because it contains injected secrets in memory.
|
||||
- Configuration that affects running account objects may not be fully hot-reloaded; inspect the owning service before claiming immediate effect.
|
||||
- A production `PUT /api/config` should be followed by an authenticated GET read-back. Do not infer persistence from a 2xx alone.
|
||||
- A subreddit listing 404 is persisted as disabled so every account runtime stops polling it.
|
||||
|
||||
**Error**: `Invalid config: browser.sessionDir must be a string`
|
||||
**Fix**: Ensure `browser.sessionDir` is set in `config.yaml`
|
||||
## Local safe setup
|
||||
|
||||
---
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
### Configuration Not Loading
|
||||
|
||||
**Symptom**: Bot fails to start with validation errors.
|
||||
|
||||
**Steps**:
|
||||
1. Verify `config.yaml` is in the project root
|
||||
2. Check YAML syntax (use a YAML linter)
|
||||
3. Verify environment variables are set correctly
|
||||
4. Check bot logs for specific validation errors
|
||||
|
||||
### Account Authentication Not Working
|
||||
|
||||
**Symptom**: Account work remains paused.
|
||||
|
||||
**Steps**:
|
||||
1. Sign in to Reddit in local Chrome
|
||||
2. Open Reddit Chat and wait for the extension badge to show `OK`
|
||||
3. Export and upload a fresh combined session dump
|
||||
4. Check dashboard logs for Reddit or Matrix identity verification errors
|
||||
|
||||
### AI Not Working
|
||||
|
||||
**Symptom**: Image analysis fails or returns errors.
|
||||
|
||||
**Steps**:
|
||||
1. Verify `OPENROUTER_API_KEY` is valid
|
||||
2. Check API key has sufficient balance
|
||||
3. Verify `openrouter.model` is available on OpenRouter
|
||||
4. Check fallback models are set correctly
|
||||
5. Review logs for specific AI errors
|
||||
|
||||
### Browser Session Expired
|
||||
|
||||
**Symptom**: Bot stops with "Browser session expired" error.
|
||||
|
||||
**Steps**:
|
||||
1. Open Reddit Chat in the local Chrome profile containing the extension
|
||||
2. Export and upload a fresh session dump
|
||||
3. Delete the downloaded dump after verification
|
||||
|
||||
### Port Already in Use
|
||||
|
||||
**Symptom**: Dashboard fails to start with "Port already in use".
|
||||
|
||||
**Steps**:
|
||||
1. Change `web.port` in `config.yaml`
|
||||
2. Or kill the process using the port: `lsof -i :3000`
|
||||
Install dependencies and run builds/tests without starting the application. If a local backend is explicitly needed, use a dry-run configuration and verify the database path/port before starting. Importing a real session or calling the start control is a separate live-capable action.
|
||||
+85
-872
@@ -1,892 +1,105 @@
|
||||
# Data Flow Documentation
|
||||
# Runtime flows
|
||||
|
||||
Complete documentation of data flows through the RedditBot system.
|
||||
## Candidate funnel
|
||||
|
||||
---
|
||||
The production multi-account path is three independent loops backed by `candidate_queue`.
|
||||
|
||||
## Table of Contents
|
||||
|
||||
1. [Overview](#overview)
|
||||
2. [Post Processing Flow](#post-processing-flow)
|
||||
3. [Response Monitoring Flow](#response-monitoring-flow)
|
||||
4. [Chat Interaction Flow](#chat-interaction-flow)
|
||||
5. [Strategy Selection Flow](#strategy-selection-flow)
|
||||
6. [Media Sync Flow](#media-sync-flow)
|
||||
7. [Auto-Reply Flow](#auto-reply-flow)
|
||||
8. [Dashboard Data Flow](#dashboard-data-flow)
|
||||
9. [Error Recovery Flows](#error-recovery-flows)
|
||||
10. [Data Lifecycle](#data-lifecycle)
|
||||
|
||||
---
|
||||
|
||||
## Overview
|
||||
|
||||
RedditBot processes data through several interconnected pipelines:
|
||||
|
||||
```
|
||||
┌─────────────┐ ┌──────────────┐ ┌─────────────┐
|
||||
│ Reddit API │────►│ Monitor │────►│ Processor │
|
||||
│ (Posts) │ │ Service │ │ Service │
|
||||
└─────────────┘ └──────────────┘ └──────┬──────┘
|
||||
│
|
||||
┌──────────────┐ ▼
|
||||
│ Response │ ┌──────────────┐
|
||||
│ Checker │◄────│ OpenRouter │
|
||||
└──────────────┘ │ (AI) │
|
||||
└──────────────┘
|
||||
│
|
||||
┌──────────────┐ ▼
|
||||
│ Chat │ ┌──────────────┐
|
||||
│ Service │◄────│ Message │
|
||||
└──────────────┘ │ Sender │
|
||||
└──────────────┘
|
||||
│
|
||||
┌──────────────┐ ▼
|
||||
│ Media │ ┌──────────────┐
|
||||
│ Service │◄────│ SQLite DB │
|
||||
└──────────────┘ └──────────────┘
|
||||
│
|
||||
┌──────────────┐ │
|
||||
│ Dashboard │◄──────────┘
|
||||
│ (Express) │
|
||||
└──────────────┘
|
||||
```mermaid
|
||||
stateDiagram-v2
|
||||
[*] --> discovered: listing intake
|
||||
discovered --> screening: leased by screening worker
|
||||
screening --> ready: eligibility and preparation pass
|
||||
screening --> retry_wait: transient failure
|
||||
screening --> rejected: terminal eligibility or AI failure
|
||||
ready --> sending: leased to available account
|
||||
sending --> sent: Matrix confirms conversation
|
||||
sending --> retry_wait: account cooldown or transient failure
|
||||
sending --> rejected: terminal delivery result
|
||||
retry_wait --> discovered: screening retry due
|
||||
retry_wait --> ready: delivery retry due
|
||||
discovered --> expired: outside configured recent window
|
||||
ready --> expired: outside configured recent window
|
||||
```
|
||||
|
||||
---
|
||||
### 1. Discovery
|
||||
|
||||
## Post Processing Flow
|
||||
The discovery loop rotates through enabled subreddits and authenticated runtimes. A runtime fetches Reddit `/new` data using the process-shared `RedditRequestCoordinator`. Intake deduplicates post IDs and authors, excludes processed/contacted history, stores the full post snapshot, source type, weight, and timestamps, and logs exact intake counts.
|
||||
|
||||
The primary data flow for discovering, analyzing, and messaging about new posts.
|
||||
The queue has a bounded active size. Screening also pauses at configured `reddit.maxReadyCandidates`, and due delivery retries are promoted only while that ready capacity exists. Reddit request pacing, not a tight application poll, is the primary limiter.
|
||||
|
||||
### Stage 1: Discovery
|
||||
### 2. Screening and preparation
|
||||
|
||||
```
|
||||
┌─────────────────────────────────────────────────────────────────┐
|
||||
│ MonitorService │
|
||||
│ │
|
||||
│ 1. Shuffle subreddit order │
|
||||
│ 2. For each subreddit: │
|
||||
│ ┌─────────────────────────────────────────────────────┐ │
|
||||
│ │ RedditApi.fetchPosts(subreddit) │ │
|
||||
│ │ │ │
|
||||
│ │ HTTP: GET /r/{subreddit}/new.json?limit=100 │ │
|
||||
│ │ │ │
|
||||
│ │ Response: { │ │
|
||||
│ │ data: { │ │
|
||||
│ │ children: [ │ │
|
||||
│ │ { │ │
|
||||
│ │ data: { │ │
|
||||
│ │ id: "abc123", │ │
|
||||
│ │ subreddit: "gonewild", │ │
|
||||
│ │ author: "user123", │ │
|
||||
│ │ title: "Summer vibes", │ │
|
||||
│ │ url: "https://...", │ │
|
||||
│ │ preview: { reddits: [...] }, │ │
|
||||
│ │ thumbnail: "https://...", │ │
|
||||
│ │ post_hint: "image", │ │
|
||||
│ │ over_18: true │ │
|
||||
│ │ } │ │
|
||||
│ │ } │ │
|
||||
│ │ ] │ │
|
||||
│ │ } │ │
|
||||
│ │ } │ │
|
||||
│ └─────────────────────────────────────────────────────┘ │
|
||||
│ │
|
||||
│ 3. Filter posts: │
|
||||
│ ├─► post_hint === "image" │
|
||||
│ ├─► author exists and is not deleted │
|
||||
│ ├─► image_url can be extracted │
|
||||
│ └─► Author not in seller/spam list │
|
||||
│ │
|
||||
│ 4. Cross-reference with DB: │
|
||||
│ ├─► PostsRepository.getById(postId) → skip if exists │
|
||||
│ └─► MessagesRepository.getByAuthor(author) → skip if sent │
|
||||
│ │
|
||||
│ 5. Return newest valid post │
|
||||
└─────────────────────────────────────────────────────────────────┘
|
||||
A screening worker leases one discovered candidate. `MonitorService.screenCandidate()` applies cheap rules before expensive/networked rules:
|
||||
|
||||
1. Global processed-post and recipient-history checks.
|
||||
2. For chat sources, self-declared gender and adult age parsing.
|
||||
3. Seller evidence from the post, profile/about, and recent activity.
|
||||
4. Source-specific preparation in `ProcessorService`.
|
||||
|
||||
For image posts, preparation downloads and validates the image, detects target gender, performs model analysis, selects an image strategy, and generates an opener. For chat posts, it selects a chat strategy and generates from the post text. Both paths apply outbound safety checks and return either a prepared immutable result or an explicit skip reason.
|
||||
|
||||
Terminal results become rejected candidates. Rate limits and transient failures become retryable candidates. Prepared candidates become ready; the persisted snapshot means a different account can deliver without repeating AI work.
|
||||
|
||||
### 3. Dispatch
|
||||
|
||||
An account is eligible for new outreach only when its runtime is running, both identities verify, it has no actionable reply backlog, there is no invitation cooldown, and its rate limiter permits a send. Available accounts are ordered by their current hourly send count.
|
||||
|
||||
Matrix cooldowns retain the operation that triggered them. A room-creation or invitation limit pauses new outreach but does not suppress replies in existing rooms; a limit returned while sending into an existing room pauses those sends as well.
|
||||
|
||||
The dispatch worker leases a ready candidate for an account, materializes the processed post in that account's repository, applies the account rate limiter, checks dry-run, and atomically claims the recipient globally. `ChatService.startConversationDetailed()` returns one of:
|
||||
|
||||
- `sent`: persist the message and outcome metadata, commit recipient history, update activity, and mark the candidate sent;
|
||||
- `terminal`: release the recipient reservation and reject the candidate;
|
||||
- `retryable`: release/defer so another healthy account can claim it soon.
|
||||
|
||||
Only Matrix confirmation makes a send successful. A healthy HTTP process, room identifier lookup, or attempted request does not.
|
||||
|
||||
## Reddit request and cooldown flow
|
||||
|
||||
Every runtime shares one `RedditRequestCoordinator`, because Reddit can enforce limits by origin as well as account. JSON requests are serialized with a minimum interval; failed requests consume spacing too. A 429 extends the global cooldown using the server delay plus a buffer. Broad 403 behavior can also result in coordinated backoff. A listing 404 is treated as terminal for that subreddit and persisted as `enabled=false`.
|
||||
|
||||
Do not add local retries around this layer without preserving the shared schedule. Per-account bursts defeat the coordinator.
|
||||
|
||||
## Reply-first conversation automation
|
||||
|
||||
Each account runtime refreshes recent chats and processes actionable unread conversations oldest-first before it can accept new outreach.
|
||||
|
||||
```mermaid
|
||||
flowchart TD
|
||||
IN["Unread inbound Matrix event"] --> VERIFIED{"Confirmed after a recorded opener?"}
|
||||
VERIFIED -- no --> IGNORE["Ignore as unverified"]
|
||||
VERIFIED -- yes --> CLAIM{"Durable event claim exists?"}
|
||||
CLAIM -- sent --> IGNORE
|
||||
CLAIM -- pending --> RECON["Reconcile against Matrix history"]
|
||||
CLAIM -- no --> STATE["Load or snapshot conversation policy"]
|
||||
STATE --> STOP{"Manual, seller, image, boundary, unsafe, or turn limit?"}
|
||||
STOP -- yes --> HANDOFF["Persist terminal/human-required outcome"]
|
||||
STOP -- no --> GEN["Generate policy-specific reply"]
|
||||
GEN --> SAFE{"Outbound checks pass?"}
|
||||
SAFE -- no --> HANDOFF
|
||||
SAFE -- yes --> PERSIST["Claim event and persist intended state"]
|
||||
PERSIST --> SEND["Rate-limited Matrix send"]
|
||||
SEND --> MARK["Mark claim sent and record latency"]
|
||||
```
|
||||
|
||||
### Stage 2: Image Processing
|
||||
The inbound event key, intended reply, claim status, and conversation state make retries idempotent. If the process dies after Matrix accepted a reply, reconciliation finds the outbound event and marks the old claim sent instead of sending it again.
|
||||
|
||||
```
|
||||
┌─────────────────────────────────────────────────────────────────┐
|
||||
│ ProcessorService │
|
||||
│ │
|
||||
│ 1. Download Image │
|
||||
│ ┌─────────────────────────────────────────────────────┐ │
|
||||
│ │ ImageService.downloadImage(url) │ │
|
||||
│ │ │ │
|
||||
│ │ HTTP: GET {image_url} │ │
|
||||
│ │ │ │
|
||||
│ │ Response: Binary image data │ │
|
||||
│ │ │ │
|
||||
│ │ Validation: │ │
|
||||
│ │ ├─► MIME type: image/jpeg | png | webp │ │
|
||||
│ │ ├─► File size: < 10MB │ │
|
||||
│ │ └─► Not empty/broken │ │
|
||||
│ └─────────────────────────────────────────────────────┘ │
|
||||
│ │
|
||||
│ 2. Convert to base64 │
|
||||
│ ┌─────────────────────────────────────────────────────┐ │
|
||||
│ │ Buffer.toString('base64') │ │
|
||||
│ │ │ │
|
||||
│ │ Result: "data:image/jpeg;base64,/9j/4AAQSkZJR..." │ │
|
||||
│ └─────────────────────────────────────────────────────┘ │
|
||||
└─────────────────────────────────────────────────────────────────┘
|
||||
```
|
||||
## Chat and media synchronization
|
||||
|
||||
### Stage 3: Gender Detection (Pre-Filter)
|
||||
`ChatService` communicates with Reddit's Matrix homeserver, maintains an account-scoped SQLite-backed cache, resolves `t2_` identities to Reddit usernames, and emits sync progress/new-reply callbacks. Startup chooses full or delta sync from persisted cursor state. Operators can explicitly run tracked sync jobs through the versioned API.
|
||||
|
||||
```
|
||||
┌─────────────────────────────────────────────────────────────────┐
|
||||
│ OpenRouterService │
|
||||
│ │
|
||||
│ 1. detectGenderOnly(imageBase64) │
|
||||
│ ┌─────────────────────────────────────────────────────┐ │
|
||||
│ │ HTTP: POST https://openrouter.ai/api/v1/chat/ │ │
|
||||
│ │ completions │ │
|
||||
│ │ │ │
|
||||
│ │ Request: { │ │
|
||||
│ │ model: "fast-model", │ │
|
||||
│ │ messages: [ │ │
|
||||
│ │ { │ │
|
||||
│ │ role: "user", │ │
|
||||
│ │ content: [ │ │
|
||||
│ │ { type: "text", text: "Is this F or M?" }, │ │
|
||||
│ │ { type: "image_url", image_url: {...} } │ │
|
||||
│ │ ] │ │
|
||||
│ │ } │ │
|
||||
│ │ ] │ │
|
||||
│ │ } │ │
|
||||
│ │ │ │
|
||||
│ │ Response: { │ │
|
||||
│ │ choices: [{ message: { content: "F" } }] │ │
|
||||
│ │ } │ │
|
||||
│ └─────────────────────────────────────────────────────┘ │
|
||||
│ │
|
||||
│ 3. Compare with config.targetGender │
|
||||
│ ├─► Match: continue to full analysis │
|
||||
│ └─► No match: SKIP post (log: "Gender mismatch, skipping") │
|
||||
└─────────────────────────────────────────────────────────────────┘
|
||||
```
|
||||
`MediaService` resolves Matrix media, downloads it into the persistent data directory, and writes an account-scoped media index. `MediaSyncService` schedules incremental work only while the account is authenticated. Private media is served behind the same authorization boundary as the dashboard and APIs.
|
||||
|
||||
### Stage 4: Strategy Selection
|
||||
## Dashboard data and events
|
||||
|
||||
```
|
||||
┌─────────────────────────────────────────────────────────────────┐
|
||||
│ StrategySelector │
|
||||
│ │
|
||||
│ 1. Get strategy statistics from DB │
|
||||
│ ┌─────────────────────────────────────────────────────┐ │
|
||||
│ │ MessagesRepository.getStrategyStats() │ │
|
||||
│ │ │ │
|
||||
│ │ Result: { │ │
|
||||
│ │ variant_a: { sent: 50, responded: 15, rate: 0.30 },│
|
||||
│ │ variant_b: { sent: 30, responded: 12, rate: 0.40 },│
|
||||
│ │ variant_c: { sent: 5, responded: 1, rate: 0.20 } │
|
||||
│ │ } │ │
|
||||
│ └─────────────────────────────────────────────────────┘ │
|
||||
│ │
|
||||
│ 2. Apply exploration/exploit algorithm │
|
||||
│ ├─► Random roll < explorationRate → pick random variant │
|
||||
│ └─► Otherwise → pick highest response rate (min samples) │
|
||||
│ │
|
||||
│ 3. Return selected variant name │
|
||||
│ Example: "variant_b" │
|
||||
└─────────────────────────────────────────────────────────────────┘
|
||||
```
|
||||
Routes assemble snapshots from repositories and injected services. `LogBroadcaster`/the realtime event hub publishes typed events for bot state, sends, failures, conversations, replies, sync, sessions, rate limits, and logs. The React client validates incoming events, invalidates matching TanStack Query keys, and reconnects with replay support. Cached screens remain visible but stale during a disconnect.
|
||||
|
||||
### Stage 5: Full AI Analysis
|
||||
## Failure classification
|
||||
|
||||
```
|
||||
┌─────────────────────────────────────────────────────────────────┐
|
||||
│ OpenRouterService │
|
||||
│ │
|
||||
│ 1. analyzeImage(imageBase64) │
|
||||
│ ┌─────────────────────────────────────────────────────┐ │
|
||||
│ │ HTTP: POST https://openrouter.ai/api/v1/chat/ │ │
|
||||
│ │ completions │ │
|
||||
│ │ │ │
|
||||
│ │ Request: { │ │
|
||||
│ │ model: "primary-model", │ │
|
||||
│ │ messages: [ │ │
|
||||
│ │ { │ │
|
||||
│ │ role: "system", │ │
|
||||
│ │ content: config.openrouter.systemPrompt │ │
|
||||
│ │ }, │ │
|
||||
│ │ { │ │
|
||||
│ │ role: "user", │ │
|
||||
│ │ content: [ │ │
|
||||
│ │ { type: "text", text: "Describe this image" },│
|
||||
│ │ { type: "image_url", image_url: {...} } │ │
|
||||
│ │ ] │ │
|
||||
│ │ } │ │
|
||||
│ │ ] │ │
|
||||
│ │ } │ │
|
||||
│ │ │ │
|
||||
│ │ Response: { │ │
|
||||
│ │ choices: [{ │ │
|
||||
│ │ message: { │ │
|
||||
│ │ content: '{"gender":"F","description":"..."}' │ │
|
||||
│ │ } │ │
|
||||
│ │ }] │ │
|
||||
│ │ } │ │
|
||||
│ └─────────────────────────────────────────────────────┘ │
|
||||
│ │
|
||||
│ 2. Check for refusal │
|
||||
│ ├─► If refused → try fallback models │
|
||||
│ └─► If all failed → mark post with error, skip │
|
||||
│ │
|
||||
│ 3. Parse and return ImageAnalysis │
|
||||
│ { gender: "F", description: "Young woman, blonde hair..." }│
|
||||
└─────────────────────────────────────────────────────────────────┘
|
||||
```
|
||||
Use three categories consistently:
|
||||
|
||||
### Stage 6: Message Generation
|
||||
- Terminal: the candidate/action is unsuitable or impossible; persist a reason and do not retry automatically.
|
||||
- Retryable: service limits, temporary auth/network errors, or account cooldown; preserve the candidate and retry time.
|
||||
- Human-required: policy or safety cannot continue autonomously; stop automation and surface an attention item.
|
||||
|
||||
```
|
||||
┌─────────────────────────────────────────────────────────────────┐
|
||||
│ OpenRouterService │
|
||||
│ │
|
||||
│ 1. Retrieve few-shot examples │
|
||||
│ ┌─────────────────────────────────────────────────────┐ │
|
||||
│ │ SuccessfulMessagesRepository.getExamples( │ │
|
||||
│ │ subreddit, count: 3 │ │
|
||||
│ │ ) │ │
|
||||
│ │ │ │
|
||||
│ │ Result: [ │ │
|
||||
│ │ { │ │
|
||||
│ │ original_message: "Hey, you look amazing...", │ │
|
||||
│ │ response_snippet: "Thank you so much!", │ │
|
||||
│ │ subreddit: "gonewild" │ │
|
||||
│ │ }, │ │
|
||||
│ │ // ... more examples │ │
|
||||
│ │ ] │ │
|
||||
│ └─────────────────────────────────────────────────────┘ │
|
||||
│ │
|
||||
│ 2. Generate message │
|
||||
│ ┌─────────────────────────────────────────────────────┐ │
|
||||
│ │ HTTP: POST https://openrouter.ai/api/v1/chat/ │ │
|
||||
│ │ completions │ │
|
||||
│ │ │ │
|
||||
│ │ Request: { │ │
|
||||
│ │ model: "primary-model", │ │
|
||||
│ │ messages: [ │ │
|
||||
│ │ { role: "system", content: messagePrompt }, │ │
|
||||
│ │ { role: "user", content: [ │ │
|
||||
│ │ `Image: ${imageAnalysis.description}`, │ │
|
||||
│ │ `Post title: ${postTitle}`, │ │
|
||||
│ │ `Strategy: ${selectedVariant}`, │ │
|
||||
│ │ `Examples: ${JSON.stringify(examples)}` │ │
|
||||
│ │ ] │ │
|
||||
│ │ } │ │
|
||||
│ │ ] │ │
|
||||
│ │ } │ │
|
||||
│ └─────────────────────────────────────────────────────┘ │
|
||||
│ │
|
||||
│ 3. Return generated message │
|
||||
│ "Hey, I came across your post and..." │
|
||||
└─────────────────────────────────────────────────────────────────┘
|
||||
```
|
||||
|
||||
### Stage 7: DM Sending
|
||||
|
||||
```
|
||||
┌─────────────────────────────────────────────────────────────────┐
|
||||
│ MessagePage │
|
||||
│ │
|
||||
│ 1. Navigate to compose page │
|
||||
│ ┌─────────────────────────────────────────────────────┐ │
|
||||
│ │ page.goto("https://www.reddit.com/compose?to= │ │
|
||||
│ │ user123") │ │
|
||||
│ └─────────────────────────────────────────────────────┘ │
|
||||
│ │
|
||||
│ 2. Fill recipient field │
|
||||
│ ┌─────────────────────────────────────────────────────┐ │
|
||||
│ │ page.evaluate(() => { │ │
|
||||
│ │ const input = document.querySelector( │ │
|
||||
│ │ 'faceplate-autocomplete input' │ │
|
||||
│ │ ); │ │
|
||||
│ │ input.value = "user123"; │ │
|
||||
│ │ input.dispatchEvent(new Event('input', │ │
|
||||
│ │ { bubbles: true })); │ │
|
||||
│ │ }); │ │
|
||||
│ └─────────────────────────────────────────────────────┘ │
|
||||
│ │
|
||||
│ 3. Fill subject field ("Stunning") │
|
||||
│ 4. Fill body field (generated message) │
|
||||
│ 5. Click send button │
|
||||
│ 6. Verify success (check for confirmation) │
|
||||
│ │
|
||||
│ Result: true (sent) or false (failed) │
|
||||
└─────────────────────────────────────────────────────────────────┘
|
||||
```
|
||||
|
||||
### Stage 8: Database Recording
|
||||
|
||||
```
|
||||
┌─────────────────────────────────────────────────────────────────┐
|
||||
│ Database │
|
||||
│ │
|
||||
│ 1. INSERT INTO processed_posts │
|
||||
│ VALUES ( │
|
||||
│ 'abc123', -- post_id │
|
||||
│ 'gonewild', -- subreddit │
|
||||
│ 'user123', -- author │
|
||||
│ 'Summer vibes', -- title │
|
||||
│ 'https://...', -- image_url │
|
||||
│ '{"gender":"F",...}', -- image_analysis │
|
||||
│ 'Hey, I came across...', -- generated_message │
|
||||
│ 1716234567, -- processed_at │
|
||||
│ 1, -- message_sent │
|
||||
│ NULL -- error │
|
||||
│ ); │
|
||||
│ │
|
||||
│ 2. INSERT INTO sent_messages │
|
||||
│ VALUES ( │
|
||||
│ NULL, -- id (auto) │
|
||||
│ 'abc123', -- post_id │
|
||||
│ 'user123', -- recipient │
|
||||
│ 'Stunning', -- subject │
|
||||
│ 'Hey, I came across...', -- message │
|
||||
│ 1716234567, -- sent_at │
|
||||
│ 'variant_b', -- strategy_id │
|
||||
│ 0, - - got_response │
|
||||
│ NULL -- response_checked_at │
|
||||
│ ); │
|
||||
└─────────────────────────────────────────────────────────────────┘
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Response Monitoring Flow
|
||||
|
||||
How the bot detects and records user replies to sent DMs.
|
||||
|
||||
```
|
||||
┌─────────────────────────────────────────────────────────────────┐
|
||||
│ ResponseCheckerService │
|
||||
│ │
|
||||
│ 1. Navigate to inbox │
|
||||
│ ┌─────────────────────────────────────────────────────┐ │
|
||||
│ │ page.goto("https://www.reddit.com/message/inbox") │ │
|
||||
│ └─────────────────────────────────────────────────────┘ │
|
||||
│ │
|
||||
│ 2. Parse inbox items │
|
||||
│ ┌─────────────────────────────────────────────────────┐ │
|
||||
│ │ Extract from page: │ │
|
||||
│ │ [ │ │
|
||||
│ │ { │ │
|
||||
│ │ from: "user456", │ │
|
||||
│ │ subject: "Re: Stunning", │ │
|
||||
│ │ snippet: "Thank you so much! 😊", │ │
|
||||
│ │ timestamp: 1716234000, │ │
|
||||
│ │ link: "/message/xyz789" │ │
|
||||
│ │ }, │ │
|
||||
│ │ // ... more messages │ │
|
||||
│ │ ] │ │
|
||||
│ └─────────────────────────────────────────────────────┘ │
|
||||
│ │
|
||||
│ 3. Match responses to sent messages │
|
||||
│ ┌─────────────────────────────────────────────────────┐ │
|
||||
│ │ For each inbox item: │ │
|
||||
│ │ ├─► MessagesRepository.getByRecipient(from) │ │
|
||||
│ │ ├─► Find unmatched message within time window │ │
|
||||
│ │ ├─► Update: got_response = 1 │ │
|
||||
│ │ ├─► Update: response_checked_at = now │ │
|
||||
│ │ └─► Store in successful_messages │ │
|
||||
│ └─────────────────────────────────────────────────────┘ │
|
||||
│ │
|
||||
│ 4. Update strategy statistics │
|
||||
│ ┌─────────────────────────────────────────────────────┐ │
|
||||
│ │ MessagesRepository.updateStrategyStats( │ │
|
||||
│ │ strategy_id, got_response: true │ │
|
||||
│ │ ) │ │
|
||||
│ └─────────────────────────────────────────────────────┘ │
|
||||
│ │
|
||||
│ 5. Log results │
|
||||
│ └─► "Found 3 responses, updated 2 strategies" │
|
||||
└─────────────────────────────────────────────────────────────────┘
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Chat Interaction Flow
|
||||
|
||||
How the bot interacts with Reddit chat conversations.
|
||||
|
||||
```
|
||||
┌─────────────────────────────────────────────────────────────────┐
|
||||
│ ChatService │
|
||||
│ │
|
||||
│ ──── PHASE 1: FETCH CHATS ────────────────────────────────── │
|
||||
│ │
|
||||
│ 1. Navigate to chat page │
|
||||
│ page.goto("https://chat.reddit.com") │
|
||||
│ │
|
||||
│ 2. Traverse shadow DOM for conversations │
|
||||
│ ┌─────────────────────────────────────────────────────┐ │
|
||||
│ │ page.evaluate(() => { │ │
|
||||
│ │ // JS injection to pierce shadow DOM │ │
|
||||
│ │ const roots = document.querySelectorAll( │ │
|
||||
│ │ 'faceplate-chat-list' │ │
|
||||
│ │ ); │ │
|
||||
│ │ const chats = []; │ │
|
||||
│ │ roots.forEach(root => { │ │
|
||||
│ │ const shadow = root.shadowRoot; │ │
|
||||
│ │ const items = shadow.querySelectorAll( │ │
|
||||
│ │ '.chat-item' │ │
|
||||
│ │ ); │ │
|
||||
│ │ items.forEach(item => { │ │
|
||||
│ │ chats.push({ │ │
|
||||
│ │ id: item.dataset.chatId, │ │
|
||||
│ │ participant: item.querySelector( │ │
|
||||
│ │ '.username' │ │
|
||||
│ │ ).textContent, │ │
|
||||
│ │ lastMessage: item.querySelector( │ │
|
||||
│ │ '.message' │ │
|
||||
│ │ ).textContent, │ │
|
||||
│ │ timestamp: item.dataset.timestamp │ │
|
||||
│ │ }); │ │
|
||||
│ │ }); │ │
|
||||
│ │ }); │ │
|
||||
│ │ return chats; │ │
|
||||
│ │ }); │ │
|
||||
│ └─────────────────────────────────────────────────────┘ │
|
||||
│ │
|
||||
│ 3. Cache results │
|
||||
│ └─► Save to memory + disk JSON │
|
||||
│ │
|
||||
│ ──── PHASE 2: LOAD CONVERSATION ──────────────────────────── │
|
||||
│ │
|
||||
│ 1. Check cache │
|
||||
│ ├─► If cached and fresh: return from memory │
|
||||
│ └─► If not: fetch from disk or Reddit │
|
||||
│ │
|
||||
│ 2. Fetch full history (pagination) │
|
||||
│ ┌─────────────────────────────────────────────────────┐ │
|
||||
│ │ For each page of messages: │ │
|
||||
│ │ ├─► Navigate to chat view │ │
|
||||
│ │ ├─► Extract messages from shadow DOM │ │
|
||||
│ │ ├─► Scroll to load more (infinite scroll) │ │
|
||||
│ │ └─► Repeat until all loaded │ │
|
||||
│ └─────────────────────────────────────────────────────┘ │
|
||||
│ │
|
||||
│ 3. Return conversation object │
|
||||
│ { │
|
||||
│ chatId: "abc123", │
|
||||
│ participants: ["user456", "bot"], │
|
||||
│ messages: [ │
|
||||
│ { author: "user456", text: "Hey!", timestamp: ... }, │
|
||||
│ { author: "bot", text: "Hi there!", timestamp: ... }, │
|
||||
│ // ... more messages │
|
||||
│ ], │
|
||||
│ lastUpdated: 1716234567 │
|
||||
│ } │
|
||||
│ │
|
||||
│ ──── PHASE 3: SEND MESSAGE ───────────────────────────────── │
|
||||
│ │
|
||||
│ 1. Navigate to chat view │
|
||||
│ 2. Fill message input (shadow DOM) │
|
||||
│ 3. Apply typing delay (human-like) │
|
||||
│ ┌─────────────────────────────────────────────────────┐ │
|
||||
│ │ const typingDelay = random(500, 2000); │ │
|
||||
│ │ await page.waitForTimeout(typingDelay); │ │
|
||||
│ │ │ │
|
||||
│ │ // Simulate typing character by character │ │
|
||||
│ │ for (const char of message) { │ │
|
||||
│ │ input.value += char; │ │
|
||||
│ │ await page.waitForTimeout(random(30, 100)); │ │
|
||||
│ │ } │ │
|
||||
│ └─────────────────────────────────────────────────────┘ │
|
||||
│ 4. Submit message │
|
||||
│ 5. Verify delivery (check for sent indicator) │
|
||||
│ │
|
||||
│ Result: true (delivered) or false (failed) │
|
||||
└─────────────────────────────────────────────────────────────────┘
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Strategy Selection Flow
|
||||
|
||||
How A/B test variants are selected and updated.
|
||||
|
||||
```
|
||||
┌─────────────────────────────────────────────────────────────────┐
|
||||
│ StrategySelector │
|
||||
│ │
|
||||
│ SELECT STRATEGY │
|
||||
│ ──────────────────────────────────────────────────────────── │
|
||||
│ │
|
||||
│ Input: subreddit, author │
|
||||
│ │
|
||||
│ 1. Get stats from DB │
|
||||
│ ┌─────────────────────────────────────────────────────┐ │
|
||||
│ │ SELECT strategy_id, │ │
|
||||
│ │ COUNT(*) as sent, │ │
|
||||
│ │ SUM(got_response) as responded │ │
|
||||
│ │ FROM sent_messages │ │
|
||||
│ │ GROUP BY strategy_id │ │
|
||||
│ └─────────────────────────────────────────────────────┘ │
|
||||
│ │
|
||||
│ 2. Calculate response rates │
|
||||
│ variant_a: 15/50 = 30% │
|
||||
│ variant_b: 12/30 = 40% │
|
||||
│ variant_c: 1/5 = 20% │
|
||||
│ │
|
||||
│ 3. Filter by minimum samples │
|
||||
│ minSamples = 10 │
|
||||
│ variant_c excluded (only 5 sends) │
|
||||
│ │
|
||||
│ 4. Explore vs Exploit │
|
||||
│ random() < explorationRate (e.g., 0.15)? │
|
||||
│ ├─► Yes: Random pick from enabled variants │
|
||||
│ └─► No: Pick highest rate (variant_b at 40%) │
|
||||
│ │
|
||||
│ Output: selected variant name │
|
||||
│ │
|
||||
│ ──── UPDATE STATS ────────────────────────────────────────── │
|
||||
│ │
|
||||
│ Input: strategy_id, got_response (boolean) │
|
||||
│ │
|
||||
│ 1. Record in sent_messages │
|
||||
│ UPDATE sent_messages │
|
||||
│ SET got_response = 1, │
|
||||
│ response_checked_at = now │
|
||||
│ WHERE id = ? │
|
||||
│ │
|
||||
│ 2. If successful: store in successful_messages │
|
||||
│ INSERT INTO successful_messages │
|
||||
│ VALUES (original_message, image_analysis, post_title, │
|
||||
│ response_snippet, subreddit, now) │
|
||||
│ │
|
||||
│ 3. Update strategy statistics │
|
||||
│ (reflected in next selectStrategy call) │
|
||||
└─────────────────────────────────────────────────────────────────┘
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Media Sync Flow
|
||||
|
||||
How media files are downloaded and synchronized from chat conversations.
|
||||
|
||||
```
|
||||
┌─────────────────────────────────────────────────────────────────┐
|
||||
│ MediaSyncService │
|
||||
│ │
|
||||
│ ──── TRIGGER ──────────────────────────────────────────────── │
|
||||
│ │
|
||||
│ Scheduled: every N minutes (config.mediaSync.intervalMinutes) │
|
||||
│ Manual: POST /api/media/sync (incremental) │
|
||||
│ Manual: POST /api/media/sync/full (full) │
|
||||
│ │
|
||||
│ ──── INCREMENTAL SYNC ────────────────────────────────────── │
|
||||
│ │
|
||||
│ 1. Get top 10 conversations from ChatService │
|
||||
│ 2. For each conversation: │
|
||||
│ ├─► Check if media already downloaded (index lookup) │
|
||||
│ ├─► If not: download media │
|
||||
│ │ └─► MediaService.downloadMedia(chatId) │
|
||||
│ └─► Update index │
|
||||
│ │
|
||||
│ ──── FULL SYNC ────────────────────────────────────────────── │
|
||||
│ │
|
||||
│ 1. Get ALL conversations from ChatService │
|
||||
│ 2. For each conversation: │
|
||||
│ ├─► Download any missing media │
|
||||
│ └─► Update index │
|
||||
│ │
|
||||
│ ──── MEDIA DOWNLOAD (MediaService) ────────────────────────── │
|
||||
│ │
|
||||
│ 1. Extract media URLs from conversation messages │
|
||||
│ 2. For each URL: │
|
||||
│ ├─► Check if Matrix thumbnail │
|
||||
│ │ └─► Resolve to full resolution URL │
|
||||
│ ├─► Download via Playwright │
|
||||
│ ├─► Compute SHA-1 hash │
|
||||
│ ├─► Check if already exists (dedup) │
|
||||
│ ├─► Save to data/media/{hash}.{ext} │
|
||||
│ └─► Update index JSON │
|
||||
│ │
|
||||
│ Output: Array of saved file paths │
|
||||
└─────────────────────────────────────────────────────────────────┘
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Auto-Reply Flow
|
||||
|
||||
How the bot automatically replies to chat messages.
|
||||
|
||||
```
|
||||
┌─────────────────────────────────────────────────────────────────┐
|
||||
│ AutoReplyService │
|
||||
│ │
|
||||
│ Scheduled: every N minutes (config.autoReply.checkInterval) │
|
||||
│ │
|
||||
│ 1. Get cached conversations │
|
||||
│ 2. For each conversation: │
|
||||
│ │ │
|
||||
│ ├─► Check for new/unread messages │
|
||||
│ │ └─► If no new messages: continue │
|
||||
│ │ │
|
||||
│ ├─► Detect upsell/spam keywords │
|
||||
│ │ ├─► "onlyfans", "link in bio", "check my..." │
|
||||
│ │ ├─► If detected: flag conversation, skip reply │
|
||||
│ │ └─► If not: continue │
|
||||
│ │ │
|
||||
│ ├─► Generate AI reply suggestion │
|
||||
│ │ ┌─────────────────────────────────────────────────┐ │
|
||||
│ │ │ OpenRouter.generateReply(conversationHistory) │ │
|
||||
│ │ │ │ │
|
||||
│ │ │ Request: { │ │
|
||||
│ │ │ messages: [ │ │
|
||||
│ │ │ { role: "system", content: replyPrompt }, │ │
|
||||
│ │ │ { role: "user", content: conversation } │ │
|
||||
│ │ │ ] │ │
|
||||
│ │ │ } │ │
|
||||
│ │ │ │ │
|
||||
│ │ │ Response: "Thanks for reaching out! 😊" │ │
|
||||
│ │ │ │ │
|
||||
│ │ │ Reply: "Thanks for reaching out! 😊" │ │
|
||||
│ │ └─────────────────────────────────────────────────┘ │
|
||||
│ │ │
|
||||
│ ├─► If autoSend = true: │
|
||||
│ │ └─► ChatService.sendMessage(chatId, reply) │
|
||||
│ │ └─► Log: "Auto-sent reply to {user}" │
|
||||
│ │ │
|
||||
│ └─► If autoSend = false: │
|
||||
│ └─► Mark as pending (visible in dashboard) │
|
||||
│ └─► User can approve/reject via dashboard │
|
||||
│ │
|
||||
│ Output: Summary of actions taken │
|
||||
│ "Checked 15 conversations: 3 new messages, 1 reply sent" │
|
||||
└─────────────────────────────────────────────────────────────────┘
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Dashboard Data Flow
|
||||
|
||||
How data flows from services to the React operator console.
|
||||
|
||||
### Current React model
|
||||
|
||||
1. Feature-local API clients request authoritative snapshots from `/api/*`.
|
||||
2. Shared Zod schemas in `packages/api-contracts` validate HTTP responses and
|
||||
realtime envelopes at runtime.
|
||||
3. TanStack Query caches snapshots, preserves them during background refresh,
|
||||
and owns mutation pending/error/success state.
|
||||
4. Operations opens `/api/events/stream`. Each typed event invalidates the
|
||||
affected query keys, such as stats, timers, conversations, media, or
|
||||
operations activity.
|
||||
5. Browser `EventSource` reconnects automatically. The client marks cached
|
||||
state stale while reconnecting instead of blanking the UI.
|
||||
|
||||
```text
|
||||
services/repositories -> Express /api snapshots -> Zod -> TanStack Query -> React routes
|
||||
| ^
|
||||
+-> realtime event hub -> /api/events/stream --------+
|
||||
```
|
||||
|
||||
### Retired legacy model
|
||||
|
||||
The historical diagram below is retained for migration context only.
|
||||
`public/index.html` has been removed and is no longer served.
|
||||
|
||||
```
|
||||
┌─────────────────────────────────────────────────────────────────┐
|
||||
│ WebServer (Express) │
|
||||
│ │
|
||||
│ ──── REST API ─────────────────────────────────────────────── │
|
||||
│ │
|
||||
│ GET /api/stats │
|
||||
│ ├─► Read from: │
|
||||
│ │ ├─► Bot running state (in-memory) │
|
||||
│ │ ├─► PostsRepository.countProcessed() │
|
||||
│ │ ├─► MessagesRepository.countSent() │
|
||||
│ │ ├─► MessagesRepository.countResponded() │
|
||||
│ │ └─► Calculate response rate │
|
||||
│ └─► Return: { running, processed, sent, responded, rate } │
|
||||
│ │
|
||||
│ GET /api/messages │
|
||||
│ └─► MessagesRepository.getRecent(limit: 50) │
|
||||
│ │
|
||||
│ GET /api/analytics/by-strategy │
|
||||
│ └─► MessagesRepository.getStrategyStats() │
|
||||
│ │
|
||||
│ PUT /api/config │
|
||||
│ ├─► Validate new config values │
|
||||
│ ├─► Update in-memory config │
|
||||
│ └─► Return updated config │
|
||||
│ │
|
||||
│ ──── LEGACY SSE LOG STREAMING ─────────────────────────────── │
|
||||
│ │
|
||||
│ GET /api/logs/stream │
|
||||
│ ├─► Client connects │
|
||||
│ ├─► Send any pending log entries │
|
||||
│ ├─► Subscribe to logger buffer │
|
||||
│ ├─► Push new entries as they arrive │
|
||||
│ │ └─► Event: log │
|
||||
│ │ └─► Data: { level, message, timestamp } │
|
||||
│ └─► On disconnect: unsubscribe │
|
||||
│ │
|
||||
│ ──── RETIRED LEGACY IMPLEMENTATION ────────────────────────── │
|
||||
│ │
|
||||
│ JavaScript polls API endpoints: │
|
||||
│ ├─► /api/stats every 5 seconds │
|
||||
│ ├─► /api/messages every 10 seconds │
|
||||
│ ├─► /api/timers every 2 seconds │
|
||||
│ └─► SSE /api/logs/stream (persistent connection) │
|
||||
│ │
|
||||
│ Updates DOM in real-time: │
|
||||
│ ├─► Stats cards (processed, sent, responded, rate) │
|
||||
│ ├─► Activity feed (scrolling list) │
|
||||
│ ├─► Message viewer (click to expand) │
|
||||
│ ├─► Chat browser (conversation list) │
|
||||
│ ├─► Media gallery (image grid) │
|
||||
│ └─► Rate limiter timers (countdown) │
|
||||
└─────────────────────────────────────────────────────────────────┘
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Error Recovery Flows
|
||||
|
||||
### AI Refusal Recovery
|
||||
|
||||
```
|
||||
1. OpenRouter.analyzeImage() returns refusal
|
||||
│
|
||||
├─► Detect refusal via heuristic patterns
|
||||
│ ├─► "cannot analyze"
|
||||
│ ├─► "against policy"
|
||||
│ └─► "unable to process"
|
||||
│
|
||||
├─► Try next model in fallback chain
|
||||
│ ├─► OpenRouter.fallbackModels[0]
|
||||
│ ├─► OpenRouter.fallbackModels[1]
|
||||
│ └─► ...
|
||||
│
|
||||
└─► If all models refuse:
|
||||
├─► Mark post as processed with error
|
||||
├─► Log: "All models refused to analyze post {id}"
|
||||
└─► Continue to next post
|
||||
```
|
||||
|
||||
### Network Error Recovery
|
||||
|
||||
```
|
||||
1. RedditApi.fetchPosts() throws error
|
||||
│
|
||||
└─► withRetry() wrapper handles recovery:
|
||||
├─► Retry 1: wait 5s
|
||||
├─► Retry 2: wait 15s
|
||||
├─► Retry 3: wait 45s
|
||||
└─► After 3 failures:
|
||||
├─► Log error
|
||||
├─► Skip this subreddit
|
||||
└─► Continue to next subreddit
|
||||
```
|
||||
|
||||
### Account Session Recovery
|
||||
|
||||
```
|
||||
1. Reddit or Matrix authentication is rejected
|
||||
│
|
||||
├─► Invalidate the shared account-auth gate
|
||||
├─► Stop account-bound schedulers
|
||||
└─► Wait for a fresh combined dump from the Chrome extension
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Data Lifecycle
|
||||
|
||||
### Post Data
|
||||
|
||||
```
|
||||
Discovered (MonitorService)
|
||||
│
|
||||
├─► Stored in processed_posts (pending)
|
||||
│
|
||||
├─► Processed (ProcessorService)
|
||||
│ ├─► image_analysis updated
|
||||
│ ├─► generated_message updated
|
||||
│ └─► message_sent updated
|
||||
│
|
||||
└─► Archived (retention policy TBD)
|
||||
└─► Kept indefinitely for analytics
|
||||
```
|
||||
|
||||
### Message Data
|
||||
|
||||
```
|
||||
Generated (ProcessorService)
|
||||
│
|
||||
├─► Stored in sent_messages (got_response = 0)
|
||||
│
|
||||
├─► Response checked (ResponseCheckerService)
|
||||
│ ├─► got_response = 1
|
||||
│ └─► response_checked_at set
|
||||
│
|
||||
├─► If responded:
|
||||
│ └─► Copied to successful_messages (for learning)
|
||||
│
|
||||
└─► Retained indefinitely for analytics
|
||||
```
|
||||
|
||||
### Learning Data
|
||||
|
||||
```
|
||||
Successful pair created (ResponseCheckerService)
|
||||
│
|
||||
├─► Stored in successful_messages
|
||||
│ ├─► original_message
|
||||
│ ├─► image_analysis
|
||||
│ ├─► post_title
|
||||
│ ├─► response_snippet
|
||||
│ └─► subreddit (for weighted selection)
|
||||
│
|
||||
├─► Retrieved for few-shot learning
|
||||
│ └─► Included in OpenRouter.generateMessage() prompt
|
||||
│
|
||||
└─► Retained indefinitely (grows bot intelligence)
|
||||
```
|
||||
|
||||
### Chat Data
|
||||
|
||||
```
|
||||
Conversation discovered (ChatService)
|
||||
│
|
||||
├─► Cached in memory
|
||||
├─► Persisted to disk JSON
|
||||
│
|
||||
├─► Messages added as they arrive
|
||||
│
|
||||
├─► Media downloaded (MediaService)
|
||||
│ └─► Stored in data/media/
|
||||
│
|
||||
└─► Retained until manually cleared
|
||||
└─► No automatic cleanup (TBD)
|
||||
```
|
||||
|
||||
### Media Data
|
||||
|
||||
```
|
||||
Media URL discovered
|
||||
│
|
||||
├─► Downloaded (MediaService)
|
||||
│ ├─► Matrix thumbnails resolved
|
||||
│ ├─► SHA-1 hash computed
|
||||
│ └─► Saved to data/media/{hash}.{ext}
|
||||
│
|
||||
├─► Indexed in data/media/index.json
|
||||
│
|
||||
├─► Deduplicated (hash check before download)
|
||||
│
|
||||
└─► Retained until manually cleared
|
||||
└─► No automatic cleanup (TBD)
|
||||
```
|
||||
Avoid converting errors into generic skips. Operational diagnosis depends on preserving which stage failed and why.
|
||||
+69
-515
@@ -1,550 +1,104 @@
|
||||
# Database Schema
|
||||
# Data model and persistence
|
||||
|
||||
Complete reference for the SQLite database schema, migrations, and query patterns.
|
||||
## Storage model
|
||||
|
||||
---
|
||||
The application uses one `better-sqlite3` database, normally `data/reddit-bot.db`, in WAL mode with foreign keys enabled. The persistent Docker volume also holds browser profiles and downloaded media, so a database-only backup is not a complete account/content backup.
|
||||
|
||||
## Table of Contents
|
||||
`src/database/client.ts` is the schema authority. Migrations are ordered functions guarded by SQLite `PRAGMA user_version` and run inside transactions. The repository does not use an ORM.
|
||||
|
||||
1. [Overview](#overview)
|
||||
2. [Database File](#database-file)
|
||||
3. [Schema](#schema)
|
||||
4. [Indexes](#indexes)
|
||||
5. [Migrations](#migrations)
|
||||
6. [Query Patterns](#query-patterns)
|
||||
7. [Repository Methods](#repository-methods)
|
||||
8. [Data Retention](#data-retention)
|
||||
|
||||
---
|
||||
|
||||
## Overview
|
||||
|
||||
RedditBot uses SQLite (via `better-sqlite3`) for persistent storage. The database tracks processed posts, sent messages, and successful message/response pairs.
|
||||
|
||||
**Features**:
|
||||
- WAL mode for better concurrent read performance
|
||||
- Foreign key constraints enabled
|
||||
- Progressive migrations for schema evolution
|
||||
- Repository pattern for data access
|
||||
|
||||
---
|
||||
|
||||
## Database File
|
||||
|
||||
**Default path**: `./data/reddit-bot.db`
|
||||
|
||||
Configured in `config.yaml`:
|
||||
|
||||
```yaml
|
||||
database:
|
||||
path: "./data/reddit-bot.db"
|
||||
```
|
||||
|
||||
The database file is auto-created on first run.
|
||||
|
||||
---
|
||||
|
||||
## Schema
|
||||
|
||||
### Table: `processed_posts`
|
||||
|
||||
Tracks all posts that have been discovered and processed.
|
||||
|
||||
```sql
|
||||
CREATE TABLE processed_posts (
|
||||
id TEXT PRIMARY KEY, -- Reddit post ID
|
||||
subreddit TEXT NOT NULL, -- Source subreddit
|
||||
author TEXT NOT NULL, -- Post author username
|
||||
title TEXT, -- Post title
|
||||
image_url TEXT, -- Original image URL
|
||||
image_analysis TEXT, -- AI-generated JSON description
|
||||
generated_message TEXT, -- AI-generated DM text
|
||||
processed_at INTEGER NOT NULL, -- Unix timestamp
|
||||
message_sent INTEGER DEFAULT 0, -- 0 = not sent, 1 = sent
|
||||
error TEXT -- Error message if processing failed
|
||||
);
|
||||
```
|
||||
|
||||
**Column Details**:
|
||||
|
||||
| Column | Type | Constraints | Description |
|
||||
|---|---|---|---|
|
||||
| `id` | TEXT | PRIMARY KEY | Reddit post ID (e.g., "abc123") |
|
||||
| `subreddit` | TEXT | NOT NULL | Source subreddit name |
|
||||
| `author` | TEXT | NOT NULL | Post author username |
|
||||
| `title` | TEXT | | Post title (nullable if unavailable) |
|
||||
| `image_url` | TEXT | | Original image URL |
|
||||
| `image_analysis` | TEXT | | JSON string from AI analysis |
|
||||
| `generated_message` | TEXT | | The DM text that was generated |
|
||||
| `processed_at` | INTEGER | NOT NULL | Unix timestamp when processed |
|
||||
| `message_sent` | INTEGER | DEFAULT 0 | 0 = not sent, 1 = successfully sent |
|
||||
| `error` | TEXT | | Error message if processing failed |
|
||||
|
||||
**Example Row**:
|
||||
|
||||
```json
|
||||
{
|
||||
"id": "abc123",
|
||||
"subreddit": "gonewild",
|
||||
"author": "user123",
|
||||
"title": "Summer vibes",
|
||||
"image_url": "https://i.redd.it/abc123.jpg",
|
||||
"image_analysis": "{\"gender\":\"F\",\"description\":\"Young woman...\"}",
|
||||
"generated_message": "Hey, you look amazing...",
|
||||
"processed_at": 1716234567,
|
||||
"message_sent": 1,
|
||||
"error": null
|
||||
}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### Table: `sent_messages`
|
||||
|
||||
Tracks all DMs sent by the bot.
|
||||
|
||||
```sql
|
||||
CREATE TABLE sent_messages (
|
||||
id INTEGER PRIMARY KEY AUTOINCREMENT,
|
||||
post_id TEXT NOT NULL,
|
||||
recipient TEXT NOT NULL,
|
||||
subject TEXT DEFAULT 'Stunning',
|
||||
message TEXT NOT NULL,
|
||||
sent_at INTEGER NOT NULL,
|
||||
strategy_id TEXT,
|
||||
got_response INTEGER DEFAULT 0,
|
||||
response_checked_at INTEGER,
|
||||
FOREIGN KEY (post_id) REFERENCES processed_posts(id)
|
||||
);
|
||||
```
|
||||
|
||||
**Column Details**:
|
||||
|
||||
| Column | Type | Constraints | Description |
|
||||
|---|---|---|---|
|
||||
| `id` | INTEGER | PRIMARY KEY AUTOINCREMENT | Internal ID |
|
||||
| `post_id` | TEXT | NOT NULL, FK | Source post ID |
|
||||
| `recipient` | TEXT | NOT NULL | Recipient username |
|
||||
| `subject` | TEXT | DEFAULT 'Stunning' | DM subject line |
|
||||
| `message` | TEXT | NOT NULL | DM body text |
|
||||
| `sent_at` | INTEGER | NOT NULL | Unix timestamp when sent |
|
||||
| `strategy_id` | TEXT | | A/B test variant name |
|
||||
| `got_response` | INTEGER | DEFAULT 0 | 0 = no response, 1 = responded |
|
||||
| `response_checked_at` | INTEGER | | Timestamp of last response check |
|
||||
|
||||
**Example Row**:
|
||||
|
||||
```json
|
||||
{
|
||||
"id": 1,
|
||||
"post_id": "abc123",
|
||||
"recipient": "user123",
|
||||
"subject": "Stunning",
|
||||
"message": "Hey, you look amazing...",
|
||||
"sent_at": 1716234567,
|
||||
"strategy_id": "compliment",
|
||||
"got_response": 1,
|
||||
"response_checked_at": 1716238167
|
||||
}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### Table: `successful_messages`
|
||||
|
||||
Stores successful message/response pairs for few-shot learning.
|
||||
|
||||
```sql
|
||||
CREATE TABLE successful_messages (
|
||||
id INTEGER PRIMARY KEY AUTOINCREMENT,
|
||||
sent_message_id INTEGER,
|
||||
original_message TEXT NOT NULL,
|
||||
image_analysis TEXT,
|
||||
post_title TEXT,
|
||||
response_snippet TEXT,
|
||||
subreddit TEXT,
|
||||
created_at INTEGER NOT NULL,
|
||||
FOREIGN KEY (sent_message_id) REFERENCES sent_messages(id)
|
||||
);
|
||||
```
|
||||
|
||||
**Column Details**:
|
||||
|
||||
| Column | Type | Constraints | Description |
|
||||
|---|---|---|---|
|
||||
| `id` | INTEGER | PRIMARY KEY AUTOINCREMENT | Internal ID |
|
||||
| `sent_message_id` | INTEGER | FK, nullable | Related sent message ID |
|
||||
| `original_message` | TEXT | NOT NULL | The DM that got a response |
|
||||
| `image_analysis` | TEXT | | Context from image analysis |
|
||||
| `post_title` | TEXT | | Original post title |
|
||||
| `response_snippet` | TEXT | | First reply text |
|
||||
| `subreddit` | TEXT | | For weighted example selection |
|
||||
| `created_at` | INTEGER | NOT NULL | Unix timestamp |
|
||||
|
||||
**Example Row**:
|
||||
|
||||
```json
|
||||
{
|
||||
"id": 1,
|
||||
"sent_message_id": 1,
|
||||
"original_message": "Hey, you look amazing...",
|
||||
"image_analysis": "{\"gender\":\"F\",\"description\":\"...\"}",
|
||||
"post_title": "Summer vibes",
|
||||
"response_snippet": "Thank you so much! 😊",
|
||||
"subreddit": "gonewild",
|
||||
"created_at": 1716238167
|
||||
}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Indexes
|
||||
|
||||
### `processed_posts` Indexes
|
||||
|
||||
```sql
|
||||
CREATE INDEX idx_posts_subreddit ON processed_posts(subreddit);
|
||||
CREATE INDEX idx_posts_author ON processed_posts(author);
|
||||
```
|
||||
|
||||
**Purpose**: Fast lookup by subreddit (for processing stats) and by author (for skip-checking).
|
||||
|
||||
---
|
||||
|
||||
### `sent_messages` Indexes
|
||||
|
||||
```sql
|
||||
CREATE INDEX idx_messages_recipient ON sent_messages(recipient);
|
||||
CREATE INDEX idx_messages_strategy ON sent_messages(strategy_id);
|
||||
CREATE INDEX idx_messages_response ON sent_messages(got_response);
|
||||
```
|
||||
|
||||
**Purpose**:
|
||||
- `idx_messages_recipient`: Fast lookup when matching responses
|
||||
- `idx_messages_strategy`: Analytics by strategy variant
|
||||
- `idx_messages_response`: Filter for responded/not-responded
|
||||
|
||||
---
|
||||
## Ownership categories
|
||||
|
||||
### `successful_messages` Indexes
|
||||
### Account registry and account content
|
||||
|
||||
```sql
|
||||
CREATE INDEX idx_success_subreddit ON successful_messages(subreddit);
|
||||
CREATE INDEX idx_success_message ON successful_messages(sent_message_id);
|
||||
```
|
||||
- `reddit_accounts`: verified Reddit/Matrix identity, encrypted-at-rest only to the degree provided by the host/volume, cookie JSON, Matrix credentials, active/status timestamps.
|
||||
- `reddit_conversations`: account-scoped cached Matrix conversation payloads.
|
||||
- `reddit_account_sync_state`: account-scoped sync cursor, full/delta timestamps, and invitation cooldown.
|
||||
- `reddit_media`: account-scoped private-media metadata and relative file path.
|
||||
|
||||
**Purpose**:
|
||||
- `idx_success_subreddit`: Weighted example selection by subreddit
|
||||
- `idx_success_message`: Link back to sent message
|
||||
The special legacy-unassigned account exists only to preserve rows created before account ownership was introduced. New runtime data must use a verified account ID.
|
||||
|
||||
---
|
||||
### Outreach and outcomes
|
||||
|
||||
## Migrations
|
||||
- `processed_posts`: account/post key, source metadata, analysis, generated opener, send marker, and terminal error/rejection reason.
|
||||
- `sent_messages`: account-owned opener/reply attribution, strategy/experiment/model metadata, Matrix room, response/image flags, and source type.
|
||||
- `successful_messages`: examples derived from positive outcomes for prompt context.
|
||||
- `conversation_outcomes`: normalized outcome for a sent message (`engaged`, `declined`, `boundary`, `upsell`, or `unknown`).
|
||||
- `activity_log`: operator-facing events with optional recipient/post context.
|
||||
- `reddit_users`: global mapping between Reddit fullname/username and Matrix identity.
|
||||
- `recipient_suppressions`: process-global recipient reservation/contact record used to prevent cross-account duplicates.
|
||||
|
||||
Migrations are defined in `src/database/client.ts` and run automatically on every startup.
|
||||
### Durable work and idempotency
|
||||
|
||||
### Migration Logic
|
||||
- `candidate_queue`: global candidate snapshot, source/priority, state, prepared payload, worker lease, retry timing, selected account, and rejection/error details.
|
||||
- `delivery_outbox`: account-scoped prepared delivery and idempotency status retained for legacy/recovery paths.
|
||||
- `conversation_states`: account/conversation policy and derived state JSON.
|
||||
- `conversation_automation_claims`: account/conversation/inbound-event claim with intended reply and pending/sent state.
|
||||
- `runtime_state`: small process controls such as whether bot work is running.
|
||||
|
||||
```typescript
|
||||
// Pseudo-code for migration process
|
||||
function runMigrations(db: Database) {
|
||||
// Enable features
|
||||
db.exec("PRAGMA journal_mode=WAL;");
|
||||
db.exec("PRAGMA foreign_keys=ON;");
|
||||
### Configuration and telemetry
|
||||
|
||||
// Create tables if they don't exist
|
||||
db.exec(`CREATE TABLE IF NOT EXISTS processed_posts (...)`);
|
||||
db.exec(`CREATE TABLE IF NOT EXISTS sent_messages (...)`);
|
||||
db.exec(`CREATE TABLE IF NOT EXISTS successful_messages (...)`);
|
||||
- `app_config`: one secret-free validated application configuration record.
|
||||
- `ai_usage`: requested/actual model, token/cost, latency, success/error, purpose, and account attribution.
|
||||
- `legacy_content_imports`: records one-time imports from legacy content stores.
|
||||
|
||||
// Create indexes if they don't exist
|
||||
db.exec(`CREATE INDEX IF NOT EXISTS idx_posts_subreddit ON processed_posts(subreddit)`);
|
||||
// ... more indexes
|
||||
## Critical invariants
|
||||
|
||||
// Safe column additions (for schema evolution)
|
||||
db.exec(`ALTER TABLE processed_posts ADD COLUMN error TEXT`);
|
||||
}
|
||||
```
|
||||
1. Account-produced rows carry `account_id`; cross-account views must opt into global queries explicitly.
|
||||
2. Recipient identity comparisons are case-insensitive and global for outreach suppression.
|
||||
3. A sent message is persisted only after remote confirmation.
|
||||
4. Candidate and automation claims are lease/claim based; a worker must own the expected lease before it changes state.
|
||||
5. Conversation policy is snapshotted in state JSON when automation starts for that conversation.
|
||||
6. Secrets and the database path are not stored in `app_config`.
|
||||
7. Private media paths in SQLite are relative to the persistent data root; HTTP access remains authenticated.
|
||||
|
||||
### Migration Safety
|
||||
## Repository map
|
||||
|
||||
- All `CREATE TABLE` statements use `IF NOT EXISTS`
|
||||
- All `CREATE INDEX` statements use `IF NOT EXISTS`
|
||||
- Column additions use `ALTER TABLE ADD COLUMN IF NOT EXISTS` pattern
|
||||
- Migrations are idempotent (safe to run multiple times)
|
||||
`src/database/repositories.ts` contains the domain repositories for posts/messages/outcomes/activity/outbox/state/claims/users/recipient history. `account-repositories.ts` owns account records and cached account content. Candidate and AI telemetry have dedicated repositories.
|
||||
|
||||
### Adding New Columns
|
||||
Prefer narrow repository methods such as `claimReady`, `markSent`, or `getByChatRoomId` to exposing SQL or generic CRUD. This keeps transaction and ownership rules near the schema.
|
||||
|
||||
When adding new columns to existing tables:
|
||||
## Candidate leases
|
||||
|
||||
```typescript
|
||||
// In src/database/client.ts
|
||||
const tableInfo = db.prepare(`PRAGMA table_info(processed_posts)`).all();
|
||||
const hasErrorColumn = tableInfo.some((col: any) => col.name === 'error');
|
||||
Discovery inserts candidates without overwriting terminal work. Screening and dispatch use a worker ID plus lease expiry. On startup all interrupted leases can be recovered; during operation expired leases are recovered and due retries promoted. A transient account failure belongs to the selected account, so delivery can be made visible quickly for another eligible runtime without losing the prepared candidate.
|
||||
|
||||
if (!hasErrorColumn) {
|
||||
db.exec("ALTER TABLE processed_posts ADD COLUMN error TEXT");
|
||||
}
|
||||
```
|
||||
Any new candidate state needs:
|
||||
|
||||
---
|
||||
- an allowed predecessor and terminal/retry semantics;
|
||||
- a lease ownership rule;
|
||||
- recovery behavior after process death;
|
||||
- counts/reasons for the operator funnel;
|
||||
- tests for stale lease, wrong owner, restart, and duplicate post/author cases.
|
||||
|
||||
## Query Patterns
|
||||
## Conversation reply claims
|
||||
|
||||
### Post Processing
|
||||
The unique identity is account + conversation + inbound message key. The claim persists the intended reply before the remote send. A pending claim is not blindly retried: Matrix history is checked first. This closes the process-crash window between remote acceptance and local completion.
|
||||
|
||||
**Check if post was already processed**:
|
||||
Changes to this flow must prove at-most-once observable replies across retry and restart tests.
|
||||
|
||||
```sql
|
||||
SELECT id FROM processed_posts WHERE id = ?;
|
||||
```
|
||||
## Configuration persistence
|
||||
|
||||
**Mark post as processed**:
|
||||
On an empty database, `config.yaml` is parsed, environment-owned values are injected, the result is validated, and the application-owned subset is saved to `app_config`. Later starts load SQLite first. Dashboard changes update that record.
|
||||
|
||||
```sql
|
||||
INSERT OR REPLACE INTO processed_posts
|
||||
(id, subreddit, author, title, image_url, image_analysis,
|
||||
generated_message, processed_at, message_sent, error)
|
||||
VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?);
|
||||
```
|
||||
The persisted record excludes:
|
||||
|
||||
**Mark post as sent**:
|
||||
- `OPENROUTER_API_KEY`;
|
||||
- dashboard/agent tokens;
|
||||
- `DATABASE_PATH`.
|
||||
|
||||
```sql
|
||||
UPDATE processed_posts
|
||||
SET message_sent = 1
|
||||
WHERE id = ?;
|
||||
```
|
||||
Those values are supplied by the environment on every load.
|
||||
|
||||
**Update post with analysis**:
|
||||
## Adding a migration
|
||||
|
||||
```sql
|
||||
UPDATE processed_posts
|
||||
SET image_analysis = ?, generated_message = ?
|
||||
WHERE id = ?;
|
||||
```
|
||||
1. Inspect the final schema and all repository assumptions.
|
||||
2. Append one migration function; never reorder or modify an already-released migration.
|
||||
3. Make the operation transactional and compatible with existing data.
|
||||
4. Use `addColumnIfNotExists` only where repeatability is appropriate; table rebuilds need explicit foreign-key handling and `foreign_key_check`.
|
||||
5. Update repository types and operations.
|
||||
6. Add tests for a fresh in-memory database and, when relevant, an older schema upgraded in place.
|
||||
7. Update this document and affected API/contracts.
|
||||
|
||||
---
|
||||
Do not edit a live SQLite file by hand. Back it up, stop writers when required, apply code-owned migrations, and verify `integrity_check`, `foreign_key_check`, row counts, and application reads.
|
||||
|
||||
### Message Tracking
|
||||
## Backup and recovery
|
||||
|
||||
**Record a sent message**:
|
||||
|
||||
```sql
|
||||
INSERT INTO sent_messages
|
||||
(post_id, recipient, subject, message, sent_at, strategy_id)
|
||||
VALUES (?, ?, 'Stunning', ?, ?, ?);
|
||||
```
|
||||
|
||||
**Get recent messages**:
|
||||
|
||||
```sql
|
||||
SELECT * FROM sent_messages
|
||||
ORDER BY sent_at DESC
|
||||
LIMIT ?;
|
||||
```
|
||||
|
||||
**Update response status**:
|
||||
|
||||
```sql
|
||||
UPDATE sent_messages
|
||||
SET got_response = 1, response_checked_at = ?
|
||||
WHERE id = ?;
|
||||
```
|
||||
|
||||
**Check if user was previously messaged**:
|
||||
|
||||
```sql
|
||||
SELECT COUNT(*) FROM sent_messages WHERE recipient = ?;
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### Strategy Analytics
|
||||
|
||||
**Get strategy statistics**:
|
||||
|
||||
```sql
|
||||
SELECT
|
||||
strategy_id,
|
||||
COUNT(*) as sent,
|
||||
SUM(got_response) as responded
|
||||
FROM sent_messages
|
||||
WHERE strategy_id IS NOT NULL
|
||||
GROUP BY strategy_id;
|
||||
```
|
||||
|
||||
**Get response rate by subreddit**:
|
||||
|
||||
```sql
|
||||
SELECT
|
||||
p.subreddit,
|
||||
COUNT(*) as sent,
|
||||
SUM(s.got_response) as responded
|
||||
FROM sent_messages s
|
||||
JOIN processed_posts p ON s.post_id = p.id
|
||||
GROUP BY p.subreddit;
|
||||
```
|
||||
|
||||
**Get daily response trend**:
|
||||
|
||||
```sql
|
||||
SELECT
|
||||
DATE(sent_at, 'unixepoch') as date,
|
||||
COUNT(*) as sent,
|
||||
SUM(got_response) as responded
|
||||
FROM sent_messages
|
||||
GROUP BY DATE(sent_at, 'unixepoch')
|
||||
ORDER BY date DESC
|
||||
LIMIT 30;
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### Few-Shot Learning
|
||||
|
||||
**Get successful examples for a subreddit**:
|
||||
|
||||
```sql
|
||||
SELECT
|
||||
original_message,
|
||||
image_analysis,
|
||||
post_title,
|
||||
response_snippet
|
||||
FROM successful_messages
|
||||
WHERE subreddit = ?
|
||||
ORDER BY created_at DESC
|
||||
LIMIT 3;
|
||||
```
|
||||
|
||||
**Store successful pair**:
|
||||
|
||||
```sql
|
||||
INSERT INTO successful_messages
|
||||
(sent_message_id, original_message, image_analysis,
|
||||
post_title, response_snippet, subreddit, created_at)
|
||||
VALUES (?, ?, ?, ?, ?, ?, ?);
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Repository Methods
|
||||
|
||||
### PostsRepository
|
||||
|
||||
```typescript
|
||||
class PostsRepository {
|
||||
/** Check if a post was already processed */
|
||||
getById(id: string): ProcessedPost | undefined
|
||||
|
||||
/** Mark a post as processed */
|
||||
insert(post: ProcessedPostInsert): void
|
||||
|
||||
/** Mark a post as having a message sent */
|
||||
markSent(id: string): void
|
||||
|
||||
/** Update post with AI analysis results */
|
||||
updateAnalysis(id: string, analysis: string, message: string): void
|
||||
|
||||
/** Get total count of processed posts */
|
||||
count(): number
|
||||
|
||||
/** Get count by subreddit */
|
||||
countBySubreddit(subreddit: string): number
|
||||
}
|
||||
```
|
||||
|
||||
### MessagesRepository
|
||||
|
||||
```typescript
|
||||
class MessagesRepository {
|
||||
/** Record a sent message */
|
||||
insert(message: MessageInsert): number
|
||||
|
||||
/** Get recent sent messages */
|
||||
getRecent(limit: number): SentMessage[]
|
||||
|
||||
/** Get messages by recipient */
|
||||
getByRecipient(recipient: string): SentMessage[]
|
||||
|
||||
/** Update response status */
|
||||
markResponded(id: number, checkedAt: number): void
|
||||
|
||||
/** Get strategy statistics */
|
||||
getStrategyStats(): StrategyStat[]
|
||||
|
||||
/** Get response rate by subreddit */
|
||||
getSubredditStats(): SubredditStat[]
|
||||
|
||||
/** Get daily response trend */
|
||||
getDailyTrend(days: number): DailyTrend[]
|
||||
|
||||
/** Get total count of sent messages */
|
||||
count(): number
|
||||
|
||||
/** Get count of responded messages */
|
||||
countResponded(): number
|
||||
}
|
||||
```
|
||||
|
||||
### SuccessfulMessagesRepository
|
||||
|
||||
```typescript
|
||||
class SuccessfulMessagesRepository {
|
||||
/** Store a successful message/response pair */
|
||||
insert(pair: SuccessfulMessageInsert): void
|
||||
|
||||
/** Get examples for few-shot learning */
|
||||
getExamples(subreddit: string, count: number): Example[]
|
||||
|
||||
/** Get total count */
|
||||
count(): number
|
||||
|
||||
/** Get recent examples */
|
||||
getRecent(limit: number): SuccessfulMessage[]
|
||||
}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Data Retention
|
||||
|
||||
### Current Behavior
|
||||
|
||||
All data is retained indefinitely. There is no automatic cleanup or retention policy.
|
||||
|
||||
| Table | Retention | Notes |
|
||||
|---|---|---|
|
||||
| `processed_posts` | Indefinite | Grows with each post processed |
|
||||
| `sent_messages` | Indefinite | Grows with each message sent |
|
||||
| `successful_messages` | Indefinite | Grows with each response received |
|
||||
|
||||
### Estimated Growth
|
||||
|
||||
With default settings (20 messages/hour, 24/7 operation):
|
||||
|
||||
| Timeframe | Posts | Messages | Responses (est. 30%) |
|
||||
|---|---|---|---|
|
||||
| 1 day | ~480 | ~288 | ~86 |
|
||||
| 1 week | ~3,360 | ~2,016 | ~605 |
|
||||
| 1 month | ~14,400 | ~8,640 | ~2,592 |
|
||||
| 1 year | ~175,200 | ~105,120 | ~31,536 |
|
||||
|
||||
### Recommended Cleanup (Future)
|
||||
|
||||
Consider implementing automated cleanup:
|
||||
|
||||
```sql
|
||||
-- Archive old processed posts
|
||||
CREATE TABLE processed_posts_archive AS
|
||||
SELECT * FROM processed_posts
|
||||
WHERE processed_at < ?;
|
||||
|
||||
-- Delete archived posts
|
||||
DELETE FROM processed_posts
|
||||
WHERE processed_at < ?;
|
||||
|
||||
-- Keep last N months of data
|
||||
```
|
||||
|
||||
> **Note**: Successful messages should NEVER be deleted as they are used for few-shot learning.
|
||||
For a running WAL database, use SQLite's online backup mechanism or stop the container before copying the database plus `-wal`/`-shm` state. A full production backup should include the entire persistent volume because sessions and media live beside the database. Validate that the backup can be opened and contains expected tables before any destructive reset.
|
||||
+89
-354
@@ -1,397 +1,132 @@
|
||||
# Deployment Guide
|
||||
# Deployment and operations
|
||||
|
||||
Complete guide for deploying RedditBot in various environments.
|
||||
## Production model
|
||||
|
||||
---
|
||||
The production unit is the `reddit-bot` Docker Compose service. The image uses the Playwright base, installs/builds the Node application, serves the compiled React dashboard and Express API on port 3000, and stores state under `/app/data` on the named `reddit_bot_data` volume.
|
||||
|
||||
## Table of Contents
|
||||
The container does not run VNC, accept Reddit passwords, or scrape a Matrix token from a login browser. Account authentication enters through a locally exported session dump and is persisted in the sensitive volume.
|
||||
|
||||
1. [Prerequisites](#prerequisites)
|
||||
2. [Local Development](#local-development)
|
||||
3. [Docker Deployment](#docker-deployment)
|
||||
4. [Docker Compose](#docker-compose)
|
||||
5. [Dokploy Deployment](#dokploy-deployment)
|
||||
6. [Production Considerations](#production-considerations)
|
||||
7. [Monitoring & Maintenance](#monitoring--maintenance)
|
||||
8. [Troubleshooting](#troubleshooting-1)
|
||||
## Authority and safety
|
||||
|
||||
---
|
||||
Deployment, restart, start/resume, session import, configuration mutation, send actions, and data deletion are separate operations. Only perform the ones the user explicitly requested. In particular, “commit and push” does not authorize a Dokploy deployment or bot restart.
|
||||
|
||||
## Prerequisites
|
||||
Before any live operation:
|
||||
|
||||
### Required
|
||||
1. Confirm the target is the RedditBot Compose service, not the whole Dokploy host/instance.
|
||||
2. Inspect current Git revision, worktree/remote state, deployment state, bot running/paused state, account runtimes, and recent errors.
|
||||
3. Identify whether the action can send messages or change account state.
|
||||
4. Back up affected persistent state before destructive or migration-heavy work.
|
||||
5. Define observable success criteria and a recovery path.
|
||||
|
||||
| Requirement | Version | Notes |
|
||||
|---|---|---|
|
||||
| Node.js | 20.19+ or 22.12+ | Required by the locked Vite toolchain |
|
||||
| npm | Bundled with Node | Use `npm ci` for lockfile-exact installs |
|
||||
| OpenRouter API Key | - | Sign up at [openrouter.ai](https://openrouter.ai) |
|
||||
| Reddit Account | - | With DM permissions enabled |
|
||||
## Required environment
|
||||
|
||||
### Recommended
|
||||
The Compose service passes:
|
||||
|
||||
| Requirement | Notes |
|
||||
|---|---|
|
||||
| 1 GB RAM | Minimum for browser automation |
|
||||
| 1 CPU core | More for higher message throughput |
|
||||
| 5 GB disk | For database, media, and browser data |
|
||||
- `OPENROUTER_API_KEY`;
|
||||
- `WEB_DASHBOARD_TOKEN`;
|
||||
- optional `REDDIT_BOT_AGENT_READ_TOKEN`;
|
||||
- optional `REDDIT_BOT_AGENT_CONTROL_TOKEN`.
|
||||
|
||||
---
|
||||
Dokploy access credentials are stored in the ignored repository `.env`. Use them only when access is explicitly requested. Never print the file, include tokens in commands that will be echoed, commit them, or copy response metadata containing credentials. Follow redirects for Dokploy API calls.
|
||||
|
||||
## Local Development
|
||||
## Persistence
|
||||
|
||||
### 1. Clone and Install
|
||||
`reddit_bot_data:/app/data` contains:
|
||||
|
||||
```bash
|
||||
git clone <repository-url>
|
||||
cd RedditBot
|
||||
npm install
|
||||
```
|
||||
- SQLite database and WAL files;
|
||||
- imported browser/account sessions;
|
||||
- cached Matrix credentials/state;
|
||||
- downloaded private media;
|
||||
- per-account browser profiles.
|
||||
|
||||
### 2. Configure
|
||||
A volume replacement or deletion is therefore a destructive account-and-content reset. Verify a readable backup before touching it and state exactly what will be lost. Removing a Compose application and deleting its volume are not the same action.
|
||||
|
||||
Create `config.yaml`:
|
||||
## Deployment procedure
|
||||
|
||||
```yaml
|
||||
reddit:
|
||||
subreddits:
|
||||
- gonewild
|
||||
pollInterval: 15
|
||||
targetGender: "F"
|
||||
sellerKeywords:
|
||||
- seller
|
||||
- onlyfans
|
||||
Use the actual current Dokploy API/Compose identifiers from a read-only lookup; do not copy identifiers into documentation or assume an older deployment still has the same ID.
|
||||
|
||||
openrouter:
|
||||
apiKey: "sk-or-v1-your-key-here"
|
||||
model: "meta-llama/llama-3.2-90b-vision-preview"
|
||||
fallbackModels:
|
||||
- "google/gemini-2.0-flash-001"
|
||||
1. Confirm the intended commit exists on the tracked remote branch.
|
||||
2. Read current Compose/deployment status and capture the prior image/revision if available.
|
||||
3. Confirm whether the bot must remain paused through deployment. Do not change persisted running state unless authorized.
|
||||
4. Trigger the bounded Compose deployment.
|
||||
5. Poll the deployment record to a terminal success/failure state; an accepted trigger is not completion.
|
||||
6. Verify the running container is the new deployment and is not restarting.
|
||||
7. Verify public `GET /healthz` and authenticated `GET /api/v1/readyz`/overview as appropriate.
|
||||
8. Verify account runtime count/state, bot paused/running state, sync/cooldown state, and recent sanitized logs.
|
||||
9. Verify the persistent volume is mounted and pre-deployment data is readable.
|
||||
|
||||
rateLimit:
|
||||
minDelayMs: 30000
|
||||
maxMessagesPerHour: 20
|
||||
jitterMs: 15000
|
||||
Do not use a green container health badge as the only evidence. The process may be healthy while account runtimes are logged out, the bot is paused, Matrix is degraded, or candidate/reply work is blocked.
|
||||
|
||||
browser:
|
||||
sessionDir: "./data/browser-session"
|
||||
dryRun: true
|
||||
## Health semantics
|
||||
|
||||
database:
|
||||
path: "./data/reddit-bot.db"
|
||||
- `/healthz`: unauthenticated process liveness only; intentionally reveals no private state.
|
||||
- `/api/v1/readyz`: authenticated component readiness and degraded authentication state.
|
||||
- `/api/v1/overview`: consolidated runtime, account, sync, rate-limit, candidate/reply, and recent-error view.
|
||||
- `/api/stats`, `/api/attention`, `/api/outreach`: operator-level workload detail.
|
||||
- `/api/events/stream`: realtime signal, not a replacement for snapshot checks.
|
||||
|
||||
web:
|
||||
port: 3000
|
||||
Success should distinguish:
|
||||
|
||||
dryRun: true # Start in dry run mode
|
||||
```
|
||||
- process/container running;
|
||||
- API reachable and authenticated;
|
||||
- database/volume mounted;
|
||||
- accounts restored and verified;
|
||||
- bot intentionally paused or running;
|
||||
- no crash loop or new high-severity errors;
|
||||
- requested workload actually completed.
|
||||
|
||||
Set environment variables:
|
||||
## Account session operations
|
||||
|
||||
```bash
|
||||
export OPENROUTER_API_KEY="sk-or-v1-your-key"
|
||||
export WEB_DASHBOARD_TOKEN="long-random-secret"
|
||||
export REDDIT_BOT_AGENT_READ_TOKEN="separate-read-secret"
|
||||
export REDDIT_BOT_AGENT_CONTROL_TOKEN="separate-control-secret"
|
||||
```
|
||||
The supported workflow is:
|
||||
|
||||
### 3. Import account session
|
||||
1. Load `extensions/reddit-session-exporter` into the operator's local Chrome.
|
||||
2. Sign in to Reddit locally and open Reddit Chat.
|
||||
3. Export a combined Reddit cookie + Matrix session JSON.
|
||||
4. Upload it through Settings > Access.
|
||||
5. Wait for both Reddit and Matrix identities to verify and for the matching runtime to be replaced/started.
|
||||
6. Delete the local export file.
|
||||
|
||||
Start the dashboard, then upload a combined Reddit and Matrix dump created by the local Chrome extension. Until both identities verify, account work remains paused.
|
||||
The dump is a bearer credential bundle. Do not inspect or paste it into logs. An account with one valid half is still unavailable for work.
|
||||
|
||||
### 4. Start the Bot
|
||||
## Incident diagnosis
|
||||
|
||||
```bash
|
||||
# Terminal 1: backend with TypeScript execution
|
||||
npm run dev
|
||||
### No new outreach
|
||||
|
||||
# Terminal 2: Vite dashboard; proxies /api to port 3000
|
||||
npm run dashboard:dev
|
||||
Check, in order: persisted bot running state, the per-account delivery states on Overview, enabled/running account count, full authentication, reply backlog, account send timer/hourly cap, account-specific Matrix cooldown, global Reddit cooldown, candidate funnel counts/reasons, enabled subreddit count, recent intake/screening logs, and OpenRouter failures.
|
||||
|
||||
# Production: compile server and dashboard, then start
|
||||
npm run build
|
||||
npm start
|
||||
```
|
||||
Do not infer “no candidates” from “fetched N posts.” Inspect already processed/contacted, age/gender, seller, expired, AI skip, ready, retry, and rejected counts.
|
||||
|
||||
### 5. Access Dashboard
|
||||
### Conversations appear stale
|
||||
|
||||
Open the Vite URL (normally http://localhost:5173) during frontend development.
|
||||
Production dashboard checks use http://localhost:3000.
|
||||
Separate Matrix remote acceptance from cached/dashboard visibility. Inspect Matrix identity/whoami, sync cursor/status, last full/delta sync, new event IDs, username enrichment, HTTP conversation snapshot, SSE validation/reconnect, and client query invalidation. A stale dashboard can coexist with a successful remote send.
|
||||
|
||||
---
|
||||
### Reddit 403/429/404
|
||||
|
||||
## Docker Deployment
|
||||
- 429: honor `retryAfterMs` through the shared coordinator and avoid additional account requests.
|
||||
- repeated/broad 403: treat as a possible service-wide anti-abuse block; inspect successful requests and shared cooldown rather than declaring every subreddit private.
|
||||
- listing 404: the code disables that configured subreddit persistently.
|
||||
|
||||
### Build the Image
|
||||
### Database problems
|
||||
|
||||
```bash
|
||||
docker build -t reddit-bot .
|
||||
```
|
||||
Stop writers if recovery needs file operations. Back up the entire data directory, inspect SQLite `integrity_check`/`foreign_key_check`, WAL state, disk capacity/permissions, migration version, and recent logs. Do not create an empty replacement database over the original.
|
||||
|
||||
### Run the Container
|
||||
## Backup and destructive recovery
|
||||
|
||||
```bash
|
||||
docker run -d \
|
||||
--name reddit-bot \
|
||||
-p 3000:3000 \
|
||||
-v $(pwd)/data:/app/data \
|
||||
-e OPENROUTER_API_KEY=sk-or-v1-your-key \
|
||||
-e WEB_DASHBOARD_TOKEN=long-random-secret \
|
||||
-e REDDIT_BOT_AGENT_READ_TOKEN=separate-read-secret \
|
||||
-e REDDIT_BOT_AGENT_CONTROL_TOKEN=separate-control-secret \
|
||||
reddit-bot:latest
|
||||
```
|
||||
|
||||
### Dockerfile behavior
|
||||
|
||||
The image is based on the Playwright Jammy image because the remaining
|
||||
authenticated browser operations need Chromium and its system libraries. It
|
||||
does not install a display server, VNC, or consumer Google Chrome.
|
||||
|
||||
That build produces both `dist/` for the Express server and
|
||||
`apps/dashboard/dist/` for Vite assets. Port 3000 serves the dashboard/API and
|
||||
the container starts the Node process directly. This is a single-stage image and retains development dependencies
|
||||
needed by the current build/runtime layout.
|
||||
|
||||
---
|
||||
|
||||
## Docker Compose
|
||||
|
||||
### docker-compose.yml
|
||||
|
||||
```yaml
|
||||
version: '3.8'
|
||||
|
||||
services:
|
||||
reddit-bot:
|
||||
build: .
|
||||
container_name: reddit-bot
|
||||
ports:
|
||||
- "3000:3000"
|
||||
volumes:
|
||||
- ./data:/app/data
|
||||
environment:
|
||||
- OPENROUTER_API_KEY=${OPENROUTER_API_KEY}
|
||||
- WEB_DASHBOARD_TOKEN=${WEB_DASHBOARD_TOKEN}
|
||||
- REDDIT_BOT_AGENT_READ_TOKEN=${REDDIT_BOT_AGENT_READ_TOKEN:-}
|
||||
- REDDIT_BOT_AGENT_CONTROL_TOKEN=${REDDIT_BOT_AGENT_CONTROL_TOKEN:-}
|
||||
restart: unless-stopped
|
||||
healthcheck:
|
||||
test: ["CMD", "wget", "--spider", "http://localhost:3000/healthz"]
|
||||
interval: 30s
|
||||
timeout: 10s
|
||||
retries: 3
|
||||
start_period: 60s
|
||||
```
|
||||
|
||||
### Start with Compose
|
||||
For routine backups, prefer a volume-level snapshot or a consistent SQLite online backup plus browser/media directories. For a reset:
|
||||
|
||||
```bash
|
||||
# Set environment variables
|
||||
cp .env.example .env
|
||||
# Edit .env with your credentials
|
||||
1. State whether scope is one account, selected content, the RedditBot volume, the Compose application, or the Dokploy instance.
|
||||
2. Capture and validate a backup.
|
||||
3. Stop relevant writers.
|
||||
4. Resolve explicit volume/container/app identifiers with read-only inspection.
|
||||
5. Perform only the authorized deletion.
|
||||
6. Recreate/deploy, then verify health, schema, accounts, and persistence.
|
||||
7. Report what was removed and whether/how it can be restored.
|
||||
|
||||
# Build and start
|
||||
docker compose up -d --build
|
||||
## Routine checks
|
||||
|
||||
# View logs
|
||||
docker compose logs -f reddit-bot
|
||||
- Review attention items, failed/retrying candidates, reply backlog age, account authentication, invitation/global cooldowns, model errors/cost, and disk/volume health.
|
||||
- Review response/image rates only with mature sample counts and by source/strategy/account context.
|
||||
- Back up before migrations, account resets, or Compose/volume changes.
|
||||
- Rotate dashboard, agent, OpenRouter, Reddit, and Matrix credentials after suspected exposure.
|
||||
|
||||
# Stop
|
||||
docker compose down
|
||||
|
||||
# Restart
|
||||
docker compose restart
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Dokploy Deployment
|
||||
|
||||
Dokploy is a self-hosted platform for deploying applications. See `DOKPLOY_DEPLOY.md` for detailed instructions.
|
||||
|
||||
### Quick Start
|
||||
|
||||
1. **Create Application**: In Dokploy dashboard, create a new Docker application
|
||||
2. **Set Environment Variables**:
|
||||
- `OPENROUTER_API_KEY`
|
||||
- `WEB_DASHBOARD_TOKEN`
|
||||
3. **Mount Volume**: Mount `./data` for persistent storage
|
||||
4. **Set Port**: Map container port 3000 to host port
|
||||
5. **Deploy**: Push to repository and trigger deployment
|
||||
|
||||
### Dokploy Configuration
|
||||
|
||||
```yaml
|
||||
# Dokploy-specific settings
|
||||
app:
|
||||
type: docker
|
||||
repository: <your-repo>
|
||||
branch: main
|
||||
dockerfilePath: Dockerfile
|
||||
|
||||
environment:
|
||||
OPENROUTER_API_KEY: $OPENROUTER_API_KEY
|
||||
WEB_DASHBOARD_TOKEN: $WEB_DASHBOARD_TOKEN
|
||||
|
||||
ports:
|
||||
- container: 3000
|
||||
host: 3000
|
||||
|
||||
volumes:
|
||||
- source: reddit-bot-data
|
||||
target: /app/data
|
||||
type: bind
|
||||
|
||||
restartPolicy: unless-stopped
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Production Considerations
|
||||
|
||||
### Security
|
||||
|
||||
1. **Network Isolation**: Keep the dashboard on a trusted network during the
|
||||
migration cutover
|
||||
2. **Environment Variables**: Never commit credentials; use secrets management
|
||||
3. **Dashboard Access**: Authenticated Express hosting for React assets, API,
|
||||
SSE, debug routes, and private media is a required cutover gate, not a
|
||||
currently proven guarantee
|
||||
4. **Browser Data**: Protect `data/browser/` directory
|
||||
|
||||
### Performance
|
||||
|
||||
1. **Account dumps**: Delete downloaded dumps after the dashboard verifies them
|
||||
2. **WAL Mode**: SQLite WAL mode is enabled by default for concurrent reads
|
||||
3. **Rate Limiting**: Adjust based on Reddit account age and trust level
|
||||
4. **Message Volume**: Start low (10-20/hour) and increase gradually
|
||||
|
||||
### Reliability
|
||||
|
||||
1. **Auto-restart**: Use `restart: unless-stopped` in Docker
|
||||
2. **Health Checks**: Monitor `/api/stats` endpoint
|
||||
3. **Logging**: Review logs regularly for errors
|
||||
4. **Backups**: Regularly backup `data/reddit-bot.db`
|
||||
|
||||
### Monitoring
|
||||
|
||||
1. **Dashboard**: Check web dashboard for real-time stats
|
||||
2. **Response Rate**: Monitor response rates per strategy/subreddit
|
||||
3. **Error Rate**: Watch for AI refusals, send failures
|
||||
4. **Browser Health**: Verify session isn't expired
|
||||
|
||||
### Scaling
|
||||
|
||||
RedditBot is designed as a single-instance application. For scaling:
|
||||
|
||||
1. **Multiple Accounts**: Run separate instances per Reddit account
|
||||
2. **Subreddit Split**: Distribute subreddits across instances
|
||||
3. **Centralized DB**: Consider moving to PostgreSQL for multi-instance
|
||||
|
||||
---
|
||||
|
||||
## Monitoring & Maintenance
|
||||
|
||||
### Daily Checks
|
||||
|
||||
- [ ] Dashboard shows bot is running
|
||||
- [ ] Response rate is reasonable (>10%)
|
||||
- [ ] No repeated errors in logs
|
||||
- [ ] Browser session is active
|
||||
|
||||
### Weekly Tasks
|
||||
|
||||
- [ ] Review analytics (strategy performance, subreddit performance)
|
||||
- [ ] Check disk usage (media files, database)
|
||||
- [ ] Verify backup is working
|
||||
- [ ] Review successful messages for quality
|
||||
|
||||
### Monthly Tasks
|
||||
|
||||
- [ ] Backup database
|
||||
- [ ] Review and update strategy variants
|
||||
- [ ] Check for OpenRouter model updates
|
||||
- [ ] Review and update seller keywords
|
||||
- [ ] Verify browser session is still valid
|
||||
|
||||
### Backup Procedure
|
||||
|
||||
```bash
|
||||
# Stop the bot
|
||||
docker compose stop
|
||||
|
||||
# Backup database
|
||||
cp data/reddit-bot.db data/reddit-bot.db.backup.$(date +%Y%m%d)
|
||||
|
||||
# Backup browser session (optional)
|
||||
tar czf data/browser.backup.$(date +%Y%m%d).tar.gz data/browser/
|
||||
|
||||
# Restart the bot
|
||||
docker compose start
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
### Bot Won't Start
|
||||
|
||||
**Symptom**: Bot exits immediately on startup.
|
||||
|
||||
**Steps**:
|
||||
1. Check logs: `docker compose logs reddit-bot`
|
||||
2. Verify `config.yaml` syntax
|
||||
3. Verify environment variables are set
|
||||
4. Check for Zod validation errors in logs
|
||||
|
||||
### Account Import Fails
|
||||
|
||||
**Symptom**: Account work stays paused or import verification fails.
|
||||
|
||||
**Steps**:
|
||||
1. Sign in to Reddit in local Chrome
|
||||
2. Open Reddit Chat and wait for the extension badge to show `OK`
|
||||
3. Export and upload a new combined session dump
|
||||
4. Check logs for Reddit or Matrix identity errors
|
||||
|
||||
### AI Analysis Fails
|
||||
|
||||
**Symptom**: All messages show "AI refusal" or analysis errors.
|
||||
|
||||
**Steps**:
|
||||
1. Verify API key is valid and has balance
|
||||
2. Check model availability on OpenRouter
|
||||
3. Try changing to a different model
|
||||
4. Review logs for specific error messages
|
||||
|
||||
### High Error Rate
|
||||
|
||||
**Symptom**: Many messages fail to send.
|
||||
|
||||
**Steps**:
|
||||
1. Check rate limiting (reduce maxMessagesPerHour)
|
||||
2. Increase minDelayMs and jitterMs
|
||||
3. Check for IP bans (try different network)
|
||||
4. Review Reddit account standing
|
||||
|
||||
### Dashboard Not Accessible
|
||||
|
||||
**Symptom**: Cannot reach http://localhost:3000.
|
||||
|
||||
**Steps**:
|
||||
1. Verify bot is running (`docker compose ps`)
|
||||
2. Check port is not in use: `lsof -i :3000`
|
||||
3. Verify firewall allows port 3000
|
||||
4. Check web.port in config.yaml
|
||||
|
||||
### Database Issues
|
||||
|
||||
**Symptom**: Database errors or corruption.
|
||||
|
||||
**Steps**:
|
||||
1. Stop the bot
|
||||
2. Backup the database
|
||||
3. Try opening with SQLite viewer to check integrity
|
||||
4. If corrupted, restore from backup
|
||||
5. Consider enabling automatic backups
|
||||
The concise account activation instructions in [`../DOKPLOY_DEPLOY.md`](../DOKPLOY_DEPLOY.md) complement this operational guide.
|
||||
+84
-470
@@ -1,510 +1,124 @@
|
||||
# Development Guide
|
||||
# Development and testing
|
||||
|
||||
Guide for contributing to RedditBot development.
|
||||
## Prerequisites
|
||||
|
||||
---
|
||||
- Node.js compatible with the lockfile/native `better-sqlite3` build.
|
||||
- npm.
|
||||
- Playwright system/browser dependencies for browser-backed tests or runtime work.
|
||||
- No live account is required for unit, contract, route, or dashboard tests.
|
||||
|
||||
## Table of Contents
|
||||
NodeNext imports use `.js` suffixes even when the source file is `.ts`. Vitest globals are disabled, so tests import `describe`, `it`, `expect`, and hooks explicitly.
|
||||
|
||||
1. [Project Structure](#project-structure)
|
||||
2. [Development Setup](#development-setup)
|
||||
3. [Coding Standards](#coding-standards)
|
||||
4. [Adding a New Service](#adding-a-new-service)
|
||||
5. [Adding a New Strategy Variant](#adding-a-new-strategy-variant)
|
||||
6. [Debugging](#debugging)
|
||||
7. [Common Tasks](#common-tasks)
|
||||
8. [Known Issues](#known-issues)
|
||||
|
||||
---
|
||||
|
||||
## Project Structure
|
||||
|
||||
```
|
||||
RedditBot/
|
||||
├── config.yaml # Main configuration
|
||||
├── package.json # Dependencies and scripts
|
||||
├── tsconfig.json # TypeScript config
|
||||
├── Dockerfile # Container build
|
||||
├── docker-compose.yml # Docker Compose
|
||||
├── apps/
|
||||
│ └── dashboard/ # React + Vite operator console
|
||||
├── packages/
|
||||
│ └── api-contracts/ # Shared Zod transport contracts
|
||||
├── scripts/
|
||||
│ ├── login.ts # Manual login helper
|
||||
│ └── inspect-chat-dom.ts # DOM inspection debug script
|
||||
└── src/
|
||||
├── index.ts # Entry point / orchestrator
|
||||
├── config/
|
||||
│ └── index.ts # Config loader + Zod schema
|
||||
├── database/
|
||||
│ ├── client.ts # SQLite setup + migrations
|
||||
│ └── repositories.ts # Repository classes
|
||||
├── reddit/
|
||||
│ └── api.ts # Reddit API client
|
||||
├── ai/
|
||||
│ └── openrouter.ts # OpenRouter AI client
|
||||
├── browser/
|
||||
│ ├── client.ts # Managed Playwright context
|
||||
│ └── media-downloader.ts # Media download
|
||||
├── services/
|
||||
│ ├── monitor.ts # Post discovery
|
||||
│ ├── processor.ts # Processing pipeline
|
||||
│ ├── response-checker.ts # Response monitoring
|
||||
│ ├── strategy-selector.ts # A/B testing
|
||||
│ ├── chat.ts # Chat scraping + messaging
|
||||
│ ├── media.ts # Media management
|
||||
│ ├── media-sync.ts # Media sync scheduler
|
||||
│ └── auto-reply.ts # Auto-reply service
|
||||
├── utils/
|
||||
│ ├── logger.ts # Winston logger
|
||||
│ ├── rate-limiter.ts # Rate limiting
|
||||
│ ├── retry.ts # Retry wrapper
|
||||
│ └── image.ts # Image utilities
|
||||
└── web/ # Express API, events, and dashboard hosting
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Development Setup
|
||||
|
||||
### 1. Install Dependencies
|
||||
## Safe setup
|
||||
|
||||
```bash
|
||||
npm install
|
||||
npm ci
|
||||
npm run build
|
||||
npm test
|
||||
```
|
||||
|
||||
### 2. Configure
|
||||
Do not start `npm run dev` merely to verify a build. The repository `.env` and persisted database can make the process live-capable. Starting the backend, importing sessions, or toggling the bot to running requires explicit authorization and a confirmed dry-run/live intent.
|
||||
|
||||
Copy and edit `config.yaml`:
|
||||
If native SQLite bindings fail after a Node upgrade, distinguish pure TypeScript/unit results from database verification. Rebuild/install a compatible `better-sqlite3`; `--ignore-scripts` is not a valid database test environment.
|
||||
|
||||
```bash
|
||||
cp config.yaml config.yaml.backup # Backup existing
|
||||
# Edit config.yaml with your settings
|
||||
```
|
||||
## Commands
|
||||
|
||||
### 3. Set Environment Variables
|
||||
| Command | Purpose |
|
||||
| --- | --- |
|
||||
| `npm run build` | Compile server and production dashboard. |
|
||||
| `npm run build:server` | Run backend TypeScript compiler. |
|
||||
| `npm run dashboard:typecheck` | Type-check the React application. |
|
||||
| `npm run dashboard:build` | Build dashboard assets. |
|
||||
| `npm test` | Run the complete Vitest suite once. |
|
||||
| `npm run test:watch` | Watch tests during local development. |
|
||||
| `npm run test:coverage` | Run with coverage. |
|
||||
| `npm run docs:api` | Regenerate `docs/api.md`. |
|
||||
| `npm run docs:api:check` | Fail if generated API documentation is stale. |
|
||||
| `npm run mcp` | Start the stdio MCP adapter; requires an authorized running API. |
|
||||
| `npm run dev` | Start the live-capable backend watcher. |
|
||||
| `npm run dashboard:dev` | Start Vite, proxying API requests to port 3000. |
|
||||
|
||||
```bash
|
||||
export OPENROUTER_API_KEY="sk-or-v1-your-key"
|
||||
export WEB_DASHBOARD_TOKEN="long-random-secret"
|
||||
```
|
||||
## Change workflow
|
||||
|
||||
### 4. Account authentication
|
||||
1. Read `agents.md`, `docs/README.md`, the routed topic docs, owning source, and focused tests.
|
||||
2. Inspect `git status` before editing. Preserve unrelated user changes and do not broadly stage a dirty worktree.
|
||||
3. Establish a baseline appropriate to the change. Code-changing tasks follow the repository test-first instructions; docs-only changes use documentation/link checks.
|
||||
4. Identify the source-of-truth boundary: config schema, migration, service, shared transport contract, route, or UI feature.
|
||||
5. Implement the smallest coherent behavior using dependency injection and single-purpose modules.
|
||||
6. Add behavior-focused tests for observable inputs, outputs, state transitions, and side effects.
|
||||
7. Update every affected document and regenerate API docs when the catalog changes.
|
||||
8. Run focused tests while iterating, then full required checks before publishing.
|
||||
9. Review the exact staged diff, commit only the coherent set, push, and verify branch/remote sync.
|
||||
|
||||
Start the app, then import a combined Reddit and Matrix session dump from the local Chrome extension under **Settings > Access**.
|
||||
## Engineering practices
|
||||
|
||||
### 5. Start Development Server
|
||||
### Types and validation
|
||||
|
||||
```bash
|
||||
# Terminal 1: backend reload with tsx
|
||||
npm run dev
|
||||
- No production `any`; use `unknown`, Zod, discriminated unions, or type guards.
|
||||
- Keep database/service-private types separate from transport contracts.
|
||||
- Parse at boundaries and pass trusted domain values inward.
|
||||
- Return explicit terminal/retryable/human-required results where callers must decide policy.
|
||||
|
||||
# Terminal 2: Vite frontend with /api proxied to port 3000
|
||||
npm run dashboard:dev
|
||||
### Service design
|
||||
|
||||
# Production build: server plus apps/dashboard/dist
|
||||
npm run build && npm start
|
||||
```
|
||||
|
||||
### 6. Access Dashboard
|
||||
|
||||
During frontend development, open the URL printed by Vite (normally
|
||||
http://localhost:5173). The Express dashboard remains on http://localhost:3000
|
||||
for production/fallback verification.
|
||||
|
||||
Useful frontend checks:
|
||||
|
||||
```bash
|
||||
npm run dashboard:typecheck
|
||||
npm run dashboard:build
|
||||
npx vitest run tests/dashboard
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Coding Standards
|
||||
|
||||
### TypeScript
|
||||
|
||||
- **Strict mode**: No `any` types (use `unknown` with type guards)
|
||||
- **Interfaces**: Define interfaces for all data structures
|
||||
- **Null checks**: Use optional chaining (`?.`) and nullish coalescing (`??`)
|
||||
- **Async/Await**: Use async/await, not `.then()` chains
|
||||
|
||||
```typescript
|
||||
// Good
|
||||
interface PostData {
|
||||
id: string;
|
||||
subreddit: string;
|
||||
author: string;
|
||||
}
|
||||
|
||||
async function fetchPost(id: string): Promise<PostData> {
|
||||
const response = await api.get(`/posts/${id}`);
|
||||
return response.data;
|
||||
}
|
||||
|
||||
// Bad
|
||||
async function fetchPost(id: any): Promise<any> {
|
||||
return api.get(`/posts/${id}`).then(r => r.data);
|
||||
}
|
||||
```
|
||||
- Constructor-inject external dependencies, clocks, fetchers, schedulers, and persistence seams.
|
||||
- Keep process-global coordination separate from account-scoped mutable state.
|
||||
- Prefer a small policy/service over another conditional in a composition root or route.
|
||||
- Preserve abort signals through queues, sleeps, and remote operations.
|
||||
|
||||
### Logging
|
||||
|
||||
Use `winston` logger, never `console.log`:
|
||||
Use the injected Winston logger. Include stage, account context (through child loggers), post/conversation IDs, and safe rejection/retry reasons. Never serialize raw Axios/Playwright errors that may include cookies or authorization headers; extract a safe message/status. Avoid high-frequency unchanged status logs.
|
||||
|
||||
```typescript
|
||||
// Good
|
||||
this.logger.info("Processed post {id}", { id: post.id });
|
||||
this.logger.warn("AI refusal detected", { postId: post.id });
|
||||
this.logger.error("Failed to send DM", { error: err.message });
|
||||
### Remote effects
|
||||
|
||||
// Bad
|
||||
console.log("Processed post", post.id);
|
||||
```
|
||||
- Check dry-run at the final side-effect boundary.
|
||||
- Make retries idempotent and distinguish remote acceptance from local persistence.
|
||||
- Reserve recipients/claims before sending and release only on safe failure paths.
|
||||
- Treat 403, 404, and 429 according to their typed domain meaning rather than one generic retry.
|
||||
|
||||
### Error Handling
|
||||
## Test layout and patterns
|
||||
|
||||
Use try/catch with structured error handling:
|
||||
Tests live in `tests/`; dashboard tests are under `tests/dashboard/`, route/event tests under `tests/web/`, and shared contract tests under `packages/api-contracts/tests/`.
|
||||
|
||||
```typescript
|
||||
try {
|
||||
await this.processPost(post);
|
||||
} catch (err) {
|
||||
this.logger.error("Post processing failed", {
|
||||
postId: post.id,
|
||||
error: err instanceof Error ? err.message : String(err)
|
||||
});
|
||||
// Don't throw - allow other posts to be processed
|
||||
}
|
||||
```
|
||||
Preferred patterns:
|
||||
|
||||
### Service Structure
|
||||
- in-memory SQLite with real migrations/repositories;
|
||||
- fake clock/sleep/fetch functions injected into services;
|
||||
- small service fakes implementing narrow interfaces;
|
||||
- Testing Library for user-observable React behavior;
|
||||
- representative payload parsing for strict Zod contracts;
|
||||
- lifecycle/restart tests for queues, claims, and account runtimes.
|
||||
|
||||
Follow single responsibility:
|
||||
Avoid deleting coverage to make a suite pass, assertions that pin private call order without behavioral value, ESM namespace mocking when injection is available, or tests that read the real `.env`/production data.
|
||||
|
||||
```typescript
|
||||
class NewService {
|
||||
constructor(
|
||||
private config: Config,
|
||||
private dependency: ExistingService,
|
||||
private logger: Logger
|
||||
) {}
|
||||
## Verification matrix
|
||||
|
||||
async doSomething(input: string): Promise<Result> {
|
||||
// Validate input
|
||||
if (!input) {
|
||||
throw new Error("Input is required");
|
||||
}
|
||||
| Change | Minimum focused verification | Final verification |
|
||||
| --- | --- | --- |
|
||||
| Pure backend/service | owning Vitest files | `npm test`, `npm run build:server` |
|
||||
| Database/migration | migration + repository tests | full tests and server build |
|
||||
| API/contract | route + contract + auth tests; regenerate docs | full tests, build, `docs:api:check` |
|
||||
| Dashboard | feature tests + typecheck | full tests and full build |
|
||||
| Docker/deployment | compose/Docker tests + image build as appropriate | health, API, container, logs, and persistence after authorized deploy |
|
||||
| Documentation only | generated-doc check + internal link scan | staged diff review |
|
||||
|
||||
// Do work
|
||||
const result = await this.dependency.process(input);
|
||||
If the worktree has pre-existing failures or concurrent changes, record the exact baseline and do not attribute them to your change. Never hide a failed required check behind a passing focused test.
|
||||
|
||||
// Return result
|
||||
return result;
|
||||
}
|
||||
}
|
||||
```
|
||||
## Common implementation recipes
|
||||
|
||||
### File Organization
|
||||
### Add an endpoint
|
||||
|
||||
- One class per file (except repositories which share one file)
|
||||
- Services in `src/services/`
|
||||
- Utilities in `src/utils/`
|
||||
- External integrations in their own folders (`src/ai/`, `src/reddit/`, `src/browser/`)
|
||||
Create or extend a focused router, validate input, add shared output contracts, register metadata in `api-catalog.ts`, add authorization/behavior tests, consume via dashboard query helpers if needed, regenerate `docs/api.md`, and update the relevant narrative doc.
|
||||
|
||||
---
|
||||
### Add persisted state
|
||||
|
||||
## Adding a New Service
|
||||
Append a migration, add a narrow repository, wire it through the composition root/runtime, test fresh and restart/upgrade cases, and update `database.md`.
|
||||
|
||||
### 1. Create the Service File
|
||||
### Change candidate or reply behavior
|
||||
|
||||
```typescript
|
||||
// src/services/new-service.ts
|
||||
import { Config } from '../config';
|
||||
import { Logger } from '../utils/logger';
|
||||
Start with the durable state transition and idempotency rule. Test rejection/retry/success, process interruption, account ownership, and operator-visible reason counts. Update product, flow, service, and data docs as applicable.
|
||||
|
||||
interface NewServiceDeps {
|
||||
config: Config;
|
||||
logger: Logger;
|
||||
// other dependencies
|
||||
}
|
||||
### Add dashboard state
|
||||
|
||||
interface Result {
|
||||
success: boolean;
|
||||
data?: any;
|
||||
error?: string;
|
||||
}
|
||||
|
||||
export class NewService {
|
||||
constructor(private deps: NewServiceDeps) {}
|
||||
|
||||
async execute(input: string): Promise<Result> {
|
||||
const { config, logger } = this.deps;
|
||||
|
||||
logger.info("NewService executing", { input });
|
||||
|
||||
// Implementation here
|
||||
|
||||
return { success: true };
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
### 2. Add to Dependencies Interface
|
||||
|
||||
If the service needs other services, add them to the interface:
|
||||
|
||||
```typescript
|
||||
interface NewServiceDeps {
|
||||
config: Config;
|
||||
logger: Logger;
|
||||
redditApi: RedditApi;
|
||||
openrouter: OpenRouterService;
|
||||
}
|
||||
```
|
||||
|
||||
### 3. Wire Up in main()
|
||||
|
||||
In `src/index.ts`, add initialization:
|
||||
|
||||
```typescript
|
||||
const newService = new NewService({
|
||||
config,
|
||||
logger,
|
||||
redditApi,
|
||||
openrouter
|
||||
});
|
||||
|
||||
// If it needs to run periodically:
|
||||
setInterval(() => {
|
||||
newService.execute(someInput).catch(err => {
|
||||
logger.error("NewService error", { error: err.message });
|
||||
});
|
||||
}, intervalMs);
|
||||
```
|
||||
|
||||
### 4. Add to the Web API and Dashboard (Optional)
|
||||
|
||||
If the service needs an API, add a focused router under `src/web/routes/` and
|
||||
mount it from `src/web/server.ts`. Define its transport schema in
|
||||
`packages/api-contracts`; do not expose repository rows directly.
|
||||
|
||||
```typescript
|
||||
router.get('/new-service', async (req, res) => {
|
||||
try {
|
||||
const result = await newService.execute(req.query.input as string);
|
||||
res.json(result);
|
||||
} catch (err) {
|
||||
res.status(500).json({ error: (err as Error).message });
|
||||
}
|
||||
});
|
||||
```
|
||||
|
||||
Add frontend behavior under `apps/dashboard/src/features/<feature>/` and drive
|
||||
it with Testing Library and MSW tests under `tests/dashboard/`.
|
||||
|
||||
### 5. Update Config (If Needed)
|
||||
|
||||
Add new configuration options to:
|
||||
- `config.yaml`
|
||||
- `src/config/index.ts` Zod schema
|
||||
- Configuration documentation
|
||||
|
||||
---
|
||||
|
||||
## Adding a New Strategy Variant
|
||||
|
||||
### 1. Add to config.yaml
|
||||
|
||||
```yaml
|
||||
strategyVariants:
|
||||
- name: "existing_variant"
|
||||
prompt: "Existing prompt..."
|
||||
enabled: true
|
||||
|
||||
- name: "new_variant"
|
||||
prompt: |
|
||||
Write a friendly message that references the image content.
|
||||
Be specific and genuine. Keep it under 2 sentences.
|
||||
enabled: true
|
||||
```
|
||||
|
||||
### 2. No Code Changes Needed
|
||||
|
||||
The `StrategySelector` automatically picks up new variants from the config. The algorithm will:
|
||||
|
||||
1. Include the new variant in random exploration
|
||||
2. Track its response rate once messages are sent
|
||||
3. Promote it to exploitation if it has good stats
|
||||
|
||||
### 3. Monitor Performance
|
||||
|
||||
Check performance via the dashboard or API:
|
||||
|
||||
```bash
|
||||
curl http://localhost:3000/api/analytics/by-strategy
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Debugging
|
||||
|
||||
### Account debugging
|
||||
|
||||
Use dashboard session status and structured logs. The backend does not expose or launch an interactive login browser.
|
||||
|
||||
### DOM Inspection
|
||||
|
||||
Use the debug endpoint to inspect Reddit's shadow DOM:
|
||||
|
||||
```bash
|
||||
curl -X POST http://localhost:3000/api/debug/inspect-dom
|
||||
```
|
||||
|
||||
Or run the standalone script:
|
||||
|
||||
```bash
|
||||
npx tsx scripts/inspect-chat-dom.ts
|
||||
```
|
||||
|
||||
### Logging
|
||||
|
||||
Adjust log level by modifying `src/utils/logger.ts`:
|
||||
|
||||
```typescript
|
||||
const logger = winston.createLogger({
|
||||
level: 'debug', // Change from 'info' to 'debug'
|
||||
// ...
|
||||
});
|
||||
```
|
||||
|
||||
### Database Inspection
|
||||
|
||||
Open the database with SQLite viewer:
|
||||
|
||||
```bash
|
||||
# Install sqlite3 if needed
|
||||
brew install sqlite3
|
||||
|
||||
# Open database
|
||||
sqlite3 data/reddit-bot.db
|
||||
|
||||
# Query
|
||||
SELECT * FROM sent_messages LIMIT 5;
|
||||
```
|
||||
|
||||
### Network Debugging
|
||||
|
||||
Enable axios debug:
|
||||
|
||||
```typescript
|
||||
// In src/reddit/api.ts
|
||||
import axios from 'axios';
|
||||
axios.defaults.debug = true;
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Common Tasks
|
||||
|
||||
### Add a New Subreddit
|
||||
|
||||
```yaml
|
||||
# config.yaml
|
||||
reddit:
|
||||
subreddits:
|
||||
- gonewild
|
||||
- ratemyboobs
|
||||
- new_subreddit_name # Add here
|
||||
```
|
||||
|
||||
### Change AI Model
|
||||
|
||||
```yaml
|
||||
# config.yaml
|
||||
openrouter:
|
||||
model: "new-model-name"
|
||||
fallbackModels:
|
||||
- "fallback-model-1"
|
||||
- "fallback-model-2"
|
||||
```
|
||||
|
||||
### Adjust Rate Limits
|
||||
|
||||
```yaml
|
||||
# config.yaml
|
||||
rateLimit:
|
||||
minDelayMs: 45000 # Increase delay
|
||||
maxMessagesPerHour: 15 # Reduce cap
|
||||
jitterMs: 20000 # More jitter
|
||||
```
|
||||
|
||||
### Enable Dry Run
|
||||
|
||||
```yaml
|
||||
# config.yaml
|
||||
dryRun: true # Logs all actions without sending
|
||||
```
|
||||
|
||||
Or via dashboard: `PUT /api/config` with `{"dryRun": true}`
|
||||
|
||||
### Add Database Column
|
||||
|
||||
```typescript
|
||||
// In src/database/client.ts
|
||||
function runMigrations(db: Database) {
|
||||
// ... existing migrations ...
|
||||
|
||||
// Check if column exists
|
||||
const tableInfo = db.prepare(`PRAGMA table_info(sent_messages)`).all();
|
||||
const hasNewColumn = tableInfo.some((col: any) => col.name === 'new_column');
|
||||
|
||||
if (!hasNewColumn) {
|
||||
db.exec("ALTER TABLE sent_messages ADD COLUMN new_column TEXT");
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
### Add New API Endpoint
|
||||
|
||||
```typescript
|
||||
// In src/web/server.ts
|
||||
import { Request, Response } from 'express';
|
||||
|
||||
server.app.get('/api/new-endpoint', async (req: Request, res: Response) => {
|
||||
try {
|
||||
const result = await someService.doSomething();
|
||||
res.json({ success: true, data: result });
|
||||
} catch (err) {
|
||||
res.status(500).json({ error: (err as Error).message });
|
||||
}
|
||||
});
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Known Issues
|
||||
|
||||
### Hardcoded Username in ChatService
|
||||
|
||||
`ChatService` has `myUsername = 'Upset_Marsupial1006'` hardcoded in JavaScript evaluation scripts. This should be parameterized from config.
|
||||
|
||||
**Fix needed**: Pass username to `page.evaluate()` calls.
|
||||
|
||||
### Shadow DOM Fragility
|
||||
|
||||
Reddit frequently updates their `faceplate-*` components. Shadow DOM traversal code may break without warning.
|
||||
|
||||
**Mitigation**: Use `scripts/inspect-chat-dom.ts` to debug when selectors change.
|
||||
|
||||
### Dashboard Migration Boundary
|
||||
|
||||
The React application lives in `apps/dashboard`. A production build is required
|
||||
before the Express server starts; there is no legacy HTML fallback.
|
||||
|
||||
The repository has automated service, contract, component, and transport tests.
|
||||
Manual browser verification is still required for Reddit DOM changes and the
|
||||
final dashboard cutover.
|
||||
Define the shared schema first, add a typed query/mutation, keep route/filter state in the URL where useful, invalidate snapshots from relevant SSE events, and verify loading/error/stale/accessibility states.
|
||||
@@ -1,111 +0,0 @@
|
||||
# React Dashboard Migration
|
||||
|
||||
## Outcome
|
||||
|
||||
The inline dashboard has been replaced by a typed, accessible React application.
|
||||
Express remains the authenticated API, SSE, private-media, and production
|
||||
static-asset host.
|
||||
|
||||
## Current status
|
||||
|
||||
The current routes are Overview, Conversations (including route-addressable
|
||||
detail), Media, Performance, Operations, and validated Settings and Access.
|
||||
Shared HTTP and realtime schemas live in `packages/api-contracts`, and
|
||||
`/api/events/stream` supplies the typed SSE transport.
|
||||
|
||||
`npm run build` compiles the server and writes the Vite application to
|
||||
`apps/dashboard/dist`. `src/web/dashboard-assets.ts` requires that build and
|
||||
fails startup with an actionable error if it is absent. The legacy HTML has
|
||||
been removed. Root pages, assets, APIs, SSE, and private media share the same
|
||||
timing-safe dashboard authentication boundary.
|
||||
|
||||
## Architecture
|
||||
|
||||
- `apps/dashboard`: React, TypeScript, Vite, React Router, and TanStack Query.
|
||||
- `packages/api-contracts`: shared Zod schemas for HTTP and realtime transport.
|
||||
- `src/web`: existing Express host, gradually adapted to the shared contracts.
|
||||
- Backend service and database types stay private to the backend. Transport
|
||||
contracts must not expose database rows as an accidental public API.
|
||||
- Server state lives in TanStack Query. Route, filter, and selection state lives
|
||||
in URLs. Local component state is reserved for ephemeral presentation state.
|
||||
- Do not add a global client-state library until a demonstrated cross-feature
|
||||
state requirement cannot be met by those boundaries.
|
||||
|
||||
## Navigation
|
||||
|
||||
The application shell provides a responsive sidebar or drawer with these routes:
|
||||
|
||||
| Route | Responsibility |
|
||||
| --- | --- |
|
||||
| `/` | Health, metrics, timers, limits, and important recent activity |
|
||||
| `/conversations` | Searchable conversation inbox |
|
||||
| `/conversations/:id` | Conversation history, context, suggestions, and sending |
|
||||
| `/media` | Private blurred-by-default gallery and sync controls |
|
||||
| `/performance` | Trends, strategies, subreddits, and successful examples |
|
||||
| `/operations` | Domain activity, background jobs, and diagnostics |
|
||||
| `/settings` | Validated runtime configuration and Reddit access state |
|
||||
|
||||
Overview surfaces bot state, dry-run state, cooldowns, timers, and the guarded
|
||||
start/stop control. Operations owns connection health, sync state, activity,
|
||||
and the bounded technical console.
|
||||
|
||||
## Realtime model
|
||||
|
||||
HTTP snapshots are authoritative. A typed SSE event stream updates or invalidates
|
||||
the relevant query cache. Every event has an ID, type, occurrence timestamp, and
|
||||
validated payload. The stream supports bounded replay and `Last-Event-ID` resume.
|
||||
Relevant events invalidate TanStack Query keys so the next HTTP read remains
|
||||
authoritative. Connection loss marks cached state stale while `EventSource`
|
||||
reconnects.
|
||||
|
||||
Required event families are bot status, message sent or failed, conversation
|
||||
updated, reply received, sync lifecycle and progress, rate-limit state, Reddit
|
||||
session state, and technical logs.
|
||||
|
||||
Important domain events appear in a readable activity timeline. Raw logs remain
|
||||
in Operations. Toasts are reserved for user-triggered outcomes requiring timely
|
||||
feedback. Realtime disconnection never blanks cached content; it marks it stale
|
||||
and falls back to visibility-aware polling where necessary.
|
||||
|
||||
## Feature slices
|
||||
|
||||
1. Capture and validate existing HTTP contracts and critical legacy journeys.
|
||||
2. Add the frontend build, application shell, authentication handling, routing,
|
||||
error boundaries, design tokens, query client, and test harness.
|
||||
3. Deliver Overview as the first production vertical slice.
|
||||
4. Add the typed realtime event core, SSE transport, reconnection, and query
|
||||
invalidation.
|
||||
5. Migrate Conversations, including explicit suggestion and send mutations.
|
||||
6. Migrate Media, preserving authentication and blur-by-default privacy.
|
||||
7. Migrate Performance with sample sizes and low-confidence warnings.
|
||||
8. Migrate Operations, access/login, and route-backed validated Settings.
|
||||
9. Run feature parity and security audits, switch Express to the Vite output,
|
||||
remove the legacy HTML, and update operator and deployment documentation.
|
||||
|
||||
All slices now run from the React application; the migration phases above are
|
||||
retained as implementation history.
|
||||
|
||||
## Engineering gates
|
||||
|
||||
- New behavior follows red-green-refactor with Vitest.
|
||||
- Component tests use Testing Library and test observable behavior.
|
||||
- HTTP tests use a controlled boundary such as MSW rather than mocking `fetch`
|
||||
throughout feature tests.
|
||||
- Contract tests prove representative current responses parse and malformed
|
||||
payloads fail.
|
||||
- Realtime tests cover ordering, replay, duplicate handling, reconnection, and
|
||||
cache repair.
|
||||
- Browser tests cover authentication, start/stop, conversation inspection,
|
||||
sending confirmation, sync, settings persistence, and private media.
|
||||
- Interactive controls are keyboard accessible and restore focus after dialogs.
|
||||
- No production frontend or backend code uses `any` or `console.log`.
|
||||
- Mutations display the confirmed server outcome and do not rely on optimistic
|
||||
success for operational controls.
|
||||
- Production builds are deterministic. The server serves only generated React
|
||||
assets plus authenticated private content.
|
||||
|
||||
## Cutover evidence
|
||||
|
||||
Cutover verification covers API and SSE authentication, private media access,
|
||||
SSE recovery, responsive navigation, frontend type-check/build, backend build,
|
||||
the complete Vitest suite, and browser smoke journeys.
|
||||
@@ -0,0 +1,105 @@
|
||||
# Dashboard architecture
|
||||
|
||||
## Stack and boundary
|
||||
|
||||
The operator console is a React 19 + TypeScript + Vite application under `apps/dashboard`. Express serves its production build, private media, API, authentication endpoints, and SSE streams from the same origin. TanStack Query owns server state; React Router owns navigation and route-addressable state.
|
||||
|
||||
Shared HTTP/SSE schemas live in `packages/api-contracts`. The server and client should not duplicate transport interfaces by hand.
|
||||
|
||||
## Routes
|
||||
|
||||
| Route | Responsibility |
|
||||
| --- | --- |
|
||||
| `/` | Health, per-account delivery/cooldown state, timers, workload, and recent activity. |
|
||||
| `/attention` | Human handoffs, delivery failures, account risks, and other actionable items. |
|
||||
| `/outreach` | Durable candidate funnel and active candidate detail. |
|
||||
| `/conversations` | Searchable account-aware conversation inbox. |
|
||||
| `/conversations/:conversationId` | Conversation history, context, suggestions, and guarded sending. |
|
||||
| `/media` | Authenticated, privacy-conscious media gallery and sync controls. |
|
||||
| `/performance` | Mature outcome trends, strategy/subreddit results, and model usage. |
|
||||
| `/operations` | Background jobs, sync state, events, rate limits, and technical logs. |
|
||||
| `/settings` | Validated runtime configuration and account/session management. |
|
||||
|
||||
Navigation is declared in `app/navigation.ts`; route elements are composed in `app/router.tsx`.
|
||||
|
||||
## State model
|
||||
|
||||
- Remote state: TanStack Query snapshots parsed by shared Zod schemas.
|
||||
- Route/filter/selection state: URL/search params when it should survive navigation or be shareable.
|
||||
- Ephemeral UI state: component/provider state, such as dialogs and notification presentation.
|
||||
- Persistent presentation state: guarded browser storage only when explicitly useful (for example notification history).
|
||||
|
||||
Do not introduce a general global state store unless a demonstrated cross-feature need cannot fit these boundaries.
|
||||
|
||||
## HTTP and contracts
|
||||
|
||||
`api/request.ts` owns authenticated request mechanics and failure conversion. `api/client.ts`/`api/queries.ts` expose typed calls and query keys. Parse every response at the boundary. Strict schemas are deliberate: when the server adds a field, update the shared contract and its consumers in the same change.
|
||||
|
||||
Mutations should display the confirmed server result. Start/stop, config, sync, import, and send actions must not use optimistic success for operational state.
|
||||
|
||||
Account delivery state is reported per runtime (`ready`, stopped/failed/login-required, Matrix cooldown, replies prioritized, or pacing). Preserve account attribution when presenting cooldowns and failures; a blocked account does not imply every account is blocked.
|
||||
|
||||
## Realtime model
|
||||
|
||||
`RealtimeProvider` connects to `/api/events/stream`. Event IDs support bounded replay/resume. Each event family either updates a narrowly safe cache value or invalidates the relevant HTTP snapshot. Reconnect and stale detection live in the app layer, not individual feature pages.
|
||||
|
||||
Rules:
|
||||
|
||||
- HTTP remains authoritative.
|
||||
- Duplicate/replayed events must be harmless.
|
||||
- A disconnected stream marks data stale but does not blank useful cached content.
|
||||
- Invalid event payloads are contained and repaired with a snapshot refresh.
|
||||
- User-triggered outcomes may create toasts; durable/history-worthy events also appear in the notification center/activity views.
|
||||
- Raw technical logs belong in Operations, not as general user notifications.
|
||||
|
||||
## Authentication and privacy
|
||||
|
||||
The login form exchanges the administrator token for an HttpOnly, SameSite=Strict cookie. API clients may use Bearer tokens. Express applies auth before static dashboard assets, APIs, SSE, and `/media`; `/healthz` is the intentional public exception.
|
||||
|
||||
Media may be private conversation content. Preserve authenticated URLs, blur/privacy defaults, explicit reveal behavior, safe new-tab links, and no token leakage into query strings or local storage.
|
||||
|
||||
## Feature structure
|
||||
|
||||
Each route is a slice under `features/<name>/` with page/route components, local styles, and feature-specific helpers. Cross-feature visual/domain components live under `components/`; application lifecycle belongs under `app/`; transport belongs under `api/`.
|
||||
|
||||
Keep feature components focused:
|
||||
|
||||
- pages compose queries and sections;
|
||||
- presentational components render typed props;
|
||||
- mutation hooks own cache invalidation;
|
||||
- parsing/formatting lives in small pure helpers;
|
||||
- destructive or external-message actions require explicit confirmation and accessible focus restoration.
|
||||
|
||||
## Accessibility and presentation
|
||||
|
||||
- All actions must be keyboard accessible and have visible focus.
|
||||
- Dialogs label purpose, trap/restore focus, and distinguish cancel from destructive confirmation.
|
||||
- Status must not rely on color alone.
|
||||
- Relative times include an exact timestamp where operationally useful.
|
||||
- Lists are explicitly sorted after filters rather than relying on backend/incidental order.
|
||||
- Loading, empty, stale, partial, and error states are distinct.
|
||||
- Show sample sizes and low-confidence context for performance metrics.
|
||||
|
||||
## Testing
|
||||
|
||||
Dashboard behavior is tested with Vitest, jsdom, Testing Library, and controlled HTTP boundaries. Focus on observable behavior:
|
||||
|
||||
- representative payloads parse and malformed payloads fail;
|
||||
- route/search state and navigation;
|
||||
- loading/error/empty/stale views;
|
||||
- mutation confirmation and server read-back;
|
||||
- SSE ordering, replay, reconnect, duplicate handling, and snapshot repair;
|
||||
- authentication/private media boundaries;
|
||||
- keyboard/focus behavior for dialogs and controls.
|
||||
|
||||
Run focused `npx vitest run tests/dashboard/<file>`, then `npm run dashboard:typecheck`, `npm run dashboard:build`, and the full suite before publishing a frontend change.
|
||||
|
||||
## Adding a dashboard capability
|
||||
|
||||
1. Define/extend the shared Zod contract.
|
||||
2. Make the server route return that contract and add a route test.
|
||||
3. Add a typed client/query key.
|
||||
4. Build the feature slice and observable UI states.
|
||||
5. Map relevant realtime events to cache updates/invalidations.
|
||||
6. Add component/realtime/contract coverage.
|
||||
7. Update `api.md` when the operation catalog changed and update this or the relevant product/flow document.
|
||||
@@ -0,0 +1,77 @@
|
||||
# Product goals and safety boundaries
|
||||
|
||||
## Product purpose
|
||||
|
||||
RedditBot helps an operator discover suitable adult Reddit users who have explicitly posted in configured communities, start relevant conversations, prioritize existing replies, and learn which outreach approaches lead to genuine engagement. It is an operator-controlled system, not an unconstrained bulk messenger.
|
||||
|
||||
The product has five connected goals:
|
||||
|
||||
1. Find candidates from explicitly configured subreddits.
|
||||
2. Reject unsuitable candidates before expensive AI work or any send.
|
||||
3. Coordinate multiple accounts without contacting the same person repeatedly.
|
||||
4. Continue conversations only within configured, bounded automation policies.
|
||||
5. Make every material decision and outcome visible to the operator.
|
||||
|
||||
## What good behavior looks like
|
||||
|
||||
- Eligibility is completed before selecting a recipient: already-processed posts, recipient history, self-declared age/gender for chat posts, seller signals, content suitability, and AI safety all have explicit outcomes.
|
||||
- Existing replies take priority over new outreach. An account with actionable inbound messages is not available for a new candidate.
|
||||
- Discovery is global while authentication, rate limits, conversations, and content remain account-owned.
|
||||
- Recipient deduplication is case-insensitive, durable across restarts, and protected by a live reservation during sends.
|
||||
- Remote side effects are fail-closed: a send is recorded only after the chat service confirms it; interrupted reply claims reconcile against Matrix history before retrying.
|
||||
- Dry-run prepares and records decisions but never sends.
|
||||
- Operators can see funnel stages, account state, rate limits, failures, handoffs, and recent activity rather than inferring success from a process-health check.
|
||||
|
||||
## Candidate policy
|
||||
|
||||
Configured subreddits have a source type:
|
||||
|
||||
- `image`: image-led posts are downloaded and analyzed before an opener is generated.
|
||||
- `chat`: text-led posts require a parseable self-declared gender and adult age, then use the chat-opener prompt.
|
||||
|
||||
Seller filtering combines the candidate post with Reddit profile/about and recent activity evidence. The configured keyword/domain lists are policy inputs, not a substitute for the surrounding screening flow. A profile lookup failure is not itself proof that a person is a seller; network and rate-limit failures must remain distinguishable from a rejection.
|
||||
|
||||
The system must not manufacture personal facts. Generated messages must not claim shared location or proximity, invite an in-person meeting, invent identity/history, or ignore a conversation boundary. The final outbound safety check is intentionally separate from prompt wording.
|
||||
|
||||
## Conversation policy modes
|
||||
|
||||
Each subreddit snapshots a policy into a conversation when automation first handles it:
|
||||
|
||||
| Mode | Meaning |
|
||||
| --- | --- |
|
||||
| `manual` | Never auto-reply; surface a human handoff. |
|
||||
| `legacy` | A short, bounded media-request flow retained for existing behavior. |
|
||||
| `r4r` | Bounded rapport-style replies with autonomous safety checks. |
|
||||
| `r4r-media` | Rapport first, one bounded media ask, then a handoff/terminal boundary. |
|
||||
|
||||
The snapshot matters: changing configuration later must not silently rewrite the rules for an active conversation. Automation stops for seller/upsell evidence, an image received, a detected boundary, unsafe content, an exhausted turn budget, or a manual policy.
|
||||
|
||||
## Success and analytics
|
||||
|
||||
Do not collapse all progress into “messages sent.” Useful states include:
|
||||
|
||||
- candidates discovered, screened, ready, leased, retrying, rejected, expired, and sent;
|
||||
- sends awaiting a mature response window versus mature sends;
|
||||
- replies, declined/boundary/upsell outcomes, and images received;
|
||||
- model usage, fallback behavior, strategy/sample size, and account ownership.
|
||||
|
||||
Fetched listing size is not an eligibility count. Logs and UI should quantify rejection and deferral reasons so a dry funnel can be diagnosed without guessing.
|
||||
|
||||
## Operational authority
|
||||
|
||||
The following require explicit user authorization because they can affect real accounts or production state:
|
||||
|
||||
- starting or resuming bot processing;
|
||||
- sending a message or triggering a send-capable workflow;
|
||||
- importing/replacing account sessions;
|
||||
- restarting or deploying the production service;
|
||||
- changing live configuration;
|
||||
- deleting/resetting databases, volumes, media, or account state.
|
||||
|
||||
Read-only inspection, builds, tests, and documentation work do not grant any of those permissions. If a request is only diagnostic, inspect and explain before editing or operating.
|
||||
|
||||
## Privacy and secrets
|
||||
|
||||
Treat these as sensitive: `.env`, Reddit cookies, exported session JSON, Matrix access tokens, dashboard/agent tokens, the SQLite database, browser profiles, conversation text, usernames, private media, and logs that may contain request metadata.
|
||||
|
||||
Never commit or reproduce them in documentation, fixtures, logs, issue text, or tool output. Prefer redacted structural evidence. Production backups contain account and private conversation data and need the same protection as the live volume.
|
||||
+95
-781
@@ -1,781 +1,95 @@
|
||||
# Service Catalog
|
||||
|
||||
Detailed documentation for each service in the RedditBot codebase.
|
||||
|
||||
---
|
||||
|
||||
## Table of Contents
|
||||
|
||||
1. [MonitorService](#monitorservice)
|
||||
2. [ProcessorService](#processorservice)
|
||||
3. [StrategySelector](#strategypicker)
|
||||
4. [ResponseChecker](#responsechecker)
|
||||
5. [ChatService](#chatservice)
|
||||
6. [MediaService](#mediaservice)
|
||||
7. [MediaSyncService](#mediasyncservice)
|
||||
8. [AutoReplyService](#autoreplyservice)
|
||||
9. [RedditApi](#redditapi)
|
||||
10. [OpenRouterService](#openrouterservice)
|
||||
11. [BrowserClient](#browserclient)
|
||||
12. [MessagePage](#messagepage)
|
||||
13. [RateLimiter](#ratelimiter)
|
||||
|
||||
---
|
||||
|
||||
## MonitorService
|
||||
|
||||
**File**: `src/services/monitor.ts`
|
||||
**Lines**: 72
|
||||
**Responsibility**: Discover new image posts across configured subreddits
|
||||
|
||||
### Purpose
|
||||
|
||||
Polls configured subreddits at regular intervals, discovers new image posts, filters out already-processed posts and previously-messaged users, and returns the single newest valid post for processing.
|
||||
|
||||
### Public Methods
|
||||
|
||||
```typescript
|
||||
class MonitorService {
|
||||
constructor(
|
||||
private redditApi: RedditApi,
|
||||
private postsRepo: PostsRepository,
|
||||
private messagesRepo: MessagesRepository,
|
||||
private config: Config
|
||||
)
|
||||
|
||||
/**
|
||||
* Discover the next post to process.
|
||||
* Returns null if no valid posts are available.
|
||||
*/
|
||||
discoverNextPost(): Promise<Post | null>
|
||||
}
|
||||
```
|
||||
|
||||
### Algorithm
|
||||
|
||||
1. Shuffle subreddit order (randomize polling order each cycle)
|
||||
2. For each subreddit:
|
||||
- Fetch new posts via `RedditApi.fetchPosts()`
|
||||
- Filter: image posts only
|
||||
- Filter: author not deleted
|
||||
- Filter: post not already in `processed_posts`
|
||||
- Filter: author not previously messaged (or meets re-message criteria)
|
||||
3. Return the newest valid post across all subreddits
|
||||
4. If no valid posts found, return `null`
|
||||
|
||||
### Dependencies
|
||||
|
||||
- `RedditApi` - Fetch posts from Reddit
|
||||
- `PostsRepository` - Check for already-processed posts
|
||||
- `MessagesRepository` - Check for previously-messaged users
|
||||
- `Config` - Subreddit list, poll interval, target gender
|
||||
|
||||
### Error Handling
|
||||
|
||||
- Skips subreddits that return errors (continues to next)
|
||||
- Logs errors at `warn` level
|
||||
- Returns `null` on any unrecoverable error
|
||||
|
||||
---
|
||||
|
||||
## ProcessorService
|
||||
|
||||
**File**: `src/services/processor.ts`
|
||||
**Lines**: 132
|
||||
**Responsibility**: Core processing pipeline for discovered posts
|
||||
|
||||
### Purpose
|
||||
|
||||
Orchestrates the full processing pipeline for a single discovered post: image download, gender detection, AI analysis, strategy selection, message generation, and DM sending.
|
||||
|
||||
### Public Methods
|
||||
|
||||
```typescript
|
||||
class ProcessorService {
|
||||
constructor(
|
||||
private config: Config,
|
||||
private redditApi: RedditApi,
|
||||
private openrouter: OpenRouterService,
|
||||
private strategySelector: StrategySelector,
|
||||
private messageSender: MessagePage,
|
||||
private postsRepo: PostsRepository,
|
||||
private messagesRepo: MessagesRepository,
|
||||
private successfulRepo: SuccessfulMessagesRepository,
|
||||
private logger: Logger
|
||||
)
|
||||
|
||||
/**
|
||||
* Process a single discovered post through the full pipeline.
|
||||
*/
|
||||
processPost(post: Post): Promise<void>
|
||||
}
|
||||
```
|
||||
|
||||
### Pipeline Steps
|
||||
|
||||
```
|
||||
1. Download image
|
||||
└─► ImageService.downloadImage(url)
|
||||
└─► Validate MIME type (image/jpeg, image/png, image/webp)
|
||||
└─► Validate file size
|
||||
└─► Convert to base64
|
||||
|
||||
2. Quick gender detection (cost-saving pre-filter)
|
||||
└─► OpenRouter.detectGenderOnly(image)
|
||||
└─► Skip if gender doesn't match targetGender
|
||||
|
||||
3. Select A/B strategy variant
|
||||
└─► StrategySelector.selectStrategy(subreddit, author)
|
||||
|
||||
4. Full AI analysis
|
||||
└─► OpenRouter.analyzeImage(image)
|
||||
└─► Detect gender, describe image
|
||||
└─► Handle refusal (mark post, try next)
|
||||
|
||||
5. Generate message
|
||||
└─► OpenRouter.generateMessage(analysis, title, examples, strategy)
|
||||
└─► Few-shot examples from SuccessfulMessagesRepository
|
||||
|
||||
6. Send DM
|
||||
└─► MessagePage.sendDM(author, "Stunning", message)
|
||||
└─► Handle failure (log, don't mark as sent)
|
||||
|
||||
7. Record in database
|
||||
└─► Mark post as processed
|
||||
└─► Mark message_sent = 1 if successful
|
||||
└─► Store generated_message for reference
|
||||
```
|
||||
|
||||
### Error Handling
|
||||
|
||||
- AI refusal: Post is marked as processed with error, skipped
|
||||
- Image download failure: Post is marked with error, skipped
|
||||
- DM send failure: Post is marked as not sent, logged
|
||||
- All errors logged with context (post ID, subreddit, author)
|
||||
|
||||
### Cost Optimization
|
||||
|
||||
- Quick gender detection costs less than full analysis
|
||||
- Non-matching genders are skipped before expensive analysis
|
||||
- AI refusal detection prevents wasted tokens on rejected prompts
|
||||
|
||||
---
|
||||
|
||||
## StrategySelector
|
||||
|
||||
**File**: `src/services/strategy-selector.ts`
|
||||
**Lines**: 101
|
||||
**Responsibility**: A/B testing with explore/exploit algorithm
|
||||
|
||||
### Purpose
|
||||
|
||||
Selects message variants (strategies) for each post using an explore/exploit algorithm based on historical response rates. Optimizes for maximum response rate over time while continuing to explore under-tested variants.
|
||||
|
||||
### Public Methods
|
||||
|
||||
```typescript
|
||||
class StrategySelector {
|
||||
constructor(
|
||||
private config: Config,
|
||||
private messagesRepo: MessagesRepository
|
||||
)
|
||||
|
||||
/**
|
||||
* Select the best strategy variant for a given subreddit/author.
|
||||
*/
|
||||
selectStrategy(subreddit: string, author: string): string
|
||||
}
|
||||
```
|
||||
|
||||
### Algorithm
|
||||
|
||||
1. **Get strategy stats** from `MessagesRepository`:
|
||||
- Response count per variant
|
||||
- Sent count per variant
|
||||
- Response rate per variant
|
||||
|
||||
2. **Apply minimum sample threshold**:
|
||||
- If a variant has fewer than `minSamples` sends, treat it as untested
|
||||
- Untested variants are excluded from exploitation
|
||||
|
||||
3. **Explore vs Exploit decision**:
|
||||
- With probability `explorationRate`, select a random variant (explore)
|
||||
- Otherwise, select the variant with the highest response rate (exploit)
|
||||
|
||||
4. **Fallback**:
|
||||
- If no variants have enough data, select randomly from enabled variants
|
||||
|
||||
### Configuration
|
||||
|
||||
```yaml
|
||||
strategyVariants:
|
||||
- name: "variant_a"
|
||||
prompt: "Compliment-based message..."
|
||||
enabled: true
|
||||
- name: "variant_b"
|
||||
prompt: "Question-based message..."
|
||||
enabled: true
|
||||
```
|
||||
|
||||
### Dependencies
|
||||
|
||||
- `Config` - Strategy variants, exploration rate, min samples
|
||||
- `MessagesRepository` - Response rate statistics
|
||||
|
||||
---
|
||||
|
||||
## ResponseChecker
|
||||
|
||||
**File**: `src/services/response-checker.ts`
|
||||
**Lines**: 179
|
||||
**Responsibility**: Monitor inbox for user replies
|
||||
|
||||
### Purpose
|
||||
|
||||
Background service that navigates the Reddit inbox via Playwright, identifies responses to bot-sent messages, updates the database, and stores successful message/response pairs for few-shot learning.
|
||||
|
||||
### Public Methods
|
||||
|
||||
```typescript
|
||||
class ResponseCheckerService {
|
||||
constructor(
|
||||
private browser: BrowserClient,
|
||||
private messagesRepo: MessagesRepository,
|
||||
private successfulRepo: SuccessfulMessagesRepository,
|
||||
private logger: Logger
|
||||
)
|
||||
|
||||
/**
|
||||
* Check inbox for responses to sent messages.
|
||||
*/
|
||||
checkResponses(): Promise<void>
|
||||
}
|
||||
```
|
||||
|
||||
### Algorithm
|
||||
|
||||
1. Navigate to `/message/inbox` via Playwright
|
||||
2. Parse inbox items (messages, conversations)
|
||||
3. For each response item:
|
||||
- Extract recipient username and message text
|
||||
- Match to `sent_messages` by recipient + timestamp proximity
|
||||
- Update `got_response = 1` and `response_checked_at`
|
||||
- Store the message/response pair in `successful_messages`
|
||||
4. Update strategy statistics (response rates)
|
||||
|
||||
### Dependencies
|
||||
|
||||
- `BrowserClient` - Playwright browser context
|
||||
- `MessagesRepository` - Update response flags
|
||||
- `SuccessfulMessagesRepository` - Store learning pairs
|
||||
- `Logger` - Activity logging
|
||||
|
||||
---
|
||||
|
||||
## ChatService
|
||||
|
||||
**File**: `src/services/chat.ts`
|
||||
**Lines**: 1123+
|
||||
**Responsibility**: Reddit chat scraping, messaging, and conversation management
|
||||
|
||||
### Purpose
|
||||
|
||||
The most complex service in the codebase. Manages Reddit chat interactions including scraping active conversations, loading conversation history, sending messages with human-like delays, detecting upsell/spam, and managing conversation caches.
|
||||
|
||||
### Public Methods
|
||||
|
||||
```typescript
|
||||
class ChatService {
|
||||
constructor(
|
||||
private browser: BrowserClient,
|
||||
private config: Config,
|
||||
private logger: Logger
|
||||
)
|
||||
|
||||
/** Fetch active conversations from Reddit */
|
||||
fetchChats(): Promise<ChatSummary[]>
|
||||
|
||||
/** Load full conversation history */
|
||||
loadConversation(chatId: string): Promise<Conversation>
|
||||
|
||||
/** Send a message in a conversation */
|
||||
sendMessage(chatId: string, text: string): Promise<boolean>
|
||||
|
||||
/** Get cached conversations */
|
||||
getCachedConversations(): Conversation[]
|
||||
|
||||
/** Check for new messages in conversations */
|
||||
checkForUpdates(): Promise<Conversation[]>
|
||||
|
||||
/** Detect upsell/spam in a message */
|
||||
detectUpsell(message: string): boolean
|
||||
}
|
||||
```
|
||||
|
||||
### Shadow DOM Handling
|
||||
|
||||
Reddit uses custom `faceplate-*` web components with shadow DOM. Standard CSS selectors cannot pierce these boundaries. The service uses JavaScript injection via `page.evaluate()`:
|
||||
|
||||
```javascript
|
||||
// Traverse shadow roots to find chat elements
|
||||
function traverseShadowRoots(root) {
|
||||
const elements = root.querySelectorAll('.some-class');
|
||||
const shadowChildren = root.querySelectorAll('*::shadow, ::deep');
|
||||
// ... recursive traversal
|
||||
}
|
||||
```
|
||||
|
||||
### Conversation Cache
|
||||
|
||||
Conversations are cached in two layers:
|
||||
|
||||
1. **In-memory**: Fast access for active conversations
|
||||
2. **Disk JSON**: Persistence across restarts at `data/chats/`
|
||||
|
||||
Cache structure:
|
||||
```json
|
||||
{
|
||||
"chatId": "abc123",
|
||||
"participants": ["user1", "bot"],
|
||||
"messages": [
|
||||
{ "author": "user1", "text": "Hey!", "timestamp": 1234567890 }
|
||||
],
|
||||
"lastUpdated": 1234567890,
|
||||
"hasNewMessages": false
|
||||
}
|
||||
```
|
||||
|
||||
### Human-Like Messaging
|
||||
|
||||
Messages are sent with:
|
||||
- Typing delay simulation (random delay before "typing" starts)
|
||||
- Variable typing speed
|
||||
- Random delays between messages
|
||||
- Delivery verification
|
||||
|
||||
### Known Issues
|
||||
|
||||
- **Hardcoded username**: `myUsername = 'Upset_Marsupial1006'` is hardcoded in JS evaluation scripts. Should be parameterized from `Config`.
|
||||
- **Size**: 1123+ lines suggests this should be split into smaller services.
|
||||
|
||||
### Dependencies
|
||||
|
||||
- `BrowserClient` - Playwright browser context
|
||||
- `Config` - Rate limits, username (should be), check intervals
|
||||
- `Logger` - Activity logging
|
||||
|
||||
---
|
||||
|
||||
## MediaService
|
||||
|
||||
**File**: `src/services/media.ts`
|
||||
**Lines**: 254
|
||||
**Responsibility**: Media download, caching, and indexing
|
||||
|
||||
### Purpose
|
||||
|
||||
Downloads images from chat conversations, resolves Matrix blurred thumbnails to full resolution, stores files locally, and maintains a deduplicated index.
|
||||
|
||||
### Public Methods
|
||||
|
||||
```typescript
|
||||
class MediaService {
|
||||
constructor(
|
||||
private config: Config,
|
||||
private browser: BrowserClient,
|
||||
private logger: Logger
|
||||
)
|
||||
|
||||
/** Download media from a conversation */
|
||||
downloadMedia(chatId: string): Promise<string[]>
|
||||
|
||||
/** Get media index */
|
||||
getIndex(): MediaIndex
|
||||
|
||||
/** Get a specific media file path */
|
||||
getPath(hash: string): string | null
|
||||
}
|
||||
```
|
||||
|
||||
### Deduplication
|
||||
|
||||
Files are deduplicated using SHA-1 hashing:
|
||||
|
||||
```
|
||||
SHA-1(image_bytes) → data/media/{hash}.{ext}
|
||||
```
|
||||
|
||||
The index maps hashes to file paths:
|
||||
```json
|
||||
{
|
||||
"abc123def456": "abc123def456.jpg",
|
||||
"789xyz012abc": "789xyz012abc.png"
|
||||
}
|
||||
```
|
||||
|
||||
### Matrix Thumbnail Resolution
|
||||
|
||||
Reddit's Matrix system uses blurred placeholder thumbnails. The service resolves these to full resolution by:
|
||||
|
||||
1. Detecting Matrix thumbnail URLs
|
||||
2. Extracting the full-resolution URL from the response
|
||||
3. Downloading the full-resolution image
|
||||
|
||||
### Dependencies
|
||||
|
||||
- `Config` - Media directory path
|
||||
- `BrowserClient` - Playwright for Matrix resolution
|
||||
- `Logger` - Download activity
|
||||
|
||||
---
|
||||
|
||||
## MediaSyncService
|
||||
|
||||
**File**: `src/services/media-sync.ts`
|
||||
**Lines**: 119
|
||||
**Responsibility**: Periodic background media synchronization
|
||||
|
||||
### Purpose
|
||||
|
||||
Runs periodic background syncs to download media from chat conversations. Supports both incremental (top 10 chats) and full (all conversations) sync modes.
|
||||
|
||||
### Public Methods
|
||||
|
||||
```typescript
|
||||
class MediaSyncService {
|
||||
constructor(
|
||||
private config: Config,
|
||||
private mediaService: MediaService,
|
||||
private chatService: ChatService,
|
||||
private logger: Logger
|
||||
)
|
||||
|
||||
/** Start periodic sync */
|
||||
start(): void
|
||||
|
||||
/** Stop periodic sync */
|
||||
stop(): void
|
||||
|
||||
/** Trigger incremental sync */
|
||||
syncIncremental(): Promise<void>
|
||||
|
||||
/** Trigger full sync */
|
||||
syncFull(): Promise<void>
|
||||
|
||||
/** Get sync status */
|
||||
getStatus(): SyncStatus
|
||||
}
|
||||
```
|
||||
|
||||
### Sync Modes
|
||||
|
||||
| Mode | Description | Use Case |
|
||||
|---|---|---|
|
||||
| Incremental | Top 10 most recent conversations | Regular background sync |
|
||||
| Full | All conversations | Initial setup, catch-up |
|
||||
|
||||
### Dependencies
|
||||
|
||||
- `Config` - Sync interval, enabled flag
|
||||
- `MediaService` - Actual download logic
|
||||
- `ChatService` - Conversation list
|
||||
- `Logger` - Sync progress
|
||||
|
||||
---
|
||||
|
||||
## AutoReplyService
|
||||
|
||||
**File**: `src/services/auto-reply.ts`
|
||||
**Lines**: 199
|
||||
**Responsibility**: Automated reply to chat conversations
|
||||
|
||||
### Purpose
|
||||
|
||||
Monitors cached conversations for unread messages, detects upsell/spam, generates AI reply suggestions, and either auto-sends or marks as pending for manual approval.
|
||||
|
||||
### Public Methods
|
||||
|
||||
```typescript
|
||||
class AutoReplyService {
|
||||
constructor(
|
||||
private config: Config,
|
||||
private chatService: ChatService,
|
||||
private openrouter: OpenRouterService,
|
||||
private logger: Logger
|
||||
)
|
||||
|
||||
/** Start auto-reply checking */
|
||||
start(): void
|
||||
|
||||
/** Stop auto-reply checking */
|
||||
stop(): void
|
||||
|
||||
/** Check for unread messages and generate replies */
|
||||
checkAndReply(): Promise<void>
|
||||
}
|
||||
```
|
||||
|
||||
### Workflow
|
||||
|
||||
```
|
||||
1. Check cached conversations for unread messages
|
||||
│
|
||||
├─► Skip if no new messages
|
||||
│
|
||||
├─► Detect upsell/spam keywords
|
||||
│ └─► Flag conversation, skip reply
|
||||
│
|
||||
├─► Generate AI reply suggestion
|
||||
│ └─► OpenRouter.generateReply(conversation history)
|
||||
│
|
||||
├─► If autoSend: send reply immediately
|
||||
│ └─► ChatService.sendMessage()
|
||||
│
|
||||
└─► If !autoSend: mark as pending
|
||||
└─► Dashboard shows pending replies for approval
|
||||
```
|
||||
|
||||
### Dependencies
|
||||
|
||||
- `Config` - Enabled flag, autoSend flag, check interval
|
||||
- `ChatService` - Conversation access, message sending
|
||||
- `OpenRouterService` - Reply generation
|
||||
- `Logger` - Activity logging
|
||||
|
||||
---
|
||||
|
||||
## RedditApi
|
||||
|
||||
**File**: `src/reddit/api.ts`
|
||||
**Lines**: 233
|
||||
**Responsibility**: Reddit API client
|
||||
|
||||
### Purpose
|
||||
|
||||
Axios-based client for interacting with Reddit's public API. Handles post fetching, user profile checking, and authenticated requests using Playwright browser cookies.
|
||||
|
||||
### Public Methods
|
||||
|
||||
```typescript
|
||||
class RedditApi {
|
||||
constructor(
|
||||
private browser: BrowserClient,
|
||||
private logger: Logger
|
||||
)
|
||||
|
||||
/** Fetch new posts from a subreddit */
|
||||
fetchPosts(subreddit: string, limit?: number): Promise<Post[]>
|
||||
|
||||
/** Check if a user is a seller/spam */
|
||||
checkUserProfile(username: string): Promise<UserProfile>
|
||||
|
||||
/** Make authenticated request using browser cookies */
|
||||
authenticatedRequest(url: string): Promise<any>
|
||||
}
|
||||
```
|
||||
|
||||
### Image URL Resolution
|
||||
|
||||
Fallback chain for extracting image URLs from post data:
|
||||
|
||||
```
|
||||
1. post.preview.reddits[0].url (Reddit CDN preview)
|
||||
2. post.url (i.redd.it direct)
|
||||
3. post.thumbnail (if not self/spam)
|
||||
4. Scan post.selftext for imgur/image links
|
||||
```
|
||||
|
||||
### Seller/Spam Detection
|
||||
|
||||
Checks user profile for keywords:
|
||||
- "seller"
|
||||
- "onlyfans"
|
||||
- "link in bio"
|
||||
- Other configured `sellerKeywords`
|
||||
|
||||
### Dependencies
|
||||
|
||||
- `BrowserClient` - Browser cookies for authentication
|
||||
- `Logger` - Request logging
|
||||
|
||||
---
|
||||
|
||||
## OpenRouterService
|
||||
|
||||
**File**: `src/ai/openrouter.ts`
|
||||
**Lines**: 307
|
||||
**Responsibility**: OpenRouter AI client
|
||||
|
||||
### Purpose
|
||||
|
||||
Client for the OpenRouter API (OpenAI-compatible SDK). Provides image analysis, message generation, reply suggestions, and upsell detection.
|
||||
|
||||
### Public Methods
|
||||
|
||||
```typescript
|
||||
class OpenRouterService {
|
||||
constructor(private config: Config, private logger: Logger)
|
||||
|
||||
/** Full image analysis with model fallback */
|
||||
analyzeImage(imageBase64: string): Promise<ImageAnalysis>
|
||||
|
||||
/** Quick gender detection (cost-saving) */
|
||||
detectGenderOnly(imageBase64: string): Promise<string>
|
||||
|
||||
/** Generate DM message */
|
||||
generateMessage(
|
||||
imageAnalysis: ImageAnalysis,
|
||||
postTitle: string,
|
||||
strategy: string
|
||||
): Promise<string>
|
||||
|
||||
/** Generate chat reply */
|
||||
generateReply(conversationHistory: Message[]): Promise<string>
|
||||
|
||||
/** Detect upsell/spam keywords */
|
||||
detectUpsell(message: string): boolean
|
||||
}
|
||||
```
|
||||
|
||||
### Model Fallback Chain
|
||||
|
||||
```
|
||||
Primary model (config.openrouter.model)
|
||||
│
|
||||
└─► If refusal detected:
|
||||
│
|
||||
└─► Try each fallback model (config.openrouter.fallbackModels)
|
||||
│
|
||||
└─► If all fail: throw error
|
||||
```
|
||||
|
||||
### Refusal Detection
|
||||
|
||||
Heuristic patterns to detect when AI models refuse to process prompts:
|
||||
|
||||
```typescript
|
||||
const refusalPatterns = [
|
||||
/cannot.*analyze/i,
|
||||
/unable.*process/i,
|
||||
/against.*policy/i,
|
||||
/cannot.*view/i,
|
||||
// ...
|
||||
];
|
||||
```
|
||||
|
||||
### Few-Shot Learning
|
||||
|
||||
`generateMessage()` retrieves successful message/response pairs from the database and includes them as examples in the prompt:
|
||||
|
||||
```
|
||||
"Here are examples of messages that got responses:
|
||||
- Example 1: {original_message} → {response_snippet}
|
||||
- Example 2: ...
|
||||
|
||||
Generate a similar message for this image..."
|
||||
```
|
||||
|
||||
### Dependencies
|
||||
|
||||
- `Config` - API key, model selection, prompts
|
||||
- `MessagesRepository` - Successful examples for few-shot learning
|
||||
|
||||
---
|
||||
|
||||
## BrowserClient
|
||||
|
||||
**File**: `src/browser/client.ts`
|
||||
**Lines**: 302
|
||||
**Responsibility**: Playwright persistent context manager
|
||||
|
||||
### Purpose
|
||||
|
||||
Manages the headless Playwright context used after an extension dump is imported. It accepts Reddit cookies, verifies account identity, and supplies authenticated context to the remaining browser-backed operations. It never performs login.
|
||||
|
||||
### Public Methods
|
||||
|
||||
```typescript
|
||||
class BrowserClient {
|
||||
constructor(private config: Config, private logger: Logger)
|
||||
|
||||
/** Initialize browser context */
|
||||
initialize(): Promise<void>
|
||||
|
||||
/** Verify browser session is active */
|
||||
verifySession(): Promise<boolean>
|
||||
|
||||
/** Import Reddit cookies from a validated extension dump */
|
||||
importRedditCookies(cookies: Cookie[]): Promise<void>
|
||||
|
||||
/** Get page context for operations */
|
||||
getPage(): Promise<Page>
|
||||
|
||||
/** Close browser */
|
||||
close(): Promise<void>
|
||||
}
|
||||
```
|
||||
|
||||
### Import Flow
|
||||
|
||||
The dashboard validates a combined extension dump, verifies Matrix with `whoami`, imports Reddit cookies, verifies Reddit identity, and only then marks the shared account gate authenticated.
|
||||
|
||||
### Session Persistence
|
||||
|
||||
- Cookies and storage are saved to `config.browser.userDataDir`
|
||||
- Subsequent launches restore the session automatically
|
||||
- Session expiry pauses account work until a fresh extension dump is imported
|
||||
|
||||
### Dependencies
|
||||
|
||||
- `Config` - Persistent session directory
|
||||
- `Logger` - Import verification progress and errors
|
||||
|
||||
---
|
||||
|
||||
## RateLimiter
|
||||
|
||||
**File**: `src/utils/rate-limiter.ts`
|
||||
**Lines**: 114
|
||||
**Responsibility**: Token-bucket rate limiting
|
||||
|
||||
### Purpose
|
||||
|
||||
Enforces rate limits on bot actions (primarily message sending) to avoid detection and bans. Uses a combination of minimum delay, random jitter, and hourly caps.
|
||||
|
||||
### Public Methods
|
||||
|
||||
```typescript
|
||||
class RateLimiter {
|
||||
constructor(config: RateLimitConfig)
|
||||
|
||||
/** Wait until a slot is available */
|
||||
waitForSlot(): Promise<void>
|
||||
|
||||
/** Get remaining time until next slot */
|
||||
getTimeUntilSlot(): number
|
||||
|
||||
/** Get current hourly count */
|
||||
getHourlyCount(): number
|
||||
|
||||
/** Reset hourly counter */
|
||||
resetHourlyCounter(): void
|
||||
}
|
||||
```
|
||||
|
||||
### Rate Limiting Strategy
|
||||
|
||||
```
|
||||
1. Check hourly cap:
|
||||
└─► If maxMessagesPerHour reached: wait until hour rolls over
|
||||
|
||||
2. Enforce minimum delay:
|
||||
└─► lastActionTime + minDelayMs + random(0, jitterMs)
|
||||
|
||||
3. Track action timestamp
|
||||
4. Increment hourly counter
|
||||
```
|
||||
|
||||
### Dependencies
|
||||
|
||||
- `Config` - Rate limit settings
|
||||
- `p-queue` - Concurrency control
|
||||
# Service and module map
|
||||
|
||||
This is an ownership map, not an API dump. Read the owning implementation and tests before changing behavior.
|
||||
|
||||
## Lifecycle and account ownership
|
||||
|
||||
| Module | Owns | Important constraints |
|
||||
| --- | --- | --- |
|
||||
| `src/bootstrap.ts` | Dependency composition and startup wiring | Keep business policy in services; preserve initialization order and test seams. |
|
||||
| `src/main-loop.ts` | Shared discovery/screening/dispatch workers and legacy loop | Loops are abortable; queue transitions and shutdown must remain recoverable. |
|
||||
| `services/reddit-account-runtime.ts` | Immutable account runtime/factory/manager lifecycle | Same-account operations serialize; session refresh replaces one runtime. |
|
||||
| `services/reddit-account-runtime-builder.ts` | Concrete account dependency graph and account loop | Account-specific browser, Matrix, repos, limiter, automation, chat, and media live here. |
|
||||
| `services/reddit-accounts.ts` | Account import/activation orchestration | Verification must bind Reddit cookies and Matrix identity before activation. |
|
||||
| `services/account-auth-context.ts` | Current verified identity snapshot | Account work pauses unless both sides authenticate. |
|
||||
| `services/session-bound-runtime.ts` | Legacy auth-bound background lifecycle | Prevents background work while logged out. |
|
||||
|
||||
## Discovery, preparation, and delivery
|
||||
|
||||
| Module | Owns | Important constraints |
|
||||
| --- | --- | --- |
|
||||
| `services/monitor.ts` | Subreddit ordering, candidate fetching, age/gender/seller screening, rejection summaries | Cheap checks first; preserve exact rejection/deferred counts; listing 404 disables the subreddit. |
|
||||
| `reddit/api.ts` | Reddit JSON calls, post parsing, seller/profile evidence, username resolution, error types | All reads must use the coordinator; redact error metadata. |
|
||||
| `reddit/request-coordinator.ts` | Process-wide Reddit request spacing and cooldown | Space failed requests too; 429 cooldown is global. |
|
||||
| `services/processor.ts` | Source-specific AI preparation and outbound safety | Returns prepared or explicit skip; performs no remote chat send. |
|
||||
| `services/strategy-selector.ts` | Weighted explore/exploit selection and experiment metadata | Separate image/chat strategy sections; respect mature response window and sample floor. |
|
||||
| `services/recipient-deduplicator.ts` | Global recipient history plus live reservation | Case-insensitive; release failures, commit successes, recover stale claims. |
|
||||
| `utils/rate-limiter.ts` | Per-account send spacing/hour limits/jitter | Increment only for confirmed sends; reply priority may use the same limiter. |
|
||||
| `database/candidate-queue-repository.ts` | Durable candidate state machine, leases, retries, counts | Mutations require the expected worker lease; recover expired/interrupted work. |
|
||||
|
||||
## Conversation, chat, and media
|
||||
|
||||
| Module | Owns | Important constraints |
|
||||
| --- | --- | --- |
|
||||
| `services/chat/service.ts` | Matrix conversation sync, cache updates, starts/replies, scoped cooldowns | Room-creation limits pause new outreach without blocking existing-room replies; Matrix acceptance and UI visibility are distinct. |
|
||||
| `services/chat/matrix/client.ts` | Matrix HTTP protocol and error classification | Authentication/rate-limit errors need typed handling. |
|
||||
| `services/chat/cache.ts` | Account-scoped cached conversations and sync state | SQLite is durable; do not share caches across accounts. |
|
||||
| `services/chat/matrix/username-resolver.ts` | Matrix/Reddit identity mapping | Preserve unresolved internal IDs without leaking them in operator text. |
|
||||
| `services/chat/matrix/token-refresher.ts` | Explicit refresh-unavailable behavior | New credentials come from verified extension dumps, not token sniffing. |
|
||||
| `services/conversation-automation.ts` | Reply-first policy, claims, retries, handoffs, turn budgets | Snapshot policy; claim before send; reconcile pending claims; fail closed. |
|
||||
| `services/conversation-state.ts` | Pure conversation parsing/state/safety helpers | Conversation text is untrusted data; keep prompt boundaries explicit. |
|
||||
| `services/outbound-message-safety.ts` | Final no-proximity/no-meeting rule | This is defense in depth beyond prompts. |
|
||||
| `services/conversation-outcome*.ts` | Response outcome classification/recording | Keep analytics attribution account/message scoped. |
|
||||
| `services/response-checker.ts` | Inbox/performance response checks and successful examples | Non-overlapping; do not confuse chat sync with outcome maturity. |
|
||||
| `services/media.ts` | Download, deduplicate, index, and list private media | Preserve account ownership and authenticated serving. |
|
||||
| `services/media-sync.ts` | Periodic authenticated media sync | Stop while auth is unavailable and on runtime shutdown. |
|
||||
|
||||
## Persistence
|
||||
|
||||
| Module | Owns |
|
||||
| --- | --- |
|
||||
| `database/client.ts` | Database open settings and ordered migrations. |
|
||||
| `database/repositories.ts` | Posts, messages, successes, activity, outbox, states, claims, outcomes, users, recipient history. |
|
||||
| `database/account-repositories.ts` | Verified account records, conversations, sync cursors, and media content. |
|
||||
| `database/ai-usage-repository.ts` | Model request/cost/reliability records. |
|
||||
| `database/legacy-content-import.ts` | One-time ownership of legacy file-backed chat/media data. |
|
||||
| `config/persistence.ts` | Secret-free application configuration in SQLite. |
|
||||
|
||||
Repositories should expose domain operations rather than raw SQL to callers. Multi-step state transitions belong in SQLite transactions when partial completion would break invariants.
|
||||
|
||||
## Web and agent surfaces
|
||||
|
||||
| Module | Owns |
|
||||
| --- | --- |
|
||||
| `web/server.ts` | Express lifecycle, middleware order, route registration, static dashboard/media. |
|
||||
| `web/auth.ts` | Dashboard session and bearer-token authorization. |
|
||||
| `web/context.ts` | Narrow route dependency container. |
|
||||
| `web/routes/*.ts` | Focused endpoint families. |
|
||||
| `web/api-catalog.ts` | Operation metadata, OpenAPI, and capability discovery. |
|
||||
| `web/agent-job-manager.ts` | Bounded async sync jobs and job status. |
|
||||
| `web/events/*` | Typed event publication, replay, subscriptions, and Express SSE. |
|
||||
| `web/log-broadcaster.ts` | Recent logs plus legacy/typed stream adapters. |
|
||||
| `mcp/api-client.ts`, `mcp/server.ts` | HTTP-backed stdio MCP tools. |
|
||||
|
||||
Authorization is semantic, not just authentication: read and control agent tokens intentionally cannot import sessions, send messages, activate accounts, mutate legacy config, or use debug routes.
|
||||
|
||||
## Frontend
|
||||
|
||||
| Area | Owns |
|
||||
| --- | --- |
|
||||
| `apps/dashboard/src/app/` | Shell, routing, providers, auth status, notifications, realtime repair. |
|
||||
| `apps/dashboard/src/api/` | HTTP client, schema parsing, query keys/hooks. |
|
||||
| `apps/dashboard/src/features/` | Route-level feature slices. |
|
||||
| `apps/dashboard/src/components/` | Reusable domain-aware UI elements. |
|
||||
| `packages/api-contracts/` | Shared strict transport schemas and inferred types. |
|
||||
|
||||
## Utility boundaries
|
||||
|
||||
- `utils/logger.ts`: Winston creation, child context, and attachable sinks. No production `console.log`.
|
||||
- `utils/retry.ts`: bounded retry behavior for operations that are safe to repeat.
|
||||
- `utils/image.ts`: download/MIME/size helpers.
|
||||
- `utils/gender-tag.ts`: deterministic self-declared tag parsing.
|
||||
- `utils/fetch-with-timeout.ts`: abortable HTTP timeout wrapper.
|
||||
- `utils/json-safe.ts`: safe serialization for log/event data.
|
||||
|
||||
When a module starts owning two independent reasons to change, extract a narrow collaborator and inject it. Do not solve size alone by moving methods into a generic utility file.
|
||||
Reference in new issue
Block a user