jjbrunton/RedditBot
1
0 Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
Files
65a8fcb16329017a2d5de66584304ce5d08cc8b9
RedditBot/docs/architecture.md
T
jjbrunton 633bb0ec04 Restructure chat service, web routes, and add test suite 
Split monolithic chat service into modular structure with Matrix client
support, extract web server into per-resource route modules, introduce
bootstrap/main-loop separation, and add comprehensive vitest coverage
across services, repositories, and utilities. Adds project docs and
CLAUDE.md agent instructions.
2026-05-21 01:32:10 +01:00

899 lines
32 KiB
Markdown
Raw Blame History

# RedditBot - Software Architecture Documentation
## Table of Contents
1. [Project Overview](#project-overview)
2. [System Architecture](#system-architecture)
3. [Directory Structure](#directory-structure)
4. [Component Reference](#component-reference)
5. [Data Flow](#data-flow)
6. [Database Schema](#database-schema)
7. [API Reference](#api-reference)
8. [Configuration](#configuration)
9. [Design Patterns](#design-patterns)
10. [External Integrations](#external-integrations)
11. [Deployment](#deployment)
12. [Technical Debt & Recommendations](#technical-debt--recommendations)
---
## Project Overview
**RedditBot** is an automated Reddit monitoring and messaging system that:
- Monitors 30+ NSFW subreddits for image posts
- Uses AI (OpenRouter) to analyze images and detect gender
- Generates personalized direct messages to post authors
- Sends DMs via browser automation (Playwright)
- Monitors for responses and tracks success rates
- Implements A/B testing with explore/exploit strategy selection
- Learns from successful message/response pairs (few-shot learning)
- Provides a real-time web dashboard for monitoring and control
### Key Capabilities
| Capability | Description |
|---|---|
| Post Discovery | Polls subreddits for new image posts, filters by gender and spam profiles |
| AI Analysis | Multi-model image analysis with fallback chain via OpenRouter |
| Message Generation | AI-crafted DMs using successful examples as few-shot prompts |
| A/B Testing | Explore/exploit strategy selector optimizes response rates over time |
| Response Monitoring | Background service checks inbox for user replies |
| Chat Automation | Reddit chat scraping via shadow DOM traversal, auto-reply support |
| Media Management | Download, cache, deduplicate, and sync media from conversations |
| Web Dashboard | Real-time monitoring with SSE log streaming, analytics, and controls |
---
## System Architecture
### High-Level Architecture
```
┌─────────────────────────────────────────────────────────────────────┐
│ main (index.ts) │
│ Orchestrator / Entry Point │
└────────────────────────┬────────────────────────────────────────────┘
│
┌────────────────┼────────────────┐
│ │ │
▼ ▼ ▼
┌───────────────┐ ┌──────────────┐ ┌──────────────┐
│ Monitor │ │ Processor │ │ Response │
│ Service │ │ │ │ Checker │
│ │ │ 1. Download │ │ │
│ Poll subreddits│ │ 2. Validate │ │ Check inbox │
│ for new posts │ │ 3. Gender │ │ for replies │
└───────┬───────┘ │ Filter │ └──────────────┘
│ │ 4. AI Analysis│ │
│ │ 5. Strategy │ │
│ │ Selection │ │
│ │ 6. Message │ │
│ │ Generation │ │
└────┬────┘ └─────────────┘ │
│ │
▼ ▼
┌───────────────────────┐ ┌───────────────────────┐
│ Strategy Selector │ │ Chat Service │
│ │ │ │
│ A/B Testing │ │ Scrapes chat (shadow │
│ Explore/Exploit │ │ DOM traversal) │
│ Variant Selection │ │ Auto-reply, upsell │
└───────────────────────┘ │ detection │
└───────────────────────┘
│ │
▼ ▼
┌───────────────────────┐ ┌───────────────────────┐
│ Message Sender │ │ Media Service │
│ (Browser) │ │ │
│ │ │ Download, cache, │
│ Playwright DM compose │ │ deduplicate (SHA-1) │
│ Shadow DOM handling │ │ Matrix thumbnail │
└───────────────────────┘ │ resolution │
└───────────────────────┘
│ │
▼ ▼
┌─────────────────────────────────────────────────────────┐
│ SQLite Database │
│ │
│ processed_posts │ sent_messages │ successful_messages │
└─────────────────────────────────────────────────────────┘
│
▼
┌─────────────────────────────────────────────────────────┐
│ Web Dashboard (Express) │
│ │
│ 25+ API endpoints │ SSE log streaming │ Analytics │
│ Runtime config │ Bot controls │ Media gallery │
└─────────────────────────────────────────────────────────┘
```
### Service Communication
```
main.ts (Orchestrator)
│
├──► Config (Zod-validated YAML + env vars)
│
├──► Database (Better-SQLite3)
│ │
│ └──► PostsRepository
│ └──► MessagesRepository
│ └──► SuccessfulMessagesRepository
│
├──► MonitorService ──► RedditApi ──► Playwright (cookies)
│
├──► ProcessorService
│ ├──► Image download (axios)
│ ├──► Gender detection (OpenRouter)
│ ├──► Full analysis (OpenRouter)
│ └──► StrategySelector ──► MessagesRepository
│
├──► MessageSender (Browser - Playwright)
│
├──► ResponseChecker ──► Playwright (inbox navigation)
│
├──► ChatService ──► Playwright (chat.reddit.com shadow DOM)
│
├──► MediaService ──► Playwright (media download)
│
├──► MediaSyncService (background periodic sync)
│
├──► AutoReplyService ──► OpenRouter (reply generation)
│
├──► RateLimiter (p-queue, jitter)
│
└──► WebServer (Express, SSE)
```
---
## Directory Structure
```
RedditBot/
├── .dockerignore # Docker ignore rules
├── .gitignore # Git ignore rules
├── agents.md # Engineering standards & agent guidelines
├── config.yaml # Main YAML configuration file
├── docker-compose.yml # Docker Compose setup
├── DOKPLOY_DEPLOY.md # Dokploy deployment guide
├── Dockerfile # Container build definition
├── package.json # Node.js dependencies & scripts
├── tsconfig.json # TypeScript configuration
├── public/
│ └── index.html # Single-file web dashboard (HTML+CSS+JS)
├── scripts/
│ ├── inspect-chat-dom.ts # Debug: inspect chat DOM structure
│ └── login.ts # Manual browser login helper
└── src/
├── index.ts # Main entry point / orchestrator (391 lines)
├── config/
│ └── index.ts # Config loader + Zod validation schema (157 lines)
├── database/
│ ├── client.ts # SQLite setup + migration runner (121 lines)
│ └── repositories.ts # Repository pattern: Posts, Messages, Success (433 lines)
├── reddit/
│ └── api.ts # Reddit API client (axios + Playwright cookies) (233 lines)
├── ai/
│ └── openrouter.ts # OpenRouter AI client (image analysis, message gen) (307 lines)
├── browser/
│ ├── client.ts # Playwright persistent context + login (302 lines)
│ ├── message-page.ts # DM compose page automation + shadow DOM (560 lines)
│ └── media-downloader.ts # Playwright-based media downloader (29 lines)
├── services/
│ ├── monitor.ts # Post discovery service (72 lines)
│ ├── processor.ts # Image download + AI + message pipeline (132 lines)
│ ├── response-checker.ts # Inbox response checker (179 lines)
│ ├── strategy-selector.ts # A/B testing explore/exploit (101 lines)
│ ├── chat.ts # Chat scraping + messaging + auto-reply (1123+ lines)
│ ├── media.ts # Media download, caching, indexing (254 lines)
│ ├── media-sync.ts # Periodic background media sync (119 lines)
│ └── auto-reply.ts # Auto-reply to chat conversations (199 lines)
├── utils/
│ ├── logger.ts # Winston structured logging (32 lines)
│ ├── rate-limiter.ts # Token-bucket rate limiting (114 lines)
│ ├── retry.ts # Exponential backoff wrapper (28 lines)
│ └── image.ts # Image download + base64 + MIME validation (34 lines)
└── web/
└── server.ts # Express dashboard + SSE streaming (794 lines)
```
---
## Component Reference
### Core Services
#### MonitorService (`src/services/monitor.ts`)
Discovers new image posts across configured subreddits.
- Shuffles subreddit order each cycle
- Fetches new posts via `RedditApi`
- Filters out already-processed posts and previously-messaged users
- Returns the single newest valid post per cycle
#### ProcessorService (`src/services/processor.ts`)
Core processing pipeline for each discovered post.
1. Downloads image via `ImageService`
2. Validates MIME type and file size
3. Runs quick gender detection (cost-saving pre-filter)
4. Selects A/B strategy variant via `StrategySelector`
5. Runs full AI analysis + message generation via `OpenRouterService`
6. Handles AI refusals gracefully (marks post, tries next)
#### StrategySelector (`src/services/strategy-selector.ts`)
Implements explore/exploit A/B testing for message variants.
- Tracks response rates per strategy variant
- Uses minimum sample thresholds before trusting statistics
- Applies random exploration rate for continued discovery
- Selects best variant based on weighted response rates
#### ResponseChecker (`src/services/response-checker.ts`)
Background service that monitors for user replies.
- Navigates Reddit inbox via Playwright
- Identifies responses to sent messages
- Updates `got_response` flag in database
- Stores successful message/response pairs for few-shot learning
#### ChatService (`src/services/chat.ts`)
Most complex service - manages Reddit chat interactions.
- Scrapes `chat.reddit.com` using JavaScript injection (shadow DOM traversal)
- Manages conversation cache (in-memory + disk JSON)
- Loads full conversation history with pagination
- Sends messages with human-like typing delays
- Verifies message delivery
- Detects upsell/spam keywords in chat
- Parameter: `myUsername` is hardcoded (should be from config)
#### MediaService (`src/services/media.ts`)
Media download, caching, and indexing.
- Downloads images from chat conversations
- Resolves Matrix blurred thumbnails to full resolution
- Stores files in `data/media/` directory
- Maintains JSON index with SHA-1 deduplication
- Tracks download status per conversation
#### MediaSyncService (`src/services/media-sync.ts`)
Periodic background media synchronization.
- Runs incremental syncs (top 10 chats) or full syncs (all conversations)
- Configurable interval via `mediaSyncInterval`
- Tracks sync status in memory
- Exposed via web dashboard endpoints
#### AutoReplyService (`src/services/auto-reply.ts`)
Automated reply to Reddit chat conversations.
- Checks for unread messages in cached conversations
- Detects upsell/spam keywords
- Generates AI reply suggestions via `OpenRouterService`
- Sends replies (auto or pending for manual approval)
- Event-based architecture for dashboard integration
### External Integrations
#### RedditApi (`src/reddit/api.ts`)
Axios-based Reddit API client.
- Fetches posts from subreddits (`/r/{name}/new.json`)
- Validates posts (image-only, skips deleted authors)
- Extracts image URLs with fallback chain: `preview` > `i.redd.it` > `imgur` > `thumbnail`
- Checks user profiles against seller/spam keywords
- Uses Playwright browser cookies for authenticated requests
#### OpenRouterService (`src/ai/openrouter.ts`)
OpenRouter API client using OpenAI-compatible SDK.
- `analyzeImage()`: Multi-model image analysis with fallback chain
- `detectGenderOnly()`: Quick cost-saving gender detection
- `generateMessage()`: DM text with few-shot examples from DB
- `generateReply()`: AI chat reply from conversation history
- `detectUpsell()`: Keyword-based spam/OnlyFans detection
- Refusal detection via heuristic patterns
#### Browser Client (`src/browser/client.ts`)
Playwright persistent context manager.
- Browser session persistence (cookies/storage)
- Login verification (multiple UI checks + API fallback)
- Automated login flow (form fill, 2FA/captcha detection)
- Debug screenshot capture on login failure
#### MessagePage (`src/browser/message-page.ts`)
Reddit DM compose page automation.
- Fills recipient/subject/body fields
- Extensive Shadow DOM support for `faceplate-*` components
- Detects recipient errors, verifies submission success
- Supports dry-run mode for testing
### Data Layer
#### Database Client (`src/database/client.ts`)
Better-SQLite3 database initialization.
- Creates database at configured path
- Enables WAL mode and foreign keys
- Runs migrations on startup
- Safe column-addition for schema evolution
#### Repositories (`src/database/repositories.ts`)
Three repository classes abstracting SQLite operations.
- **PostsRepository**: Processed post CRUD, mark sent, update analysis
- **MessagesRepository**: Sent message CRUD, response tracking, strategy stats, subreddit stats, daily trends
- **SuccessfulMessagesRepository**: Successful pairs for few-shot learning, subreddit-weighted retrieval
### Utilities
#### Logger (`src/utils/logger.ts`)
Winston structured logging.
- JSON format for log files
- Colored console output
- File logging: `error.log` and combined log
#### RateLimiter (`src/utils/rate-limiter.ts`)
Token-bucket rate limiting.
- p-queue with concurrency=1
- Minimum delay between messages with random jitter
- Hourly message count tracking
- Timer status exposed to dashboard
#### Retry (`src/utils/retry.ts`)
Exponential backoff wrapper.
- p-retry with configurable retries (default 3)
- Backoff range: 5s to 60s
- Logging on retries and failures
#### Image (`src/utils/image.ts`)
Image processing utilities.
- Download via axios
- Base64 conversion for AI API
- MIME type validation (image/jpeg, image/png, image/webp)
### Web Dashboard
#### WebServer (`src/web/server.ts`)
Express server with 25+ API endpoints.
- Stats, activity, messages, chats, conversations, media endpoints
- Bot controls (start/stop)
- SSE realtime log streaming
- Analytics: strategy, subreddit, daily trends
- Runtime config management (GET/PUT)
- Debug endpoints
---
## Data Flow
### Post Processing Pipeline
```
1. MonitorService polls subreddits
│
├─► RedditApi.fetchPosts()
│ └─► GET /r/{subreddit}/new.json
│ └─► Filter: image posts only, valid author
│
├─► Filter: skip processed posts (DB check)
├─► Filter: skip previously messaged users
│
└─► Return newest valid post
│
2. ProcessorService.processPost(post)
│
├─► ImageService.downloadImage(url)
│ └─► Validate MIME type, file size
│
├─► OpenRouter.detectGenderOnly()
│ └─► Quick pre-filter (cost-saving)
│ └─► Skip if gender doesn't match target
│
├─► StrategySelector.selectStrategy(subreddit, author)
│ └─► Explore/exploit algorithm
│
├─► OpenRouter.analyzeImage(image)
│ └─► Multi-model fallback chain
│ └─► Detect gender, describe image
│
├─► OpenRouter.generateMessage(analysis, title, examples, strategy)
│ └─► Few-shot learning from successful_messages
│
└─► Mark post as processed in DB
│
3. MessagePage.sendDM(author, subject, message)
│
├─► Navigate to /compose?to={author}
├─► Fill fields (with shadow DOM handling)
├─► Submit message
└─► Verify success
│
└─► Record sent_messages in DB
```
### Response Monitoring Flow
```
1. ResponseChecker.checkResponses()
│
├─► Navigate to /message/inbox
├─► Parse response items
│
├─► For each response:
│ ├─► Match to sent_messages by recipient + timestamp
│ ├─► Update got_response = 1
│ ├─► Store message/response pair
│ └─► Add to successful_messages (for few-shot learning)
│
└─► Update strategy stats in DB
```
### Chat Interaction Flow
```
1. ChatService.fetchChats()
│
├─► Navigate to chat.reddit.com
├─► Traverse shadow DOM via JS injection
├─► Extract active conversations
└─► Cache in memory + disk JSON
│
2. ChatService.loadConversation(chatId)
│
├─► Load from cache
├─► Fetch full history (pagination)
└─► Return conversation object
│
3. ChatService.sendMessage(chatId, text)
│
├─► Navigate to chat view
├─► Fill message input (shadow DOM)
├─► Apply typing delay (human-like)
├─► Submit
└─► Verify delivery
```
---
## Database Schema
### Table: `processed_posts`
Tracks all posts that have been discovered and processed.
| Column | Type | Constraints | Description |
|---|---|---|---|
| `id` | TEXT | PRIMARY KEY | Reddit post ID |
| `subreddit` | TEXT | NOT NULL | Source subreddit |
| `author` | TEXT | NOT NULL | Post author username |
| `title` | TEXT | | Post title |
| `image_url` | TEXT | | Original image URL |
| `image_analysis` | TEXT | | AI-generated image description |
| `generated_message` | TEXT | | AI-generated DM text |
| `processed_at` | INTEGER | NOT NULL | Unix timestamp |
| `message_sent` | INTEGER | DEFAULT 0 | 0 = not sent, 1 = sent |
| `error` | TEXT | | Error message if processing failed |
**Indexes**: `idx_posts_subreddit`, `idx_posts_author`
### Table: `sent_messages`
Tracks all DMs sent by the bot.
| Column | Type | Constraints | Description |
|---|---|---|---|
| `id` | INTEGER | PRIMARY KEY AUTOINCREMENT | Internal ID |
| `post_id` | TEXT | FK → processed_posts.id | Source post |
| `recipient` | TEXT | NOT NULL | Recipient username |
| `subject` | TEXT | DEFAULT 'Stunning' | DM subject line |
| `message` | TEXT | NOT NULL | DM body |
| `sent_at` | INTEGER | NOT NULL | Unix timestamp |
| `strategy_id` | TEXT | | A/B test variant identifier |
| `got_response` | INTEGER | DEFAULT 0 | 0 = no response, 1 = responded |
| `response_checked_at` | INTEGER | | Timestamp of last response check |
**Indexes**: `idx_messages_recipient`, `idx_messages_strategy`, `idx_messages_response`
### Table: `successful_messages`
Stores successful message/response pairs for few-shot learning.
| Column | Type | Constraints | Description |
|---|---|---|---|
| `id` | INTEGER | PRIMARY KEY AUTOINCREMENT | Internal ID |
| `sent_message_id` | INTEGER | FK → sent_messages.id (nullable) | Related message |
| `original_message` | TEXT | NOT NULL | The DM that got a response |
| `image_analysis` | TEXT | | Context from image analysis |
| `post_title` | TEXT | | Original post title |
| `response_snippet` | TEXT | | First reply text |
| `subreddit` | TEXT | | For weighted example selection |
| `created_at` | INTEGER | NOT NULL | Unix timestamp |
**Indexes**: `idx_success_subreddit`, `idx_success_message`
### Migration Strategy
The database uses progressive migrations that:
- Run on every startup
- Use `ALTER TABLE ADD COLUMN IF NOT EXISTS` for safe schema evolution
- Create indexes with `IF NOT EXISTS` for idempotency
- Enable WAL mode and foreign keys for performance and integrity
---
## API Reference
### Web Dashboard Endpoints
All endpoints are served by the Express server at `http://localhost:{port}`.
#### Bot Status & Control
| Method | Endpoint | Description |
|---|---|---|
| GET | `/api/stats` | Bot statistics (running state, post/message/response counts, response rate) |
| POST | `/api/start` | Start the bot |
| POST | `/api/stop` | Stop the bot |
#### Activity & Messages
| Method | Endpoint | Description |
|---|---|---|
| GET | `/api/activity` | Recent activity log entries |
| GET | `/api/posts/by-author/:username` | Look up processed posts by user |
| GET | `/api/messages` | Recent sent messages |
| GET | `/api/successful` | Successful message/response pairs |
#### Chat & Conversations
| Method | Endpoint | Description |
|---|---|---|
| GET | `/api/chats` | Active cached conversations |
| POST | `/api/chats/fetch-all` | Fetch all chats from Reddit |
| GET | `/api/conversations` | All cached conversations |
| GET | `/api/conversations/:id` | Single conversation detail |
| POST | `/api/conversations/:id/suggest-reply` | AI suggest reply for conversation |
| POST | `/api/conversations/:id/send-reply` | Send chat reply |
| POST | `/api/conversations/:id/approve-reply` | Approve pending auto-reply |
| POST | `/api/conversations/:id/reject-reply` | Reject pending auto-reply |
| POST | `/api/conversations/:id/dismiss-upsell` | Dismiss upsell flag |
#### Media
| Method | Endpoint | Description |
|---|---|---|
| GET | `/api/media` | Downloaded media gallery |
| GET | `/api/media/sync` | Media sync status |
| POST | `/api/media/sync` | Trigger incremental sync (top 10 chats) |
| POST | `/api/media/sync/full` | Trigger full sync (all conversations) |
#### Analytics
| Method | Endpoint | Description |
|---|---|---|
| GET | `/api/analytics/by-subreddit` | Response rate by subreddit |
| GET | `/api/analytics/by-strategy` | Response rate by strategy variant |
| GET | `/api/analytics/trend` | Daily response rate trend |
#### Configuration
| Method | Endpoint | Description |
|---|---|---|
| GET | `/api/config` | Current runtime configuration |
| PUT | `/api/config` | Update runtime config (rate limits, dry run, poll interval) |
#### Logs & Debugging
| Method | Endpoint | Description |
|---|---|---|
| GET | `/api/logs/stream` | SSE realtime log stream |
| GET | `/api/logs` | Recent logs (HTTP) |
| GET | `/api/timers` | Rate limiter timer status |
| POST | `/api/debug/inspect-dom` | Debug: inspect chat DOM structure |
| POST | `/api/debug/check-responses` | Debug: manual response check |
---
## Configuration
### Configuration Sources (Priority Order)
1. **`config.yaml`** - Primary configuration file (project root)
2. **Environment Variables** - Override specific fields:
- `OPENROUTER_API_KEY` - OpenRouter API key
- `REDDIT_USERNAME` - Reddit username
- `REDDIT_PASSWORD` - Reddit password
- `BROWSER_HEADLESS` - Browser headless mode (`true`/`false`)
### Configuration Schema (Zod-Validated)
```typescript
{
reddit: {
subreddits: string[] // Subreddits to monitor
pollInterval: number // Minutes between polls
targetGender: string // Target gender filter
sellerKeywords: string[] // Keywords for spam/seller detection
}
openrouter: {
apiKey: string // OpenRouter API key
model: string // Primary AI model
fallbackModels: string[] // Fallback models on refusal
systemPrompt: string // System prompt for analysis
messagePrompt: string // System prompt for message generation
}
rateLimit: {
minDelayMs: number // Minimum delay between messages
maxMessagesPerHour: number // Hourly message cap
jitterMs: number // Random jitter range
}
browser: {
headless: boolean // Headless mode
userDataDir: string // Persistent session directory
}
database: {
path: string // SQLite database path
}
mediaSync: {
enabled: boolean // Enable media sync
intervalMinutes: number // Sync interval
}
autoReply: {
enabled: boolean // Enable auto-reply
autoSend: boolean // Auto-send or pending approval
checkInterval: number // Check interval in minutes
}
strategyVariants: {
name: string // Variant name
prompt: string // Message style prompt
enabled: boolean // Enable/disable variant
}[]
web: {
port: number // Dashboard port
}
}
```
### Runtime Configuration
The following settings can be modified via the web dashboard without restarting:
- `rateLimit.minDelayMs` - Message delay
- `rateLimit.maxMessagesPerHour` - Hourly cap
- `rateLimit.jitterMs` - Jitter range
- `reddit.pollInterval` - Poll frequency
- `dryRun` - Dry run mode (log only, no actual sends)
---
## Design Patterns
### Repository Pattern
All database operations are abstracted behind three repository classes:
- `PostsRepository` - Post lifecycle management
- `MessagesRepository` - Message tracking and analytics
- `SuccessfulMessagesRepository` - Few-shot learning data
Benefits:
- Single source of truth for data access
- Easy to swap implementations (e.g., different databases)
- Consistent error handling and query patterns
### Dependency Injection
Services receive dependencies via constructor parameters:
```typescript
class ProcessorService {
constructor(
private config: Config,
private redditApi: RedditApi,
private openrouter: OpenRouterService,
private strategySelector: StrategySelector,
private messageSender: MessagePage,
private postsRepo: PostsRepository,
private messagesRepo: MessagesRepository,
private successfulRepo: SuccessfulMessagesRepository,
private logger: Logger
) {}
}
```
Benefits:
- Loose coupling between components
- Easy to mock dependencies for testing
- Clear dependency graph
### Strategy Pattern
`StrategySelector` implements A/B testing with multiple message variants:
```typescript
class StrategySelector {
selectStrategy(subreddit: string, author: string): string {
// Explore/exploit algorithm
// Returns variant name based on response rates
}
}
```
Benefits:
- Swappable message strategies
- Measurable performance per variant
- Configurable exploration rate
### Factory Pattern
Configuration and database creation use factory functions:
- `loadConfig()` - Loads, merges env vars, validates with Zod
- `createDatabase()` - Initializes SQLite, runs migrations
### Observer Pattern
SSE (Server-Sent Events) for realtime log streaming:
```typescript
// Logger writes to a shared buffer
logger.info("Message sent to user123");
// WebServer subscribes clients via SSE
app.get("/api/logs/stream", (req, res) => {
// Push new log entries to connected clients
});
```
### Circuit Breaker / Retry
`withRetry()` wrapper for resilient external calls:
```typescript
await withRetry(
() => redditApi.fetchPosts(subreddit),
{ retries: 3, logger }
);
```
Multiple fallback chains:
- AI model fallback (primary → fallback models)
- Image URL fallback (preview → i.redd.it → imgur → thumbnail)
### Rate Limiter Pattern
Token-bucket-like implementation:
```typescript
class RateLimiter {
async waitForSlot(): Promise<void> {
// Enforce minimum delay + jitter
// Track hourly counts
// Block if hourly cap reached
}
}
```
---
## External Integrations
### Reddit API
- **Authentication**: Playwright browser cookies (persistent session)
- **Endpoints Used**:
- `GET /r/{name}/new.json` - Fetch new posts
- `GET /user/{username}/profile` - Check for seller/spam indicators
- `GET /message/inbox` - Check for responses
- `GET chat.reddit.com` - Chat scraping
- **Rate Limits**: Enforced by `RateLimiter` utility
### OpenRouter AI
- **SDK**: OpenAI-compatible (`openai` npm package)
- **Base URL**: `https://openrouter.ai/api/v1`
- **Endpoints Used**:
- `POST /chat/completions` - Image analysis, message generation, replies
- **Model Fallback**: Primary model → fallback models on refusal
- **Cost Optimization**: Quick gender detection before full analysis
### Playwright Browser
- **Context**: Persistent (cookies/session storage saved to disk)
- **Usage**:
- Automated Reddit login
- DM compose page automation
- Inbox navigation for response checking
- Chat page scraping (shadow DOM traversal)
- Media file downloads
- **Shadow DOM**: JavaScript injection for `faceplate-*` component traversal
---
## Deployment
### Docker
The project includes full Docker support:
- **`Dockerfile`** - Multi-stage build with Node.js
- **`docker-compose.yml`** - Service orchestration
- **`.dockerignore`** - Excludes unnecessary files
### Dokploy
Deployment documentation is provided in `DOKPLOY_DEPLOY.md`.
### Required Environment Variables
| Variable | Required | Description |
|---|---|---|
| `OPENROUTER_API_KEY` | Yes | OpenRouter API key for AI |
| `REDDIT_USERNAME` | Yes | Reddit username |
| `REDDIT_PASSWORD` | Yes | Reddit password |
| `BROWSER_HEADLESS` | No | Browser headless mode (default: `true`) |
### Pre-Deployment Checklist
1. Set up `config.yaml` with target subreddits and settings
2. Set environment variables (or use `.env` file)
3. Run `npm install` to install dependencies
4. Run `npm run build` to compile TypeScript
5. Run `scripts/login.ts` once to establish browser session
6. Start the bot (via `npm start` or Docker)
---
## Technical Debt & Recommendations
### Critical
1. **No Tests** - Zero test coverage. Add unit tests for services, especially:
- `StrategySelector` (complex algorithm)
- `ChatService` (shadow DOM logic)
- `OpenRouterService` (model fallback logic)
2. **No README.md** - No project documentation for new developers.
3. **Hardcoded Username** - `ChatService` has `myUsername = 'Upset_Marsupial1006'` hardcoded in JS evaluation scripts. Should be parameterized from config.
### High Priority
4. **God Class** - `main.ts` (391 lines) creates all dependencies and wires everything. Consider extracting an `AppContainer` or `DependencyContainer`.
5. **ChatService Size** - 1123+ lines. Consider splitting into:
- `ChatScraper` (navigation + DOM traversal)
- `ConversationCache` (cache management)
- `ChatMessenger` (sending + verification)
6. **No .env.example** - No template showing required environment variables.
### Medium Priority
7. **Single-Page Dashboard** - `public/index.html` is ~2000+ lines with embedded CSS/JS. Consider a proper frontend framework.
8. **WebServer Size** - 794 lines. Consider splitting routes into separate files.
9. **Config Mutability** - Runtime config updates via `PUT /api/config` modify the in-memory config object. Consider a proper config service with validation on updates.
10. **Error Handling** - Some services have basic try/catch. Consider structured error types with retry logic.
### Low Priority
11. **Logging** - Winston is well-configured, but consider adding request IDs for tracing.
12. **Type Safety** - Some `page.evaluate()` calls use `any` types. Consider stricter typing for DOM structures.
13. **Media Storage** - Media files stored in `data/media/` with no cleanup policy. Consider retention policies.
Reference in new issue View Git Blame Copy Permalink