Restructure chat service, web routes, and add test suite
Split monolithic chat service into modular structure with Matrix client support, extract web server into per-resource route modules, introduce bootstrap/main-loop separation, and add comprehensive vitest coverage across services, repositories, and utilities. Adds project docs and CLAUDE.md agent instructions.
This commit is contained in:
jjbrunton committed
1 parent
dfa0580859
commit
633bb0ec04
84 files changed
+19278
-3888
No files matched your search
@@ -0,0 +1,898 @@
|
||||
# RedditBot - Software Architecture Documentation
|
||||
|
||||
## Table of Contents
|
||||
|
||||
1. [Project Overview](#project-overview)
|
||||
2. [System Architecture](#system-architecture)
|
||||
3. [Directory Structure](#directory-structure)
|
||||
4. [Component Reference](#component-reference)
|
||||
5. [Data Flow](#data-flow)
|
||||
6. [Database Schema](#database-schema)
|
||||
7. [API Reference](#api-reference)
|
||||
8. [Configuration](#configuration)
|
||||
9. [Design Patterns](#design-patterns)
|
||||
10. [External Integrations](#external-integrations)
|
||||
11. [Deployment](#deployment)
|
||||
12. [Technical Debt & Recommendations](#technical-debt--recommendations)
|
||||
|
||||
---
|
||||
|
||||
## Project Overview
|
||||
|
||||
**RedditBot** is an automated Reddit monitoring and messaging system that:
|
||||
|
||||
- Monitors 30+ NSFW subreddits for image posts
|
||||
- Uses AI (OpenRouter) to analyze images and detect gender
|
||||
- Generates personalized direct messages to post authors
|
||||
- Sends DMs via browser automation (Playwright)
|
||||
- Monitors for responses and tracks success rates
|
||||
- Implements A/B testing with explore/exploit strategy selection
|
||||
- Learns from successful message/response pairs (few-shot learning)
|
||||
- Provides a real-time web dashboard for monitoring and control
|
||||
|
||||
### Key Capabilities
|
||||
|
||||
| Capability | Description |
|
||||
|---|---|
|
||||
| Post Discovery | Polls subreddits for new image posts, filters by gender and spam profiles |
|
||||
| AI Analysis | Multi-model image analysis with fallback chain via OpenRouter |
|
||||
| Message Generation | AI-crafted DMs using successful examples as few-shot prompts |
|
||||
| A/B Testing | Explore/exploit strategy selector optimizes response rates over time |
|
||||
| Response Monitoring | Background service checks inbox for user replies |
|
||||
| Chat Automation | Reddit chat scraping via shadow DOM traversal, auto-reply support |
|
||||
| Media Management | Download, cache, deduplicate, and sync media from conversations |
|
||||
| Web Dashboard | Real-time monitoring with SSE log streaming, analytics, and controls |
|
||||
|
||||
---
|
||||
|
||||
## System Architecture
|
||||
|
||||
### High-Level Architecture
|
||||
|
||||
```
|
||||
┌─────────────────────────────────────────────────────────────────────┐
|
||||
│ main (index.ts) │
|
||||
│ Orchestrator / Entry Point │
|
||||
└────────────────────────┬────────────────────────────────────────────┘
|
||||
│
|
||||
┌────────────────┼────────────────┐
|
||||
│ │ │
|
||||
▼ ▼ ▼
|
||||
┌───────────────┐ ┌──────────────┐ ┌──────────────┐
|
||||
│ Monitor │ │ Processor │ │ Response │
|
||||
│ Service │ │ │ │ Checker │
|
||||
│ │ │ 1. Download │ │ │
|
||||
│ Poll subreddits│ │ 2. Validate │ │ Check inbox │
|
||||
│ for new posts │ │ 3. Gender │ │ for replies │
|
||||
└───────┬───────┘ │ Filter │ └──────────────┘
|
||||
│ │ 4. AI Analysis│ │
|
||||
│ │ 5. Strategy │ │
|
||||
│ │ Selection │ │
|
||||
│ │ 6. Message │ │
|
||||
│ │ Generation │ │
|
||||
└────┬────┘ └─────────────┘ │
|
||||
│ │
|
||||
▼ ▼
|
||||
┌───────────────────────┐ ┌───────────────────────┐
|
||||
│ Strategy Selector │ │ Chat Service │
|
||||
│ │ │ │
|
||||
│ A/B Testing │ │ Scrapes chat (shadow │
|
||||
│ Explore/Exploit │ │ DOM traversal) │
|
||||
│ Variant Selection │ │ Auto-reply, upsell │
|
||||
└───────────────────────┘ │ detection │
|
||||
└───────────────────────┘
|
||||
│ │
|
||||
▼ ▼
|
||||
┌───────────────────────┐ ┌───────────────────────┐
|
||||
│ Message Sender │ │ Media Service │
|
||||
│ (Browser) │ │ │
|
||||
│ │ │ Download, cache, │
|
||||
│ Playwright DM compose │ │ deduplicate (SHA-1) │
|
||||
│ Shadow DOM handling │ │ Matrix thumbnail │
|
||||
└───────────────────────┘ │ resolution │
|
||||
└───────────────────────┘
|
||||
│ │
|
||||
▼ ▼
|
||||
┌─────────────────────────────────────────────────────────┐
|
||||
│ SQLite Database │
|
||||
│ │
|
||||
│ processed_posts │ sent_messages │ successful_messages │
|
||||
└─────────────────────────────────────────────────────────┘
|
||||
│
|
||||
▼
|
||||
┌─────────────────────────────────────────────────────────┐
|
||||
│ Web Dashboard (Express) │
|
||||
│ │
|
||||
│ 25+ API endpoints │ SSE log streaming │ Analytics │
|
||||
│ Runtime config │ Bot controls │ Media gallery │
|
||||
└─────────────────────────────────────────────────────────┘
|
||||
```
|
||||
|
||||
### Service Communication
|
||||
|
||||
```
|
||||
main.ts (Orchestrator)
|
||||
│
|
||||
├──► Config (Zod-validated YAML + env vars)
|
||||
│
|
||||
├──► Database (Better-SQLite3)
|
||||
│ │
|
||||
│ └──► PostsRepository
|
||||
│ └──► MessagesRepository
|
||||
│ └──► SuccessfulMessagesRepository
|
||||
│
|
||||
├──► MonitorService ──► RedditApi ──► Playwright (cookies)
|
||||
│
|
||||
├──► ProcessorService
|
||||
│ ├──► Image download (axios)
|
||||
│ ├──► Gender detection (OpenRouter)
|
||||
│ ├──► Full analysis (OpenRouter)
|
||||
│ └──► StrategySelector ──► MessagesRepository
|
||||
│
|
||||
├──► MessageSender (Browser - Playwright)
|
||||
│
|
||||
├──► ResponseChecker ──► Playwright (inbox navigation)
|
||||
│
|
||||
├──► ChatService ──► Playwright (chat.reddit.com shadow DOM)
|
||||
│
|
||||
├──► MediaService ──► Playwright (media download)
|
||||
│
|
||||
├──► MediaSyncService (background periodic sync)
|
||||
│
|
||||
├──► AutoReplyService ──► OpenRouter (reply generation)
|
||||
│
|
||||
├──► RateLimiter (p-queue, jitter)
|
||||
│
|
||||
└──► WebServer (Express, SSE)
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Directory Structure
|
||||
|
||||
```
|
||||
RedditBot/
|
||||
├── .dockerignore # Docker ignore rules
|
||||
├── .gitignore # Git ignore rules
|
||||
├── agents.md # Engineering standards & agent guidelines
|
||||
├── config.yaml # Main YAML configuration file
|
||||
├── docker-compose.yml # Docker Compose setup
|
||||
├── DOKPLOY_DEPLOY.md # Dokploy deployment guide
|
||||
├── Dockerfile # Container build definition
|
||||
├── package.json # Node.js dependencies & scripts
|
||||
├── tsconfig.json # TypeScript configuration
|
||||
├── public/
|
||||
│ └── index.html # Single-file web dashboard (HTML+CSS+JS)
|
||||
├── scripts/
|
||||
│ ├── inspect-chat-dom.ts # Debug: inspect chat DOM structure
|
||||
│ └── login.ts # Manual browser login helper
|
||||
└── src/
|
||||
├── index.ts # Main entry point / orchestrator (391 lines)
|
||||
├── config/
|
||||
│ └── index.ts # Config loader + Zod validation schema (157 lines)
|
||||
├── database/
|
||||
│ ├── client.ts # SQLite setup + migration runner (121 lines)
|
||||
│ └── repositories.ts # Repository pattern: Posts, Messages, Success (433 lines)
|
||||
├── reddit/
|
||||
│ └── api.ts # Reddit API client (axios + Playwright cookies) (233 lines)
|
||||
├── ai/
|
||||
│ └── openrouter.ts # OpenRouter AI client (image analysis, message gen) (307 lines)
|
||||
├── browser/
|
||||
│ ├── client.ts # Playwright persistent context + login (302 lines)
|
||||
│ ├── message-page.ts # DM compose page automation + shadow DOM (560 lines)
|
||||
│ └── media-downloader.ts # Playwright-based media downloader (29 lines)
|
||||
├── services/
|
||||
│ ├── monitor.ts # Post discovery service (72 lines)
|
||||
│ ├── processor.ts # Image download + AI + message pipeline (132 lines)
|
||||
│ ├── response-checker.ts # Inbox response checker (179 lines)
|
||||
│ ├── strategy-selector.ts # A/B testing explore/exploit (101 lines)
|
||||
│ ├── chat.ts # Chat scraping + messaging + auto-reply (1123+ lines)
|
||||
│ ├── media.ts # Media download, caching, indexing (254 lines)
|
||||
│ ├── media-sync.ts # Periodic background media sync (119 lines)
|
||||
│ └── auto-reply.ts # Auto-reply to chat conversations (199 lines)
|
||||
├── utils/
|
||||
│ ├── logger.ts # Winston structured logging (32 lines)
|
||||
│ ├── rate-limiter.ts # Token-bucket rate limiting (114 lines)
|
||||
│ ├── retry.ts # Exponential backoff wrapper (28 lines)
|
||||
│ └── image.ts # Image download + base64 + MIME validation (34 lines)
|
||||
└── web/
|
||||
└── server.ts # Express dashboard + SSE streaming (794 lines)
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Component Reference
|
||||
|
||||
### Core Services
|
||||
|
||||
#### MonitorService (`src/services/monitor.ts`)
|
||||
Discovers new image posts across configured subreddits.
|
||||
|
||||
- Shuffles subreddit order each cycle
|
||||
- Fetches new posts via `RedditApi`
|
||||
- Filters out already-processed posts and previously-messaged users
|
||||
- Returns the single newest valid post per cycle
|
||||
|
||||
#### ProcessorService (`src/services/processor.ts`)
|
||||
Core processing pipeline for each discovered post.
|
||||
|
||||
1. Downloads image via `ImageService`
|
||||
2. Validates MIME type and file size
|
||||
3. Runs quick gender detection (cost-saving pre-filter)
|
||||
4. Selects A/B strategy variant via `StrategySelector`
|
||||
5. Runs full AI analysis + message generation via `OpenRouterService`
|
||||
6. Handles AI refusals gracefully (marks post, tries next)
|
||||
|
||||
#### StrategySelector (`src/services/strategy-selector.ts`)
|
||||
Implements explore/exploit A/B testing for message variants.
|
||||
|
||||
- Tracks response rates per strategy variant
|
||||
- Uses minimum sample thresholds before trusting statistics
|
||||
- Applies random exploration rate for continued discovery
|
||||
- Selects best variant based on weighted response rates
|
||||
|
||||
#### ResponseChecker (`src/services/response-checker.ts`)
|
||||
Background service that monitors for user replies.
|
||||
|
||||
- Navigates Reddit inbox via Playwright
|
||||
- Identifies responses to sent messages
|
||||
- Updates `got_response` flag in database
|
||||
- Stores successful message/response pairs for few-shot learning
|
||||
|
||||
#### ChatService (`src/services/chat.ts`)
|
||||
Most complex service - manages Reddit chat interactions.
|
||||
|
||||
- Scrapes `chat.reddit.com` using JavaScript injection (shadow DOM traversal)
|
||||
- Manages conversation cache (in-memory + disk JSON)
|
||||
- Loads full conversation history with pagination
|
||||
- Sends messages with human-like typing delays
|
||||
- Verifies message delivery
|
||||
- Detects upsell/spam keywords in chat
|
||||
- Parameter: `myUsername` is hardcoded (should be from config)
|
||||
|
||||
#### MediaService (`src/services/media.ts`)
|
||||
Media download, caching, and indexing.
|
||||
|
||||
- Downloads images from chat conversations
|
||||
- Resolves Matrix blurred thumbnails to full resolution
|
||||
- Stores files in `data/media/` directory
|
||||
- Maintains JSON index with SHA-1 deduplication
|
||||
- Tracks download status per conversation
|
||||
|
||||
#### MediaSyncService (`src/services/media-sync.ts`)
|
||||
Periodic background media synchronization.
|
||||
|
||||
- Runs incremental syncs (top 10 chats) or full syncs (all conversations)
|
||||
- Configurable interval via `mediaSyncInterval`
|
||||
- Tracks sync status in memory
|
||||
- Exposed via web dashboard endpoints
|
||||
|
||||
#### AutoReplyService (`src/services/auto-reply.ts`)
|
||||
Automated reply to Reddit chat conversations.
|
||||
|
||||
- Checks for unread messages in cached conversations
|
||||
- Detects upsell/spam keywords
|
||||
- Generates AI reply suggestions via `OpenRouterService`
|
||||
- Sends replies (auto or pending for manual approval)
|
||||
- Event-based architecture for dashboard integration
|
||||
|
||||
### External Integrations
|
||||
|
||||
#### RedditApi (`src/reddit/api.ts`)
|
||||
Axios-based Reddit API client.
|
||||
|
||||
- Fetches posts from subreddits (`/r/{name}/new.json`)
|
||||
- Validates posts (image-only, skips deleted authors)
|
||||
- Extracts image URLs with fallback chain: `preview` > `i.redd.it` > `imgur` > `thumbnail`
|
||||
- Checks user profiles against seller/spam keywords
|
||||
- Uses Playwright browser cookies for authenticated requests
|
||||
|
||||
#### OpenRouterService (`src/ai/openrouter.ts`)
|
||||
OpenRouter API client using OpenAI-compatible SDK.
|
||||
|
||||
- `analyzeImage()`: Multi-model image analysis with fallback chain
|
||||
- `detectGenderOnly()`: Quick cost-saving gender detection
|
||||
- `generateMessage()`: DM text with few-shot examples from DB
|
||||
- `generateReply()`: AI chat reply from conversation history
|
||||
- `detectUpsell()`: Keyword-based spam/OnlyFans detection
|
||||
- Refusal detection via heuristic patterns
|
||||
|
||||
#### Browser Client (`src/browser/client.ts`)
|
||||
Playwright persistent context manager.
|
||||
|
||||
- Browser session persistence (cookies/storage)
|
||||
- Login verification (multiple UI checks + API fallback)
|
||||
- Automated login flow (form fill, 2FA/captcha detection)
|
||||
- Debug screenshot capture on login failure
|
||||
|
||||
#### MessagePage (`src/browser/message-page.ts`)
|
||||
Reddit DM compose page automation.
|
||||
|
||||
- Fills recipient/subject/body fields
|
||||
- Extensive Shadow DOM support for `faceplate-*` components
|
||||
- Detects recipient errors, verifies submission success
|
||||
- Supports dry-run mode for testing
|
||||
|
||||
### Data Layer
|
||||
|
||||
#### Database Client (`src/database/client.ts`)
|
||||
Better-SQLite3 database initialization.
|
||||
|
||||
- Creates database at configured path
|
||||
- Enables WAL mode and foreign keys
|
||||
- Runs migrations on startup
|
||||
- Safe column-addition for schema evolution
|
||||
|
||||
#### Repositories (`src/database/repositories.ts`)
|
||||
Three repository classes abstracting SQLite operations.
|
||||
|
||||
- **PostsRepository**: Processed post CRUD, mark sent, update analysis
|
||||
- **MessagesRepository**: Sent message CRUD, response tracking, strategy stats, subreddit stats, daily trends
|
||||
- **SuccessfulMessagesRepository**: Successful pairs for few-shot learning, subreddit-weighted retrieval
|
||||
|
||||
### Utilities
|
||||
|
||||
#### Logger (`src/utils/logger.ts`)
|
||||
Winston structured logging.
|
||||
|
||||
- JSON format for log files
|
||||
- Colored console output
|
||||
- File logging: `error.log` and combined log
|
||||
|
||||
#### RateLimiter (`src/utils/rate-limiter.ts`)
|
||||
Token-bucket rate limiting.
|
||||
|
||||
- p-queue with concurrency=1
|
||||
- Minimum delay between messages with random jitter
|
||||
- Hourly message count tracking
|
||||
- Timer status exposed to dashboard
|
||||
|
||||
#### Retry (`src/utils/retry.ts`)
|
||||
Exponential backoff wrapper.
|
||||
|
||||
- p-retry with configurable retries (default 3)
|
||||
- Backoff range: 5s to 60s
|
||||
- Logging on retries and failures
|
||||
|
||||
#### Image (`src/utils/image.ts`)
|
||||
Image processing utilities.
|
||||
|
||||
- Download via axios
|
||||
- Base64 conversion for AI API
|
||||
- MIME type validation (image/jpeg, image/png, image/webp)
|
||||
|
||||
### Web Dashboard
|
||||
|
||||
#### WebServer (`src/web/server.ts`)
|
||||
Express server with 25+ API endpoints.
|
||||
|
||||
- Stats, activity, messages, chats, conversations, media endpoints
|
||||
- Bot controls (start/stop)
|
||||
- SSE realtime log streaming
|
||||
- Analytics: strategy, subreddit, daily trends
|
||||
- Runtime config management (GET/PUT)
|
||||
- Debug endpoints
|
||||
|
||||
---
|
||||
|
||||
## Data Flow
|
||||
|
||||
### Post Processing Pipeline
|
||||
|
||||
```
|
||||
1. MonitorService polls subreddits
|
||||
│
|
||||
├─► RedditApi.fetchPosts()
|
||||
│ └─► GET /r/{subreddit}/new.json
|
||||
│ └─► Filter: image posts only, valid author
|
||||
│
|
||||
├─► Filter: skip processed posts (DB check)
|
||||
├─► Filter: skip previously messaged users
|
||||
│
|
||||
└─► Return newest valid post
|
||||
│
|
||||
2. ProcessorService.processPost(post)
|
||||
│
|
||||
├─► ImageService.downloadImage(url)
|
||||
│ └─► Validate MIME type, file size
|
||||
│
|
||||
├─► OpenRouter.detectGenderOnly()
|
||||
│ └─► Quick pre-filter (cost-saving)
|
||||
│ └─► Skip if gender doesn't match target
|
||||
│
|
||||
├─► StrategySelector.selectStrategy(subreddit, author)
|
||||
│ └─► Explore/exploit algorithm
|
||||
│
|
||||
├─► OpenRouter.analyzeImage(image)
|
||||
│ └─► Multi-model fallback chain
|
||||
│ └─► Detect gender, describe image
|
||||
│
|
||||
├─► OpenRouter.generateMessage(analysis, title, examples, strategy)
|
||||
│ └─► Few-shot learning from successful_messages
|
||||
│
|
||||
└─► Mark post as processed in DB
|
||||
│
|
||||
3. MessagePage.sendDM(author, subject, message)
|
||||
│
|
||||
├─► Navigate to /compose?to={author}
|
||||
├─► Fill fields (with shadow DOM handling)
|
||||
├─► Submit message
|
||||
└─► Verify success
|
||||
│
|
||||
└─► Record sent_messages in DB
|
||||
```
|
||||
|
||||
### Response Monitoring Flow
|
||||
|
||||
```
|
||||
1. ResponseChecker.checkResponses()
|
||||
│
|
||||
├─► Navigate to /message/inbox
|
||||
├─► Parse response items
|
||||
│
|
||||
├─► For each response:
|
||||
│ ├─► Match to sent_messages by recipient + timestamp
|
||||
│ ├─► Update got_response = 1
|
||||
│ ├─► Store message/response pair
|
||||
│ └─► Add to successful_messages (for few-shot learning)
|
||||
│
|
||||
└─► Update strategy stats in DB
|
||||
```
|
||||
|
||||
### Chat Interaction Flow
|
||||
|
||||
```
|
||||
1. ChatService.fetchChats()
|
||||
│
|
||||
├─► Navigate to chat.reddit.com
|
||||
├─► Traverse shadow DOM via JS injection
|
||||
├─► Extract active conversations
|
||||
└─► Cache in memory + disk JSON
|
||||
│
|
||||
2. ChatService.loadConversation(chatId)
|
||||
│
|
||||
├─► Load from cache
|
||||
├─► Fetch full history (pagination)
|
||||
└─► Return conversation object
|
||||
│
|
||||
3. ChatService.sendMessage(chatId, text)
|
||||
│
|
||||
├─► Navigate to chat view
|
||||
├─► Fill message input (shadow DOM)
|
||||
├─► Apply typing delay (human-like)
|
||||
├─► Submit
|
||||
└─► Verify delivery
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Database Schema
|
||||
|
||||
### Table: `processed_posts`
|
||||
|
||||
Tracks all posts that have been discovered and processed.
|
||||
|
||||
| Column | Type | Constraints | Description |
|
||||
|---|---|---|---|
|
||||
| `id` | TEXT | PRIMARY KEY | Reddit post ID |
|
||||
| `subreddit` | TEXT | NOT NULL | Source subreddit |
|
||||
| `author` | TEXT | NOT NULL | Post author username |
|
||||
| `title` | TEXT | | Post title |
|
||||
| `image_url` | TEXT | | Original image URL |
|
||||
| `image_analysis` | TEXT | | AI-generated image description |
|
||||
| `generated_message` | TEXT | | AI-generated DM text |
|
||||
| `processed_at` | INTEGER | NOT NULL | Unix timestamp |
|
||||
| `message_sent` | INTEGER | DEFAULT 0 | 0 = not sent, 1 = sent |
|
||||
| `error` | TEXT | | Error message if processing failed |
|
||||
|
||||
**Indexes**: `idx_posts_subreddit`, `idx_posts_author`
|
||||
|
||||
### Table: `sent_messages`
|
||||
|
||||
Tracks all DMs sent by the bot.
|
||||
|
||||
| Column | Type | Constraints | Description |
|
||||
|---|---|---|---|
|
||||
| `id` | INTEGER | PRIMARY KEY AUTOINCREMENT | Internal ID |
|
||||
| `post_id` | TEXT | FK → processed_posts.id | Source post |
|
||||
| `recipient` | TEXT | NOT NULL | Recipient username |
|
||||
| `subject` | TEXT | DEFAULT 'Stunning' | DM subject line |
|
||||
| `message` | TEXT | NOT NULL | DM body |
|
||||
| `sent_at` | INTEGER | NOT NULL | Unix timestamp |
|
||||
| `strategy_id` | TEXT | | A/B test variant identifier |
|
||||
| `got_response` | INTEGER | DEFAULT 0 | 0 = no response, 1 = responded |
|
||||
| `response_checked_at` | INTEGER | | Timestamp of last response check |
|
||||
|
||||
**Indexes**: `idx_messages_recipient`, `idx_messages_strategy`, `idx_messages_response`
|
||||
|
||||
### Table: `successful_messages`
|
||||
|
||||
Stores successful message/response pairs for few-shot learning.
|
||||
|
||||
| Column | Type | Constraints | Description |
|
||||
|---|---|---|---|
|
||||
| `id` | INTEGER | PRIMARY KEY AUTOINCREMENT | Internal ID |
|
||||
| `sent_message_id` | INTEGER | FK → sent_messages.id (nullable) | Related message |
|
||||
| `original_message` | TEXT | NOT NULL | The DM that got a response |
|
||||
| `image_analysis` | TEXT | | Context from image analysis |
|
||||
| `post_title` | TEXT | | Original post title |
|
||||
| `response_snippet` | TEXT | | First reply text |
|
||||
| `subreddit` | TEXT | | For weighted example selection |
|
||||
| `created_at` | INTEGER | NOT NULL | Unix timestamp |
|
||||
|
||||
**Indexes**: `idx_success_subreddit`, `idx_success_message`
|
||||
|
||||
### Migration Strategy
|
||||
|
||||
The database uses progressive migrations that:
|
||||
- Run on every startup
|
||||
- Use `ALTER TABLE ADD COLUMN IF NOT EXISTS` for safe schema evolution
|
||||
- Create indexes with `IF NOT EXISTS` for idempotency
|
||||
- Enable WAL mode and foreign keys for performance and integrity
|
||||
|
||||
---
|
||||
|
||||
## API Reference
|
||||
|
||||
### Web Dashboard Endpoints
|
||||
|
||||
All endpoints are served by the Express server at `http://localhost:{port}`.
|
||||
|
||||
#### Bot Status & Control
|
||||
|
||||
| Method | Endpoint | Description |
|
||||
|---|---|---|
|
||||
| GET | `/api/stats` | Bot statistics (running state, post/message/response counts, response rate) |
|
||||
| POST | `/api/start` | Start the bot |
|
||||
| POST | `/api/stop` | Stop the bot |
|
||||
|
||||
#### Activity & Messages
|
||||
|
||||
| Method | Endpoint | Description |
|
||||
|---|---|---|
|
||||
| GET | `/api/activity` | Recent activity log entries |
|
||||
| GET | `/api/posts/by-author/:username` | Look up processed posts by user |
|
||||
| GET | `/api/messages` | Recent sent messages |
|
||||
| GET | `/api/successful` | Successful message/response pairs |
|
||||
|
||||
#### Chat & Conversations
|
||||
|
||||
| Method | Endpoint | Description |
|
||||
|---|---|---|
|
||||
| GET | `/api/chats` | Active cached conversations |
|
||||
| POST | `/api/chats/fetch-all` | Fetch all chats from Reddit |
|
||||
| GET | `/api/conversations` | All cached conversations |
|
||||
| GET | `/api/conversations/:id` | Single conversation detail |
|
||||
| POST | `/api/conversations/:id/suggest-reply` | AI suggest reply for conversation |
|
||||
| POST | `/api/conversations/:id/send-reply` | Send chat reply |
|
||||
| POST | `/api/conversations/:id/approve-reply` | Approve pending auto-reply |
|
||||
| POST | `/api/conversations/:id/reject-reply` | Reject pending auto-reply |
|
||||
| POST | `/api/conversations/:id/dismiss-upsell` | Dismiss upsell flag |
|
||||
|
||||
#### Media
|
||||
|
||||
| Method | Endpoint | Description |
|
||||
|---|---|---|
|
||||
| GET | `/api/media` | Downloaded media gallery |
|
||||
| GET | `/api/media/sync` | Media sync status |
|
||||
| POST | `/api/media/sync` | Trigger incremental sync (top 10 chats) |
|
||||
| POST | `/api/media/sync/full` | Trigger full sync (all conversations) |
|
||||
|
||||
#### Analytics
|
||||
|
||||
| Method | Endpoint | Description |
|
||||
|---|---|---|
|
||||
| GET | `/api/analytics/by-subreddit` | Response rate by subreddit |
|
||||
| GET | `/api/analytics/by-strategy` | Response rate by strategy variant |
|
||||
| GET | `/api/analytics/trend` | Daily response rate trend |
|
||||
|
||||
#### Configuration
|
||||
|
||||
| Method | Endpoint | Description |
|
||||
|---|---|---|
|
||||
| GET | `/api/config` | Current runtime configuration |
|
||||
| PUT | `/api/config` | Update runtime config (rate limits, dry run, poll interval) |
|
||||
|
||||
#### Logs & Debugging
|
||||
|
||||
| Method | Endpoint | Description |
|
||||
|---|---|---|
|
||||
| GET | `/api/logs/stream` | SSE realtime log stream |
|
||||
| GET | `/api/logs` | Recent logs (HTTP) |
|
||||
| GET | `/api/timers` | Rate limiter timer status |
|
||||
| POST | `/api/debug/inspect-dom` | Debug: inspect chat DOM structure |
|
||||
| POST | `/api/debug/check-responses` | Debug: manual response check |
|
||||
|
||||
---
|
||||
|
||||
## Configuration
|
||||
|
||||
### Configuration Sources (Priority Order)
|
||||
|
||||
1. **`config.yaml`** - Primary configuration file (project root)
|
||||
2. **Environment Variables** - Override specific fields:
|
||||
- `OPENROUTER_API_KEY` - OpenRouter API key
|
||||
- `REDDIT_USERNAME` - Reddit username
|
||||
- `REDDIT_PASSWORD` - Reddit password
|
||||
- `BROWSER_HEADLESS` - Browser headless mode (`true`/`false`)
|
||||
|
||||
### Configuration Schema (Zod-Validated)
|
||||
|
||||
```typescript
|
||||
{
|
||||
reddit: {
|
||||
subreddits: string[] // Subreddits to monitor
|
||||
pollInterval: number // Minutes between polls
|
||||
targetGender: string // Target gender filter
|
||||
sellerKeywords: string[] // Keywords for spam/seller detection
|
||||
}
|
||||
openrouter: {
|
||||
apiKey: string // OpenRouter API key
|
||||
model: string // Primary AI model
|
||||
fallbackModels: string[] // Fallback models on refusal
|
||||
systemPrompt: string // System prompt for analysis
|
||||
messagePrompt: string // System prompt for message generation
|
||||
}
|
||||
rateLimit: {
|
||||
minDelayMs: number // Minimum delay between messages
|
||||
maxMessagesPerHour: number // Hourly message cap
|
||||
jitterMs: number // Random jitter range
|
||||
}
|
||||
browser: {
|
||||
headless: boolean // Headless mode
|
||||
userDataDir: string // Persistent session directory
|
||||
}
|
||||
database: {
|
||||
path: string // SQLite database path
|
||||
}
|
||||
mediaSync: {
|
||||
enabled: boolean // Enable media sync
|
||||
intervalMinutes: number // Sync interval
|
||||
}
|
||||
autoReply: {
|
||||
enabled: boolean // Enable auto-reply
|
||||
autoSend: boolean // Auto-send or pending approval
|
||||
checkInterval: number // Check interval in minutes
|
||||
}
|
||||
strategyVariants: {
|
||||
name: string // Variant name
|
||||
prompt: string // Message style prompt
|
||||
enabled: boolean // Enable/disable variant
|
||||
}[]
|
||||
web: {
|
||||
port: number // Dashboard port
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
### Runtime Configuration
|
||||
|
||||
The following settings can be modified via the web dashboard without restarting:
|
||||
|
||||
- `rateLimit.minDelayMs` - Message delay
|
||||
- `rateLimit.maxMessagesPerHour` - Hourly cap
|
||||
- `rateLimit.jitterMs` - Jitter range
|
||||
- `reddit.pollInterval` - Poll frequency
|
||||
- `dryRun` - Dry run mode (log only, no actual sends)
|
||||
|
||||
---
|
||||
|
||||
## Design Patterns
|
||||
|
||||
### Repository Pattern
|
||||
|
||||
All database operations are abstracted behind three repository classes:
|
||||
|
||||
- `PostsRepository` - Post lifecycle management
|
||||
- `MessagesRepository` - Message tracking and analytics
|
||||
- `SuccessfulMessagesRepository` - Few-shot learning data
|
||||
|
||||
Benefits:
|
||||
- Single source of truth for data access
|
||||
- Easy to swap implementations (e.g., different databases)
|
||||
- Consistent error handling and query patterns
|
||||
|
||||
### Dependency Injection
|
||||
|
||||
Services receive dependencies via constructor parameters:
|
||||
|
||||
```typescript
|
||||
class ProcessorService {
|
||||
constructor(
|
||||
private config: Config,
|
||||
private redditApi: RedditApi,
|
||||
private openrouter: OpenRouterService,
|
||||
private strategySelector: StrategySelector,
|
||||
private messageSender: MessagePage,
|
||||
private postsRepo: PostsRepository,
|
||||
private messagesRepo: MessagesRepository,
|
||||
private successfulRepo: SuccessfulMessagesRepository,
|
||||
private logger: Logger
|
||||
) {}
|
||||
}
|
||||
```
|
||||
|
||||
Benefits:
|
||||
- Loose coupling between components
|
||||
- Easy to mock dependencies for testing
|
||||
- Clear dependency graph
|
||||
|
||||
### Strategy Pattern
|
||||
|
||||
`StrategySelector` implements A/B testing with multiple message variants:
|
||||
|
||||
```typescript
|
||||
class StrategySelector {
|
||||
selectStrategy(subreddit: string, author: string): string {
|
||||
// Explore/exploit algorithm
|
||||
// Returns variant name based on response rates
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
Benefits:
|
||||
- Swappable message strategies
|
||||
- Measurable performance per variant
|
||||
- Configurable exploration rate
|
||||
|
||||
### Factory Pattern
|
||||
|
||||
Configuration and database creation use factory functions:
|
||||
|
||||
- `loadConfig()` - Loads, merges env vars, validates with Zod
|
||||
- `createDatabase()` - Initializes SQLite, runs migrations
|
||||
|
||||
### Observer Pattern
|
||||
|
||||
SSE (Server-Sent Events) for realtime log streaming:
|
||||
|
||||
```typescript
|
||||
// Logger writes to a shared buffer
|
||||
logger.info("Message sent to user123");
|
||||
|
||||
// WebServer subscribes clients via SSE
|
||||
app.get("/api/logs/stream", (req, res) => {
|
||||
// Push new log entries to connected clients
|
||||
});
|
||||
```
|
||||
|
||||
### Circuit Breaker / Retry
|
||||
|
||||
`withRetry()` wrapper for resilient external calls:
|
||||
|
||||
```typescript
|
||||
await withRetry(
|
||||
() => redditApi.fetchPosts(subreddit),
|
||||
{ retries: 3, logger }
|
||||
);
|
||||
```
|
||||
|
||||
Multiple fallback chains:
|
||||
- AI model fallback (primary → fallback models)
|
||||
- Image URL fallback (preview → i.redd.it → imgur → thumbnail)
|
||||
|
||||
### Rate Limiter Pattern
|
||||
|
||||
Token-bucket-like implementation:
|
||||
|
||||
```typescript
|
||||
class RateLimiter {
|
||||
async waitForSlot(): Promise<void> {
|
||||
// Enforce minimum delay + jitter
|
||||
// Track hourly counts
|
||||
// Block if hourly cap reached
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## External Integrations
|
||||
|
||||
### Reddit API
|
||||
|
||||
- **Authentication**: Playwright browser cookies (persistent session)
|
||||
- **Endpoints Used**:
|
||||
- `GET /r/{name}/new.json` - Fetch new posts
|
||||
- `GET /user/{username}/profile` - Check for seller/spam indicators
|
||||
- `GET /message/inbox` - Check for responses
|
||||
- `GET chat.reddit.com` - Chat scraping
|
||||
- **Rate Limits**: Enforced by `RateLimiter` utility
|
||||
|
||||
### OpenRouter AI
|
||||
|
||||
- **SDK**: OpenAI-compatible (`openai` npm package)
|
||||
- **Base URL**: `https://openrouter.ai/api/v1`
|
||||
- **Endpoints Used**:
|
||||
- `POST /chat/completions` - Image analysis, message generation, replies
|
||||
- **Model Fallback**: Primary model → fallback models on refusal
|
||||
- **Cost Optimization**: Quick gender detection before full analysis
|
||||
|
||||
### Playwright Browser
|
||||
|
||||
- **Context**: Persistent (cookies/session storage saved to disk)
|
||||
- **Usage**:
|
||||
- Automated Reddit login
|
||||
- DM compose page automation
|
||||
- Inbox navigation for response checking
|
||||
- Chat page scraping (shadow DOM traversal)
|
||||
- Media file downloads
|
||||
- **Shadow DOM**: JavaScript injection for `faceplate-*` component traversal
|
||||
|
||||
---
|
||||
|
||||
## Deployment
|
||||
|
||||
### Docker
|
||||
|
||||
The project includes full Docker support:
|
||||
|
||||
- **`Dockerfile`** - Multi-stage build with Node.js
|
||||
- **`docker-compose.yml`** - Service orchestration
|
||||
- **`.dockerignore`** - Excludes unnecessary files
|
||||
|
||||
### Dokploy
|
||||
|
||||
Deployment documentation is provided in `DOKPLOY_DEPLOY.md`.
|
||||
|
||||
### Required Environment Variables
|
||||
|
||||
| Variable | Required | Description |
|
||||
|---|---|---|
|
||||
| `OPENROUTER_API_KEY` | Yes | OpenRouter API key for AI |
|
||||
| `REDDIT_USERNAME` | Yes | Reddit username |
|
||||
| `REDDIT_PASSWORD` | Yes | Reddit password |
|
||||
| `BROWSER_HEADLESS` | No | Browser headless mode (default: `true`) |
|
||||
|
||||
### Pre-Deployment Checklist
|
||||
|
||||
1. Set up `config.yaml` with target subreddits and settings
|
||||
2. Set environment variables (or use `.env` file)
|
||||
3. Run `npm install` to install dependencies
|
||||
4. Run `npm run build` to compile TypeScript
|
||||
5. Run `scripts/login.ts` once to establish browser session
|
||||
6. Start the bot (via `npm start` or Docker)
|
||||
|
||||
---
|
||||
|
||||
## Technical Debt & Recommendations
|
||||
|
||||
### Critical
|
||||
|
||||
1. **No Tests** - Zero test coverage. Add unit tests for services, especially:
|
||||
- `StrategySelector` (complex algorithm)
|
||||
- `ChatService` (shadow DOM logic)
|
||||
- `OpenRouterService` (model fallback logic)
|
||||
|
||||
2. **No README.md** - No project documentation for new developers.
|
||||
|
||||
3. **Hardcoded Username** - `ChatService` has `myUsername = 'Upset_Marsupial1006'` hardcoded in JS evaluation scripts. Should be parameterized from config.
|
||||
|
||||
### High Priority
|
||||
|
||||
4. **God Class** - `main.ts` (391 lines) creates all dependencies and wires everything. Consider extracting an `AppContainer` or `DependencyContainer`.
|
||||
|
||||
5. **ChatService Size** - 1123+ lines. Consider splitting into:
|
||||
- `ChatScraper` (navigation + DOM traversal)
|
||||
- `ConversationCache` (cache management)
|
||||
- `ChatMessenger` (sending + verification)
|
||||
|
||||
6. **No .env.example** - No template showing required environment variables.
|
||||
|
||||
### Medium Priority
|
||||
|
||||
7. **Single-Page Dashboard** - `public/index.html` is ~2000+ lines with embedded CSS/JS. Consider a proper frontend framework.
|
||||
|
||||
8. **WebServer Size** - 794 lines. Consider splitting routes into separate files.
|
||||
|
||||
9. **Config Mutability** - Runtime config updates via `PUT /api/config` modify the in-memory config object. Consider a proper config service with validation on updates.
|
||||
|
||||
10. **Error Handling** - Some services have basic try/catch. Consider structured error types with retry logic.
|
||||
|
||||
### Low Priority
|
||||
|
||||
11. **Logging** - Winston is well-configured, but consider adding request IDs for tracing.
|
||||
|
||||
12. **Type Safety** - Some `page.evaluate()` calls use `any` types. Consider stricter typing for DOM structures.
|
||||
|
||||
13. **Media Storage** - Media files stored in `data/media/` with no cleanup policy. Consider retention policies.
|
||||
Reference in new issue
Block a user