jjbrunton/RedditBot
1
0 Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

feat: persist settings and enrich conversation context

Browse Source
This commit is contained in:
jjbrunton committed 2026-08-08 18:18:28 +01:00
1 parent a36723a325
commit 2c100d72b6
84 files changed
+2248 -644
Download Patch File Download Diff File Expand all files Collapse all files

No files matched your search

docs/api.md
+2 -2
View File
@@ -761,8 +761,8 @@ GET /api/config
"jitterMs": 15000
},
"browser": {
"headless": true,
"userDataDir": "./data/browser"
"sessionDir": "./data/browser-session",
"dryRun": false
},
"database": {
"path": "./data/reddit-bot.db"
docs/architecture.md
+7 -19
View File
@@ -182,8 +182,7 @@ RedditBot/
├── ai/
│ └── openrouter.ts # OpenRouter AI client (image analysis, message gen) (307 lines)
├── browser/
│ ├── client.ts # Playwright persistent context + login (302 lines)
│ ├── message-page.ts # DM compose page automation + shadow DOM (560 lines)
│ ├── client.ts # Managed Playwright context + imported cookies
│ └── media-downloader.ts # Playwright-based media downloader (29 lines)
├── services/
│ ├── monitor.ts # Post discovery service (72 lines)
@@ -319,14 +318,7 @@ Playwright persistent context manager.
- Browser session persistence (cookies/storage)
- Login verification (multiple UI checks + API fallback)
- Automated login flow (form fill, 2FA/captcha detection)
- Debug screenshot capture on login failure
#### MessagePage (`src/browser/message-page.ts`)
Reddit DM compose page automation.
- Fills recipient/subject/body fields
- Extension-dump cookie import; no login form automation
- Extensive Shadow DOM support for `faceplate-*` components
- Detects recipient errors, verifies submission success
- Supports dry-run mode for testing
@@ -653,9 +645,7 @@ All endpoints are served by the Express server at `http://localhost:{port}`.
1. **`config.yaml`** - Primary configuration file (project root)
2. **Environment Variables** - Override specific fields:
- `OPENROUTER_API_KEY` - OpenRouter API key
- `REDDIT_USERNAME` - Reddit username
- `REDDIT_PASSWORD` - Reddit password
- `BROWSER_HEADLESS` - Browser headless mode (`true`/`false`)
- `WEB_DASHBOARD_TOKEN` - Dashboard access token
### Configuration Schema (Zod-Validated)
@@ -680,8 +670,8 @@ All endpoints are served by the Express server at `http://localhost:{port}`.
jitterMs: number // Random jitter range
}
browser: {
headless: boolean // Headless mode
userDataDir: string // Persistent session directory
sessionDir: string // Persistent imported session directory
dryRun: boolean // Prevent outbound delivery
}
database: {
path: string // SQLite database path
@@ -866,7 +856,7 @@ class RateLimiter {
The project includes full Docker support:
- **`Dockerfile`** - Playwright image with browser, VNC, and frontend/server build tooling
- **`Dockerfile`** - Playwright image with frontend/server build tooling; no VNC or interactive Chrome process
- **`docker-compose.yml`** - Service orchestration
- **`.dockerignore`** - Excludes unnecessary files
@@ -879,9 +869,7 @@ Deployment documentation is provided in `DOKPLOY_DEPLOY.md`.
| Variable | Required | Description |
|---|---|---|
| `OPENROUTER_API_KEY` | Yes | OpenRouter API key for AI |
| `REDDIT_USERNAME` | Yes | Reddit username |
| `REDDIT_PASSWORD` | Yes | Reddit password |
| `BROWSER_HEADLESS` | No | Browser headless mode (default: `true`) |
| `WEB_DASHBOARD_TOKEN` | Yes when dashboard enabled | Dashboard access token |
### Pre-Deployment Checklist
docs/configuration.md
+37 -40
View File
@@ -7,9 +7,9 @@ Complete reference for configuring RedditBot.
## Table of Contents
1. [Configuration Overview](#configuration-overview)
2. [Configuration Files](#configuration-files)
2. [Configuration Storage](#configuration-storage)
3. [Environment Variables](#environment-variables)
4. [config.yaml Reference](#configyaml-reference)
4. [Application Settings](#application-settings)
5. [Runtime Configuration](#runtime-configuration)
6. [Configuration Validation](#configuration-validation)
7. [Troubleshooting](#troubleshooting)
@@ -25,7 +25,7 @@ RedditBot uses a layered configuration approach:
│ Priority Order (High → Low) │
├──────────────────────────────────────────────────────────────┤
│ 1. Environment Variables │
│ (OPENROUTER_API_KEY, REDDIT_USERNAME, etc.) │
│ (OPENROUTER_API_KEY, WEB_DASHBOARD_TOKEN) │
│ │
│ 2. config.yaml │
│ (Primary configuration file) │
@@ -37,18 +37,20 @@ RedditBot uses a layered configuration approach:
Configuration is loaded at startup by `loadConfig()` in `src/config/index.ts`, which:
1. Reads `config.yaml` from the project root
2. Merges environment variable overrides
1. Reads the singleton `app_config` record from SQLite
2. Imports `config.yaml` only when upgrading an existing database that has no settings record
3. Validates the entire configuration tree using Zod
4. Returns a strongly-typed `Config` object
Application-owned settings are edited from the dashboard and saved to SQLite. `OPENROUTER_API_KEY`, `WEB_DASHBOARD_TOKEN`, and `DATABASE_PATH` remain environment-owned bootstrap values and are never stored in `app_config`.
---
## Configuration Files
## Configuration Storage
### config.yaml
### Legacy config.yaml
The primary configuration file, located at the project root:
`config.yaml` is retained temporarily as a one-time migration source for existing installations. Once every deployed database contains an `app_config` row, it can be removed in a follow-up release.
```
RedditBot/
@@ -79,14 +81,13 @@ data/
## Environment Variables
Environment variables override corresponding values in `config.yaml`.
Environment variables provide secrets and bootstrap settings that are intentionally excluded from SQLite.
| Variable | Required | Type | Description | Example |
|---|---|---|---|---|
| `OPENROUTER_API_KEY` | Yes | string | OpenRouter API key for AI services | `sk-or-v1-abc123...` |
| `REDDIT_USERNAME` | Yes | string | Reddit username for authentication | `MyBotUser` |
| `REDDIT_PASSWORD` | Yes | string | Reddit password for authentication | `secret123` |
| `BROWSER_HEADLESS` | No | boolean | Run browser in headless mode | `true` or `false` |
| `WEB_DASHBOARD_TOKEN` | Yes when dashboard enabled | string | Dashboard access token | `long-random-secret` |
| `DATABASE_PATH` | No | string | SQLite database location | `./data/reddit-bot.db` |
### Setting Environment Variables
@@ -96,9 +97,7 @@ Create a `.env` file in the project root:
```env
OPENROUTER_API_KEY=sk-or-v1-abc123...
REDDIT_USERNAME=MyBotUser
REDDIT_PASSWORD=secret123
BROWSER_HEADLESS=true
WEB_DASHBOARD_TOKEN=long-random-secret
```
> **Note**: Add `.env` to `.gitignore` - never commit credentials.
@@ -107,8 +106,7 @@ BROWSER_HEADLESS=true
```bash
export OPENROUTER_API_KEY=sk-or-v1-abc123...
export REDDIT_USERNAME=MyBotUser
export REDDIT_PASSWORD=secret123
export WEB_DASHBOARD_TOKEN=long-random-secret
```
**Option 3: Docker**
@@ -118,13 +116,16 @@ Pass via `docker-compose.yml` or Docker CLI:
```yaml
environment:
- OPENROUTER_API_KEY=sk-or-v1-abc123...
- REDDIT_USERNAME=MyBotUser
- REDDIT_PASSWORD=secret123
- WEB_DASHBOARD_TOKEN=long-random-secret
```
---
## config.yaml Reference
## Application Settings
Use **Settings** in the dashboard for monitoring, rate limits, dry-run behaviour, subreddits, models, prompts, filtering, media sync, and strategy configuration. The advanced JSON editor exposes settings that do not yet have dedicated controls; its payload is validated by the server before it is saved.
The YAML examples below document the legacy field layout used during one-time migration.
Complete reference for every configuration option.
@@ -235,13 +236,9 @@ Max messages: 20 per hour
```yaml
browser:
# Run browser in headless mode (no UI)
# Set to false for debugging
headless: true
# Directory for persistent browser data
# Cookies and session storage are saved here
userDataDir: "./data/browser"
# Persistent managed-browser data populated only by verified imports
sessionDir: "./data/browser-session"
dryRun: false
```
**Browser session persistence:**
@@ -412,8 +409,8 @@ All configuration is validated using Zod at startup. Invalid configuration will
| `rateLimit.minDelayMs` | Must be positive integer |
| `rateLimit.maxMessagesPerHour` | Must be positive integer |
| `rateLimit.jitterMs` | Must be non-negative integer |
| `browser.headless` | Must be boolean |
| `browser.userDataDir` | Must be non-empty string |
| `browser.sessionDir` | Must be non-empty string |
| `browser.dryRun` | Must be boolean |
| `database.path` | Must be non-empty string |
| `mediaSync.enabled` | Must be boolean |
| `mediaSync.intervalMinutes` | Must be positive integer (if enabled) |
@@ -432,8 +429,8 @@ All configuration is validated using Zod at startup. Invalid configuration will
**Error**: `Invalid config: reddit.subreddits must be a non-empty array`
**Fix**: Add at least one subreddit to `reddit.subreddits` in `config.yaml`
**Error**: `Invalid config: browser.userDataDir must be a string`
**Fix**: Ensure `browser.userDataDir` is set in `config.yaml`
**Error**: `Invalid config: browser.sessionDir must be a string`
**Fix**: Ensure `browser.sessionDir` is set in `config.yaml`
---
@@ -449,15 +446,15 @@ All configuration is validated using Zod at startup. Invalid configuration will
3. Verify environment variables are set correctly
4. Check bot logs for specific validation errors
### Credentials Not Working
### Account Authentication Not Working
**Symptom**: Browser login fails.
**Symptom**: Account work remains paused.
**Steps**:
1. Verify `REDDIT_USERNAME` and `REDDIT_PASSWORD` are correct
2. Check that the account is not locked or requires 2FA
3. Try manual login via `npm run login`
4. Ensure `BROWSER_HEADLESS=false` for debugging login issues
1. Sign in to Reddit in local Chrome
2. Open Reddit Chat and wait for the extension badge to show `OK`
3. Export and upload a fresh combined session dump
4. Check dashboard logs for Reddit or Matrix identity verification errors
### AI Not Working
@@ -475,9 +472,9 @@ All configuration is validated using Zod at startup. Invalid configuration will
**Symptom**: Bot stops with "Browser session expired" error.
**Steps**:
1. Delete `data/browser/` directory
2. Run `npm run login` to re-authenticate
3. Restart the bot
1. Open Reddit Chat in the local Chrome profile containing the extension
2. Export and upload a fresh session dump
3. Delete the downloaded dump after verification
### Port Already in Use
docs/data-flow.md
+5 -12
View File
@@ -791,21 +791,14 @@ The historical diagram below is retained for migration context only.
└─► Continue to next subreddit
```
### Browser Session Recovery
### Account Session Recovery
```
1. BrowserClient.verifySession() returns false
1. Reddit or Matrix authentication is rejected
│
├─► Check for expired cookies
│
├─► Attempt automated re-login
│ ├─► BrowserClient.login()
│ └─► If success: continue
│
└─► If re-login fails:
├─► Log error: "Browser session expired"
├─► Stop bot
└─► Require manual login (scripts/login.ts)
├─► Invalidate the shared account-auth gate
├─► Stop account-bound schedulers
└─► Wait for a fresh combined dump from the Chrome extension
```
---
docs/deployment.md
+22 -42
View File
@@ -74,8 +74,8 @@ rateLimit:
jitterMs: 15000
browser:
headless: true
userDataDir: "./data/browser"
sessionDir: "./data/browser-session"
dryRun: true
database:
path: "./data/reddit-bot.db"
@@ -89,23 +89,13 @@ dryRun: true # Start in dry run mode
Set environment variables:
```bash
export REDDIT_USERNAME="your-username"
export REDDIT_PASSWORD="your-password"
export OPENROUTER_API_KEY="sk-or-v1-your-key"
export WEB_DASHBOARD_TOKEN="long-random-secret"
```
### 3. Initial Login
### 3. Import account session
Run the login script to establish a browser session:
```bash
npm run login
```
This will:
1. Open a browser window
2. Navigate to Reddit login
3. Wait for you to complete login (including 2FA if enabled)
4. Save cookies to `data/browser/`
Start the dashboard, then upload a combined Reddit and Matrix dump created by the local Chrome extension. Until both identities verify, account work remains paused.
### 4. Start the Bot
@@ -144,23 +134,19 @@ docker run -d \
-p 3000:3000 \
-v $(pwd)/data:/app/data \
-e OPENROUTER_API_KEY=sk-or-v1-your-key \
-e REDDIT_USERNAME=your-username \
-e REDDIT_PASSWORD=your-password \
-e BROWSER_HEADLESS=true \
-e WEB_DASHBOARD_TOKEN=long-random-secret \
reddit-bot:latest
```
### Dockerfile behavior
The image is based on the Playwright Jammy image rather than Alpine because the
runtime needs Chromium and its system libraries. It installs Xvfb, x11vnc,
noVNC, Fluxbox, and Websockify for interactive Reddit login; installs the locked
Node dependencies; copies the repository; and runs `npm run build`.
The image is based on the Playwright Jammy image because the remaining
authenticated browser operations need Chromium and its system libraries. It
does not install a display server, VNC, or consumer Google Chrome.
That build produces both `dist/` for the Express server and
`apps/dashboard/dist/` for Vite assets. Ports 3000 (dashboard/API) and 6080
(noVNC) are exposed, and `docker/entrypoint.sh` starts the display services and
application. This is a single-stage image and retains development dependencies
`apps/dashboard/dist/` for Vite assets. Port 3000 serves the dashboard/API and
the container starts the Node process directly. This is a single-stage image and retains development dependencies
needed by the current build/runtime layout.
---
@@ -182,9 +168,7 @@ services:
- ./data:/app/data
environment:
- OPENROUTER_API_KEY=${OPENROUTER_API_KEY}
- REDDIT_USERNAME=${REDDIT_USERNAME}
- REDDIT_PASSWORD=${REDDIT_PASSWORD}
- BROWSER_HEADLESS=true
- WEB_DASHBOARD_TOKEN=${WEB_DASHBOARD_TOKEN}
restart: unless-stopped
healthcheck:
test: ["CMD", "wget", "--spider", "http://localhost:3000/api/stats"]
@@ -225,9 +209,7 @@ Dokploy is a self-hosted platform for deploying applications. See `DOKPLOY_DEPLO
1. **Create Application**: In Dokploy dashboard, create a new Docker application
2. **Set Environment Variables**:
- `OPENROUTER_API_KEY`
- `REDDIT_USERNAME`
- `REDDIT_PASSWORD`
- `BROWSER_HEADLESS`
- `WEB_DASHBOARD_TOKEN`
3. **Mount Volume**: Mount `./data` for persistent storage
4. **Set Port**: Map container port 3000 to host port
5. **Deploy**: Push to repository and trigger deployment
@@ -244,9 +226,7 @@ app:
environment:
OPENROUTER_API_KEY: $OPENROUTER_API_KEY
REDDIT_USERNAME: $REDDIT_USERNAME
REDDIT_PASSWORD: $REDDIT_PASSWORD
BROWSER_HEADLESS: "true"
WEB_DASHBOARD_TOKEN: $WEB_DASHBOARD_TOKEN
ports:
- container: 3000
@@ -276,7 +256,7 @@ app:
### Performance
1. **Browser Headless**: Always use headless mode in production
1. **Account dumps**: Delete downloaded dumps after the dashboard verifies them
2. **WAL Mode**: SQLite WAL mode is enabled by default for concurrent reads
3. **Rate Limiting**: Adjust based on Reddit account age and trust level
4. **Message Volume**: Start low (10-20/hour) and increase gradually
@@ -359,15 +339,15 @@ docker compose start
3. Verify environment variables are set
4. Check for Zod validation errors in logs
### Browser Login Fails
### Account Import Fails
**Symptom**: "Browser session expired" or login fails.
**Symptom**: Account work stays paused or import verification fails.
**Steps**:
1. Delete browser data: `rm -rf data/browser/`
2. Re-run login: `npm run login`
3. Check Reddit account is not locked
4. Try `BROWSER_HEADLESS=false` for debugging
1. Sign in to Reddit in local Chrome
2. Open Reddit Chat and wait for the extension badge to show `OK`
3. Export and upload a new combined session dump
4. Check logs for Reddit or Matrix identity errors
### AI Analysis Fails
docs/development.md
+6 -23
View File
@@ -45,8 +45,7 @@ RedditBot/
├── ai/
│ └── openrouter.ts # OpenRouter AI client
├── browser/
│ ├── client.ts # Playwright browser manager
│ ├── message-page.ts # DM compose automation
│ ├── client.ts # Managed Playwright context
│ └── media-downloader.ts # Media download
├── services/
│ ├── monitor.ts # Post discovery
@@ -87,17 +86,13 @@ cp config.yaml config.yaml.backup # Backup existing
### 3. Set Environment Variables
```bash
export REDDIT_USERNAME="your-username"
export REDDIT_PASSWORD="your-password"
export OPENROUTER_API_KEY="sk-or-v1-your-key"
export WEB_DASHBOARD_TOKEN="long-random-secret"
```
### 4. Initial Login
### 4. Account authentication
```bash
# Open browser for manual login
npm run login
```
Start the app, then import a combined Reddit and Matrix session dump from the local Chrome extension under **Settings > Access**.
### 5. Start Development Server
@@ -357,21 +352,9 @@ curl http://localhost:3000/api/analytics/by-strategy
## Debugging
### Browser Debugging
### Account debugging
Run with visible browser:
```yaml
# config.yaml
browser:
headless: false
```
Or set environment variable:
```bash
BROWSER_HEADLESS=false npm run dev
```
Use dashboard session status and structured logs. The backend does not expose or launch an interactive login browser.
### DOM Inspection
docs/services.md
+8 -81
View File
@@ -690,7 +690,7 @@ Generate a similar message for this image..."
### Purpose
Manages the Playwright browser session with persistent cookies/storage. Handles automated login, session verification, and 2FA/captcha detection.
Manages the headless Playwright context used after an extension dump is imported. It accepts Reddit cookies, verifies account identity, and supplies authenticated context to the remaining browser-backed operations. It never performs login.
### Public Methods
@@ -704,8 +704,8 @@ class BrowserClient {
/** Verify browser session is active */
verifySession(): Promise<boolean>
/** Perform automated login */
login(): Promise<void>
/** Import Reddit cookies from a validated extension dump */
importRedditCookies(cookies: Cookie[]): Promise<void>
/** Get page context for operations */
getPage(): Promise<Page>
@@ -715,93 +715,20 @@ class BrowserClient {
}
```
### Login Flow
### Import Flow
```
1. Navigate to Reddit login page
2. Fill username and password
3. Submit form
4. Check for 2FA prompt
└─► If present: wait for manual input (with timeout)
5. Check for captcha
└─► If present: log warning, wait
6. Verify login success:
└─► Check for user menu indicator
└─► Check for profile avatar
└─► Fallback: API request to /api/me
7. Save cookies to persistent directory
8. On failure: capture debug screenshot
```
The dashboard validates a combined extension dump, verifies Matrix with `whoami`, imports Reddit cookies, verifies Reddit identity, and only then marks the shared account gate authenticated.
### Session Persistence
- Cookies and storage are saved to `config.browser.userDataDir`
- Subsequent launches restore the session automatically
- Session expires when Reddit cookies expire (requires re-login)
- Session expiry pauses account work until a fresh extension dump is imported
### Dependencies
- `Config` - Credentials, headless mode, user data dir
- `Logger` - Login progress, errors
---
## MessagePage
**File**: `src/browser/message-page.ts`
**Lines**: 560
**Responsibility**: Reddit DM compose page automation
### Purpose
Automates the Reddit DM compose page: filling recipient/subject/body fields, handling shadow DOM components, submitting messages, and verifying success.
### Public Methods
```typescript
class MessagePage {
constructor(
private browser: BrowserClient,
private config: Config,
private logger: Logger
)
/** Send a DM to a recipient */
sendDM(recipient: string, subject: string, body: string): Promise<boolean>
/** Check if recipient field shows an error */
hasRecipientError(page: Page): Promise<boolean>
}
```
### Shadow DOM Handling
Reddit's compose form uses `faceplate-*` web components:
```javascript
// Fill shadow DOM input fields
page.evaluate(({ selector, value }) => {
const shadowInput = document.querySelector(`${selector} input`);
if (shadowInput) {
shadowInput.value = value;
shadowInput.dispatchEvent(new Event('input', { bubbles: true }));
}
}, { selector: 'faceplate-textarea[name="body"]', value: body });
```
### Dry Run Mode
When `config.dryRun` is enabled:
- All fields are filled as normal
- Submit button is NOT clicked
- Message is logged but not sent
- Returns `true` (simulates success)
### Dependencies
- `BrowserClient` - Playwright page access
- `Config` - Dry run mode
- `Logger` - Send activity
- `Config` - Persistent session directory
- `Logger` - Import verification progress and errors
---