Restructure chat service, web routes, and add test suite
Split monolithic chat service into modular structure with Matrix client support, extract web server into per-resource route modules, introduce bootstrap/main-loop separation, and add comprehensive vitest coverage across services, repositories, and utilities. Adds project docs and CLAUDE.md agent instructions.
This commit is contained in:
jjbrunton committed
1 parent
dfa0580859
commit
633bb0ec04
84 files changed
+19278
-3888
No files matched your search
@@ -0,0 +1,514 @@
|
||||
# Development Guide
|
||||
|
||||
Guide for contributing to RedditBot development.
|
||||
|
||||
---
|
||||
|
||||
## Table of Contents
|
||||
|
||||
1. [Project Structure](#project-structure)
|
||||
2. [Development Setup](#development-setup)
|
||||
3. [Coding Standards](#coding-standards)
|
||||
4. [Adding a New Service](#adding-a-new-service)
|
||||
5. [Adding a New Strategy Variant](#adding-a-new-strategy-variant)
|
||||
6. [Debugging](#debugging)
|
||||
7. [Common Tasks](#common-tasks)
|
||||
8. [Known Issues](#known-issues)
|
||||
|
||||
---
|
||||
|
||||
## Project Structure
|
||||
|
||||
```
|
||||
RedditBot/
|
||||
├── config.yaml # Main configuration
|
||||
├── package.json # Dependencies and scripts
|
||||
├── tsconfig.json # TypeScript config
|
||||
├── Dockerfile # Container build
|
||||
├── docker-compose.yml # Docker Compose
|
||||
├── public/
|
||||
│ └── index.html # Web dashboard (single file)
|
||||
├── scripts/
|
||||
│ ├── login.ts # Manual login helper
|
||||
│ └── inspect-chat-dom.ts # DOM inspection debug script
|
||||
└── src/
|
||||
├── index.ts # Entry point / orchestrator
|
||||
├── config/
|
||||
│ └── index.ts # Config loader + Zod schema
|
||||
├── database/
|
||||
│ ├── client.ts # SQLite setup + migrations
|
||||
│ └── repositories.ts # Repository classes
|
||||
├── reddit/
|
||||
│ └── api.ts # Reddit API client
|
||||
├── ai/
|
||||
│ └── openrouter.ts # OpenRouter AI client
|
||||
├── browser/
|
||||
│ ├── client.ts # Playwright browser manager
|
||||
│ ├── message-page.ts # DM compose automation
|
||||
│ └── media-downloader.ts # Media download
|
||||
├── services/
|
||||
│ ├── monitor.ts # Post discovery
|
||||
│ ├── processor.ts # Processing pipeline
|
||||
│ ├── response-checker.ts # Response monitoring
|
||||
│ ├── strategy-selector.ts # A/B testing
|
||||
│ ├── chat.ts # Chat scraping + messaging
|
||||
│ ├── media.ts # Media management
|
||||
│ ├── media-sync.ts # Media sync scheduler
|
||||
│ └── auto-reply.ts # Auto-reply service
|
||||
├── utils/
|
||||
│ ├── logger.ts # Winston logger
|
||||
│ ├── rate-limiter.ts # Rate limiting
|
||||
│ ├── retry.ts # Retry wrapper
|
||||
│ └── image.ts # Image utilities
|
||||
└── web/
|
||||
└── server.ts # Express dashboard server
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Development Setup
|
||||
|
||||
### 1. Install Dependencies
|
||||
|
||||
```bash
|
||||
npm install
|
||||
```
|
||||
|
||||
### 2. Configure
|
||||
|
||||
Copy and edit `config.yaml`:
|
||||
|
||||
```bash
|
||||
cp config.yaml config.yaml.backup # Backup existing
|
||||
# Edit config.yaml with your settings
|
||||
```
|
||||
|
||||
### 3. Set Environment Variables
|
||||
|
||||
```bash
|
||||
export REDDIT_USERNAME="your-username"
|
||||
export REDDIT_PASSWORD="your-password"
|
||||
export OPENROUTER_API_KEY="sk-or-v1-your-key"
|
||||
```
|
||||
|
||||
### 4. Initial Login
|
||||
|
||||
```bash
|
||||
# Open browser for manual login
|
||||
npm run login
|
||||
```
|
||||
|
||||
### 5. Start Development Server
|
||||
|
||||
```bash
|
||||
# Development mode (hot reload with tsx)
|
||||
npm run dev
|
||||
|
||||
# Production mode (compiled JS)
|
||||
npm run build && npm start
|
||||
```
|
||||
|
||||
### 6. Access Dashboard
|
||||
|
||||
Open http://localhost:3000
|
||||
|
||||
---
|
||||
|
||||
## Coding Standards
|
||||
|
||||
### TypeScript
|
||||
|
||||
- **Strict mode**: No `any` types (use `unknown` with type guards)
|
||||
- **Interfaces**: Define interfaces for all data structures
|
||||
- **Null checks**: Use optional chaining (`?.`) and nullish coalescing (`??`)
|
||||
- **Async/Await**: Use async/await, not `.then()` chains
|
||||
|
||||
```typescript
|
||||
// Good
|
||||
interface PostData {
|
||||
id: string;
|
||||
subreddit: string;
|
||||
author: string;
|
||||
}
|
||||
|
||||
async function fetchPost(id: string): Promise<PostData> {
|
||||
const response = await api.get(`/posts/${id}`);
|
||||
return response.data;
|
||||
}
|
||||
|
||||
// Bad
|
||||
async function fetchPost(id: any): Promise<any> {
|
||||
return api.get(`/posts/${id}`).then(r => r.data);
|
||||
}
|
||||
```
|
||||
|
||||
### Logging
|
||||
|
||||
Use `winston` logger, never `console.log`:
|
||||
|
||||
```typescript
|
||||
// Good
|
||||
this.logger.info("Processed post {id}", { id: post.id });
|
||||
this.logger.warn("AI refusal detected", { postId: post.id });
|
||||
this.logger.error("Failed to send DM", { error: err.message });
|
||||
|
||||
// Bad
|
||||
console.log("Processed post", post.id);
|
||||
```
|
||||
|
||||
### Error Handling
|
||||
|
||||
Use try/catch with structured error handling:
|
||||
|
||||
```typescript
|
||||
try {
|
||||
await this.processPost(post);
|
||||
} catch (err) {
|
||||
this.logger.error("Post processing failed", {
|
||||
postId: post.id,
|
||||
error: err instanceof Error ? err.message : String(err)
|
||||
});
|
||||
// Don't throw - allow other posts to be processed
|
||||
}
|
||||
```
|
||||
|
||||
### Service Structure
|
||||
|
||||
Follow single responsibility:
|
||||
|
||||
```typescript
|
||||
class NewService {
|
||||
constructor(
|
||||
private config: Config,
|
||||
private dependency: ExistingService,
|
||||
private logger: Logger
|
||||
) {}
|
||||
|
||||
async doSomething(input: string): Promise<Result> {
|
||||
// Validate input
|
||||
if (!input) {
|
||||
throw new Error("Input is required");
|
||||
}
|
||||
|
||||
// Do work
|
||||
const result = await this.dependency.process(input);
|
||||
|
||||
// Return result
|
||||
return result;
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
### File Organization
|
||||
|
||||
- One class per file (except repositories which share one file)
|
||||
- Services in `src/services/`
|
||||
- Utilities in `src/utils/`
|
||||
- External integrations in their own folders (`src/ai/`, `src/reddit/`, `src/browser/`)
|
||||
|
||||
---
|
||||
|
||||
## Adding a New Service
|
||||
|
||||
### 1. Create the Service File
|
||||
|
||||
```typescript
|
||||
// src/services/new-service.ts
|
||||
import { Config } from '../config';
|
||||
import { Logger } from '../utils/logger';
|
||||
|
||||
interface NewServiceDeps {
|
||||
config: Config;
|
||||
logger: Logger;
|
||||
// other dependencies
|
||||
}
|
||||
|
||||
interface Result {
|
||||
success: boolean;
|
||||
data?: any;
|
||||
error?: string;
|
||||
}
|
||||
|
||||
export class NewService {
|
||||
constructor(private deps: NewServiceDeps) {}
|
||||
|
||||
async execute(input: string): Promise<Result> {
|
||||
const { config, logger } = this.deps;
|
||||
|
||||
logger.info("NewService executing", { input });
|
||||
|
||||
// Implementation here
|
||||
|
||||
return { success: true };
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
### 2. Add to Dependencies Interface
|
||||
|
||||
If the service needs other services, add them to the interface:
|
||||
|
||||
```typescript
|
||||
interface NewServiceDeps {
|
||||
config: Config;
|
||||
logger: Logger;
|
||||
redditApi: RedditApi;
|
||||
openrouter: OpenRouterService;
|
||||
}
|
||||
```
|
||||
|
||||
### 3. Wire Up in main()
|
||||
|
||||
In `src/index.ts`, add initialization:
|
||||
|
||||
```typescript
|
||||
const newService = new NewService({
|
||||
config,
|
||||
logger,
|
||||
redditApi,
|
||||
openrouter
|
||||
});
|
||||
|
||||
// If it needs to run periodically:
|
||||
setInterval(() => {
|
||||
newService.execute(someInput).catch(err => {
|
||||
logger.error("NewService error", { error: err.message });
|
||||
});
|
||||
}, intervalMs);
|
||||
```
|
||||
|
||||
### 4. Add to Web Dashboard (Optional)
|
||||
|
||||
If the service needs API endpoints, add them in `src/web/server.ts`:
|
||||
|
||||
```typescript
|
||||
server.app.get('/api/new-service', async (req, res) => {
|
||||
try {
|
||||
const result = await newService.execute(req.query.input as string);
|
||||
res.json(result);
|
||||
} catch (err) {
|
||||
res.status(500).json({ error: (err as Error).message });
|
||||
}
|
||||
});
|
||||
```
|
||||
|
||||
### 5. Update Config (If Needed)
|
||||
|
||||
Add new configuration options to:
|
||||
- `config.yaml`
|
||||
- `src/config/index.ts` Zod schema
|
||||
- Configuration documentation
|
||||
|
||||
---
|
||||
|
||||
## Adding a New Strategy Variant
|
||||
|
||||
### 1. Add to config.yaml
|
||||
|
||||
```yaml
|
||||
strategyVariants:
|
||||
- name: "existing_variant"
|
||||
prompt: "Existing prompt..."
|
||||
enabled: true
|
||||
|
||||
- name: "new_variant"
|
||||
prompt: |
|
||||
Write a friendly message that references the image content.
|
||||
Be specific and genuine. Keep it under 2 sentences.
|
||||
enabled: true
|
||||
```
|
||||
|
||||
### 2. No Code Changes Needed
|
||||
|
||||
The `StrategySelector` automatically picks up new variants from the config. The algorithm will:
|
||||
|
||||
1. Include the new variant in random exploration
|
||||
2. Track its response rate once messages are sent
|
||||
3. Promote it to exploitation if it has good stats
|
||||
|
||||
### 3. Monitor Performance
|
||||
|
||||
Check performance via the dashboard or API:
|
||||
|
||||
```bash
|
||||
curl http://localhost:3000/api/analytics/by-strategy
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Debugging
|
||||
|
||||
### Browser Debugging
|
||||
|
||||
Run with visible browser:
|
||||
|
||||
```yaml
|
||||
# config.yaml
|
||||
browser:
|
||||
headless: false
|
||||
```
|
||||
|
||||
Or set environment variable:
|
||||
|
||||
```bash
|
||||
BROWSER_HEADLESS=false npm run dev
|
||||
```
|
||||
|
||||
### DOM Inspection
|
||||
|
||||
Use the debug endpoint to inspect Reddit's shadow DOM:
|
||||
|
||||
```bash
|
||||
curl -X POST http://localhost:3000/api/debug/inspect-dom
|
||||
```
|
||||
|
||||
Or run the standalone script:
|
||||
|
||||
```bash
|
||||
npx tsx scripts/inspect-chat-dom.ts
|
||||
```
|
||||
|
||||
### Logging
|
||||
|
||||
Adjust log level by modifying `src/utils/logger.ts`:
|
||||
|
||||
```typescript
|
||||
const logger = winston.createLogger({
|
||||
level: 'debug', // Change from 'info' to 'debug'
|
||||
// ...
|
||||
});
|
||||
```
|
||||
|
||||
### Database Inspection
|
||||
|
||||
Open the database with SQLite viewer:
|
||||
|
||||
```bash
|
||||
# Install sqlite3 if needed
|
||||
brew install sqlite3
|
||||
|
||||
# Open database
|
||||
sqlite3 data/reddit-bot.db
|
||||
|
||||
# Query
|
||||
SELECT * FROM sent_messages LIMIT 5;
|
||||
```
|
||||
|
||||
### Network Debugging
|
||||
|
||||
Enable axios debug:
|
||||
|
||||
```typescript
|
||||
// In src/reddit/api.ts
|
||||
import axios from 'axios';
|
||||
axios.defaults.debug = true;
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Common Tasks
|
||||
|
||||
### Add a New Subreddit
|
||||
|
||||
```yaml
|
||||
# config.yaml
|
||||
reddit:
|
||||
subreddits:
|
||||
- gonewild
|
||||
- ratemyboobs
|
||||
- new_subreddit_name # Add here
|
||||
```
|
||||
|
||||
### Change AI Model
|
||||
|
||||
```yaml
|
||||
# config.yaml
|
||||
openrouter:
|
||||
model: "new-model-name"
|
||||
fallbackModels:
|
||||
- "fallback-model-1"
|
||||
- "fallback-model-2"
|
||||
```
|
||||
|
||||
### Adjust Rate Limits
|
||||
|
||||
```yaml
|
||||
# config.yaml
|
||||
rateLimit:
|
||||
minDelayMs: 45000 # Increase delay
|
||||
maxMessagesPerHour: 15 # Reduce cap
|
||||
jitterMs: 20000 # More jitter
|
||||
```
|
||||
|
||||
### Enable Dry Run
|
||||
|
||||
```yaml
|
||||
# config.yaml
|
||||
dryRun: true # Logs all actions without sending
|
||||
```
|
||||
|
||||
Or via dashboard: `PUT /api/config` with `{"dryRun": true}`
|
||||
|
||||
### Add Database Column
|
||||
|
||||
```typescript
|
||||
// In src/database/client.ts
|
||||
function runMigrations(db: Database) {
|
||||
// ... existing migrations ...
|
||||
|
||||
// Check if column exists
|
||||
const tableInfo = db.prepare(`PRAGMA table_info(sent_messages)`).all();
|
||||
const hasNewColumn = tableInfo.some((col: any) => col.name === 'new_column');
|
||||
|
||||
if (!hasNewColumn) {
|
||||
db.exec("ALTER TABLE sent_messages ADD COLUMN new_column TEXT");
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
### Add New API Endpoint
|
||||
|
||||
```typescript
|
||||
// In src/web/server.ts
|
||||
import { Request, Response } from 'express';
|
||||
|
||||
server.app.get('/api/new-endpoint', async (req: Request, res: Response) => {
|
||||
try {
|
||||
const result = await someService.doSomething();
|
||||
res.json({ success: true, data: result });
|
||||
} catch (err) {
|
||||
res.status(500).json({ error: (err as Error).message });
|
||||
}
|
||||
});
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Known Issues
|
||||
|
||||
### Hardcoded Username in ChatService
|
||||
|
||||
`ChatService` has `myUsername = 'Upset_Marsupial1006'` hardcoded in JavaScript evaluation scripts. This should be parameterized from config.
|
||||
|
||||
**Fix needed**: Pass username to `page.evaluate()` calls.
|
||||
|
||||
### Shadow DOM Fragility
|
||||
|
||||
Reddit frequently updates their `faceplate-*` components. Shadow DOM traversal code may break without warning.
|
||||
|
||||
**Mitigation**: Use `scripts/inspect-chat-dom.ts` to debug when selectors change.
|
||||
|
||||
### No Test Coverage
|
||||
|
||||
The project has no automated tests. Any changes should be manually tested.
|
||||
|
||||
**Recommendation**: Add tests for critical paths:
|
||||
- StrategySelector algorithm
|
||||
- ChatService shadow DOM parsing
|
||||
- OpenRouterService model fallback
|
||||
|
||||
### Single-File Dashboard
|
||||
|
||||
The entire dashboard is in `public/index.html` (~2000+ lines). Making changes requires editing a single large file.
|
||||
|
||||
**Recommendation**: Consider migrating to a proper frontend framework for maintainability.
|
||||
Reference in new issue
Block a user