docs: add agent codebase guide
This commit is contained in:
jjbrunton committed
1 parent
742c06d0c4
commit
c630303b47
12 files changed
+881
-4483
No files matched your search
+84
-470
@@ -1,510 +1,124 @@
|
||||
# Development Guide
|
||||
# Development and testing
|
||||
|
||||
Guide for contributing to RedditBot development.
|
||||
## Prerequisites
|
||||
|
||||
---
|
||||
- Node.js compatible with the lockfile/native `better-sqlite3` build.
|
||||
- npm.
|
||||
- Playwright system/browser dependencies for browser-backed tests or runtime work.
|
||||
- No live account is required for unit, contract, route, or dashboard tests.
|
||||
|
||||
## Table of Contents
|
||||
NodeNext imports use `.js` suffixes even when the source file is `.ts`. Vitest globals are disabled, so tests import `describe`, `it`, `expect`, and hooks explicitly.
|
||||
|
||||
1. [Project Structure](#project-structure)
|
||||
2. [Development Setup](#development-setup)
|
||||
3. [Coding Standards](#coding-standards)
|
||||
4. [Adding a New Service](#adding-a-new-service)
|
||||
5. [Adding a New Strategy Variant](#adding-a-new-strategy-variant)
|
||||
6. [Debugging](#debugging)
|
||||
7. [Common Tasks](#common-tasks)
|
||||
8. [Known Issues](#known-issues)
|
||||
|
||||
---
|
||||
|
||||
## Project Structure
|
||||
|
||||
```
|
||||
RedditBot/
|
||||
├── config.yaml # Main configuration
|
||||
├── package.json # Dependencies and scripts
|
||||
├── tsconfig.json # TypeScript config
|
||||
├── Dockerfile # Container build
|
||||
├── docker-compose.yml # Docker Compose
|
||||
├── apps/
|
||||
│ └── dashboard/ # React + Vite operator console
|
||||
├── packages/
|
||||
│ └── api-contracts/ # Shared Zod transport contracts
|
||||
├── scripts/
|
||||
│ ├── login.ts # Manual login helper
|
||||
│ └── inspect-chat-dom.ts # DOM inspection debug script
|
||||
└── src/
|
||||
├── index.ts # Entry point / orchestrator
|
||||
├── config/
|
||||
│ └── index.ts # Config loader + Zod schema
|
||||
├── database/
|
||||
│ ├── client.ts # SQLite setup + migrations
|
||||
│ └── repositories.ts # Repository classes
|
||||
├── reddit/
|
||||
│ └── api.ts # Reddit API client
|
||||
├── ai/
|
||||
│ └── openrouter.ts # OpenRouter AI client
|
||||
├── browser/
|
||||
│ ├── client.ts # Managed Playwright context
|
||||
│ └── media-downloader.ts # Media download
|
||||
├── services/
|
||||
│ ├── monitor.ts # Post discovery
|
||||
│ ├── processor.ts # Processing pipeline
|
||||
│ ├── response-checker.ts # Response monitoring
|
||||
│ ├── strategy-selector.ts # A/B testing
|
||||
│ ├── chat.ts # Chat scraping + messaging
|
||||
│ ├── media.ts # Media management
|
||||
│ ├── media-sync.ts # Media sync scheduler
|
||||
│ └── auto-reply.ts # Auto-reply service
|
||||
├── utils/
|
||||
│ ├── logger.ts # Winston logger
|
||||
│ ├── rate-limiter.ts # Rate limiting
|
||||
│ ├── retry.ts # Retry wrapper
|
||||
│ └── image.ts # Image utilities
|
||||
└── web/ # Express API, events, and dashboard hosting
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Development Setup
|
||||
|
||||
### 1. Install Dependencies
|
||||
## Safe setup
|
||||
|
||||
```bash
|
||||
npm install
|
||||
npm ci
|
||||
npm run build
|
||||
npm test
|
||||
```
|
||||
|
||||
### 2. Configure
|
||||
Do not start `npm run dev` merely to verify a build. The repository `.env` and persisted database can make the process live-capable. Starting the backend, importing sessions, or toggling the bot to running requires explicit authorization and a confirmed dry-run/live intent.
|
||||
|
||||
Copy and edit `config.yaml`:
|
||||
If native SQLite bindings fail after a Node upgrade, distinguish pure TypeScript/unit results from database verification. Rebuild/install a compatible `better-sqlite3`; `--ignore-scripts` is not a valid database test environment.
|
||||
|
||||
```bash
|
||||
cp config.yaml config.yaml.backup # Backup existing
|
||||
# Edit config.yaml with your settings
|
||||
```
|
||||
## Commands
|
||||
|
||||
### 3. Set Environment Variables
|
||||
| Command | Purpose |
|
||||
| --- | --- |
|
||||
| `npm run build` | Compile server and production dashboard. |
|
||||
| `npm run build:server` | Run backend TypeScript compiler. |
|
||||
| `npm run dashboard:typecheck` | Type-check the React application. |
|
||||
| `npm run dashboard:build` | Build dashboard assets. |
|
||||
| `npm test` | Run the complete Vitest suite once. |
|
||||
| `npm run test:watch` | Watch tests during local development. |
|
||||
| `npm run test:coverage` | Run with coverage. |
|
||||
| `npm run docs:api` | Regenerate `docs/api.md`. |
|
||||
| `npm run docs:api:check` | Fail if generated API documentation is stale. |
|
||||
| `npm run mcp` | Start the stdio MCP adapter; requires an authorized running API. |
|
||||
| `npm run dev` | Start the live-capable backend watcher. |
|
||||
| `npm run dashboard:dev` | Start Vite, proxying API requests to port 3000. |
|
||||
|
||||
```bash
|
||||
export OPENROUTER_API_KEY="sk-or-v1-your-key"
|
||||
export WEB_DASHBOARD_TOKEN="long-random-secret"
|
||||
```
|
||||
## Change workflow
|
||||
|
||||
### 4. Account authentication
|
||||
1. Read `agents.md`, `docs/README.md`, the routed topic docs, owning source, and focused tests.
|
||||
2. Inspect `git status` before editing. Preserve unrelated user changes and do not broadly stage a dirty worktree.
|
||||
3. Establish a baseline appropriate to the change. Code-changing tasks follow the repository test-first instructions; docs-only changes use documentation/link checks.
|
||||
4. Identify the source-of-truth boundary: config schema, migration, service, shared transport contract, route, or UI feature.
|
||||
5. Implement the smallest coherent behavior using dependency injection and single-purpose modules.
|
||||
6. Add behavior-focused tests for observable inputs, outputs, state transitions, and side effects.
|
||||
7. Update every affected document and regenerate API docs when the catalog changes.
|
||||
8. Run focused tests while iterating, then full required checks before publishing.
|
||||
9. Review the exact staged diff, commit only the coherent set, push, and verify branch/remote sync.
|
||||
|
||||
Start the app, then import a combined Reddit and Matrix session dump from the local Chrome extension under **Settings > Access**.
|
||||
## Engineering practices
|
||||
|
||||
### 5. Start Development Server
|
||||
### Types and validation
|
||||
|
||||
```bash
|
||||
# Terminal 1: backend reload with tsx
|
||||
npm run dev
|
||||
- No production `any`; use `unknown`, Zod, discriminated unions, or type guards.
|
||||
- Keep database/service-private types separate from transport contracts.
|
||||
- Parse at boundaries and pass trusted domain values inward.
|
||||
- Return explicit terminal/retryable/human-required results where callers must decide policy.
|
||||
|
||||
# Terminal 2: Vite frontend with /api proxied to port 3000
|
||||
npm run dashboard:dev
|
||||
### Service design
|
||||
|
||||
# Production build: server plus apps/dashboard/dist
|
||||
npm run build && npm start
|
||||
```
|
||||
|
||||
### 6. Access Dashboard
|
||||
|
||||
During frontend development, open the URL printed by Vite (normally
|
||||
http://localhost:5173). The Express dashboard remains on http://localhost:3000
|
||||
for production/fallback verification.
|
||||
|
||||
Useful frontend checks:
|
||||
|
||||
```bash
|
||||
npm run dashboard:typecheck
|
||||
npm run dashboard:build
|
||||
npx vitest run tests/dashboard
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Coding Standards
|
||||
|
||||
### TypeScript
|
||||
|
||||
- **Strict mode**: No `any` types (use `unknown` with type guards)
|
||||
- **Interfaces**: Define interfaces for all data structures
|
||||
- **Null checks**: Use optional chaining (`?.`) and nullish coalescing (`??`)
|
||||
- **Async/Await**: Use async/await, not `.then()` chains
|
||||
|
||||
```typescript
|
||||
// Good
|
||||
interface PostData {
|
||||
id: string;
|
||||
subreddit: string;
|
||||
author: string;
|
||||
}
|
||||
|
||||
async function fetchPost(id: string): Promise<PostData> {
|
||||
const response = await api.get(`/posts/${id}`);
|
||||
return response.data;
|
||||
}
|
||||
|
||||
// Bad
|
||||
async function fetchPost(id: any): Promise<any> {
|
||||
return api.get(`/posts/${id}`).then(r => r.data);
|
||||
}
|
||||
```
|
||||
- Constructor-inject external dependencies, clocks, fetchers, schedulers, and persistence seams.
|
||||
- Keep process-global coordination separate from account-scoped mutable state.
|
||||
- Prefer a small policy/service over another conditional in a composition root or route.
|
||||
- Preserve abort signals through queues, sleeps, and remote operations.
|
||||
|
||||
### Logging
|
||||
|
||||
Use `winston` logger, never `console.log`:
|
||||
Use the injected Winston logger. Include stage, account context (through child loggers), post/conversation IDs, and safe rejection/retry reasons. Never serialize raw Axios/Playwright errors that may include cookies or authorization headers; extract a safe message/status. Avoid high-frequency unchanged status logs.
|
||||
|
||||
```typescript
|
||||
// Good
|
||||
this.logger.info("Processed post {id}", { id: post.id });
|
||||
this.logger.warn("AI refusal detected", { postId: post.id });
|
||||
this.logger.error("Failed to send DM", { error: err.message });
|
||||
### Remote effects
|
||||
|
||||
// Bad
|
||||
console.log("Processed post", post.id);
|
||||
```
|
||||
- Check dry-run at the final side-effect boundary.
|
||||
- Make retries idempotent and distinguish remote acceptance from local persistence.
|
||||
- Reserve recipients/claims before sending and release only on safe failure paths.
|
||||
- Treat 403, 404, and 429 according to their typed domain meaning rather than one generic retry.
|
||||
|
||||
### Error Handling
|
||||
## Test layout and patterns
|
||||
|
||||
Use try/catch with structured error handling:
|
||||
Tests live in `tests/`; dashboard tests are under `tests/dashboard/`, route/event tests under `tests/web/`, and shared contract tests under `packages/api-contracts/tests/`.
|
||||
|
||||
```typescript
|
||||
try {
|
||||
await this.processPost(post);
|
||||
} catch (err) {
|
||||
this.logger.error("Post processing failed", {
|
||||
postId: post.id,
|
||||
error: err instanceof Error ? err.message : String(err)
|
||||
});
|
||||
// Don't throw - allow other posts to be processed
|
||||
}
|
||||
```
|
||||
Preferred patterns:
|
||||
|
||||
### Service Structure
|
||||
- in-memory SQLite with real migrations/repositories;
|
||||
- fake clock/sleep/fetch functions injected into services;
|
||||
- small service fakes implementing narrow interfaces;
|
||||
- Testing Library for user-observable React behavior;
|
||||
- representative payload parsing for strict Zod contracts;
|
||||
- lifecycle/restart tests for queues, claims, and account runtimes.
|
||||
|
||||
Follow single responsibility:
|
||||
Avoid deleting coverage to make a suite pass, assertions that pin private call order without behavioral value, ESM namespace mocking when injection is available, or tests that read the real `.env`/production data.
|
||||
|
||||
```typescript
|
||||
class NewService {
|
||||
constructor(
|
||||
private config: Config,
|
||||
private dependency: ExistingService,
|
||||
private logger: Logger
|
||||
) {}
|
||||
## Verification matrix
|
||||
|
||||
async doSomething(input: string): Promise<Result> {
|
||||
// Validate input
|
||||
if (!input) {
|
||||
throw new Error("Input is required");
|
||||
}
|
||||
| Change | Minimum focused verification | Final verification |
|
||||
| --- | --- | --- |
|
||||
| Pure backend/service | owning Vitest files | `npm test`, `npm run build:server` |
|
||||
| Database/migration | migration + repository tests | full tests and server build |
|
||||
| API/contract | route + contract + auth tests; regenerate docs | full tests, build, `docs:api:check` |
|
||||
| Dashboard | feature tests + typecheck | full tests and full build |
|
||||
| Docker/deployment | compose/Docker tests + image build as appropriate | health, API, container, logs, and persistence after authorized deploy |
|
||||
| Documentation only | generated-doc check + internal link scan | staged diff review |
|
||||
|
||||
// Do work
|
||||
const result = await this.dependency.process(input);
|
||||
If the worktree has pre-existing failures or concurrent changes, record the exact baseline and do not attribute them to your change. Never hide a failed required check behind a passing focused test.
|
||||
|
||||
// Return result
|
||||
return result;
|
||||
}
|
||||
}
|
||||
```
|
||||
## Common implementation recipes
|
||||
|
||||
### File Organization
|
||||
### Add an endpoint
|
||||
|
||||
- One class per file (except repositories which share one file)
|
||||
- Services in `src/services/`
|
||||
- Utilities in `src/utils/`
|
||||
- External integrations in their own folders (`src/ai/`, `src/reddit/`, `src/browser/`)
|
||||
Create or extend a focused router, validate input, add shared output contracts, register metadata in `api-catalog.ts`, add authorization/behavior tests, consume via dashboard query helpers if needed, regenerate `docs/api.md`, and update the relevant narrative doc.
|
||||
|
||||
---
|
||||
### Add persisted state
|
||||
|
||||
## Adding a New Service
|
||||
Append a migration, add a narrow repository, wire it through the composition root/runtime, test fresh and restart/upgrade cases, and update `database.md`.
|
||||
|
||||
### 1. Create the Service File
|
||||
### Change candidate or reply behavior
|
||||
|
||||
```typescript
|
||||
// src/services/new-service.ts
|
||||
import { Config } from '../config';
|
||||
import { Logger } from '../utils/logger';
|
||||
Start with the durable state transition and idempotency rule. Test rejection/retry/success, process interruption, account ownership, and operator-visible reason counts. Update product, flow, service, and data docs as applicable.
|
||||
|
||||
interface NewServiceDeps {
|
||||
config: Config;
|
||||
logger: Logger;
|
||||
// other dependencies
|
||||
}
|
||||
### Add dashboard state
|
||||
|
||||
interface Result {
|
||||
success: boolean;
|
||||
data?: any;
|
||||
error?: string;
|
||||
}
|
||||
|
||||
export class NewService {
|
||||
constructor(private deps: NewServiceDeps) {}
|
||||
|
||||
async execute(input: string): Promise<Result> {
|
||||
const { config, logger } = this.deps;
|
||||
|
||||
logger.info("NewService executing", { input });
|
||||
|
||||
// Implementation here
|
||||
|
||||
return { success: true };
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
### 2. Add to Dependencies Interface
|
||||
|
||||
If the service needs other services, add them to the interface:
|
||||
|
||||
```typescript
|
||||
interface NewServiceDeps {
|
||||
config: Config;
|
||||
logger: Logger;
|
||||
redditApi: RedditApi;
|
||||
openrouter: OpenRouterService;
|
||||
}
|
||||
```
|
||||
|
||||
### 3. Wire Up in main()
|
||||
|
||||
In `src/index.ts`, add initialization:
|
||||
|
||||
```typescript
|
||||
const newService = new NewService({
|
||||
config,
|
||||
logger,
|
||||
redditApi,
|
||||
openrouter
|
||||
});
|
||||
|
||||
// If it needs to run periodically:
|
||||
setInterval(() => {
|
||||
newService.execute(someInput).catch(err => {
|
||||
logger.error("NewService error", { error: err.message });
|
||||
});
|
||||
}, intervalMs);
|
||||
```
|
||||
|
||||
### 4. Add to the Web API and Dashboard (Optional)
|
||||
|
||||
If the service needs an API, add a focused router under `src/web/routes/` and
|
||||
mount it from `src/web/server.ts`. Define its transport schema in
|
||||
`packages/api-contracts`; do not expose repository rows directly.
|
||||
|
||||
```typescript
|
||||
router.get('/new-service', async (req, res) => {
|
||||
try {
|
||||
const result = await newService.execute(req.query.input as string);
|
||||
res.json(result);
|
||||
} catch (err) {
|
||||
res.status(500).json({ error: (err as Error).message });
|
||||
}
|
||||
});
|
||||
```
|
||||
|
||||
Add frontend behavior under `apps/dashboard/src/features/<feature>/` and drive
|
||||
it with Testing Library and MSW tests under `tests/dashboard/`.
|
||||
|
||||
### 5. Update Config (If Needed)
|
||||
|
||||
Add new configuration options to:
|
||||
- `config.yaml`
|
||||
- `src/config/index.ts` Zod schema
|
||||
- Configuration documentation
|
||||
|
||||
---
|
||||
|
||||
## Adding a New Strategy Variant
|
||||
|
||||
### 1. Add to config.yaml
|
||||
|
||||
```yaml
|
||||
strategyVariants:
|
||||
- name: "existing_variant"
|
||||
prompt: "Existing prompt..."
|
||||
enabled: true
|
||||
|
||||
- name: "new_variant"
|
||||
prompt: |
|
||||
Write a friendly message that references the image content.
|
||||
Be specific and genuine. Keep it under 2 sentences.
|
||||
enabled: true
|
||||
```
|
||||
|
||||
### 2. No Code Changes Needed
|
||||
|
||||
The `StrategySelector` automatically picks up new variants from the config. The algorithm will:
|
||||
|
||||
1. Include the new variant in random exploration
|
||||
2. Track its response rate once messages are sent
|
||||
3. Promote it to exploitation if it has good stats
|
||||
|
||||
### 3. Monitor Performance
|
||||
|
||||
Check performance via the dashboard or API:
|
||||
|
||||
```bash
|
||||
curl http://localhost:3000/api/analytics/by-strategy
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Debugging
|
||||
|
||||
### Account debugging
|
||||
|
||||
Use dashboard session status and structured logs. The backend does not expose or launch an interactive login browser.
|
||||
|
||||
### DOM Inspection
|
||||
|
||||
Use the debug endpoint to inspect Reddit's shadow DOM:
|
||||
|
||||
```bash
|
||||
curl -X POST http://localhost:3000/api/debug/inspect-dom
|
||||
```
|
||||
|
||||
Or run the standalone script:
|
||||
|
||||
```bash
|
||||
npx tsx scripts/inspect-chat-dom.ts
|
||||
```
|
||||
|
||||
### Logging
|
||||
|
||||
Adjust log level by modifying `src/utils/logger.ts`:
|
||||
|
||||
```typescript
|
||||
const logger = winston.createLogger({
|
||||
level: 'debug', // Change from 'info' to 'debug'
|
||||
// ...
|
||||
});
|
||||
```
|
||||
|
||||
### Database Inspection
|
||||
|
||||
Open the database with SQLite viewer:
|
||||
|
||||
```bash
|
||||
# Install sqlite3 if needed
|
||||
brew install sqlite3
|
||||
|
||||
# Open database
|
||||
sqlite3 data/reddit-bot.db
|
||||
|
||||
# Query
|
||||
SELECT * FROM sent_messages LIMIT 5;
|
||||
```
|
||||
|
||||
### Network Debugging
|
||||
|
||||
Enable axios debug:
|
||||
|
||||
```typescript
|
||||
// In src/reddit/api.ts
|
||||
import axios from 'axios';
|
||||
axios.defaults.debug = true;
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Common Tasks
|
||||
|
||||
### Add a New Subreddit
|
||||
|
||||
```yaml
|
||||
# config.yaml
|
||||
reddit:
|
||||
subreddits:
|
||||
- gonewild
|
||||
- ratemyboobs
|
||||
- new_subreddit_name # Add here
|
||||
```
|
||||
|
||||
### Change AI Model
|
||||
|
||||
```yaml
|
||||
# config.yaml
|
||||
openrouter:
|
||||
model: "new-model-name"
|
||||
fallbackModels:
|
||||
- "fallback-model-1"
|
||||
- "fallback-model-2"
|
||||
```
|
||||
|
||||
### Adjust Rate Limits
|
||||
|
||||
```yaml
|
||||
# config.yaml
|
||||
rateLimit:
|
||||
minDelayMs: 45000 # Increase delay
|
||||
maxMessagesPerHour: 15 # Reduce cap
|
||||
jitterMs: 20000 # More jitter
|
||||
```
|
||||
|
||||
### Enable Dry Run
|
||||
|
||||
```yaml
|
||||
# config.yaml
|
||||
dryRun: true # Logs all actions without sending
|
||||
```
|
||||
|
||||
Or via dashboard: `PUT /api/config` with `{"dryRun": true}`
|
||||
|
||||
### Add Database Column
|
||||
|
||||
```typescript
|
||||
// In src/database/client.ts
|
||||
function runMigrations(db: Database) {
|
||||
// ... existing migrations ...
|
||||
|
||||
// Check if column exists
|
||||
const tableInfo = db.prepare(`PRAGMA table_info(sent_messages)`).all();
|
||||
const hasNewColumn = tableInfo.some((col: any) => col.name === 'new_column');
|
||||
|
||||
if (!hasNewColumn) {
|
||||
db.exec("ALTER TABLE sent_messages ADD COLUMN new_column TEXT");
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
### Add New API Endpoint
|
||||
|
||||
```typescript
|
||||
// In src/web/server.ts
|
||||
import { Request, Response } from 'express';
|
||||
|
||||
server.app.get('/api/new-endpoint', async (req: Request, res: Response) => {
|
||||
try {
|
||||
const result = await someService.doSomething();
|
||||
res.json({ success: true, data: result });
|
||||
} catch (err) {
|
||||
res.status(500).json({ error: (err as Error).message });
|
||||
}
|
||||
});
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Known Issues
|
||||
|
||||
### Hardcoded Username in ChatService
|
||||
|
||||
`ChatService` has `myUsername = 'Upset_Marsupial1006'` hardcoded in JavaScript evaluation scripts. This should be parameterized from config.
|
||||
|
||||
**Fix needed**: Pass username to `page.evaluate()` calls.
|
||||
|
||||
### Shadow DOM Fragility
|
||||
|
||||
Reddit frequently updates their `faceplate-*` components. Shadow DOM traversal code may break without warning.
|
||||
|
||||
**Mitigation**: Use `scripts/inspect-chat-dom.ts` to debug when selectors change.
|
||||
|
||||
### Dashboard Migration Boundary
|
||||
|
||||
The React application lives in `apps/dashboard`. A production build is required
|
||||
before the Express server starts; there is no legacy HTML fallback.
|
||||
|
||||
The repository has automated service, contract, component, and transport tests.
|
||||
Manual browser verification is still required for Reddit DOM changes and the
|
||||
final dashboard cutover.
|
||||
Define the shared schema first, add a typed query/mutation, keep route/filter state in the URL where useful, invalidate snapshots from relevant SSE events, and verify loading/error/stale/accessibility states.
|
||||
Reference in new issue
Block a user