jjbrunton/RedditBot
1
0 Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Restructure chat service, web routes, and add test suite

Browse Source 
Split monolithic chat service into modular structure with Matrix client
support, extract web server into per-resource route modules, introduce
bootstrap/main-loop separation, and add comprehensive vitest coverage
across services, repositories, and utilities. Adds project docs and
CLAUDE.md agent instructions.
This commit is contained in:
jjbrunton committed 2026-05-21 01:32:10 +01:00
1 parent dfa0580859
commit 633bb0ec04
84 files changed
+19182 -3792
Download Patch File Download Diff File Expand all files Collapse all files

No files matched your search

CLAUDE.md
+48
View File
@@ -0,0 +1,48 @@
# RedditBot — agent instructions
## Test-first workflow (MANDATORY)
This applies to every code-changing task in this repo. Sub-agents inherit it.
### 1. Before implementation — establish the baseline
Run the full test suite **first**:
```
npm test
```
Note the count of passing / failing tests. Do not start implementing until you have this baseline. If something is already failing, surface it to the user before touching it — don't silently inherit pre-existing failures into your scope.
### 2. During implementation — update tests alongside code
- When you change a function's behavior, update or add the test for it in the same change.
- When you change a function's signature, fix the call sites in tests in the same change.
- When you introduce a new module, add a test for it in `tests/<name>.test.ts`.
- If a test fails because the *test* is wrong (stale assertion, ESM mocking issue, real env leaking in), fix the test. If it fails because the *code* is wrong, fix the code.
- Never delete a test to make it pass. Either fix the assertion to reflect intended behavior, or fix the code.
### 3. Before reporting done — confirm green
Re-run the full suite:
```
npm test
npx tsc --noEmit
```
**Both must be clean before you say the task is done.** If either is failing, either fix it or explicitly call out the regression in your final message — do not report success and leave a broken suite.
If tests pass count went down vs. the baseline you took in step 1, that is a regression. Treat it as a blocker.
### Why this matters
The test suite is the only automated safety net for this project (no CI yet, no integration tests). Drift here means real bugs ship. Past incident: a refactor went out with 129 pre-existing test failures undetected — they had been broken for weeks because nobody re-ran the suite.
## Project notes for agents
- Runtime: TypeScript with `module: NodeNext`. Imports use `.js` extensions even for `.ts` files.
- Tests: vitest, `globals: false` so `describe`/`it`/`expect`/`beforeEach`/`afterEach` must be imported explicitly.
- The OpenRouter, MediaService, ProcessorService, RateLimiter, and MonitorService constructors accept optional last parameters for test-injectable dependencies (mock clients, temp dirs, fake fetchers, shortened timing). Use them rather than `vi.mock()` of ESM modules — the namespace-mocking pattern fails under NodeNext.
- Shadow-DOM scraping for `chat.reddit.com` lives in `src/services/chat/scrapers.ts` as typed `page.evaluate` functions. The `myUsername` parameter is threaded through from `config.reddit.username`.
- Some `.env` vars (notably `OPENROUTER_API_KEY`) are auto-loaded via `dotenv.config()` at module import. Tests that touch `loadConfig` must snapshot+clear these in `beforeEach` and restore in `afterEach`.
Dockerfile
+26 -9
View File
@@ -1,7 +1,17 @@
# Use the official Playwright image which includes all browser dependencies
FROM mcr.microsoft.com/playwright:v1.57.0-jammy
# Set working directory
# Install Xvfb (virtual display), x11vnc (VNC server) and noVNC (web-based VNC client)
# so a human can log in to Reddit through the dashboard.
RUN apt-get update && apt-get install -y --no-install-recommends \
xvfb \
x11vnc \
novnc \
websockify \
fluxbox \
procps \
&& rm -rf /var/lib/apt/lists/*
WORKDIR /app
# Copy package files
@@ -16,15 +26,22 @@ COPY . .
# Build the TypeScript project
RUN npm run build
# Prune dev dependencies to reduce image size (optional, but good practice)
# Note: we need to keep some deps if they are runtime requirements
# npm prune --production
# Create data directory
RUN mkdir -p data
# Expose the port defined in config (default 3000)
EXPOSE 3000
# Copy entrypoint
COPY docker/entrypoint.sh /usr/local/bin/entrypoint.sh
RUN chmod +x /usr/local/bin/entrypoint.sh
# Start the application using the compiled code
CMD ["node", "dist/index.js"]
# Web dashboard
EXPOSE 3000
# noVNC (browser-accessible VNC for human login)
EXPOSE 6080
ENV DISPLAY=:99 \
VNC_DISPLAY_NUM=99 \
VNC_PORT=5900 \
NOVNC_PORT=6080 \
SCREEN_GEOMETRY=1280x800x24
CMD ["/usr/local/bin/entrypoint.sh"]
agents.md
+23 -5
View File
@@ -19,16 +19,34 @@ Refactor code to improve maintainability, testability, and clarity. Break down l
- **Zod Validation**: Use `zod` (already in deps) to validate external data (config files, API responses).
- **Async/Await**: properly handle promises. Use `p-queue` or `p-retry` for rate-limited operations.
### 4. Project Specifics
- **Tech Stack**: Node.js, TypeScript, Playwright, Express, SQLite.
### 4. Test-Driven Development (TDD) — REQUIRED
All new functionality and bug fixes MUST follow the red-green-refactor cycle. This is non-negotiable.
- **Red**: Write a failing test FIRST that describes the desired behavior. Run it and confirm it fails for the expected reason.
- **Green**: Write the minimum production code needed to make the test pass. No extra features, no speculative abstractions.
- **Refactor**: With tests green, clean up implementation and tests. Re-run the suite after each change.
**Rules**:
- Never commit production code without an accompanying test that fails without it.
- Tests live in `tests/` (mirroring `src/` structure) and run via `vitest` (`npm test`).
- Use dependency injection so collaborators (DB, HTTP, browser, clock) can be faked — do not mock what you don't own; wrap third-party APIs in a thin interface and fake the interface.
- Prefer behavior-focused tests (inputs → outputs / observable side effects) over tests that pin internal structure.
- Bug fixes start with a regression test that reproduces the bug.
- If a change is genuinely untestable (e.g. a one-line config tweak), state that explicitly in the PR/commit rather than skipping silently.
### 5. Project Specifics
- **Tech Stack**: Node.js, TypeScript, Playwright, Express, SQLite, Vitest.
- **Directories**:
- `src/services/` for business logic.
- `src/web/` for Express server and UI.
- `src/browser/` for Playwright interactions.
- `tests/` for vitest specs, mirroring `src/` layout.
- **Logging**: Use `winston` for structured logging. Do not use `console.log` in production code.
## Workflow for Agents
1. **Analyze**: Before writing code, read the file and understand the context.
2. **Plan**: Identify what needs to change. Check if the change violates SRP.
3. **Refactor**: Create new files if necessary. Don't be afraid to create new small files.
4. **Verify**: Ensure types check out (`npm run build` or `tsc`).
2. **Plan**: Identify what needs to change. Check if the change violates SRP. Identify the test(s) that will drive the change.
3. **Write failing test (Red)**: Add a vitest spec that captures the new behavior or reproduces the bug. Run `npm test` and confirm it fails for the right reason.
4. **Implement (Green)**: Write the smallest change that makes the test pass. Create new small files when SRP calls for it.
5. **Refactor**: Tidy the implementation and the test with the suite green. Re-run tests after each refactor step.
6. **Verify**: Ensure the full suite passes (`npm test`) and types check (`npm run build` or `tsc --noEmit`).
config.yaml
+123 -40
View File
@@ -1,36 +1,44 @@
reddit:
# Weights bias how often each sub is picked each poll cycle. Higher weight
# = more likely to appear earlier in the check order. Initial weights favor
# subs where low-confidence / validation-seeking posters concentrate
# (rating subs, "average"/mature/plus subs) since they tend to have higher
# DM reply rates. Default is 1 if omitted.
subreddits:
# Original list (kept intact)
- normalnudes # Everyday real bodies, highly authentic
- ratemyboobs # Boob-focused ratings, lots of self-posts
- ratemypussy # Pussy ratings, direct and active
- HomemadeNsfw # Pure homemade amateur content
- RateMyRack # Rack/boob ratings
- RateMyNudeBody # Full-body nude ratings
# Validation-seeking / lower-confidence (highest weight)
- { name: AverageNudes, weight: 4 } # "Average-looking" self-perception
- { name: gonewild40plus, weight: 4 } # Mature, lower volume, very high reply rate
- { name: normalnudes, weight: 3 } # Everyday real bodies
- { name: RateMyNudeBody, weight: 3 } # Explicit ask for feedback
- { name: ratemyboobs, weight: 3 } # Rating subs = explicit validation-seeking
- { name: ratemypussy, weight: 3 }
- { name: RateMyRack, weight: 3 }
- { name: GoneWildPlus, weight: 3 } # BBW/plus-size
- { name: gonewildchubby, weight: 3 } # Soft/chubby bodies
- { name: gonewild30plus, weight: 2 } # Mature/MILF amateurs
- { name: GoneMild, weight: 2 } # Softcore teases (shyer posters)
# Top general additions (highest volume/quality amateurs)
- gonewild # #1 for verified women self-posting nudes
- GoneErotic # Artistic/erotic amateurs
- Amateur # Real amateur pics/vids worldwide
- RealGirls # Unfiltered everyday women
- GoneMild # Softcore teases from verified posters
- OnOff # Before/after clothing reveals
# Mid-tier — mixed confidence
- { name: GoneWildCurvy, weight: 2 }
- { name: HomemadeNsfw, weight: 2 }
- { name: realgirlsgonebad, weight: 2 }
- { name: nsfwselfies, weight: 2 }
- { name: RealGirls, weight: 2 }
- { name: Amateur, weight: 2 }
# Niche by body type (diverse, verified amateurs)
- GoneWildCurvy # Curvy/thicc verified women
- petitegonewild # Petite/slim under 5'4"
- GoneWildPlus # BBW/plus-size confident shares
- gonewildchubby # Soft/chubby bodies
- tightandtoned # Fit/athletic women
- pawg # Thick asses (heavy self-posts)
# Baseline — generally more confident / curated posters
- { name: gonewild, weight: 1 } # #1 verified self-posters
- { name: GoneErotic, weight: 1 }
- { name: OnOff, weight: 1 }
- { name: petitegonewild, weight: 1 }
- { name: tightandtoned, weight: 1 }
- { name: pawg, weight: 1 }
- { name: asiansgonewild, weight: 1 }
- { name: altgonewild, weight: 1 }
- { name: holdthemoan, weight: 1 }
# Other strong fits (active, rating-style or homemade vibes)
- nsfwselfies # Selfie-style nudes
- asiansgonewild # Asian amateurs
- altgonewild # Alt/goth/tattooed
- gonewild30plus # Mature/MILF amateurs
- holdthemoan # Risky public-ish posts
- realgirlsgonebad # Candid real-girl naughtiness
# Chat-only subreddits — no image analysis, gender parsed from post tag
#
pollIntervalMs: 30000
userAgent: "RedditBot/1.0 (by /u/yourusername)"
targetGenders:
@@ -39,11 +47,21 @@ reddit:
# sellerKeywords: [...] # Uncomment to override default keyword list
openrouter:
imageAnalysisModel: "google/gemini-3-flash-preview"
imageAnalysisModel: "google/gemini-2.5-flash-lite"
imageAnalysisFallbackModels:
- "x-ai/grok-4.1-fast"
- "meta-llama/llama-4-maverick"
chatModel: "x-ai/grok-4.1-fast"
- "x-ai/grok-4.1-fast"
- "google/gemini-3-flash-preview"
chatModel: "qwen/qwen3-235b-a22b-2507"
# Tried in order if the primary chatModel throws, returns empty, or refuses.
# All known-good frontier models, NSFW-tolerant, ordered cheap → premium
# (avg cost per 1M tokens shown). Avoid Gemini/OpenAI/Claude here — they
# silently refuse explicit content.
chatFallbackModels:
- "deepseek/deepseek-v4-flash" # ~$0.17/M
- "meta-llama/llama-4-scout" # ~$0.19/M
- "meta-llama/llama-4-maverick" # ~$0.38/M
- "deepseek/deepseek-chat-v3.1" # ~$0.50/M
imageAnalysisPrompt: |
You're analyzing images.
@@ -58,13 +76,28 @@ openrouter:
Be graphic. Describe exactly what you see. No clinical terms.
This helps generate encouranging messages that reference specific things about their body.
Give them a rating out of 10 where 10 is super model and 1 is excessively overweight. This helps us send encouragement to the user. You MUST provide a reason with your rating. Be critical and honest.
Act within guidelines.
Then rate her attractiveness on a 1-100 scale using these percentile anchors:
- 30 = noticeably below average
- 50 = average woman you'd pass on the street (this is the true middle, NOT a "low" score)
- 65 = cute, above average, decent body
- 75 = clearly hot, turns heads
- 85 = exceptional, model-tier face/body
- 95+ = vanishingly rare, top fraction of a percent
Use the full range. Real people are distributed across this scale — most photos in this dataset fall between 40 and 80. Do NOT cluster scores in the 65-75 band as a safe default. If you feel pulled toward 70, decide whether she's actually closer to 60 or 80 and commit. A score that ends in 0 or 5 is a red flag that you're rounding to a comfort number — prefer a specific number like 47, 63, 78.
Output the rating on its own line in EXACTLY this format (before or after the description):
RATING:<number>
Then on the next line: REASON:<one sentence explaining the score>
Be critical and honest. Act within guidelines.
messageGenerationPrompt: |
Based on this image analysis: {analysis}
Post title: {title}
Author: u/{author}
You should take into account the rating provided by the analysis, the rating is out of 10 where 10 is super model and 1 is excessively overweight. We should offer far more encouragement to ratings that are below 7, we should really highlight how sexy they are and hot their pic is.
Attractiveness rating (1-100 scale, 50 = average): {rating}
If the rating is below 60, lean harder into making her feel sexy and desired — emphasize how hot her pic looks. If the rating is 75+, you can be more confidently flirty without overdoing the flattery.
Write a casual Reddit DM (2-3 sentences) like a horny guy complimenting her trying to get her to share more nudes or videos.
@@ -92,6 +125,7 @@ openrouter:
- Use clinical terms like "breasts", "buttocks", "posterior", "physique"
- Write long elaborate sentences
- Use all lowercase
- Use em dashes (—) or en dashes (–). Use commas, full stops, or separate sentences instead. This is a dead giveaway for AI writing.
Examples of good messages:
- "Damn those tits are perfect. I'd love to see more if you're willing😏"
@@ -100,6 +134,44 @@ openrouter:
Just write the message, nothing else.
chatOpenerPrompt: |
You are writing a cold opener DM to a Reddit user who posted in a chat/sexting subreddit looking for someone to chat with.
Post title: {title}
Post body: {selftext}
Author: u/{author}
Detected gender of poster: {gender}
YOUR PERSONA:
- You are male, based in the UK, pick a random age between 30 and 35 for this message
- Roughly half the time, mention (or imply) you're married/have a wife and are looking for something on the side; the other half, don't mention relationship status at all
- Don't force the married angle if their post wouldn't vibe with it, read the room
- Use British English spelling and phrasing (e.g. "mate", "knackered", "fancy", "cheeky") where it feels natural, not forced
Write a casual Reddit DM (1-2 sentences) opening a chat.
MUST follow:
- Lead with your age/gender/location tag (e.g. "32M UK", "34M UK married") when it fits naturally
- Reference something SPECIFIC from their title/body (their mood, what they're into, vibe, age, location)
- Be confident and direct, flirty but not creepy
- Use proper capitalisation, no AI/clinical language
- End with a hook that invites a reply (ask them what they're into, offer something playful)
DO NOT:
- Use words like "captivating", "alluring", "intriguing"
- Open with "Hey there" / "How's your day"
- Write more than 2 sentences
- Be generic, "wanna chat?" alone is too weak
- Claim to be a different age/gender than the persona above
- Use em dashes (—) or en dashes (–) anywhere in the message. Use commas, full stops, or separate sentences instead. This is a dead giveaway for AI writing.
Examples of good openers:
- "32M UK, married and bored while the wife's out. Your post caught my eye, what are you into tonight?"
- "34M UK here, love that you're up late looking for trouble. Fancy telling me what gets you going?"
- "33M from the UK, your vibe is exactly my type. Tell me what you'd want a stranger to do to you."
Just write the message, nothing else.
strategies:
explorationRate: 0.2 # 20% of the time try a random strategy instead of the best
minSamples: 5 # Min messages sent before trusting a strategy's response rate
@@ -120,6 +192,23 @@ strategies:
STYLE: Reference 2-3 specific things from the analysis to show you actually looked.
Be enthusiastic but not over the top. End with asking for more.
chatStrategies:
explorationRate: 0.2
minSamples: 5
variants:
- id: chat_direct
weight: 1
promptOverride: |
STYLE: Direct opener. Open with your own age/gender, reference theirs, and ask one specific question about what they're into. No em dashes.
- id: chat_playful
weight: 1
promptOverride: |
STYLE: Playful and teasing. Light humour, mild flirty challenge, end with a hook that's hard to ignore. No em dashes.
- id: chat_mirror
weight: 1
promptOverride: |
STYLE: Mirror their energy. Match the vibe of their post (shy/horny/bored/dom/sub) and respond in kind. Be specific to what they wrote. No em dashes.
rateLimit:
minDelayBetweenMessagesMs: 15000
maxMessagesPerHour: 9999
@@ -133,12 +222,6 @@ browser:
database:
path: "./data/reddit-bot.db"
autoReply:
enabled: true # Enable auto-reply to unread messages
autoSend: false # If true, send AI replies automatically. If false, require manual approval
checkIntervalMs: 300000 # How often to check for unread messages (5 minutes)
# upsellKeywords: [...] # Uncomment to override default upsell detection keywords
web:
enabled: true
port: 3000
docker-compose.yml
+7 -2
View File
@@ -8,9 +8,14 @@ services:
# Mount the data directory to persist the database and browser session
volumes:
- ./data:/app/data
# Environment variables
ports:
- "3000:3000" # Web dashboard
- "6080:6080" # noVNC for human login
environment:
- OPENROUTER_API_KEY=${OPENROUTER_API_KEY}
- REDDIT_USERNAME=${REDDIT_USERNAME}
- REDDIT_PASSWORD=${REDDIT_PASSWORD}
- BROWSER_HEADLESS=true
# Chromium runs headed under Xvfb so a human can complete login over noVNC.
- BROWSER_HEADLESS=false
# Required to lock down the VNC endpoint.
- VNC_PASSWORD=${VNC_PASSWORD}
docker/entrypoint.sh
Executable
+70
View File
@@ -0,0 +1,70 @@
#!/usr/bin/env bash
# Start Xvfb + x11vnc + noVNC so a human can complete Reddit login
# through the dashboard, then exec the Node bot which drives Chromium
# on that virtual display.
set -euo pipefail
DISPLAY_NUM="${VNC_DISPLAY_NUM:-99}"
VNC_PORT="${VNC_PORT:-5900}"
NOVNC_PORT="${NOVNC_PORT:-6080}"
GEOMETRY="${SCREEN_GEOMETRY:-1280x800x24}"
export DISPLAY=":${DISPLAY_NUM}"
# noVNC ships under different paths depending on the distro; find the web root once.
if [ -d /usr/share/novnc ]; then
NOVNC_WEB_DIR=/usr/share/novnc
elif [ -d /usr/share/webapps/novnc ]; then
NOVNC_WEB_DIR=/usr/share/webapps/novnc
else
echo "[entrypoint] could not locate noVNC web root" >&2
exit 1
fi
# Make / on the noVNC server land on vnc.html with autoconnect.
if [ ! -e "${NOVNC_WEB_DIR}/index.html" ] && [ -e "${NOVNC_WEB_DIR}/vnc.html" ]; then
ln -sf vnc.html "${NOVNC_WEB_DIR}/index.html"
fi
echo "[entrypoint] starting Xvfb on ${DISPLAY} (${GEOMETRY})"
Xvfb "${DISPLAY}" -screen 0 "${GEOMETRY}" -ac +extension RANDR &
XVFB_PID=$!
# Give Xvfb a moment to come up.
for _ in 1 2 3 4 5 6 7 8 9 10; do
if xdpyinfo -display "${DISPLAY}" >/dev/null 2>&1; then break; fi
sleep 0.2
done
# Minimal window manager so dialogs and tabs aren't stuck behind the root window.
if command -v fluxbox >/dev/null 2>&1; then
fluxbox >/dev/null 2>&1 &
fi
VNC_ARGS=(-display "${DISPLAY}" -forever -shared -rfbport "${VNC_PORT}" -nopw_warn)
if [ -n "${VNC_PASSWORD:-}" ]; then
mkdir -p /root/.vnc
x11vnc -storepasswd "${VNC_PASSWORD}" /root/.vnc/passwd >/dev/null
VNC_ARGS+=(-rfbauth /root/.vnc/passwd)
echo "[entrypoint] x11vnc starting with password auth"
else
VNC_ARGS+=(-nopw)
echo "[entrypoint] WARNING: VNC_PASSWORD not set — VNC has no auth"
fi
x11vnc "${VNC_ARGS[@]}" >/var/log/x11vnc.log 2>&1 &
X11VNC_PID=$!
echo "[entrypoint] starting noVNC (websockify) on :${NOVNC_PORT}"
websockify --web "${NOVNC_WEB_DIR}" "${NOVNC_PORT}" "localhost:${VNC_PORT}" \
>/var/log/websockify.log 2>&1 &
NOVNC_PID=$!
cleanup() {
echo "[entrypoint] shutting down"
kill "${NOVNC_PID}" "${X11VNC_PID}" "${XVFB_PID}" 2>/dev/null || true
wait || true
}
trap cleanup TERM INT EXIT
echo "[entrypoint] launching bot"
exec node dist/index.js
docs/api.md
+1002
View File
File diff suppressed because it is too large. Load diff
docs/architecture.md
+898
View File
@@ -0,0 +1,898 @@
# RedditBot - Software Architecture Documentation
## Table of Contents
1. [Project Overview](#project-overview)
2. [System Architecture](#system-architecture)
3. [Directory Structure](#directory-structure)
4. [Component Reference](#component-reference)
5. [Data Flow](#data-flow)
6. [Database Schema](#database-schema)
7. [API Reference](#api-reference)
8. [Configuration](#configuration)
9. [Design Patterns](#design-patterns)
10. [External Integrations](#external-integrations)
11. [Deployment](#deployment)
12. [Technical Debt & Recommendations](#technical-debt--recommendations)
---
## Project Overview
**RedditBot** is an automated Reddit monitoring and messaging system that:
- Monitors 30+ NSFW subreddits for image posts
- Uses AI (OpenRouter) to analyze images and detect gender
- Generates personalized direct messages to post authors
- Sends DMs via browser automation (Playwright)
- Monitors for responses and tracks success rates
- Implements A/B testing with explore/exploit strategy selection
- Learns from successful message/response pairs (few-shot learning)
- Provides a real-time web dashboard for monitoring and control
### Key Capabilities
| Capability | Description |
|---|---|
| Post Discovery | Polls subreddits for new image posts, filters by gender and spam profiles |
| AI Analysis | Multi-model image analysis with fallback chain via OpenRouter |
| Message Generation | AI-crafted DMs using successful examples as few-shot prompts |
| A/B Testing | Explore/exploit strategy selector optimizes response rates over time |
| Response Monitoring | Background service checks inbox for user replies |
| Chat Automation | Reddit chat scraping via shadow DOM traversal, auto-reply support |
| Media Management | Download, cache, deduplicate, and sync media from conversations |
| Web Dashboard | Real-time monitoring with SSE log streaming, analytics, and controls |
---
## System Architecture
### High-Level Architecture
```
┌─────────────────────────────────────────────────────────────────────┐
│ main (index.ts) │
│ Orchestrator / Entry Point │
└────────────────────────┬────────────────────────────────────────────┘
│
┌────────────────┼────────────────┐
│ │ │
▼ ▼ ▼
┌───────────────┐ ┌──────────────┐ ┌──────────────┐
│ Monitor │ │ Processor │ │ Response │
│ Service │ │ │ │ Checker │
│ │ │ 1. Download │ │ │
│ Poll subreddits│ │ 2. Validate │ │ Check inbox │
│ for new posts │ │ 3. Gender │ │ for replies │
└───────┬───────┘ │ Filter │ └──────────────┘
│ │ 4. AI Analysis│ │
│ │ 5. Strategy │ │
│ │ Selection │ │
│ │ 6. Message │ │
│ │ Generation │ │
└────┬────┘ └─────────────┘ │
│ │
▼ ▼
┌───────────────────────┐ ┌───────────────────────┐
│ Strategy Selector │ │ Chat Service │
│ │ │ │
│ A/B Testing │ │ Scrapes chat (shadow │
│ Explore/Exploit │ │ DOM traversal) │
│ Variant Selection │ │ Auto-reply, upsell │
└───────────────────────┘ │ detection │
└───────────────────────┘
│ │
▼ ▼
┌───────────────────────┐ ┌───────────────────────┐
│ Message Sender │ │ Media Service │
│ (Browser) │ │ │
│ │ │ Download, cache, │
│ Playwright DM compose │ │ deduplicate (SHA-1) │
│ Shadow DOM handling │ │ Matrix thumbnail │
└───────────────────────┘ │ resolution │
└───────────────────────┘
│ │
▼ ▼
┌─────────────────────────────────────────────────────────┐
│ SQLite Database │
│ │
│ processed_posts │ sent_messages │ successful_messages │
└─────────────────────────────────────────────────────────┘
│
▼
┌─────────────────────────────────────────────────────────┐
│ Web Dashboard (Express) │
│ │
│ 25+ API endpoints │ SSE log streaming │ Analytics │
│ Runtime config │ Bot controls │ Media gallery │
└─────────────────────────────────────────────────────────┘
```
### Service Communication
```
main.ts (Orchestrator)
│
├──► Config (Zod-validated YAML + env vars)
│
├──► Database (Better-SQLite3)
│ │
│ └──► PostsRepository
│ └──► MessagesRepository
│ └──► SuccessfulMessagesRepository
│
├──► MonitorService ──► RedditApi ──► Playwright (cookies)
│
├──► ProcessorService
│ ├──► Image download (axios)
│ ├──► Gender detection (OpenRouter)
│ ├──► Full analysis (OpenRouter)
│ └──► StrategySelector ──► MessagesRepository
│
├──► MessageSender (Browser - Playwright)
│
├──► ResponseChecker ──► Playwright (inbox navigation)
│
├──► ChatService ──► Playwright (chat.reddit.com shadow DOM)
│
├──► MediaService ──► Playwright (media download)
│
├──► MediaSyncService (background periodic sync)
│
├──► AutoReplyService ──► OpenRouter (reply generation)
│
├──► RateLimiter (p-queue, jitter)
│
└──► WebServer (Express, SSE)
```
---
## Directory Structure
```
RedditBot/
├── .dockerignore # Docker ignore rules
├── .gitignore # Git ignore rules
├── agents.md # Engineering standards & agent guidelines
├── config.yaml # Main YAML configuration file
├── docker-compose.yml # Docker Compose setup
├── DOKPLOY_DEPLOY.md # Dokploy deployment guide
├── Dockerfile # Container build definition
├── package.json # Node.js dependencies & scripts
├── tsconfig.json # TypeScript configuration
├── public/
│ └── index.html # Single-file web dashboard (HTML+CSS+JS)
├── scripts/
│ ├── inspect-chat-dom.ts # Debug: inspect chat DOM structure
│ └── login.ts # Manual browser login helper
└── src/
├── index.ts # Main entry point / orchestrator (391 lines)
├── config/
│ └── index.ts # Config loader + Zod validation schema (157 lines)
├── database/
│ ├── client.ts # SQLite setup + migration runner (121 lines)
│ └── repositories.ts # Repository pattern: Posts, Messages, Success (433 lines)
├── reddit/
│ └── api.ts # Reddit API client (axios + Playwright cookies) (233 lines)
├── ai/
│ └── openrouter.ts # OpenRouter AI client (image analysis, message gen) (307 lines)
├── browser/
│ ├── client.ts # Playwright persistent context + login (302 lines)
│ ├── message-page.ts # DM compose page automation + shadow DOM (560 lines)
│ └── media-downloader.ts # Playwright-based media downloader (29 lines)
├── services/
│ ├── monitor.ts # Post discovery service (72 lines)
│ ├── processor.ts # Image download + AI + message pipeline (132 lines)
│ ├── response-checker.ts # Inbox response checker (179 lines)
│ ├── strategy-selector.ts # A/B testing explore/exploit (101 lines)
│ ├── chat.ts # Chat scraping + messaging + auto-reply (1123+ lines)
│ ├── media.ts # Media download, caching, indexing (254 lines)
│ ├── media-sync.ts # Periodic background media sync (119 lines)
│ └── auto-reply.ts # Auto-reply to chat conversations (199 lines)
├── utils/
│ ├── logger.ts # Winston structured logging (32 lines)
│ ├── rate-limiter.ts # Token-bucket rate limiting (114 lines)
│ ├── retry.ts # Exponential backoff wrapper (28 lines)
│ └── image.ts # Image download + base64 + MIME validation (34 lines)
└── web/
└── server.ts # Express dashboard + SSE streaming (794 lines)
```
---
## Component Reference
### Core Services
#### MonitorService (`src/services/monitor.ts`)
Discovers new image posts across configured subreddits.
- Shuffles subreddit order each cycle
- Fetches new posts via `RedditApi`
- Filters out already-processed posts and previously-messaged users
- Returns the single newest valid post per cycle
#### ProcessorService (`src/services/processor.ts`)
Core processing pipeline for each discovered post.
1. Downloads image via `ImageService`
2. Validates MIME type and file size
3. Runs quick gender detection (cost-saving pre-filter)
4. Selects A/B strategy variant via `StrategySelector`
5. Runs full AI analysis + message generation via `OpenRouterService`
6. Handles AI refusals gracefully (marks post, tries next)
#### StrategySelector (`src/services/strategy-selector.ts`)
Implements explore/exploit A/B testing for message variants.
- Tracks response rates per strategy variant
- Uses minimum sample thresholds before trusting statistics
- Applies random exploration rate for continued discovery
- Selects best variant based on weighted response rates
#### ResponseChecker (`src/services/response-checker.ts`)
Background service that monitors for user replies.
- Navigates Reddit inbox via Playwright
- Identifies responses to sent messages
- Updates `got_response` flag in database
- Stores successful message/response pairs for few-shot learning
#### ChatService (`src/services/chat.ts`)
Most complex service - manages Reddit chat interactions.
- Scrapes `chat.reddit.com` using JavaScript injection (shadow DOM traversal)
- Manages conversation cache (in-memory + disk JSON)
- Loads full conversation history with pagination
- Sends messages with human-like typing delays
- Verifies message delivery
- Detects upsell/spam keywords in chat
- Parameter: `myUsername` is hardcoded (should be from config)
#### MediaService (`src/services/media.ts`)
Media download, caching, and indexing.
- Downloads images from chat conversations
- Resolves Matrix blurred thumbnails to full resolution
- Stores files in `data/media/` directory
- Maintains JSON index with SHA-1 deduplication
- Tracks download status per conversation
#### MediaSyncService (`src/services/media-sync.ts`)
Periodic background media synchronization.
- Runs incremental syncs (top 10 chats) or full syncs (all conversations)
- Configurable interval via `mediaSyncInterval`
- Tracks sync status in memory
- Exposed via web dashboard endpoints
#### AutoReplyService (`src/services/auto-reply.ts`)
Automated reply to Reddit chat conversations.
- Checks for unread messages in cached conversations
- Detects upsell/spam keywords
- Generates AI reply suggestions via `OpenRouterService`
- Sends replies (auto or pending for manual approval)
- Event-based architecture for dashboard integration
### External Integrations
#### RedditApi (`src/reddit/api.ts`)
Axios-based Reddit API client.
- Fetches posts from subreddits (`/r/{name}/new.json`)
- Validates posts (image-only, skips deleted authors)
- Extracts image URLs with fallback chain: `preview` > `i.redd.it` > `imgur` > `thumbnail`
- Checks user profiles against seller/spam keywords
- Uses Playwright browser cookies for authenticated requests
#### OpenRouterService (`src/ai/openrouter.ts`)
OpenRouter API client using OpenAI-compatible SDK.
- `analyzeImage()`: Multi-model image analysis with fallback chain
- `detectGenderOnly()`: Quick cost-saving gender detection
- `generateMessage()`: DM text with few-shot examples from DB
- `generateReply()`: AI chat reply from conversation history
- `detectUpsell()`: Keyword-based spam/OnlyFans detection
- Refusal detection via heuristic patterns
#### Browser Client (`src/browser/client.ts`)
Playwright persistent context manager.
- Browser session persistence (cookies/storage)
- Login verification (multiple UI checks + API fallback)
- Automated login flow (form fill, 2FA/captcha detection)
- Debug screenshot capture on login failure
#### MessagePage (`src/browser/message-page.ts`)
Reddit DM compose page automation.
- Fills recipient/subject/body fields
- Extensive Shadow DOM support for `faceplate-*` components
- Detects recipient errors, verifies submission success
- Supports dry-run mode for testing
### Data Layer
#### Database Client (`src/database/client.ts`)
Better-SQLite3 database initialization.
- Creates database at configured path
- Enables WAL mode and foreign keys
- Runs migrations on startup
- Safe column-addition for schema evolution
#### Repositories (`src/database/repositories.ts`)
Three repository classes abstracting SQLite operations.
- **PostsRepository**: Processed post CRUD, mark sent, update analysis
- **MessagesRepository**: Sent message CRUD, response tracking, strategy stats, subreddit stats, daily trends
- **SuccessfulMessagesRepository**: Successful pairs for few-shot learning, subreddit-weighted retrieval
### Utilities
#### Logger (`src/utils/logger.ts`)
Winston structured logging.
- JSON format for log files
- Colored console output
- File logging: `error.log` and combined log
#### RateLimiter (`src/utils/rate-limiter.ts`)
Token-bucket rate limiting.
- p-queue with concurrency=1
- Minimum delay between messages with random jitter
- Hourly message count tracking
- Timer status exposed to dashboard
#### Retry (`src/utils/retry.ts`)
Exponential backoff wrapper.
- p-retry with configurable retries (default 3)
- Backoff range: 5s to 60s
- Logging on retries and failures
#### Image (`src/utils/image.ts`)
Image processing utilities.
- Download via axios
- Base64 conversion for AI API
- MIME type validation (image/jpeg, image/png, image/webp)
### Web Dashboard
#### WebServer (`src/web/server.ts`)
Express server with 25+ API endpoints.
- Stats, activity, messages, chats, conversations, media endpoints
- Bot controls (start/stop)
- SSE realtime log streaming
- Analytics: strategy, subreddit, daily trends
- Runtime config management (GET/PUT)
- Debug endpoints
---
## Data Flow
### Post Processing Pipeline
```
1. MonitorService polls subreddits
│
├─► RedditApi.fetchPosts()
│ └─► GET /r/{subreddit}/new.json
│ └─► Filter: image posts only, valid author
│
├─► Filter: skip processed posts (DB check)
├─► Filter: skip previously messaged users
│
└─► Return newest valid post
│
2. ProcessorService.processPost(post)
│
├─► ImageService.downloadImage(url)
│ └─► Validate MIME type, file size
│
├─► OpenRouter.detectGenderOnly()
│ └─► Quick pre-filter (cost-saving)
│ └─► Skip if gender doesn't match target
│
├─► StrategySelector.selectStrategy(subreddit, author)
│ └─► Explore/exploit algorithm
│
├─► OpenRouter.analyzeImage(image)
│ └─► Multi-model fallback chain
│ └─► Detect gender, describe image
│
├─► OpenRouter.generateMessage(analysis, title, examples, strategy)
│ └─► Few-shot learning from successful_messages
│
└─► Mark post as processed in DB
│
3. MessagePage.sendDM(author, subject, message)
│
├─► Navigate to /compose?to={author}
├─► Fill fields (with shadow DOM handling)
├─► Submit message
└─► Verify success
│
└─► Record sent_messages in DB
```
### Response Monitoring Flow
```
1. ResponseChecker.checkResponses()
│
├─► Navigate to /message/inbox
├─► Parse response items
│
├─► For each response:
│ ├─► Match to sent_messages by recipient + timestamp
│ ├─► Update got_response = 1
│ ├─► Store message/response pair
│ └─► Add to successful_messages (for few-shot learning)
│
└─► Update strategy stats in DB
```
### Chat Interaction Flow
```
1. ChatService.fetchChats()
│
├─► Navigate to chat.reddit.com
├─► Traverse shadow DOM via JS injection
├─► Extract active conversations
└─► Cache in memory + disk JSON
│
2. ChatService.loadConversation(chatId)
│
├─► Load from cache
├─► Fetch full history (pagination)
└─► Return conversation object
│
3. ChatService.sendMessage(chatId, text)
│
├─► Navigate to chat view
├─► Fill message input (shadow DOM)
├─► Apply typing delay (human-like)
├─► Submit
└─► Verify delivery
```
---
## Database Schema
### Table: `processed_posts`
Tracks all posts that have been discovered and processed.
| Column | Type | Constraints | Description |
|---|---|---|---|
| `id` | TEXT | PRIMARY KEY | Reddit post ID |
| `subreddit` | TEXT | NOT NULL | Source subreddit |
| `author` | TEXT | NOT NULL | Post author username |
| `title` | TEXT | | Post title |
| `image_url` | TEXT | | Original image URL |
| `image_analysis` | TEXT | | AI-generated image description |
| `generated_message` | TEXT | | AI-generated DM text |
| `processed_at` | INTEGER | NOT NULL | Unix timestamp |
| `message_sent` | INTEGER | DEFAULT 0 | 0 = not sent, 1 = sent |
| `error` | TEXT | | Error message if processing failed |
**Indexes**: `idx_posts_subreddit`, `idx_posts_author`
### Table: `sent_messages`
Tracks all DMs sent by the bot.
| Column | Type | Constraints | Description |
|---|---|---|---|
| `id` | INTEGER | PRIMARY KEY AUTOINCREMENT | Internal ID |
| `post_id` | TEXT | FK → processed_posts.id | Source post |
| `recipient` | TEXT | NOT NULL | Recipient username |
| `subject` | TEXT | DEFAULT 'Stunning' | DM subject line |
| `message` | TEXT | NOT NULL | DM body |
| `sent_at` | INTEGER | NOT NULL | Unix timestamp |
| `strategy_id` | TEXT | | A/B test variant identifier |
| `got_response` | INTEGER | DEFAULT 0 | 0 = no response, 1 = responded |
| `response_checked_at` | INTEGER | | Timestamp of last response check |
**Indexes**: `idx_messages_recipient`, `idx_messages_strategy`, `idx_messages_response`
### Table: `successful_messages`
Stores successful message/response pairs for few-shot learning.
| Column | Type | Constraints | Description |
|---|---|---|---|
| `id` | INTEGER | PRIMARY KEY AUTOINCREMENT | Internal ID |
| `sent_message_id` | INTEGER | FK → sent_messages.id (nullable) | Related message |
| `original_message` | TEXT | NOT NULL | The DM that got a response |
| `image_analysis` | TEXT | | Context from image analysis |
| `post_title` | TEXT | | Original post title |
| `response_snippet` | TEXT | | First reply text |
| `subreddit` | TEXT | | For weighted example selection |
| `created_at` | INTEGER | NOT NULL | Unix timestamp |
**Indexes**: `idx_success_subreddit`, `idx_success_message`
### Migration Strategy
The database uses progressive migrations that:
- Run on every startup
- Use `ALTER TABLE ADD COLUMN IF NOT EXISTS` for safe schema evolution
- Create indexes with `IF NOT EXISTS` for idempotency
- Enable WAL mode and foreign keys for performance and integrity
---
## API Reference
### Web Dashboard Endpoints
All endpoints are served by the Express server at `http://localhost:{port}`.
#### Bot Status & Control
| Method | Endpoint | Description |
|---|---|---|
| GET | `/api/stats` | Bot statistics (running state, post/message/response counts, response rate) |
| POST | `/api/start` | Start the bot |
| POST | `/api/stop` | Stop the bot |
#### Activity & Messages
| Method | Endpoint | Description |
|---|---|---|
| GET | `/api/activity` | Recent activity log entries |
| GET | `/api/posts/by-author/:username` | Look up processed posts by user |
| GET | `/api/messages` | Recent sent messages |
| GET | `/api/successful` | Successful message/response pairs |
#### Chat & Conversations
| Method | Endpoint | Description |
|---|---|---|
| GET | `/api/chats` | Active cached conversations |
| POST | `/api/chats/fetch-all` | Fetch all chats from Reddit |
| GET | `/api/conversations` | All cached conversations |
| GET | `/api/conversations/:id` | Single conversation detail |
| POST | `/api/conversations/:id/suggest-reply` | AI suggest reply for conversation |
| POST | `/api/conversations/:id/send-reply` | Send chat reply |
| POST | `/api/conversations/:id/approve-reply` | Approve pending auto-reply |
| POST | `/api/conversations/:id/reject-reply` | Reject pending auto-reply |
| POST | `/api/conversations/:id/dismiss-upsell` | Dismiss upsell flag |
#### Media
| Method | Endpoint | Description |
|---|---|---|
| GET | `/api/media` | Downloaded media gallery |
| GET | `/api/media/sync` | Media sync status |
| POST | `/api/media/sync` | Trigger incremental sync (top 10 chats) |
| POST | `/api/media/sync/full` | Trigger full sync (all conversations) |
#### Analytics
| Method | Endpoint | Description |
|---|---|---|
| GET | `/api/analytics/by-subreddit` | Response rate by subreddit |
| GET | `/api/analytics/by-strategy` | Response rate by strategy variant |
| GET | `/api/analytics/trend` | Daily response rate trend |
#### Configuration
| Method | Endpoint | Description |
|---|---|---|
| GET | `/api/config` | Current runtime configuration |
| PUT | `/api/config` | Update runtime config (rate limits, dry run, poll interval) |
#### Logs & Debugging
| Method | Endpoint | Description |
|---|---|---|
| GET | `/api/logs/stream` | SSE realtime log stream |
| GET | `/api/logs` | Recent logs (HTTP) |
| GET | `/api/timers` | Rate limiter timer status |
| POST | `/api/debug/inspect-dom` | Debug: inspect chat DOM structure |
| POST | `/api/debug/check-responses` | Debug: manual response check |
---
## Configuration
### Configuration Sources (Priority Order)
1. **`config.yaml`** - Primary configuration file (project root)
2. **Environment Variables** - Override specific fields:
- `OPENROUTER_API_KEY` - OpenRouter API key
- `REDDIT_USERNAME` - Reddit username
- `REDDIT_PASSWORD` - Reddit password
- `BROWSER_HEADLESS` - Browser headless mode (`true`/`false`)
### Configuration Schema (Zod-Validated)
```typescript
{
reddit: {
subreddits: string[] // Subreddits to monitor
pollInterval: number // Minutes between polls
targetGender: string // Target gender filter
sellerKeywords: string[] // Keywords for spam/seller detection
}
openrouter: {
apiKey: string // OpenRouter API key
model: string // Primary AI model
fallbackModels: string[] // Fallback models on refusal
systemPrompt: string // System prompt for analysis
messagePrompt: string // System prompt for message generation
}
rateLimit: {
minDelayMs: number // Minimum delay between messages
maxMessagesPerHour: number // Hourly message cap
jitterMs: number // Random jitter range
}
browser: {
headless: boolean // Headless mode
userDataDir: string // Persistent session directory
}
database: {
path: string // SQLite database path
}
mediaSync: {
enabled: boolean // Enable media sync
intervalMinutes: number // Sync interval
}
autoReply: {
enabled: boolean // Enable auto-reply
autoSend: boolean // Auto-send or pending approval
checkInterval: number // Check interval in minutes
}
strategyVariants: {
name: string // Variant name
prompt: string // Message style prompt
enabled: boolean // Enable/disable variant
}[]
web: {
port: number // Dashboard port
}
}
```
### Runtime Configuration
The following settings can be modified via the web dashboard without restarting:
- `rateLimit.minDelayMs` - Message delay
- `rateLimit.maxMessagesPerHour` - Hourly cap
- `rateLimit.jitterMs` - Jitter range
- `reddit.pollInterval` - Poll frequency
- `dryRun` - Dry run mode (log only, no actual sends)
---
## Design Patterns
### Repository Pattern
All database operations are abstracted behind three repository classes:
- `PostsRepository` - Post lifecycle management
- `MessagesRepository` - Message tracking and analytics
- `SuccessfulMessagesRepository` - Few-shot learning data
Benefits:
- Single source of truth for data access
- Easy to swap implementations (e.g., different databases)
- Consistent error handling and query patterns
### Dependency Injection
Services receive dependencies via constructor parameters:
```typescript
class ProcessorService {
constructor(
private config: Config,
private redditApi: RedditApi,
private openrouter: OpenRouterService,
private strategySelector: StrategySelector,
private messageSender: MessagePage,
private postsRepo: PostsRepository,
private messagesRepo: MessagesRepository,
private successfulRepo: SuccessfulMessagesRepository,
private logger: Logger
) {}
}
```
Benefits:
- Loose coupling between components
- Easy to mock dependencies for testing
- Clear dependency graph
### Strategy Pattern
`StrategySelector` implements A/B testing with multiple message variants:
```typescript
class StrategySelector {
selectStrategy(subreddit: string, author: string): string {
// Explore/exploit algorithm
// Returns variant name based on response rates
}
}
```
Benefits:
- Swappable message strategies
- Measurable performance per variant
- Configurable exploration rate
### Factory Pattern
Configuration and database creation use factory functions:
- `loadConfig()` - Loads, merges env vars, validates with Zod
- `createDatabase()` - Initializes SQLite, runs migrations
### Observer Pattern
SSE (Server-Sent Events) for realtime log streaming:
```typescript
// Logger writes to a shared buffer
logger.info("Message sent to user123");
// WebServer subscribes clients via SSE
app.get("/api/logs/stream", (req, res) => {
// Push new log entries to connected clients
});
```
### Circuit Breaker / Retry
`withRetry()` wrapper for resilient external calls:
```typescript
await withRetry(
() => redditApi.fetchPosts(subreddit),
{ retries: 3, logger }
);
```
Multiple fallback chains:
- AI model fallback (primary → fallback models)
- Image URL fallback (preview → i.redd.it → imgur → thumbnail)
### Rate Limiter Pattern
Token-bucket-like implementation:
```typescript
class RateLimiter {
async waitForSlot(): Promise<void> {
// Enforce minimum delay + jitter
// Track hourly counts
// Block if hourly cap reached
}
}
```
---
## External Integrations
### Reddit API
- **Authentication**: Playwright browser cookies (persistent session)
- **Endpoints Used**:
- `GET /r/{name}/new.json` - Fetch new posts
- `GET /user/{username}/profile` - Check for seller/spam indicators
- `GET /message/inbox` - Check for responses
- `GET chat.reddit.com` - Chat scraping
- **Rate Limits**: Enforced by `RateLimiter` utility
### OpenRouter AI
- **SDK**: OpenAI-compatible (`openai` npm package)
- **Base URL**: `https://openrouter.ai/api/v1`
- **Endpoints Used**:
- `POST /chat/completions` - Image analysis, message generation, replies
- **Model Fallback**: Primary model → fallback models on refusal
- **Cost Optimization**: Quick gender detection before full analysis
### Playwright Browser
- **Context**: Persistent (cookies/session storage saved to disk)
- **Usage**:
- Automated Reddit login
- DM compose page automation
- Inbox navigation for response checking
- Chat page scraping (shadow DOM traversal)
- Media file downloads
- **Shadow DOM**: JavaScript injection for `faceplate-*` component traversal
---
## Deployment
### Docker
The project includes full Docker support:
- **`Dockerfile`** - Multi-stage build with Node.js
- **`docker-compose.yml`** - Service orchestration
- **`.dockerignore`** - Excludes unnecessary files
### Dokploy
Deployment documentation is provided in `DOKPLOY_DEPLOY.md`.
### Required Environment Variables
| Variable | Required | Description |
|---|---|---|
| `OPENROUTER_API_KEY` | Yes | OpenRouter API key for AI |
| `REDDIT_USERNAME` | Yes | Reddit username |
| `REDDIT_PASSWORD` | Yes | Reddit password |
| `BROWSER_HEADLESS` | No | Browser headless mode (default: `true`) |
### Pre-Deployment Checklist
1. Set up `config.yaml` with target subreddits and settings
2. Set environment variables (or use `.env` file)
3. Run `npm install` to install dependencies
4. Run `npm run build` to compile TypeScript
5. Run `scripts/login.ts` once to establish browser session
6. Start the bot (via `npm start` or Docker)
---
## Technical Debt & Recommendations
### Critical
1. **No Tests** - Zero test coverage. Add unit tests for services, especially:
- `StrategySelector` (complex algorithm)
- `ChatService` (shadow DOM logic)
- `OpenRouterService` (model fallback logic)
2. **No README.md** - No project documentation for new developers.
3. **Hardcoded Username** - `ChatService` has `myUsername = 'Upset_Marsupial1006'` hardcoded in JS evaluation scripts. Should be parameterized from config.
### High Priority
4. **God Class** - `main.ts` (391 lines) creates all dependencies and wires everything. Consider extracting an `AppContainer` or `DependencyContainer`.
5. **ChatService Size** - 1123+ lines. Consider splitting into:
- `ChatScraper` (navigation + DOM traversal)
- `ConversationCache` (cache management)
- `ChatMessenger` (sending + verification)
6. **No .env.example** - No template showing required environment variables.
### Medium Priority
7. **Single-Page Dashboard** - `public/index.html` is ~2000+ lines with embedded CSS/JS. Consider a proper frontend framework.
8. **WebServer Size** - 794 lines. Consider splitting routes into separate files.
9. **Config Mutability** - Runtime config updates via `PUT /api/config` modify the in-memory config object. Consider a proper config service with validation on updates.
10. **Error Handling** - Some services have basic try/catch. Consider structured error types with retry logic.
### Low Priority
11. **Logging** - Winston is well-configured, but consider adding request IDs for tracing.
12. **Type Safety** - Some `page.evaluate()` calls use `any` types. Consider stricter typing for DOM structures.
13. **Media Storage** - Media files stored in `data/media/` with no cleanup policy. Consider retention policies.
docs/configuration.md
+488
View File
@@ -0,0 +1,488 @@
# Configuration Guide
Complete reference for configuring RedditBot.
---
## Table of Contents
1. [Configuration Overview](#configuration-overview)
2. [Configuration Files](#configuration-files)
3. [Environment Variables](#environment-variables)
4. [config.yaml Reference](#configyaml-reference)
5. [Runtime Configuration](#runtime-configuration)
6. [Configuration Validation](#configuration-validation)
7. [Troubleshooting](#troubleshooting)
---
## Configuration Overview
RedditBot uses a layered configuration approach:
```
┌──────────────────────────────────────────────────────────────┐
│ Priority Order (High → Low) │
├──────────────────────────────────────────────────────────────┤
│ 1. Environment Variables │
│ (OPENROUTER_API_KEY, REDDIT_USERNAME, etc.) │
│ │
│ 2. config.yaml │
│ (Primary configuration file) │
│ │
│ 3. Defaults │
│ (Hardcoded defaults in code) │
└──────────────────────────────────────────────────────────────┘
```
Configuration is loaded at startup by `loadConfig()` in `src/config/index.ts`, which:
1. Reads `config.yaml` from the project root
2. Merges environment variable overrides
3. Validates the entire configuration tree using Zod
4. Returns a strongly-typed `Config` object
---
## Configuration Files
### config.yaml
The primary configuration file, located at the project root:
```
RedditBot/
├── config.yaml # ← Main configuration
├── package.json
├── tsconfig.json
└── src/
```
### Data Directory
Runtime data is stored in `data/` (created automatically):
```
data/
├── reddit-bot.db # SQLite database
├── browser/ # Browser session data (cookies, storage)
├── media/ # Downloaded media files
│ ├── index.json # Media index (SHA-1 → filename)
│ ├── abc123.jpg # Media file 1
│ └── def456.png # Media file 2
└── chats/ # Cached conversation JSON files
├── chat_abc123.json
└── chat_def456.json
```
---
## Environment Variables
Environment variables override corresponding values in `config.yaml`.
| Variable | Required | Type | Description | Example |
|---|---|---|---|---|
| `OPENROUTER_API_KEY` | Yes | string | OpenRouter API key for AI services | `sk-or-v1-abc123...` |
| `REDDIT_USERNAME` | Yes | string | Reddit username for authentication | `MyBotUser` |
| `REDDIT_PASSWORD` | Yes | string | Reddit password for authentication | `secret123` |
| `BROWSER_HEADLESS` | No | boolean | Run browser in headless mode | `true` or `false` |
### Setting Environment Variables
**Option 1: `.env` file** (recommended for development)
Create a `.env` file in the project root:
```env
OPENROUTER_API_KEY=sk-or-v1-abc123...
REDDIT_USERNAME=MyBotUser
REDDIT_PASSWORD=secret123
BROWSER_HEADLESS=true
```
> **Note**: Add `.env` to `.gitignore` - never commit credentials.
**Option 2: Shell export**
```bash
export OPENROUTER_API_KEY=sk-or-v1-abc123...
export REDDIT_USERNAME=MyBotUser
export REDDIT_PASSWORD=secret123
```
**Option 3: Docker**
Pass via `docker-compose.yml` or Docker CLI:
```yaml
environment:
- OPENROUTER_API_KEY=sk-or-v1-abc123...
- REDDIT_USERNAME=MyBotUser
- REDDIT_PASSWORD=secret123
```
---
## config.yaml Reference
Complete reference for every configuration option.
### Reddit Settings
```yaml
reddit:
# Subreddits to monitor for new posts
subreddits:
- gonewild
- ratemyboobs
- HomemadeNsfw
- realgirls
- plussize_gonewild
# ... add or remove subreddits
# Minutes between polling cycles
pollInterval: 15
# Target gender for image analysis filtering
# Options: "F" (female), "M" (male)
targetGender: "F"
# Keywords used to detect sellers/spam profiles
# Posts from users matching these keywords are skipped
sellerKeywords:
- seller
- onlyfans
- link in bio
- check my profile
- promo
- discount
- affiliate
```
### OpenRouter AI Settings
```yaml
openrouter:
# OpenRouter API key (can also be set via OPENROUTER_API_KEY env var)
apiKey: "sk-or-v1-abc123..."
# Primary AI model for image analysis
# Models must be available on OpenRouter
model: "meta-llama/llama-3.2-90b-vision-preview"
# Fallback models tried if the primary model refuses or fails
fallbackModels:
- "google/gemini-2.0-flash-001"
- "meta-llama/llama-3.2-11b-vision-instruct"
# Quick detection model (cheaper/faster, used for gender pre-filter)
quickModel: "google/gemini-2.0-flash-lite-001"
# System prompt for image analysis
# The AI will describe the image and detect gender
systemPrompt: |
Analyze this image and provide:
1. Gender detected (F or M)
2. Detailed description of the image content
3. Notable features (hair color, clothing, setting, etc.)
Return as JSON: {"gender": "F", "description": "...", "features": [...]}
# System prompt for message generation
# Used when crafting DMs to post authors
messagePrompt: |
Generate a natural, friendly DM for this Reddit post.
Image analysis: {description}
Post title: {title}
Strategy: {strategy}
Guidelines:
- Be complimentary but not creepy
- Reference something specific from the image
- Keep it under 2 sentences
- Sound like a genuine person, not a bot
- Do NOT ask for anything in return
```
### Rate Limiting
```yaml
rateLimit:
# Minimum delay between messages (milliseconds)
# Higher values reduce detection risk
minDelayMs: 30000
# Maximum messages per hour
# Bot will pause when this limit is reached
maxMessagesPerHour: 20
# Random jitter added to delay (milliseconds)
# Adds unpredictability to timing
jitterMs: 15000
```
**Timing example with defaults:**
```
Minimum delay: 30,000ms (30 seconds)
Jitter: 0 to 15,000ms (0-15 seconds)
Effective delay: 30-45 seconds between messages
Max messages: 20 per hour
```
### Browser Settings
```yaml
browser:
# Run browser in headless mode (no UI)
# Set to false for debugging
headless: true
# Directory for persistent browser data
# Cookies and session storage are saved here
userDataDir: "./data/browser"
```
**Browser session persistence:**
```
data/browser/
├── Default/
│ ├── Cookies # Reddit session cookies
│ ├── Local Storage # Local storage data
│ └── Preferences # Browser preferences
├── Login Data # Saved login info
└── Preferences # Browser state
```
> **Important**: Keep this directory intact. Deleting it requires re-login.
### Database Settings
```yaml
database:
# Path to SQLite database file
path: "./data/reddit-bot.db"
```
The database is auto-created on first run. Migrations run automatically on startup.
### Media Sync Settings
```yaml
mediaSync:
# Enable periodic media synchronization
enabled: true
# Interval between sync cycles (minutes)
intervalMinutes: 30
```
**Sync behavior:**
- **Incremental sync** (automatic): Downloads media from top 10 conversations
- **Full sync** (manual via dashboard): Downloads media from all conversations
### Auto-Reply Settings
```yaml
autoReply:
# Enable automatic chat replies
enabled: true
# Auto-send replies (true) or mark as pending for approval (false)
autoSend: false
# Minutes between checking for new messages
checkInterval: 10
```
**Reply modes:**
| autoSend | Behavior |
|---|---|
| `true` | Bot sends AI-generated replies automatically |
| `false` | Bot suggests replies, user approves via dashboard |
### Strategy Variants (A/B Testing)
```yaml
strategyVariants:
# Variant 1: Compliment-based
- name: "compliment"
prompt: |
Write a friendly compliment about the person's appearance.
Be genuine and specific. Keep it short and natural.
enabled: true
# Variant 2: Question-based
- name: "question"
prompt: |
Write a friendly message that includes a question.
Engage them in conversation naturally.
enabled: true
# Variant 3: Casual observation
- name: "observation"
prompt: |
Write a casual observation about the photo or setting.
Sound like you're commenting to a friend.
enabled: true
```
### Web Dashboard Settings
```yaml
web:
# Port for the web dashboard
port: 3000
```
### Global Settings
```yaml
# Global dry run mode
# When true, all actions are logged but not actually performed
dryRun: false
# Number of successful message examples to include in prompts
fewShotExamples: 3
# Minimum responses needed before trusting strategy stats
minStrategySamples: 10
# Exploration rate for A/B testing (0.0 to 1.0)
# 0.15 = 15% of messages use random variants (exploration)
explorationRate: 0.15
```
---
## Runtime Configuration
The following settings can be modified via the web dashboard without restarting the bot:
| Setting | Endpoint | Type | Description |
|---|---|---|---|
| `rateLimit.minDelayMs` | `PUT /api/config` | number | Message delay |
| `rateLimit.maxMessagesPerHour` | `PUT /api/config` | number | Hourly cap |
| `rateLimit.jitterMs` | `PUT /api/config` | number | Jitter range |
| `reddit.pollInterval` | `PUT /api/config` | number | Poll frequency |
| `dryRun` | `PUT /api/config` | boolean | Dry run mode |
### Updating Runtime Config
Via dashboard:
1. Open the settings modal
2. Modify desired values
3. Click "Save"
Via API:
```bash
curl -X PUT http://localhost:3000/api/config \
-H "Content-Type: application/json" \
-d '{
"rateLimit": {
"minDelayMs": 45000,
"maxMessagesPerHour": 15,
"jitterMs": 20000
},
"dryRun": true
}'
```
> **Note**: Runtime changes are in-memory only. They are lost when the bot restarts. To make changes persistent, update `config.yaml`.
---
## Configuration Validation
All configuration is validated using Zod at startup. Invalid configuration will cause the bot to exit with a descriptive error.
### Validation Rules
| Field | Validation |
|---|---|
| `reddit.subreddits` | Must be non-empty array of strings |
| `reddit.pollInterval` | Must be positive integer |
| `reddit.targetGender` | Must be "F" or "M" |
| `openrouter.apiKey` | Must be non-empty string |
| `openrouter.model` | Must be non-empty string |
| `openrouter.fallbackModels` | Must be non-empty array |
| `rateLimit.minDelayMs` | Must be positive integer |
| `rateLimit.maxMessagesPerHour` | Must be positive integer |
| `rateLimit.jitterMs` | Must be non-negative integer |
| `browser.headless` | Must be boolean |
| `browser.userDataDir` | Must be non-empty string |
| `database.path` | Must be non-empty string |
| `mediaSync.enabled` | Must be boolean |
| `mediaSync.intervalMinutes` | Must be positive integer (if enabled) |
| `autoReply.enabled` | Must be boolean |
| `autoReply.autoSend` | Must be boolean |
| `autoReply.checkInterval` | Must be positive integer (if enabled) |
| `strategyVariants` | Must be array with name + prompt |
| `web.port` | Must be integer 1-65535 |
| `dryRun` | Must be boolean |
### Common Validation Errors
**Error**: `Invalid config: openrouter.apiKey must be a string`
**Fix**: Set `OPENROUTER_API_KEY` environment variable or add `apiKey` to `config.yaml`
**Error**: `Invalid config: reddit.subreddits must be a non-empty array`
**Fix**: Add at least one subreddit to `reddit.subreddits` in `config.yaml`
**Error**: `Invalid config: browser.userDataDir must be a string`
**Fix**: Ensure `browser.userDataDir` is set in `config.yaml`
---
## Troubleshooting
### Configuration Not Loading
**Symptom**: Bot fails to start with validation errors.
**Steps**:
1. Verify `config.yaml` is in the project root
2. Check YAML syntax (use a YAML linter)
3. Verify environment variables are set correctly
4. Check bot logs for specific validation errors
### Credentials Not Working
**Symptom**: Browser login fails.
**Steps**:
1. Verify `REDDIT_USERNAME` and `REDDIT_PASSWORD` are correct
2. Check that the account is not locked or requires 2FA
3. Try manual login via `npm run login`
4. Ensure `BROWSER_HEADLESS=false` for debugging login issues
### AI Not Working
**Symptom**: Image analysis fails or returns errors.
**Steps**:
1. Verify `OPENROUTER_API_KEY` is valid
2. Check API key has sufficient balance
3. Verify `openrouter.model` is available on OpenRouter
4. Check fallback models are set correctly
5. Review logs for specific AI errors
### Browser Session Expired
**Symptom**: Bot stops with "Browser session expired" error.
**Steps**:
1. Delete `data/browser/` directory
2. Run `npm run login` to re-authenticate
3. Restart the bot
### Port Already in Use
**Symptom**: Dashboard fails to start with "Port already in use".
**Steps**:
1. Change `web.port` in `config.yaml`
2. Or kill the process using the port: `lsof -i :3000`
docs/data-flow.md
+875
View File
@@ -0,0 +1,875 @@
# Data Flow Documentation
Complete documentation of data flows through the RedditBot system.
---
## Table of Contents
1. [Overview](#overview)
2. [Post Processing Flow](#post-processing-flow)
3. [Response Monitoring Flow](#response-monitoring-flow)
4. [Chat Interaction Flow](#chat-interaction-flow)
5. [Strategy Selection Flow](#strategy-selection-flow)
6. [Media Sync Flow](#media-sync-flow)
7. [Auto-Reply Flow](#auto-reply-flow)
8. [Dashboard Data Flow](#dashboard-data-flow)
9. [Error Recovery Flows](#error-recovery-flows)
10. [Data Lifecycle](#data-lifecycle)
---
## Overview
RedditBot processes data through several interconnected pipelines:
```
┌─────────────┐ ┌──────────────┐ ┌─────────────┐
│ Reddit API │────►│ Monitor │────►│ Processor │
│ (Posts) │ │ Service │ │ Service │
└─────────────┘ └──────────────┘ └──────┬──────┘
│
┌──────────────┐ ▼
│ Response │ ┌──────────────┐
│ Checker │◄────│ OpenRouter │
└──────────────┘ │ (AI) │
└──────────────┘
│
┌──────────────┐ ▼
│ Chat │ ┌──────────────┐
│ Service │◄────│ Message │
└──────────────┘ │ Sender │
└──────────────┘
│
┌──────────────┐ ▼
│ Media │ ┌──────────────┐
│ Service │◄────│ SQLite DB │
└──────────────┘ └──────────────┘
│
┌──────────────┐ │
│ Dashboard │◄──────────┘
│ (Express) │
└──────────────┘
```
---
## Post Processing Flow
The primary data flow for discovering, analyzing, and messaging about new posts.
### Stage 1: Discovery
```
┌─────────────────────────────────────────────────────────────────┐
│ MonitorService │
│ │
│ 1. Shuffle subreddit order │
│ 2. For each subreddit: │
│ ┌─────────────────────────────────────────────────────┐ │
│ │ RedditApi.fetchPosts(subreddit) │ │
│ │ │ │
│ │ HTTP: GET /r/{subreddit}/new.json?limit=100 │ │
│ │ │ │
│ │ Response: { │ │
│ │ data: { │ │
│ │ children: [ │ │
│ │ { │ │
│ │ data: { │ │
│ │ id: "abc123", │ │
│ │ subreddit: "gonewild", │ │
│ │ author: "user123", │ │
│ │ title: "Summer vibes", │ │
│ │ url: "https://...", │ │
│ │ preview: { reddits: [...] }, │ │
│ │ thumbnail: "https://...", │ │
│ │ post_hint: "image", │ │
│ │ over_18: true │ │
│ │ } │ │
│ │ } │ │
│ │ ] │ │
│ │ } │ │
│ │ } │ │
│ └─────────────────────────────────────────────────────┘ │
│ │
│ 3. Filter posts: │
│ ├─► post_hint === "image" │
│ ├─► author exists and is not deleted │
│ ├─► image_url can be extracted │
│ └─► Author not in seller/spam list │
│ │
│ 4. Cross-reference with DB: │
│ ├─► PostsRepository.getById(postId) → skip if exists │
│ └─► MessagesRepository.getByAuthor(author) → skip if sent │
│ │
│ 5. Return newest valid post │
└─────────────────────────────────────────────────────────────────┘
```
### Stage 2: Image Processing
```
┌─────────────────────────────────────────────────────────────────┐
│ ProcessorService │
│ │
│ 1. Download Image │
│ ┌─────────────────────────────────────────────────────┐ │
│ │ ImageService.downloadImage(url) │ │
│ │ │ │
│ │ HTTP: GET {image_url} │ │
│ │ │ │
│ │ Response: Binary image data │ │
│ │ │ │
│ │ Validation: │ │
│ │ ├─► MIME type: image/jpeg | png | webp │ │
│ │ ├─► File size: < 10MB │ │
│ │ └─► Not empty/broken │ │
│ └─────────────────────────────────────────────────────┘ │
│ │
│ 2. Convert to base64 │
│ ┌─────────────────────────────────────────────────────┐ │
│ │ Buffer.toString('base64') │ │
│ │ │ │
│ │ Result: "data:image/jpeg;base64,/9j/4AAQSkZJR..." │ │
│ └─────────────────────────────────────────────────────┘ │
└─────────────────────────────────────────────────────────────────┘
```
### Stage 3: Gender Detection (Pre-Filter)
```
┌─────────────────────────────────────────────────────────────────┐
│ OpenRouterService │
│ │
│ 1. detectGenderOnly(imageBase64) │
│ ┌─────────────────────────────────────────────────────┐ │
│ │ HTTP: POST https://openrouter.ai/api/v1/chat/ │ │
│ │ completions │ │
│ │ │ │
│ │ Request: { │ │
│ │ model: "fast-model", │ │
│ │ messages: [ │ │
│ │ { │ │
│ │ role: "user", │ │
│ │ content: [ │ │
│ │ { type: "text", text: "Is this F or M?" }, │ │
│ │ { type: "image_url", image_url: {...} } │ │
│ │ ] │ │
│ │ } │ │
│ │ ] │ │
│ │ } │ │
│ │ │ │
│ │ Response: { │ │
│ │ choices: [{ message: { content: "F" } }] │ │
│ │ } │ │
│ └─────────────────────────────────────────────────────┘ │
│ │
│ 3. Compare with config.targetGender │
│ ├─► Match: continue to full analysis │
│ └─► No match: SKIP post (log: "Gender mismatch, skipping") │
└─────────────────────────────────────────────────────────────────┘
```
### Stage 4: Strategy Selection
```
┌─────────────────────────────────────────────────────────────────┐
│ StrategySelector │
│ │
│ 1. Get strategy statistics from DB │
│ ┌─────────────────────────────────────────────────────┐ │
│ │ MessagesRepository.getStrategyStats() │ │
│ │ │ │
│ │ Result: { │ │
│ │ variant_a: { sent: 50, responded: 15, rate: 0.30 },│
│ │ variant_b: { sent: 30, responded: 12, rate: 0.40 },│
│ │ variant_c: { sent: 5, responded: 1, rate: 0.20 } │
│ │ } │ │
│ └─────────────────────────────────────────────────────┘ │
│ │
│ 2. Apply exploration/exploit algorithm │
│ ├─► Random roll < explorationRate → pick random variant │
│ └─► Otherwise → pick highest response rate (min samples) │
│ │
│ 3. Return selected variant name │
│ Example: "variant_b" │
└─────────────────────────────────────────────────────────────────┘
```
### Stage 5: Full AI Analysis
```
┌─────────────────────────────────────────────────────────────────┐
│ OpenRouterService │
│ │
│ 1. analyzeImage(imageBase64) │
│ ┌─────────────────────────────────────────────────────┐ │
│ │ HTTP: POST https://openrouter.ai/api/v1/chat/ │ │
│ │ completions │ │
│ │ │ │
│ │ Request: { │ │
│ │ model: "primary-model", │ │
│ │ messages: [ │ │
│ │ { │ │
│ │ role: "system", │ │
│ │ content: config.openrouter.systemPrompt │ │
│ │ }, │ │
│ │ { │ │
│ │ role: "user", │ │
│ │ content: [ │ │
│ │ { type: "text", text: "Describe this image" },│
│ │ { type: "image_url", image_url: {...} } │ │
│ │ ] │ │
│ │ } │ │
│ │ ] │ │
│ │ } │ │
│ │ │ │
│ │ Response: { │ │
│ │ choices: [{ │ │
│ │ message: { │ │
│ │ content: '{"gender":"F","description":"..."}' │ │
│ │ } │ │
│ │ }] │ │
│ │ } │ │
│ └─────────────────────────────────────────────────────┘ │
│ │
│ 2. Check for refusal │
│ ├─► If refused → try fallback models │
│ └─► If all failed → mark post with error, skip │
│ │
│ 3. Parse and return ImageAnalysis │
│ { gender: "F", description: "Young woman, blonde hair..." }│
└─────────────────────────────────────────────────────────────────┘
```
### Stage 6: Message Generation
```
┌─────────────────────────────────────────────────────────────────┐
│ OpenRouterService │
│ │
│ 1. Retrieve few-shot examples │
│ ┌─────────────────────────────────────────────────────┐ │
│ │ SuccessfulMessagesRepository.getExamples( │ │
│ │ subreddit, count: 3 │ │
│ │ ) │ │
│ │ │ │
│ │ Result: [ │ │
│ │ { │ │
│ │ original_message: "Hey, you look amazing...", │ │
│ │ response_snippet: "Thank you so much!", │ │
│ │ subreddit: "gonewild" │ │
│ │ }, │ │
│ │ // ... more examples │ │
│ │ ] │ │
│ └─────────────────────────────────────────────────────┘ │
│ │
│ 2. Generate message │
│ ┌─────────────────────────────────────────────────────┐ │
│ │ HTTP: POST https://openrouter.ai/api/v1/chat/ │ │
│ │ completions │ │
│ │ │ │
│ │ Request: { │ │
│ │ model: "primary-model", │ │
│ │ messages: [ │ │
│ │ { role: "system", content: messagePrompt }, │ │
│ │ { role: "user", content: [ │ │
│ │ `Image: ${imageAnalysis.description}`, │ │
│ │ `Post title: ${postTitle}`, │ │
│ │ `Strategy: ${selectedVariant}`, │ │
│ │ `Examples: ${JSON.stringify(examples)}` │ │
│ │ ] │ │
│ │ } │ │
│ │ ] │ │
│ │ } │ │
│ └─────────────────────────────────────────────────────┘ │
│ │
│ 3. Return generated message │
│ "Hey, I came across your post and..." │
└─────────────────────────────────────────────────────────────────┘
```
### Stage 7: DM Sending
```
┌─────────────────────────────────────────────────────────────────┐
│ MessagePage │
│ │
│ 1. Navigate to compose page │
│ ┌─────────────────────────────────────────────────────┐ │
│ │ page.goto("https://www.reddit.com/compose?to= │ │
│ │ user123") │ │
│ └─────────────────────────────────────────────────────┘ │
│ │
│ 2. Fill recipient field │
│ ┌─────────────────────────────────────────────────────┐ │
│ │ page.evaluate(() => { │ │
│ │ const input = document.querySelector( │ │
│ │ 'faceplate-autocomplete input' │ │
│ │ ); │ │
│ │ input.value = "user123"; │ │
│ │ input.dispatchEvent(new Event('input', │ │
│ │ { bubbles: true })); │ │
│ │ }); │ │
│ └─────────────────────────────────────────────────────┘ │
│ │
│ 3. Fill subject field ("Stunning") │
│ 4. Fill body field (generated message) │
│ 5. Click send button │
│ 6. Verify success (check for confirmation) │
│ │
│ Result: true (sent) or false (failed) │
└─────────────────────────────────────────────────────────────────┘
```
### Stage 8: Database Recording
```
┌─────────────────────────────────────────────────────────────────┐
│ Database │
│ │
│ 1. INSERT INTO processed_posts │
│ VALUES ( │
│ 'abc123', -- post_id │
│ 'gonewild', -- subreddit │
│ 'user123', -- author │
│ 'Summer vibes', -- title │
│ 'https://...', -- image_url │
│ '{"gender":"F",...}', -- image_analysis │
│ 'Hey, I came across...', -- generated_message │
│ 1716234567, -- processed_at │
│ 1, -- message_sent │
│ NULL -- error │
│ ); │
│ │
│ 2. INSERT INTO sent_messages │
│ VALUES ( │
│ NULL, -- id (auto) │
│ 'abc123', -- post_id │
│ 'user123', -- recipient │
│ 'Stunning', -- subject │
│ 'Hey, I came across...', -- message │
│ 1716234567, -- sent_at │
│ 'variant_b', -- strategy_id │
│ 0, - - got_response │
│ NULL -- response_checked_at │
│ ); │
└─────────────────────────────────────────────────────────────────┘
```
---
## Response Monitoring Flow
How the bot detects and records user replies to sent DMs.
```
┌─────────────────────────────────────────────────────────────────┐
│ ResponseCheckerService │
│ │
│ 1. Navigate to inbox │
│ ┌─────────────────────────────────────────────────────┐ │
│ │ page.goto("https://www.reddit.com/message/inbox") │ │
│ └─────────────────────────────────────────────────────┘ │
│ │
│ 2. Parse inbox items │
│ ┌─────────────────────────────────────────────────────┐ │
│ │ Extract from page: │ │
│ │ [ │ │
│ │ { │ │
│ │ from: "user456", │ │
│ │ subject: "Re: Stunning", │ │
│ │ snippet: "Thank you so much! 😊", │ │
│ │ timestamp: 1716234000, │ │
│ │ link: "/message/xyz789" │ │
│ │ }, │ │
│ │ // ... more messages │ │
│ │ ] │ │
│ └─────────────────────────────────────────────────────┘ │
│ │
│ 3. Match responses to sent messages │
│ ┌─────────────────────────────────────────────────────┐ │
│ │ For each inbox item: │ │
│ │ ├─► MessagesRepository.getByRecipient(from) │ │
│ │ ├─► Find unmatched message within time window │ │
│ │ ├─► Update: got_response = 1 │ │
│ │ ├─► Update: response_checked_at = now │ │
│ │ └─► Store in successful_messages │ │
│ └─────────────────────────────────────────────────────┘ │
│ │
│ 4. Update strategy statistics │
│ ┌─────────────────────────────────────────────────────┐ │
│ │ MessagesRepository.updateStrategyStats( │ │
│ │ strategy_id, got_response: true │ │
│ │ ) │ │
│ └─────────────────────────────────────────────────────┘ │
│ │
│ 5. Log results │
│ └─► "Found 3 responses, updated 2 strategies" │
└─────────────────────────────────────────────────────────────────┘
```
---
## Chat Interaction Flow
How the bot interacts with Reddit chat conversations.
```
┌─────────────────────────────────────────────────────────────────┐
│ ChatService │
│ │
│ ──── PHASE 1: FETCH CHATS ────────────────────────────────── │
│ │
│ 1. Navigate to chat page │
│ page.goto("https://chat.reddit.com") │
│ │
│ 2. Traverse shadow DOM for conversations │
│ ┌─────────────────────────────────────────────────────┐ │
│ │ page.evaluate(() => { │ │
│ │ // JS injection to pierce shadow DOM │ │
│ │ const roots = document.querySelectorAll( │ │
│ │ 'faceplate-chat-list' │ │
│ │ ); │ │
│ │ const chats = []; │ │
│ │ roots.forEach(root => { │ │
│ │ const shadow = root.shadowRoot; │ │
│ │ const items = shadow.querySelectorAll( │ │
│ │ '.chat-item' │ │
│ │ ); │ │
│ │ items.forEach(item => { │ │
│ │ chats.push({ │ │
│ │ id: item.dataset.chatId, │ │
│ │ participant: item.querySelector( │ │
│ │ '.username' │ │
│ │ ).textContent, │ │
│ │ lastMessage: item.querySelector( │ │
│ │ '.message' │ │
│ │ ).textContent, │ │
│ │ timestamp: item.dataset.timestamp │ │
│ │ }); │ │
│ │ }); │ │
│ │ }); │ │
│ │ return chats; │ │
│ │ }); │ │
│ └─────────────────────────────────────────────────────┘ │
│ │
│ 3. Cache results │
│ └─► Save to memory + disk JSON │
│ │
│ ──── PHASE 2: LOAD CONVERSATION ──────────────────────────── │
│ │
│ 1. Check cache │
│ ├─► If cached and fresh: return from memory │
│ └─► If not: fetch from disk or Reddit │
│ │
│ 2. Fetch full history (pagination) │
│ ┌─────────────────────────────────────────────────────┐ │
│ │ For each page of messages: │ │
│ │ ├─► Navigate to chat view │ │
│ │ ├─► Extract messages from shadow DOM │ │
│ │ ├─► Scroll to load more (infinite scroll) │ │
│ │ └─► Repeat until all loaded │ │
│ └─────────────────────────────────────────────────────┘ │
│ │
│ 3. Return conversation object │
│ { │
│ chatId: "abc123", │
│ participants: ["user456", "bot"], │
│ messages: [ │
│ { author: "user456", text: "Hey!", timestamp: ... }, │
│ { author: "bot", text: "Hi there!", timestamp: ... }, │
│ // ... more messages │
│ ], │
│ lastUpdated: 1716234567 │
│ } │
│ │
│ ──── PHASE 3: SEND MESSAGE ───────────────────────────────── │
│ │
│ 1. Navigate to chat view │
│ 2. Fill message input (shadow DOM) │
│ 3. Apply typing delay (human-like) │
│ ┌─────────────────────────────────────────────────────┐ │
│ │ const typingDelay = random(500, 2000); │ │
│ │ await page.waitForTimeout(typingDelay); │ │
│ │ │ │
│ │ // Simulate typing character by character │ │
│ │ for (const char of message) { │ │
│ │ input.value += char; │ │
│ │ await page.waitForTimeout(random(30, 100)); │ │
│ │ } │ │
│ └─────────────────────────────────────────────────────┘ │
│ 4. Submit message │
│ 5. Verify delivery (check for sent indicator) │
│ │
│ Result: true (delivered) or false (failed) │
└─────────────────────────────────────────────────────────────────┘
```
---
## Strategy Selection Flow
How A/B test variants are selected and updated.
```
┌─────────────────────────────────────────────────────────────────┐
│ StrategySelector │
│ │
│ SELECT STRATEGY │
│ ──────────────────────────────────────────────────────────── │
│ │
│ Input: subreddit, author │
│ │
│ 1. Get stats from DB │
│ ┌─────────────────────────────────────────────────────┐ │
│ │ SELECT strategy_id, │ │
│ │ COUNT(*) as sent, │ │
│ │ SUM(got_response) as responded │ │
│ │ FROM sent_messages │ │
│ │ GROUP BY strategy_id │ │
│ └─────────────────────────────────────────────────────┘ │
│ │
│ 2. Calculate response rates │
│ variant_a: 15/50 = 30% │
│ variant_b: 12/30 = 40% │
│ variant_c: 1/5 = 20% │
│ │
│ 3. Filter by minimum samples │
│ minSamples = 10 │
│ variant_c excluded (only 5 sends) │
│ │
│ 4. Explore vs Exploit │
│ random() < explorationRate (e.g., 0.15)? │
│ ├─► Yes: Random pick from enabled variants │
│ └─► No: Pick highest rate (variant_b at 40%) │
│ │
│ Output: selected variant name │
│ │
│ ──── UPDATE STATS ────────────────────────────────────────── │
│ │
│ Input: strategy_id, got_response (boolean) │
│ │
│ 1. Record in sent_messages │
│ UPDATE sent_messages │
│ SET got_response = 1, │
│ response_checked_at = now │
│ WHERE id = ? │
│ │
│ 2. If successful: store in successful_messages │
│ INSERT INTO successful_messages │
│ VALUES (original_message, image_analysis, post_title, │
│ response_snippet, subreddit, now) │
│ │
│ 3. Update strategy statistics │
│ (reflected in next selectStrategy call) │
└─────────────────────────────────────────────────────────────────┘
```
---
## Media Sync Flow
How media files are downloaded and synchronized from chat conversations.
```
┌─────────────────────────────────────────────────────────────────┐
│ MediaSyncService │
│ │
│ ──── TRIGGER ──────────────────────────────────────────────── │
│ │
│ Scheduled: every N minutes (config.mediaSync.intervalMinutes) │
│ Manual: POST /api/media/sync (incremental) │
│ Manual: POST /api/media/sync/full (full) │
│ │
│ ──── INCREMENTAL SYNC ────────────────────────────────────── │
│ │
│ 1. Get top 10 conversations from ChatService │
│ 2. For each conversation: │
│ ├─► Check if media already downloaded (index lookup) │
│ ├─► If not: download media │
│ │ └─► MediaService.downloadMedia(chatId) │
│ └─► Update index │
│ │
│ ──── FULL SYNC ────────────────────────────────────────────── │
│ │
│ 1. Get ALL conversations from ChatService │
│ 2. For each conversation: │
│ ├─► Download any missing media │
│ └─► Update index │
│ │
│ ──── MEDIA DOWNLOAD (MediaService) ────────────────────────── │
│ │
│ 1. Extract media URLs from conversation messages │
│ 2. For each URL: │
│ ├─► Check if Matrix thumbnail │
│ │ └─► Resolve to full resolution URL │
│ ├─► Download via Playwright │
│ ├─► Compute SHA-1 hash │
│ ├─► Check if already exists (dedup) │
│ ├─► Save to data/media/{hash}.{ext} │
│ └─► Update index JSON │
│ │
│ Output: Array of saved file paths │
└─────────────────────────────────────────────────────────────────┘
```
---
## Auto-Reply Flow
How the bot automatically replies to chat messages.
```
┌─────────────────────────────────────────────────────────────────┐
│ AutoReplyService │
│ │
│ Scheduled: every N minutes (config.autoReply.checkInterval) │
│ │
│ 1. Get cached conversations │
│ 2. For each conversation: │
│ │ │
│ ├─► Check for new/unread messages │
│ │ └─► If no new messages: continue │
│ │ │
│ ├─► Detect upsell/spam keywords │
│ │ ├─► "onlyfans", "link in bio", "check my..." │
│ │ ├─► If detected: flag conversation, skip reply │
│ │ └─► If not: continue │
│ │ │
│ ├─► Generate AI reply suggestion │
│ │ ┌─────────────────────────────────────────────────┐ │
│ │ │ OpenRouter.generateReply(conversationHistory) │ │
│ │ │ │ │
│ │ │ Request: { │ │
│ │ │ messages: [ │ │
│ │ │ { role: "system", content: replyPrompt }, │ │
│ │ │ { role: "user", content: conversation } │ │
│ │ │ ] │ │
│ │ │ } │ │
│ │ │ │ │
│ │ │ Response: "Thanks for reaching out! 😊" │ │
│ │ │ │ │
│ │ │ Reply: "Thanks for reaching out! 😊" │ │
│ │ └─────────────────────────────────────────────────┘ │
│ │ │
│ ├─► If autoSend = true: │
│ │ └─► ChatService.sendMessage(chatId, reply) │
│ │ └─► Log: "Auto-sent reply to {user}" │
│ │ │
│ └─► If autoSend = false: │
│ └─► Mark as pending (visible in dashboard) │
│ └─► User can approve/reject via dashboard │
│ │
│ Output: Summary of actions taken │
│ "Checked 15 conversations: 3 new messages, 1 reply sent" │
└─────────────────────────────────────────────────────────────────┘
```
---
## Dashboard Data Flow
How data flows from services to the web dashboard.
```
┌─────────────────────────────────────────────────────────────────┐
│ WebServer (Express) │
│ │
│ ──── REST API ─────────────────────────────────────────────── │
│ │
│ GET /api/stats │
│ ├─► Read from: │
│ │ ├─► Bot running state (in-memory) │
│ │ ├─► PostsRepository.countProcessed() │
│ │ ├─► MessagesRepository.countSent() │
│ │ ├─► MessagesRepository.countResponded() │
│ │ └─► Calculate response rate │
│ └─► Return: { running, processed, sent, responded, rate } │
│ │
│ GET /api/messages │
│ └─► MessagesRepository.getRecent(limit: 50) │
│ │
│ GET /api/analytics/by-strategy │
│ └─► MessagesRepository.getStrategyStats() │
│ │
│ PUT /api/config │
│ ├─► Validate new config values │
│ ├─► Update in-memory config │
│ └─► Return updated config │
│ │
│ ──── SSE LOG STREAMING ────────────────────────────────────── │
│ │
│ GET /api/logs/stream │
│ ├─► Client connects │
│ ├─► Send any pending log entries │
│ ├─► Subscribe to logger buffer │
│ ├─► Push new entries as they arrive │
│ │ └─► Event: log │
│ │ └─► Data: { level, message, timestamp } │
│ └─► On disconnect: unsubscribe │
│ │
│ ──── FRONTEND (public/index.html) ─────────────────────────── │
│ │
│ JavaScript polls API endpoints: │
│ ├─► /api/stats every 5 seconds │
│ ├─► /api/messages every 10 seconds │
│ ├─► /api/timers every 2 seconds │
│ └─► SSE /api/logs/stream (persistent connection) │
│ │
│ Updates DOM in real-time: │
│ ├─► Stats cards (processed, sent, responded, rate) │
│ ├─► Activity feed (scrolling list) │
│ ├─► Message viewer (click to expand) │
│ ├─► Chat browser (conversation list) │
│ ├─► Media gallery (image grid) │
│ └─► Rate limiter timers (countdown) │
└─────────────────────────────────────────────────────────────────┘
```
---
## Error Recovery Flows
### AI Refusal Recovery
```
1. OpenRouter.analyzeImage() returns refusal
│
├─► Detect refusal via heuristic patterns
│ ├─► "cannot analyze"
│ ├─► "against policy"
│ └─► "unable to process"
│
├─► Try next model in fallback chain
│ ├─► OpenRouter.fallbackModels[0]
│ ├─► OpenRouter.fallbackModels[1]
│ └─► ...
│
└─► If all models refuse:
├─► Mark post as processed with error
├─► Log: "All models refused to analyze post {id}"
└─► Continue to next post
```
### Network Error Recovery
```
1. RedditApi.fetchPosts() throws error
│
└─► withRetry() wrapper handles recovery:
├─► Retry 1: wait 5s
├─► Retry 2: wait 15s
├─► Retry 3: wait 45s
└─► After 3 failures:
├─► Log error
├─► Skip this subreddit
└─► Continue to next subreddit
```
### Browser Session Recovery
```
1. BrowserClient.verifySession() returns false
│
├─► Check for expired cookies
│
├─► Attempt automated re-login
│ ├─► BrowserClient.login()
│ └─► If success: continue
│
└─► If re-login fails:
├─► Log error: "Browser session expired"
├─► Stop bot
└─► Require manual login (scripts/login.ts)
```
---
## Data Lifecycle
### Post Data
```
Discovered (MonitorService)
│
├─► Stored in processed_posts (pending)
│
├─► Processed (ProcessorService)
│ ├─► image_analysis updated
│ ├─► generated_message updated
│ └─► message_sent updated
│
└─► Archived (retention policy TBD)
└─► Kept indefinitely for analytics
```
### Message Data
```
Generated (ProcessorService)
│
├─► Stored in sent_messages (got_response = 0)
│
├─► Response checked (ResponseCheckerService)
│ ├─► got_response = 1
│ └─► response_checked_at set
│
├─► If responded:
│ └─► Copied to successful_messages (for learning)
│
└─► Retained indefinitely for analytics
```
### Learning Data
```
Successful pair created (ResponseCheckerService)
│
├─► Stored in successful_messages
│ ├─► original_message
│ ├─► image_analysis
│ ├─► post_title
│ ├─► response_snippet
│ └─► subreddit (for weighted selection)
│
├─► Retrieved for few-shot learning
│ └─► Included in OpenRouter.generateMessage() prompt
│
└─► Retained indefinitely (grows bot intelligence)
```
### Chat Data
```
Conversation discovered (ChatService)
│
├─► Cached in memory
├─► Persisted to disk JSON
│
├─► Messages added as they arrive
│
├─► Media downloaded (MediaService)
│ └─► Stored in data/media/
│
└─► Retained until manually cleared
└─► No automatic cleanup (TBD)
```
### Media Data
```
Media URL discovered
│
├─► Downloaded (MediaService)
│ ├─► Matrix thumbnails resolved
│ ├─► SHA-1 hash computed
│ └─► Saved to data/media/{hash}.{ext}
│
├─► Indexed in data/media/index.json
│
├─► Deduplicated (hash check before download)
│
└─► Retained until manually cleared
└─► No automatic cleanup (TBD)
```
docs/database.md
+550
View File
@@ -0,0 +1,550 @@
# Database Schema
Complete reference for the SQLite database schema, migrations, and query patterns.
---
## Table of Contents
1. [Overview](#overview)
2. [Database File](#database-file)
3. [Schema](#schema)
4. [Indexes](#indexes)
5. [Migrations](#migrations)
6. [Query Patterns](#query-patterns)
7. [Repository Methods](#repository-methods)
8. [Data Retention](#data-retention)
---
## Overview
RedditBot uses SQLite (via `better-sqlite3`) for persistent storage. The database tracks processed posts, sent messages, and successful message/response pairs.
**Features**:
- WAL mode for better concurrent read performance
- Foreign key constraints enabled
- Progressive migrations for schema evolution
- Repository pattern for data access
---
## Database File
**Default path**: `./data/reddit-bot.db`
Configured in `config.yaml`:
```yaml
database:
path: "./data/reddit-bot.db"
```
The database file is auto-created on first run.
---
## Schema
### Table: `processed_posts`
Tracks all posts that have been discovered and processed.
```sql
CREATE TABLE processed_posts (
id TEXT PRIMARY KEY, -- Reddit post ID
subreddit TEXT NOT NULL, -- Source subreddit
author TEXT NOT NULL, -- Post author username
title TEXT, -- Post title
image_url TEXT, -- Original image URL
image_analysis TEXT, -- AI-generated JSON description
generated_message TEXT, -- AI-generated DM text
processed_at INTEGER NOT NULL, -- Unix timestamp
message_sent INTEGER DEFAULT 0, -- 0 = not sent, 1 = sent
error TEXT -- Error message if processing failed
);
```
**Column Details**:
| Column | Type | Constraints | Description |
|---|---|---|---|
| `id` | TEXT | PRIMARY KEY | Reddit post ID (e.g., "abc123") |
| `subreddit` | TEXT | NOT NULL | Source subreddit name |
| `author` | TEXT | NOT NULL | Post author username |
| `title` | TEXT | | Post title (nullable if unavailable) |
| `image_url` | TEXT | | Original image URL |
| `image_analysis` | TEXT | | JSON string from AI analysis |
| `generated_message` | TEXT | | The DM text that was generated |
| `processed_at` | INTEGER | NOT NULL | Unix timestamp when processed |
| `message_sent` | INTEGER | DEFAULT 0 | 0 = not sent, 1 = successfully sent |
| `error` | TEXT | | Error message if processing failed |
**Example Row**:
```json
{
"id": "abc123",
"subreddit": "gonewild",
"author": "user123",
"title": "Summer vibes",
"image_url": "https://i.redd.it/abc123.jpg",
"image_analysis": "{\"gender\":\"F\",\"description\":\"Young woman...\"}",
"generated_message": "Hey, you look amazing...",
"processed_at": 1716234567,
"message_sent": 1,
"error": null
}
```
---
### Table: `sent_messages`
Tracks all DMs sent by the bot.
```sql
CREATE TABLE sent_messages (
id INTEGER PRIMARY KEY AUTOINCREMENT,
post_id TEXT NOT NULL,
recipient TEXT NOT NULL,
subject TEXT DEFAULT 'Stunning',
message TEXT NOT NULL,
sent_at INTEGER NOT NULL,
strategy_id TEXT,
got_response INTEGER DEFAULT 0,
response_checked_at INTEGER,
FOREIGN KEY (post_id) REFERENCES processed_posts(id)
);
```
**Column Details**:
| Column | Type | Constraints | Description |
|---|---|---|---|
| `id` | INTEGER | PRIMARY KEY AUTOINCREMENT | Internal ID |
| `post_id` | TEXT | NOT NULL, FK | Source post ID |
| `recipient` | TEXT | NOT NULL | Recipient username |
| `subject` | TEXT | DEFAULT 'Stunning' | DM subject line |
| `message` | TEXT | NOT NULL | DM body text |
| `sent_at` | INTEGER | NOT NULL | Unix timestamp when sent |
| `strategy_id` | TEXT | | A/B test variant name |
| `got_response` | INTEGER | DEFAULT 0 | 0 = no response, 1 = responded |
| `response_checked_at` | INTEGER | | Timestamp of last response check |
**Example Row**:
```json
{
"id": 1,
"post_id": "abc123",
"recipient": "user123",
"subject": "Stunning",
"message": "Hey, you look amazing...",
"sent_at": 1716234567,
"strategy_id": "compliment",
"got_response": 1,
"response_checked_at": 1716238167
}
```
---
### Table: `successful_messages`
Stores successful message/response pairs for few-shot learning.
```sql
CREATE TABLE successful_messages (
id INTEGER PRIMARY KEY AUTOINCREMENT,
sent_message_id INTEGER,
original_message TEXT NOT NULL,
image_analysis TEXT,
post_title TEXT,
response_snippet TEXT,
subreddit TEXT,
created_at INTEGER NOT NULL,
FOREIGN KEY (sent_message_id) REFERENCES sent_messages(id)
);
```
**Column Details**:
| Column | Type | Constraints | Description |
|---|---|---|---|
| `id` | INTEGER | PRIMARY KEY AUTOINCREMENT | Internal ID |
| `sent_message_id` | INTEGER | FK, nullable | Related sent message ID |
| `original_message` | TEXT | NOT NULL | The DM that got a response |
| `image_analysis` | TEXT | | Context from image analysis |
| `post_title` | TEXT | | Original post title |
| `response_snippet` | TEXT | | First reply text |
| `subreddit` | TEXT | | For weighted example selection |
| `created_at` | INTEGER | NOT NULL | Unix timestamp |
**Example Row**:
```json
{
"id": 1,
"sent_message_id": 1,
"original_message": "Hey, you look amazing...",
"image_analysis": "{\"gender\":\"F\",\"description\":\"...\"}",
"post_title": "Summer vibes",
"response_snippet": "Thank you so much! 😊",
"subreddit": "gonewild",
"created_at": 1716238167
}
```
---
## Indexes
### `processed_posts` Indexes
```sql
CREATE INDEX idx_posts_subreddit ON processed_posts(subreddit);
CREATE INDEX idx_posts_author ON processed_posts(author);
```
**Purpose**: Fast lookup by subreddit (for processing stats) and by author (for skip-checking).
---
### `sent_messages` Indexes
```sql
CREATE INDEX idx_messages_recipient ON sent_messages(recipient);
CREATE INDEX idx_messages_strategy ON sent_messages(strategy_id);
CREATE INDEX idx_messages_response ON sent_messages(got_response);
```
**Purpose**:
- `idx_messages_recipient`: Fast lookup when matching responses
- `idx_messages_strategy`: Analytics by strategy variant
- `idx_messages_response`: Filter for responded/not-responded
---
### `successful_messages` Indexes
```sql
CREATE INDEX idx_success_subreddit ON successful_messages(subreddit);
CREATE INDEX idx_success_message ON successful_messages(sent_message_id);
```
**Purpose**:
- `idx_success_subreddit`: Weighted example selection by subreddit
- `idx_success_message`: Link back to sent message
---
## Migrations
Migrations are defined in `src/database/client.ts` and run automatically on every startup.
### Migration Logic
```typescript
// Pseudo-code for migration process
function runMigrations(db: Database) {
// Enable features
db.exec("PRAGMA journal_mode=WAL;");
db.exec("PRAGMA foreign_keys=ON;");
// Create tables if they don't exist
db.exec(`CREATE TABLE IF NOT EXISTS processed_posts (...)`);
db.exec(`CREATE TABLE IF NOT EXISTS sent_messages (...)`);
db.exec(`CREATE TABLE IF NOT EXISTS successful_messages (...)`);
// Create indexes if they don't exist
db.exec(`CREATE INDEX IF NOT EXISTS idx_posts_subreddit ON processed_posts(subreddit)`);
// ... more indexes
// Safe column additions (for schema evolution)
db.exec(`ALTER TABLE processed_posts ADD COLUMN error TEXT`);
}
```
### Migration Safety
- All `CREATE TABLE` statements use `IF NOT EXISTS`
- All `CREATE INDEX` statements use `IF NOT EXISTS`
- Column additions use `ALTER TABLE ADD COLUMN IF NOT EXISTS` pattern
- Migrations are idempotent (safe to run multiple times)
### Adding New Columns
When adding new columns to existing tables:
```typescript
// In src/database/client.ts
const tableInfo = db.prepare(`PRAGMA table_info(processed_posts)`).all();
const hasErrorColumn = tableInfo.some((col: any) => col.name === 'error');
if (!hasErrorColumn) {
db.exec("ALTER TABLE processed_posts ADD COLUMN error TEXT");
}
```
---
## Query Patterns
### Post Processing
**Check if post was already processed**:
```sql
SELECT id FROM processed_posts WHERE id = ?;
```
**Mark post as processed**:
```sql
INSERT OR REPLACE INTO processed_posts
(id, subreddit, author, title, image_url, image_analysis,
generated_message, processed_at, message_sent, error)
VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?);
```
**Mark post as sent**:
```sql
UPDATE processed_posts
SET message_sent = 1
WHERE id = ?;
```
**Update post with analysis**:
```sql
UPDATE processed_posts
SET image_analysis = ?, generated_message = ?
WHERE id = ?;
```
---
### Message Tracking
**Record a sent message**:
```sql
INSERT INTO sent_messages
(post_id, recipient, subject, message, sent_at, strategy_id)
VALUES (?, ?, 'Stunning', ?, ?, ?);
```
**Get recent messages**:
```sql
SELECT * FROM sent_messages
ORDER BY sent_at DESC
LIMIT ?;
```
**Update response status**:
```sql
UPDATE sent_messages
SET got_response = 1, response_checked_at = ?
WHERE id = ?;
```
**Check if user was previously messaged**:
```sql
SELECT COUNT(*) FROM sent_messages WHERE recipient = ?;
```
---
### Strategy Analytics
**Get strategy statistics**:
```sql
SELECT
strategy_id,
COUNT(*) as sent,
SUM(got_response) as responded
FROM sent_messages
WHERE strategy_id IS NOT NULL
GROUP BY strategy_id;
```
**Get response rate by subreddit**:
```sql
SELECT
p.subreddit,
COUNT(*) as sent,
SUM(s.got_response) as responded
FROM sent_messages s
JOIN processed_posts p ON s.post_id = p.id
GROUP BY p.subreddit;
```
**Get daily response trend**:
```sql
SELECT
DATE(sent_at, 'unixepoch') as date,
COUNT(*) as sent,
SUM(got_response) as responded
FROM sent_messages
GROUP BY DATE(sent_at, 'unixepoch')
ORDER BY date DESC
LIMIT 30;
```
---
### Few-Shot Learning
**Get successful examples for a subreddit**:
```sql
SELECT
original_message,
image_analysis,
post_title,
response_snippet
FROM successful_messages
WHERE subreddit = ?
ORDER BY created_at DESC
LIMIT 3;
```
**Store successful pair**:
```sql
INSERT INTO successful_messages
(sent_message_id, original_message, image_analysis,
post_title, response_snippet, subreddit, created_at)
VALUES (?, ?, ?, ?, ?, ?, ?);
```
---
## Repository Methods
### PostsRepository
```typescript
class PostsRepository {
/** Check if a post was already processed */
getById(id: string): ProcessedPost | undefined
/** Mark a post as processed */
insert(post: ProcessedPostInsert): void
/** Mark a post as having a message sent */
markSent(id: string): void
/** Update post with AI analysis results */
updateAnalysis(id: string, analysis: string, message: string): void
/** Get total count of processed posts */
count(): number
/** Get count by subreddit */
countBySubreddit(subreddit: string): number
}
```
### MessagesRepository
```typescript
class MessagesRepository {
/** Record a sent message */
insert(message: MessageInsert): number
/** Get recent sent messages */
getRecent(limit: number): SentMessage[]
/** Get messages by recipient */
getByRecipient(recipient: string): SentMessage[]
/** Update response status */
markResponded(id: number, checkedAt: number): void
/** Get strategy statistics */
getStrategyStats(): StrategyStat[]
/** Get response rate by subreddit */
getSubredditStats(): SubredditStat[]
/** Get daily response trend */
getDailyTrend(days: number): DailyTrend[]
/** Get total count of sent messages */
count(): number
/** Get count of responded messages */
countResponded(): number
}
```
### SuccessfulMessagesRepository
```typescript
class SuccessfulMessagesRepository {
/** Store a successful message/response pair */
insert(pair: SuccessfulMessageInsert): void
/** Get examples for few-shot learning */
getExamples(subreddit: string, count: number): Example[]
/** Get total count */
count(): number
/** Get recent examples */
getRecent(limit: number): SuccessfulMessage[]
}
```
---
## Data Retention
### Current Behavior
All data is retained indefinitely. There is no automatic cleanup or retention policy.
| Table | Retention | Notes |
|---|---|---|
| `processed_posts` | Indefinite | Grows with each post processed |
| `sent_messages` | Indefinite | Grows with each message sent |
| `successful_messages` | Indefinite | Grows with each response received |
### Estimated Growth
With default settings (20 messages/hour, 24/7 operation):
| Timeframe | Posts | Messages | Responses (est. 30%) |
|---|---|---|---|
| 1 day | ~480 | ~288 | ~86 |
| 1 week | ~3,360 | ~2,016 | ~605 |
| 1 month | ~14,400 | ~8,640 | ~2,592 |
| 1 year | ~175,200 | ~105,120 | ~31,536 |
### Recommended Cleanup (Future)
Consider implementing automated cleanup:
```sql
-- Archive old processed posts
CREATE TABLE processed_posts_archive AS
SELECT * FROM processed_posts
WHERE processed_at < ?;
-- Delete archived posts
DELETE FROM processed_posts
WHERE processed_at < ?;
-- Keep last N months of data
```
> **Note**: Successful messages should NEVER be deleted as they are used for few-shot learning.
docs/deployment.md
+413
View File
@@ -0,0 +1,413 @@
# Deployment Guide
Complete guide for deploying RedditBot in various environments.
---
## Table of Contents
1. [Prerequisites](#prerequisites)
2. [Local Development](#local-development)
3. [Docker Deployment](#docker-deployment)
4. [Docker Compose](#docker-compose)
5. [Dokploy Deployment](#dokploy-deployment)
6. [Production Considerations](#production-considerations)
7. [Monitoring & Maintenance](#monitoring--maintenance)
8. [Troubleshooting](#troubleshooting-1)
---
## Prerequisites
### Required
| Requirement | Version | Notes |
|---|---|---|
| Node.js | 18+ | LTS recommended |
| npm | 9+ | Or yarn/pnpm |
| OpenRouter API Key | - | Sign up at [openrouter.ai](https://openrouter.ai) |
| Reddit Account | - | With DM permissions enabled |
### Recommended
| Requirement | Notes |
|---|---|
| 1 GB RAM | Minimum for browser automation |
| 1 CPU core | More for higher message throughput |
| 5 GB disk | For database, media, and browser data |
---
## Local Development
### 1. Clone and Install
```bash
git clone <repository-url>
cd RedditBot
npm install
```
### 2. Configure
Create `config.yaml`:
```yaml
reddit:
subreddits:
- gonewild
pollInterval: 15
targetGender: "F"
sellerKeywords:
- seller
- onlyfans
openrouter:
apiKey: "sk-or-v1-your-key-here"
model: "meta-llama/llama-3.2-90b-vision-preview"
fallbackModels:
- "google/gemini-2.0-flash-001"
rateLimit:
minDelayMs: 30000
maxMessagesPerHour: 20
jitterMs: 15000
browser:
headless: true
userDataDir: "./data/browser"
database:
path: "./data/reddit-bot.db"
web:
port: 3000
dryRun: true # Start in dry run mode
```
Set environment variables:
```bash
export REDDIT_USERNAME="your-username"
export REDDIT_PASSWORD="your-password"
```
### 3. Initial Login
Run the login script to establish a browser session:
```bash
npm run login
```
This will:
1. Open a browser window
2. Navigate to Reddit login
3. Wait for you to complete login (including 2FA if enabled)
4. Save cookies to `data/browser/`
### 4. Start the Bot
```bash
# Development (with TypeScript execution)
npm run dev
# Production (compiled JavaScript)
npm start
```
### 5. Access Dashboard
Open http://localhost:3000 in your browser.
---
## Docker Deployment
### Build the Image
```bash
docker build -t reddit-bot .
```
### Run the Container
```bash
docker run -d \
--name reddit-bot \
-p 3000:3000 \
-v $(pwd)/data:/app/data \
-e OPENROUTER_API_KEY=sk-or-v1-your-key \
-e REDDIT_USERNAME=your-username \
-e REDDIT_PASSWORD=your-password \
-e BROWSER_HEADLESS=true \
reddit-bot:latest
```
### Dockerfile Structure
```dockerfile
# Build stage
FROM node:18-alpine AS builder
WORKDIR /app
COPY package*.json ./
RUN npm ci
COPY . .
RUN npm run build
# Production stage
FROM node:18-alpine
WORKDIR /app
COPY --from=builder /app/dist ./dist
COPY --from=builder /app/node_modules ./node_modules
COPY --from=builder /app/config.yaml ./config.yaml
COPY --from=builder /app/package.json ./package.json
EXPOSE 3000
CMD ["node", "dist/index.js"]
```
---
## Docker Compose
### docker-compose.yml
```yaml
version: '3.8'
services:
reddit-bot:
build: .
container_name: reddit-bot
ports:
- "3000:3000"
volumes:
- ./data:/app/data
environment:
- OPENROUTER_API_KEY=${OPENROUTER_API_KEY}
- REDDIT_USERNAME=${REDDIT_USERNAME}
- REDDIT_PASSWORD=${REDDIT_PASSWORD}
- BROWSER_HEADLESS=true
restart: unless-stopped
healthcheck:
test: ["CMD", "wget", "--spider", "http://localhost:3000/api/stats"]
interval: 30s
timeout: 10s
retries: 3
start_period: 60s
```
### Start with Compose
```bash
# Set environment variables
cp .env.example .env
# Edit .env with your credentials
# Build and start
docker compose up -d --build
# View logs
docker compose logs -f reddit-bot
# Stop
docker compose down
# Restart
docker compose restart
```
---
## Dokploy Deployment
Dokploy is a self-hosted platform for deploying applications. See `DOKPLOY_DEPLOY.md` for detailed instructions.
### Quick Start
1. **Create Application**: In Dokploy dashboard, create a new Docker application
2. **Set Environment Variables**:
- `OPENROUTER_API_KEY`
- `REDDIT_USERNAME`
- `REDDIT_PASSWORD`
- `BROWSER_HEADLESS`
3. **Mount Volume**: Mount `./data` for persistent storage
4. **Set Port**: Map container port 3000 to host port
5. **Deploy**: Push to repository and trigger deployment
### Dokploy Configuration
```yaml
# Dokploy-specific settings
app:
type: docker
repository: <your-repo>
branch: main
dockerfilePath: Dockerfile
environment:
OPENROUTER_API_KEY: $OPENROUTER_API_KEY
REDDIT_USERNAME: $REDDIT_USERNAME
REDDIT_PASSWORD: $REDDIT_PASSWORD
BROWSER_HEADLESS: "true"
ports:
- container: 3000
host: 3000
volumes:
- source: reddit-bot-data
target: /app/data
type: bind
restartPolicy: unless-stopped
```
---
## Production Considerations
### Security
1. **Network Isolation**: Run behind a reverse proxy with authentication
2. **Environment Variables**: Never commit credentials; use secrets management
3. **Dashboard Access**: Restrict to internal network or add auth
4. **Browser Data**: Protect `data/browser/` directory
### Performance
1. **Browser Headless**: Always use headless mode in production
2. **WAL Mode**: SQLite WAL mode is enabled by default for concurrent reads
3. **Rate Limiting**: Adjust based on Reddit account age and trust level
4. **Message Volume**: Start low (10-20/hour) and increase gradually
### Reliability
1. **Auto-restart**: Use `restart: unless-stopped` in Docker
2. **Health Checks**: Monitor `/api/stats` endpoint
3. **Logging**: Review logs regularly for errors
4. **Backups**: Regularly backup `data/reddit-bot.db`
### Monitoring
1. **Dashboard**: Check web dashboard for real-time stats
2. **Response Rate**: Monitor response rates per strategy/subreddit
3. **Error Rate**: Watch for AI refusals, send failures
4. **Browser Health**: Verify session isn't expired
### Scaling
RedditBot is designed as a single-instance application. For scaling:
1. **Multiple Accounts**: Run separate instances per Reddit account
2. **Subreddit Split**: Distribute subreddits across instances
3. **Centralized DB**: Consider moving to PostgreSQL for multi-instance
---
## Monitoring & Maintenance
### Daily Checks
- [ ] Dashboard shows bot is running
- [ ] Response rate is reasonable (>10%)
- [ ] No repeated errors in logs
- [ ] Browser session is active
### Weekly Tasks
- [ ] Review analytics (strategy performance, subreddit performance)
- [ ] Check disk usage (media files, database)
- [ ] Verify backup is working
- [ ] Review successful messages for quality
### Monthly Tasks
- [ ] Backup database
- [ ] Review and update strategy variants
- [ ] Check for OpenRouter model updates
- [ ] Review and update seller keywords
- [ ] Verify browser session is still valid
### Backup Procedure
```bash
# Stop the bot
docker compose stop
# Backup database
cp data/reddit-bot.db data/reddit-bot.db.backup.$(date +%Y%m%d)
# Backup browser session (optional)
tar czf data/browser.backup.$(date +%Y%m%d).tar.gz data/browser/
# Restart the bot
docker compose start
```
---
## Troubleshooting
### Bot Won't Start
**Symptom**: Bot exits immediately on startup.
**Steps**:
1. Check logs: `docker compose logs reddit-bot`
2. Verify `config.yaml` syntax
3. Verify environment variables are set
4. Check for Zod validation errors in logs
### Browser Login Fails
**Symptom**: "Browser session expired" or login fails.
**Steps**:
1. Delete browser data: `rm -rf data/browser/`
2. Re-run login: `npm run login`
3. Check Reddit account is not locked
4. Try `BROWSER_HEADLESS=false` for debugging
### AI Analysis Fails
**Symptom**: All messages show "AI refusal" or analysis errors.
**Steps**:
1. Verify API key is valid and has balance
2. Check model availability on OpenRouter
3. Try changing to a different model
4. Review logs for specific error messages
### High Error Rate
**Symptom**: Many messages fail to send.
**Steps**:
1. Check rate limiting (reduce maxMessagesPerHour)
2. Increase minDelayMs and jitterMs
3. Check for IP bans (try different network)
4. Review Reddit account standing
### Dashboard Not Accessible
**Symptom**: Cannot reach http://localhost:3000.
**Steps**:
1. Verify bot is running (`docker compose ps`)
2. Check port is not in use: `lsof -i :3000`
3. Verify firewall allows port 3000
4. Check web.port in config.yaml
### Database Issues
**Symptom**: Database errors or corruption.
**Steps**:
1. Stop the bot
2. Backup the database
3. Try opening with SQLite viewer to check integrity
4. If corrupted, restore from backup
5. Consider enabling automatic backups
docs/development.md
+514
View File
@@ -0,0 +1,514 @@
# Development Guide
Guide for contributing to RedditBot development.
---
## Table of Contents
1. [Project Structure](#project-structure)
2. [Development Setup](#development-setup)
3. [Coding Standards](#coding-standards)
4. [Adding a New Service](#adding-a-new-service)
5. [Adding a New Strategy Variant](#adding-a-new-strategy-variant)
6. [Debugging](#debugging)
7. [Common Tasks](#common-tasks)
8. [Known Issues](#known-issues)
---
## Project Structure
```
RedditBot/
├── config.yaml # Main configuration
├── package.json # Dependencies and scripts
├── tsconfig.json # TypeScript config
├── Dockerfile # Container build
├── docker-compose.yml # Docker Compose
├── public/
│ └── index.html # Web dashboard (single file)
├── scripts/
│ ├── login.ts # Manual login helper
│ └── inspect-chat-dom.ts # DOM inspection debug script
└── src/
├── index.ts # Entry point / orchestrator
├── config/
│ └── index.ts # Config loader + Zod schema
├── database/
│ ├── client.ts # SQLite setup + migrations
│ └── repositories.ts # Repository classes
├── reddit/
│ └── api.ts # Reddit API client
├── ai/
│ └── openrouter.ts # OpenRouter AI client
├── browser/
│ ├── client.ts # Playwright browser manager
│ ├── message-page.ts # DM compose automation
│ └── media-downloader.ts # Media download
├── services/
│ ├── monitor.ts # Post discovery
│ ├── processor.ts # Processing pipeline
│ ├── response-checker.ts # Response monitoring
│ ├── strategy-selector.ts # A/B testing
│ ├── chat.ts # Chat scraping + messaging
│ ├── media.ts # Media management
│ ├── media-sync.ts # Media sync scheduler
│ └── auto-reply.ts # Auto-reply service
├── utils/
│ ├── logger.ts # Winston logger
│ ├── rate-limiter.ts # Rate limiting
│ ├── retry.ts # Retry wrapper
│ └── image.ts # Image utilities
└── web/
└── server.ts # Express dashboard server
```
---
## Development Setup
### 1. Install Dependencies
```bash
npm install
```
### 2. Configure
Copy and edit `config.yaml`:
```bash
cp config.yaml config.yaml.backup # Backup existing
# Edit config.yaml with your settings
```
### 3. Set Environment Variables
```bash
export REDDIT_USERNAME="your-username"
export REDDIT_PASSWORD="your-password"
export OPENROUTER_API_KEY="sk-or-v1-your-key"
```
### 4. Initial Login
```bash
# Open browser for manual login
npm run login
```
### 5. Start Development Server
```bash
# Development mode (hot reload with tsx)
npm run dev
# Production mode (compiled JS)
npm run build && npm start
```
### 6. Access Dashboard
Open http://localhost:3000
---
## Coding Standards
### TypeScript
- **Strict mode**: No `any` types (use `unknown` with type guards)
- **Interfaces**: Define interfaces for all data structures
- **Null checks**: Use optional chaining (`?.`) and nullish coalescing (`??`)
- **Async/Await**: Use async/await, not `.then()` chains
```typescript
// Good
interface PostData {
id: string;
subreddit: string;
author: string;
}
async function fetchPost(id: string): Promise<PostData> {
const response = await api.get(`/posts/${id}`);
return response.data;
}
// Bad
async function fetchPost(id: any): Promise<any> {
return api.get(`/posts/${id}`).then(r => r.data);
}
```
### Logging
Use `winston` logger, never `console.log`:
```typescript
// Good
this.logger.info("Processed post {id}", { id: post.id });
this.logger.warn("AI refusal detected", { postId: post.id });
this.logger.error("Failed to send DM", { error: err.message });
// Bad
console.log("Processed post", post.id);
```
### Error Handling
Use try/catch with structured error handling:
```typescript
try {
await this.processPost(post);
} catch (err) {
this.logger.error("Post processing failed", {
postId: post.id,
error: err instanceof Error ? err.message : String(err)
});
// Don't throw - allow other posts to be processed
}
```
### Service Structure
Follow single responsibility:
```typescript
class NewService {
constructor(
private config: Config,
private dependency: ExistingService,
private logger: Logger
) {}
async doSomething(input: string): Promise<Result> {
// Validate input
if (!input) {
throw new Error("Input is required");
}
// Do work
const result = await this.dependency.process(input);
// Return result
return result;
}
}
```
### File Organization
- One class per file (except repositories which share one file)
- Services in `src/services/`
- Utilities in `src/utils/`
- External integrations in their own folders (`src/ai/`, `src/reddit/`, `src/browser/`)
---
## Adding a New Service
### 1. Create the Service File
```typescript
// src/services/new-service.ts
import { Config } from '../config';
import { Logger } from '../utils/logger';
interface NewServiceDeps {
config: Config;
logger: Logger;
// other dependencies
}
interface Result {
success: boolean;
data?: any;
error?: string;
}
export class NewService {
constructor(private deps: NewServiceDeps) {}
async execute(input: string): Promise<Result> {
const { config, logger } = this.deps;
logger.info("NewService executing", { input });
// Implementation here
return { success: true };
}
}
```
### 2. Add to Dependencies Interface
If the service needs other services, add them to the interface:
```typescript
interface NewServiceDeps {
config: Config;
logger: Logger;
redditApi: RedditApi;
openrouter: OpenRouterService;
}
```
### 3. Wire Up in main()
In `src/index.ts`, add initialization:
```typescript
const newService = new NewService({
config,
logger,
redditApi,
openrouter
});
// If it needs to run periodically:
setInterval(() => {
newService.execute(someInput).catch(err => {
logger.error("NewService error", { error: err.message });
});
}, intervalMs);
```
### 4. Add to Web Dashboard (Optional)
If the service needs API endpoints, add them in `src/web/server.ts`:
```typescript
server.app.get('/api/new-service', async (req, res) => {
try {
const result = await newService.execute(req.query.input as string);
res.json(result);
} catch (err) {
res.status(500).json({ error: (err as Error).message });
}
});
```
### 5. Update Config (If Needed)
Add new configuration options to:
- `config.yaml`
- `src/config/index.ts` Zod schema
- Configuration documentation
---
## Adding a New Strategy Variant
### 1. Add to config.yaml
```yaml
strategyVariants:
- name: "existing_variant"
prompt: "Existing prompt..."
enabled: true
- name: "new_variant"
prompt: |
Write a friendly message that references the image content.
Be specific and genuine. Keep it under 2 sentences.
enabled: true
```
### 2. No Code Changes Needed
The `StrategySelector` automatically picks up new variants from the config. The algorithm will:
1. Include the new variant in random exploration
2. Track its response rate once messages are sent
3. Promote it to exploitation if it has good stats
### 3. Monitor Performance
Check performance via the dashboard or API:
```bash
curl http://localhost:3000/api/analytics/by-strategy
```
---
## Debugging
### Browser Debugging
Run with visible browser:
```yaml
# config.yaml
browser:
headless: false
```
Or set environment variable:
```bash
BROWSER_HEADLESS=false npm run dev
```
### DOM Inspection
Use the debug endpoint to inspect Reddit's shadow DOM:
```bash
curl -X POST http://localhost:3000/api/debug/inspect-dom
```
Or run the standalone script:
```bash
npx tsx scripts/inspect-chat-dom.ts
```
### Logging
Adjust log level by modifying `src/utils/logger.ts`:
```typescript
const logger = winston.createLogger({
level: 'debug', // Change from 'info' to 'debug'
// ...
});
```
### Database Inspection
Open the database with SQLite viewer:
```bash
# Install sqlite3 if needed
brew install sqlite3
# Open database
sqlite3 data/reddit-bot.db
# Query
SELECT * FROM sent_messages LIMIT 5;
```
### Network Debugging
Enable axios debug:
```typescript
// In src/reddit/api.ts
import axios from 'axios';
axios.defaults.debug = true;
```
---
## Common Tasks
### Add a New Subreddit
```yaml
# config.yaml
reddit:
subreddits:
- gonewild
- ratemyboobs
- new_subreddit_name # Add here
```
### Change AI Model
```yaml
# config.yaml
openrouter:
model: "new-model-name"
fallbackModels:
- "fallback-model-1"
- "fallback-model-2"
```
### Adjust Rate Limits
```yaml
# config.yaml
rateLimit:
minDelayMs: 45000 # Increase delay
maxMessagesPerHour: 15 # Reduce cap
jitterMs: 20000 # More jitter
```
### Enable Dry Run
```yaml
# config.yaml
dryRun: true # Logs all actions without sending
```
Or via dashboard: `PUT /api/config` with `{"dryRun": true}`
### Add Database Column
```typescript
// In src/database/client.ts
function runMigrations(db: Database) {
// ... existing migrations ...
// Check if column exists
const tableInfo = db.prepare(`PRAGMA table_info(sent_messages)`).all();
const hasNewColumn = tableInfo.some((col: any) => col.name === 'new_column');
if (!hasNewColumn) {
db.exec("ALTER TABLE sent_messages ADD COLUMN new_column TEXT");
}
}
```
### Add New API Endpoint
```typescript
// In src/web/server.ts
import { Request, Response } from 'express';
server.app.get('/api/new-endpoint', async (req: Request, res: Response) => {
try {
const result = await someService.doSomething();
res.json({ success: true, data: result });
} catch (err) {
res.status(500).json({ error: (err as Error).message });
}
});
```
---
## Known Issues
### Hardcoded Username in ChatService
`ChatService` has `myUsername = 'Upset_Marsupial1006'` hardcoded in JavaScript evaluation scripts. This should be parameterized from config.
**Fix needed**: Pass username to `page.evaluate()` calls.
### Shadow DOM Fragility
Reddit frequently updates their `faceplate-*` components. Shadow DOM traversal code may break without warning.
**Mitigation**: Use `scripts/inspect-chat-dom.ts` to debug when selectors change.
### No Test Coverage
The project has no automated tests. Any changes should be manually tested.
**Recommendation**: Add tests for critical paths:
- StrategySelector algorithm
- ChatService shadow DOM parsing
- OpenRouterService model fallback
### Single-File Dashboard
The entire dashboard is in `public/index.html` (~2000+ lines). Making changes requires editing a single large file.
**Recommendation**: Consider migrating to a proper frontend framework for maintainability.
docs/services.md
+854
View File
@@ -0,0 +1,854 @@
# Service Catalog
Detailed documentation for each service in the RedditBot codebase.
---
## Table of Contents
1. [MonitorService](#monitorservice)
2. [ProcessorService](#processorservice)
3. [StrategySelector](#strategypicker)
4. [ResponseChecker](#responsechecker)
5. [ChatService](#chatservice)
6. [MediaService](#mediaservice)
7. [MediaSyncService](#mediasyncservice)
8. [AutoReplyService](#autoreplyservice)
9. [RedditApi](#redditapi)
10. [OpenRouterService](#openrouterservice)
11. [BrowserClient](#browserclient)
12. [MessagePage](#messagepage)
13. [RateLimiter](#ratelimiter)
---
## MonitorService
**File**: `src/services/monitor.ts`
**Lines**: 72
**Responsibility**: Discover new image posts across configured subreddits
### Purpose
Polls configured subreddits at regular intervals, discovers new image posts, filters out already-processed posts and previously-messaged users, and returns the single newest valid post for processing.
### Public Methods
```typescript
class MonitorService {
constructor(
private redditApi: RedditApi,
private postsRepo: PostsRepository,
private messagesRepo: MessagesRepository,
private config: Config
)
/**
* Discover the next post to process.
* Returns null if no valid posts are available.
*/
discoverNextPost(): Promise<Post | null>
}
```
### Algorithm
1. Shuffle subreddit order (randomize polling order each cycle)
2. For each subreddit:
- Fetch new posts via `RedditApi.fetchPosts()`
- Filter: image posts only
- Filter: author not deleted
- Filter: post not already in `processed_posts`
- Filter: author not previously messaged (or meets re-message criteria)
3. Return the newest valid post across all subreddits
4. If no valid posts found, return `null`
### Dependencies
- `RedditApi` - Fetch posts from Reddit
- `PostsRepository` - Check for already-processed posts
- `MessagesRepository` - Check for previously-messaged users
- `Config` - Subreddit list, poll interval, target gender
### Error Handling
- Skips subreddits that return errors (continues to next)
- Logs errors at `warn` level
- Returns `null` on any unrecoverable error
---
## ProcessorService
**File**: `src/services/processor.ts`
**Lines**: 132
**Responsibility**: Core processing pipeline for discovered posts
### Purpose
Orchestrates the full processing pipeline for a single discovered post: image download, gender detection, AI analysis, strategy selection, message generation, and DM sending.
### Public Methods
```typescript
class ProcessorService {
constructor(
private config: Config,
private redditApi: RedditApi,
private openrouter: OpenRouterService,
private strategySelector: StrategySelector,
private messageSender: MessagePage,
private postsRepo: PostsRepository,
private messagesRepo: MessagesRepository,
private successfulRepo: SuccessfulMessagesRepository,
private logger: Logger
)
/**
* Process a single discovered post through the full pipeline.
*/
processPost(post: Post): Promise<void>
}
```
### Pipeline Steps
```
1. Download image
└─► ImageService.downloadImage(url)
└─► Validate MIME type (image/jpeg, image/png, image/webp)
└─► Validate file size
└─► Convert to base64
2. Quick gender detection (cost-saving pre-filter)
└─► OpenRouter.detectGenderOnly(image)
└─► Skip if gender doesn't match targetGender
3. Select A/B strategy variant
└─► StrategySelector.selectStrategy(subreddit, author)
4. Full AI analysis
└─► OpenRouter.analyzeImage(image)
└─► Detect gender, describe image
└─► Handle refusal (mark post, try next)
5. Generate message
└─► OpenRouter.generateMessage(analysis, title, examples, strategy)
└─► Few-shot examples from SuccessfulMessagesRepository
6. Send DM
└─► MessagePage.sendDM(author, "Stunning", message)
└─► Handle failure (log, don't mark as sent)
7. Record in database
└─► Mark post as processed
└─► Mark message_sent = 1 if successful
└─► Store generated_message for reference
```
### Error Handling
- AI refusal: Post is marked as processed with error, skipped
- Image download failure: Post is marked with error, skipped
- DM send failure: Post is marked as not sent, logged
- All errors logged with context (post ID, subreddit, author)
### Cost Optimization
- Quick gender detection costs less than full analysis
- Non-matching genders are skipped before expensive analysis
- AI refusal detection prevents wasted tokens on rejected prompts
---
## StrategySelector
**File**: `src/services/strategy-selector.ts`
**Lines**: 101
**Responsibility**: A/B testing with explore/exploit algorithm
### Purpose
Selects message variants (strategies) for each post using an explore/exploit algorithm based on historical response rates. Optimizes for maximum response rate over time while continuing to explore under-tested variants.
### Public Methods
```typescript
class StrategySelector {
constructor(
private config: Config,
private messagesRepo: MessagesRepository
)
/**
* Select the best strategy variant for a given subreddit/author.
*/
selectStrategy(subreddit: string, author: string): string
}
```
### Algorithm
1. **Get strategy stats** from `MessagesRepository`:
- Response count per variant
- Sent count per variant
- Response rate per variant
2. **Apply minimum sample threshold**:
- If a variant has fewer than `minSamples` sends, treat it as untested
- Untested variants are excluded from exploitation
3. **Explore vs Exploit decision**:
- With probability `explorationRate`, select a random variant (explore)
- Otherwise, select the variant with the highest response rate (exploit)
4. **Fallback**:
- If no variants have enough data, select randomly from enabled variants
### Configuration
```yaml
strategyVariants:
- name: "variant_a"
prompt: "Compliment-based message..."
enabled: true
- name: "variant_b"
prompt: "Question-based message..."
enabled: true
```
### Dependencies
- `Config` - Strategy variants, exploration rate, min samples
- `MessagesRepository` - Response rate statistics
---
## ResponseChecker
**File**: `src/services/response-checker.ts`
**Lines**: 179
**Responsibility**: Monitor inbox for user replies
### Purpose
Background service that navigates the Reddit inbox via Playwright, identifies responses to bot-sent messages, updates the database, and stores successful message/response pairs for few-shot learning.
### Public Methods
```typescript
class ResponseCheckerService {
constructor(
private browser: BrowserClient,
private messagesRepo: MessagesRepository,
private successfulRepo: SuccessfulMessagesRepository,
private logger: Logger
)
/**
* Check inbox for responses to sent messages.
*/
checkResponses(): Promise<void>
}
```
### Algorithm
1. Navigate to `/message/inbox` via Playwright
2. Parse inbox items (messages, conversations)
3. For each response item:
- Extract recipient username and message text
- Match to `sent_messages` by recipient + timestamp proximity
- Update `got_response = 1` and `response_checked_at`
- Store the message/response pair in `successful_messages`
4. Update strategy statistics (response rates)
### Dependencies
- `BrowserClient` - Playwright browser context
- `MessagesRepository` - Update response flags
- `SuccessfulMessagesRepository` - Store learning pairs
- `Logger` - Activity logging
---
## ChatService
**File**: `src/services/chat.ts`
**Lines**: 1123+
**Responsibility**: Reddit chat scraping, messaging, and conversation management
### Purpose
The most complex service in the codebase. Manages Reddit chat interactions including scraping active conversations, loading conversation history, sending messages with human-like delays, detecting upsell/spam, and managing conversation caches.
### Public Methods
```typescript
class ChatService {
constructor(
private browser: BrowserClient,
private config: Config,
private logger: Logger
)
/** Fetch active conversations from Reddit */
fetchChats(): Promise<ChatSummary[]>
/** Load full conversation history */
loadConversation(chatId: string): Promise<Conversation>
/** Send a message in a conversation */
sendMessage(chatId: string, text: string): Promise<boolean>
/** Get cached conversations */
getCachedConversations(): Conversation[]
/** Check for new messages in conversations */
checkForUpdates(): Promise<Conversation[]>
/** Detect upsell/spam in a message */
detectUpsell(message: string): boolean
}
```
### Shadow DOM Handling
Reddit uses custom `faceplate-*` web components with shadow DOM. Standard CSS selectors cannot pierce these boundaries. The service uses JavaScript injection via `page.evaluate()`:
```javascript
// Traverse shadow roots to find chat elements
function traverseShadowRoots(root) {
const elements = root.querySelectorAll('.some-class');
const shadowChildren = root.querySelectorAll('*::shadow, ::deep');
// ... recursive traversal
}
```
### Conversation Cache
Conversations are cached in two layers:
1. **In-memory**: Fast access for active conversations
2. **Disk JSON**: Persistence across restarts at `data/chats/`
Cache structure:
```json
{
"chatId": "abc123",
"participants": ["user1", "bot"],
"messages": [
{ "author": "user1", "text": "Hey!", "timestamp": 1234567890 }
],
"lastUpdated": 1234567890,
"hasNewMessages": false
}
```
### Human-Like Messaging
Messages are sent with:
- Typing delay simulation (random delay before "typing" starts)
- Variable typing speed
- Random delays between messages
- Delivery verification
### Known Issues
- **Hardcoded username**: `myUsername = 'Upset_Marsupial1006'` is hardcoded in JS evaluation scripts. Should be parameterized from `Config`.
- **Size**: 1123+ lines suggests this should be split into smaller services.
### Dependencies
- `BrowserClient` - Playwright browser context
- `Config` - Rate limits, username (should be), check intervals
- `Logger` - Activity logging
---
## MediaService
**File**: `src/services/media.ts`
**Lines**: 254
**Responsibility**: Media download, caching, and indexing
### Purpose
Downloads images from chat conversations, resolves Matrix blurred thumbnails to full resolution, stores files locally, and maintains a deduplicated index.
### Public Methods
```typescript
class MediaService {
constructor(
private config: Config,
private browser: BrowserClient,
private logger: Logger
)
/** Download media from a conversation */
downloadMedia(chatId: string): Promise<string[]>
/** Get media index */
getIndex(): MediaIndex
/** Get a specific media file path */
getPath(hash: string): string | null
}
```
### Deduplication
Files are deduplicated using SHA-1 hashing:
```
SHA-1(image_bytes) → data/media/{hash}.{ext}
```
The index maps hashes to file paths:
```json
{
"abc123def456": "abc123def456.jpg",
"789xyz012abc": "789xyz012abc.png"
}
```
### Matrix Thumbnail Resolution
Reddit's Matrix system uses blurred placeholder thumbnails. The service resolves these to full resolution by:
1. Detecting Matrix thumbnail URLs
2. Extracting the full-resolution URL from the response
3. Downloading the full-resolution image
### Dependencies
- `Config` - Media directory path
- `BrowserClient` - Playwright for Matrix resolution
- `Logger` - Download activity
---
## MediaSyncService
**File**: `src/services/media-sync.ts`
**Lines**: 119
**Responsibility**: Periodic background media synchronization
### Purpose
Runs periodic background syncs to download media from chat conversations. Supports both incremental (top 10 chats) and full (all conversations) sync modes.
### Public Methods
```typescript
class MediaSyncService {
constructor(
private config: Config,
private mediaService: MediaService,
private chatService: ChatService,
private logger: Logger
)
/** Start periodic sync */
start(): void
/** Stop periodic sync */
stop(): void
/** Trigger incremental sync */
syncIncremental(): Promise<void>
/** Trigger full sync */
syncFull(): Promise<void>
/** Get sync status */
getStatus(): SyncStatus
}
```
### Sync Modes
| Mode | Description | Use Case |
|---|---|---|
| Incremental | Top 10 most recent conversations | Regular background sync |
| Full | All conversations | Initial setup, catch-up |
### Dependencies
- `Config` - Sync interval, enabled flag
- `MediaService` - Actual download logic
- `ChatService` - Conversation list
- `Logger` - Sync progress
---
## AutoReplyService
**File**: `src/services/auto-reply.ts`
**Lines**: 199
**Responsibility**: Automated reply to chat conversations
### Purpose
Monitors cached conversations for unread messages, detects upsell/spam, generates AI reply suggestions, and either auto-sends or marks as pending for manual approval.
### Public Methods
```typescript
class AutoReplyService {
constructor(
private config: Config,
private chatService: ChatService,
private openrouter: OpenRouterService,
private logger: Logger
)
/** Start auto-reply checking */
start(): void
/** Stop auto-reply checking */
stop(): void
/** Check for unread messages and generate replies */
checkAndReply(): Promise<void>
}
```
### Workflow
```
1. Check cached conversations for unread messages
│
├─► Skip if no new messages
│
├─► Detect upsell/spam keywords
│ └─► Flag conversation, skip reply
│
├─► Generate AI reply suggestion
│ └─► OpenRouter.generateReply(conversation history)
│
├─► If autoSend: send reply immediately
│ └─► ChatService.sendMessage()
│
└─► If !autoSend: mark as pending
└─► Dashboard shows pending replies for approval
```
### Dependencies
- `Config` - Enabled flag, autoSend flag, check interval
- `ChatService` - Conversation access, message sending
- `OpenRouterService` - Reply generation
- `Logger` - Activity logging
---
## RedditApi
**File**: `src/reddit/api.ts`
**Lines**: 233
**Responsibility**: Reddit API client
### Purpose
Axios-based client for interacting with Reddit's public API. Handles post fetching, user profile checking, and authenticated requests using Playwright browser cookies.
### Public Methods
```typescript
class RedditApi {
constructor(
private browser: BrowserClient,
private logger: Logger
)
/** Fetch new posts from a subreddit */
fetchPosts(subreddit: string, limit?: number): Promise<Post[]>
/** Check if a user is a seller/spam */
checkUserProfile(username: string): Promise<UserProfile>
/** Make authenticated request using browser cookies */
authenticatedRequest(url: string): Promise<any>
}
```
### Image URL Resolution
Fallback chain for extracting image URLs from post data:
```
1. post.preview.reddits[0].url (Reddit CDN preview)
2. post.url (i.redd.it direct)
3. post.thumbnail (if not self/spam)
4. Scan post.selftext for imgur/image links
```
### Seller/Spam Detection
Checks user profile for keywords:
- "seller"
- "onlyfans"
- "link in bio"
- Other configured `sellerKeywords`
### Dependencies
- `BrowserClient` - Browser cookies for authentication
- `Logger` - Request logging
---
## OpenRouterService
**File**: `src/ai/openrouter.ts`
**Lines**: 307
**Responsibility**: OpenRouter AI client
### Purpose
Client for the OpenRouter API (OpenAI-compatible SDK). Provides image analysis, message generation, reply suggestions, and upsell detection.
### Public Methods
```typescript
class OpenRouterService {
constructor(private config: Config, private logger: Logger)
/** Full image analysis with model fallback */
analyzeImage(imageBase64: string): Promise<ImageAnalysis>
/** Quick gender detection (cost-saving) */
detectGenderOnly(imageBase64: string): Promise<string>
/** Generate DM message */
generateMessage(
imageAnalysis: ImageAnalysis,
postTitle: string,
strategy: string
): Promise<string>
/** Generate chat reply */
generateReply(conversationHistory: Message[]): Promise<string>
/** Detect upsell/spam keywords */
detectUpsell(message: string): boolean
}
```
### Model Fallback Chain
```
Primary model (config.openrouter.model)
│
└─► If refusal detected:
│
└─► Try each fallback model (config.openrouter.fallbackModels)
│
└─► If all fail: throw error
```
### Refusal Detection
Heuristic patterns to detect when AI models refuse to process prompts:
```typescript
const refusalPatterns = [
/cannot.*analyze/i,
/unable.*process/i,
/against.*policy/i,
/cannot.*view/i,
// ...
];
```
### Few-Shot Learning
`generateMessage()` retrieves successful message/response pairs from the database and includes them as examples in the prompt:
```
"Here are examples of messages that got responses:
- Example 1: {original_message} → {response_snippet}
- Example 2: ...
Generate a similar message for this image..."
```
### Dependencies
- `Config` - API key, model selection, prompts
- `MessagesRepository` - Successful examples for few-shot learning
---
## BrowserClient
**File**: `src/browser/client.ts`
**Lines**: 302
**Responsibility**: Playwright persistent context manager
### Purpose
Manages the Playwright browser session with persistent cookies/storage. Handles automated login, session verification, and 2FA/captcha detection.
### Public Methods
```typescript
class BrowserClient {
constructor(private config: Config, private logger: Logger)
/** Initialize browser context */
initialize(): Promise<void>
/** Verify browser session is active */
verifySession(): Promise<boolean>
/** Perform automated login */
login(): Promise<void>
/** Get page context for operations */
getPage(): Promise<Page>
/** Close browser */
close(): Promise<void>
}
```
### Login Flow
```
1. Navigate to Reddit login page
2. Fill username and password
3. Submit form
4. Check for 2FA prompt
└─► If present: wait for manual input (with timeout)
5. Check for captcha
└─► If present: log warning, wait
6. Verify login success:
└─► Check for user menu indicator
└─► Check for profile avatar
└─► Fallback: API request to /api/me
7. Save cookies to persistent directory
8. On failure: capture debug screenshot
```
### Session Persistence
- Cookies and storage are saved to `config.browser.userDataDir`
- Subsequent launches restore the session automatically
- Session expires when Reddit cookies expire (requires re-login)
### Dependencies
- `Config` - Credentials, headless mode, user data dir
- `Logger` - Login progress, errors
---
## MessagePage
**File**: `src/browser/message-page.ts`
**Lines**: 560
**Responsibility**: Reddit DM compose page automation
### Purpose
Automates the Reddit DM compose page: filling recipient/subject/body fields, handling shadow DOM components, submitting messages, and verifying success.
### Public Methods
```typescript
class MessagePage {
constructor(
private browser: BrowserClient,
private config: Config,
private logger: Logger
)
/** Send a DM to a recipient */
sendDM(recipient: string, subject: string, body: string): Promise<boolean>
/** Check if recipient field shows an error */
hasRecipientError(page: Page): Promise<boolean>
}
```
### Shadow DOM Handling
Reddit's compose form uses `faceplate-*` web components:
```javascript
// Fill shadow DOM input fields
page.evaluate(({ selector, value }) => {
const shadowInput = document.querySelector(`${selector} input`);
if (shadowInput) {
shadowInput.value = value;
shadowInput.dispatchEvent(new Event('input', { bubbles: true }));
}
}, { selector: 'faceplate-textarea[name="body"]', value: body });
```
### Dry Run Mode
When `config.dryRun` is enabled:
- All fields are filled as normal
- Submit button is NOT clicked
- Message is logged but not sent
- Returns `true` (simulates success)
### Dependencies
- `BrowserClient` - Playwright page access
- `Config` - Dry run mode
- `Logger` - Send activity
---
## RateLimiter
**File**: `src/utils/rate-limiter.ts`
**Lines**: 114
**Responsibility**: Token-bucket rate limiting
### Purpose
Enforces rate limits on bot actions (primarily message sending) to avoid detection and bans. Uses a combination of minimum delay, random jitter, and hourly caps.
### Public Methods
```typescript
class RateLimiter {
constructor(config: RateLimitConfig)
/** Wait until a slot is available */
waitForSlot(): Promise<void>
/** Get remaining time until next slot */
getTimeUntilSlot(): number
/** Get current hourly count */
getHourlyCount(): number
/** Reset hourly counter */
resetHourlyCounter(): void
}
```
### Rate Limiting Strategy
```
1. Check hourly cap:
└─► If maxMessagesPerHour reached: wait until hour rolls over
2. Enforce minimum delay:
└─► lastActionTime + minDelayMs + random(0, jitterMs)
3. Track action timestamp
4. Increment hourly counter
```
### Dependencies
- `Config` - Rate limit settings
- `p-queue` - Concurrency control
package-lock.json
Generated
+1238 -1
View File
File diff suppressed because it is too large. Load diff
package.json
+6 -2
View File
@@ -8,7 +8,10 @@
"build": "tsc",
"start": "tsx src/index.ts",
"login": "tsx scripts/login.ts",
"dev": "tsx watch src/index.ts"
"dev": "tsx watch src/index.ts",
"test": "vitest run",
"test:watch": "vitest",
"test:coverage": "vitest run --coverage"
},
"keywords": [
"reddit",
@@ -35,6 +38,7 @@
"@types/express": "^5.0.6",
"@types/node": "^25.0.3",
"tsx": "^4.21.0",
"typescript": "^5.9.3"
"typescript": "^5.9.3",
"vitest": "^4.1.7"
}
}
public/index.html
+1943 -920
View File
File diff suppressed because it is too large. Load diff
scripts/inspect-chat-dom.ts
-249
View File
@@ -1,249 +0,0 @@
/**
* Diagnostic script: inspect the actual DOM structure of Reddit chat
* to understand how media images are rendered.
*
* Usage: npx tsx scripts/inspect-chat-dom.ts
*/
import { chromium } from 'playwright';
import path from 'path';
async function main() {
const sessionDir = '/tmp/reddit-session-copy';
console.log('Launching browser with session copy...');
const context = await chromium.launchPersistentContext(sessionDir, {
headless: false,
viewport: { width: 1280, height: 720 },
args: ['--no-sandbox', '--disable-blink-features=AutomationControlled'],
});
const page = await context.newPage();
await page.goto('https://chat.reddit.com/', {
waitUntil: 'domcontentloaded',
timeout: 30000,
});
await page.waitForTimeout(5000);
console.log('\n=== Finding chat rooms ===');
// Find the first chat room and click into it
const firstChatUrl = await page.evaluate(`
(function() {
var rsApp = document.querySelector('rs-app');
if (!rsApp || !rsApp.shadowRoot) return null;
// Look for room links
function findLinks(root) {
if (!root) return [];
var links = Array.from(root.querySelectorAll('a[href*="/room/"]'));
var allElements = root.querySelectorAll('*');
for (var i = 0; i < allElements.length; i++) {
if (allElements[i].shadowRoot) {
links = links.concat(findLinks(allElements[i].shadowRoot));
}
}
return links;
}
var links = findLinks(rsApp.shadowRoot);
if (links.length === 0) return null;
return links[0].href;
})()
`);
console.log('First chat URL:', firstChatUrl);
if (firstChatUrl) {
await page.goto(firstChatUrl, { waitUntil: 'domcontentloaded', timeout: 15000 });
await page.waitForTimeout(3000);
}
console.log('\n=== Inspecting message area DOM structure ===');
const domInfo = await page.evaluate(`
(function() {
var results = {
rsApp: false,
rsRoom: false,
rsRoomShadow: false,
customElements: [],
allImages: [],
messageStructure: [],
shadowRootChain: [],
};
var rsApp = document.querySelector('rs-app');
if (!rsApp || !rsApp.shadowRoot) {
results.error = 'No rs-app shadow root found';
return results;
}
results.rsApp = true;
var rsRoom = rsApp.shadowRoot.querySelector('rs-room');
if (!rsRoom) {
results.error = 'No rs-room found';
return results;
}
results.rsRoom = true;
results.rsRoomShadow = !!rsRoom.shadowRoot;
// Find ALL custom elements and their shadow roots recursively
function mapShadowDom(root, depth, path) {
if (!root || depth > 8) return;
var allEls = root.querySelectorAll('*');
for (var i = 0; i < allEls.length; i++) {
var el = allEls[i];
var tag = el.tagName.toLowerCase();
if (tag.includes('-') || el.shadowRoot) {
var entry = {
tag: tag,
depth: depth,
path: path + ' > ' + tag,
hasShadow: !!el.shadowRoot,
childCount: el.childElementCount,
className: (el.className || '').toString().slice(0, 100),
};
results.customElements.push(entry);
if (el.shadowRoot) {
mapShadowDom(el.shadowRoot, depth + 1, path + ' > ' + tag + ' [shadow]');
}
}
}
}
var searchRoot = rsRoom.shadowRoot || rsRoom;
mapShadowDom(searchRoot, 0, 'rs-room');
// Find ALL images recursively through shadow DOMs
function findAllImages(root, depth, path) {
if (!root || depth > 8) return;
var imgs = root.querySelectorAll('img');
for (var i = 0; i < imgs.length; i++) {
var img = imgs[i];
results.allImages.push({
src: (img.getAttribute('src') || '').slice(0, 200),
alt: img.getAttribute('alt') || '',
width: img.width,
height: img.height,
naturalWidth: img.naturalWidth,
naturalHeight: img.naturalHeight,
className: (img.className || '').toString().slice(0, 100),
parentTag: img.parentElement ? img.parentElement.tagName.toLowerCase() : 'none',
parentClass: img.parentElement ? (img.parentElement.className || '').toString().slice(0, 100) : '',
grandparentTag: img.parentElement && img.parentElement.parentElement ? img.parentElement.parentElement.tagName.toLowerCase() : 'none',
path: path,
depth: depth,
});
}
var allEls = root.querySelectorAll('*');
for (var j = 0; j < allEls.length; j++) {
if (allEls[j].shadowRoot) {
findAllImages(allEls[j].shadowRoot, depth + 1, path + ' > ' + allEls[j].tagName.toLowerCase() + '[shadow]');
}
}
}
findAllImages(searchRoot, 0, 'rs-room');
// Try to understand message structure: look for repeated patterns
// Walk the shadow DOM and find elements that look like messages
function findMessageElements(root, depth, path) {
if (!root || depth > 8) return;
var allEls = root.querySelectorAll('*');
for (var i = 0; i < allEls.length; i++) {
var el = allEls[i];
var tag = el.tagName.toLowerCase();
var cls = (el.className || '').toString().toLowerCase();
var text = (el.textContent || '').trim();
// Look for elements that contain message-like content
if (tag.includes('message') || cls.includes('message') ||
tag.includes('msg') || cls.includes('msg') ||
tag.includes('chat') || cls.includes('bubble')) {
results.messageStructure.push({
tag: tag,
className: cls.slice(0, 150),
path: path,
depth: depth,
textSnippet: text.slice(0, 100),
childCount: el.childElementCount,
hasImages: el.querySelectorAll('img').length,
innerHTML: el.innerHTML.slice(0, 300),
});
}
if (el.shadowRoot) {
findMessageElements(el.shadowRoot, depth + 1, path + ' > ' + tag + '[shadow]');
}
}
}
findMessageElements(searchRoot, 0, 'rs-room');
return results;
})()
`);
console.log('\n--- Shadow DOM Structure ---');
console.log('rs-app:', domInfo.rsApp);
console.log('rs-room:', domInfo.rsRoom);
console.log('rs-room has shadow:', domInfo.rsRoomShadow);
if (domInfo.error) console.log('ERROR:', domInfo.error);
console.log('\n--- Custom Elements with Shadow Roots ---');
for (const el of domInfo.customElements || []) {
console.log(`${' '.repeat(el.depth)}${el.tag} (shadow: ${el.hasShadow}, children: ${el.childCount}) ${el.className ? '[class: ' + el.className + ']' : ''}`);
}
console.log('\n--- All Images Found ---');
for (const img of domInfo.allImages || []) {
console.log(` [depth=${img.depth}] ${img.src}`);
console.log(` size: ${img.width}x${img.height} (natural: ${img.naturalWidth}x${img.naturalHeight})`);
console.log(` class: ${img.className}`);
console.log(` parent: <${img.parentTag} class="${img.parentClass}">`);
console.log(` path: ${img.path}`);
console.log('');
}
console.log('\n--- Message-like Elements ---');
for (const msg of domInfo.messageStructure || []) {
console.log(` [depth=${msg.depth}] <${msg.tag}> images=${msg.hasImages}`);
console.log(` class: ${msg.className}`);
console.log(` text: ${msg.textSnippet}`);
console.log(` path: ${msg.path}`);
console.log(` html: ${msg.innerHTML}`);
console.log('');
}
// Also dump a broader view - find what's directly inside rs-room shadow
console.log('\n=== Direct children of message area ===');
const directChildren = await page.evaluate(`
(function() {
var rsApp = document.querySelector('rs-app');
if (!rsApp || !rsApp.shadowRoot) return 'no rs-app';
var rsRoom = rsApp.shadowRoot.querySelector('rs-room');
if (!rsRoom) return 'no rs-room';
var root = rsRoom.shadowRoot || rsRoom;
var children = [];
for (var i = 0; i < root.children.length; i++) {
var child = root.children[i];
children.push({
tag: child.tagName.toLowerCase(),
className: (child.className || '').toString().slice(0, 200),
id: child.id || '',
childCount: child.childElementCount,
hasShadow: !!child.shadowRoot,
textSnippet: (child.textContent || '').trim().slice(0, 150),
});
}
return children;
})()
`);
console.log(JSON.stringify(directChildren, null, 2));
console.log('\nDone. Closing browser...');
await context.close();
}
main().catch(console.error);
scripts/matrix-sniff.ts
+217
View File
@@ -0,0 +1,217 @@
/**
* Matrix network sniffer.
*
* Watches every request to matrix.redditspace.com made by chat.reddit.com,
* captures method/URL/body for both request and response, and writes the
* sequence to data/matrix-sniff-<timestamp>.json when you exit (Ctrl-C).
*
* Usage:
* tsx scripts/matrix-sniff.ts
*
* Then in the browser that pops up: log in if needed → click "Start chat"
* → pick a recipient → send a message → come back here and press Ctrl-C.
*
* The script does NOT initiate any Matrix calls itself. It only observes
* what the UI sends, so there's no risk of an accidental outbound message.
*/
import fs from 'fs';
import path from 'path';
import { chromium, type Request, type Response } from 'playwright';
import { createLogger } from '../src/utils/logger.js';
const SESSION_DIR = path.resolve(process.cwd(), './data/browser-session');
const OUT_DIR = path.resolve(process.cwd(), './data');
const MATRIX_HOST_RE = /matrix\.(redditspace|reddit)\.com$/i;
interface CapturedCall {
index: number;
startedAt: string;
method: string;
url: string;
pathLabel: string;
requestHeaders: Record<string, string>;
requestBody: string | null;
requestBodyJson: unknown | null;
responseStatus: number | null;
responseHeaders: Record<string, string> | null;
responseBody: string | null;
responseBodyJson: unknown | null;
durationMs: number | null;
}
/** Best-effort short label so the live log is readable. */
function labelFor(method: string, urlStr: string): string {
try {
const url = new URL(urlStr);
// Strip /_matrix/client/v3 prefix for readability.
let p = url.pathname.replace(/^\/_matrix\/(client|media)\/(v3|v1)/, '');
// Truncate long IDs in path segments.
p = p
.split('/')
.map((seg) => (seg.length > 24 ? seg.slice(0, 12) + '…' + seg.slice(-6) : seg))
.join('/');
return `${method} ${p}`;
} catch {
return `${method} ${urlStr}`;
}
}
function safeParseJson(s: string | null): unknown | null {
if (!s) return null;
try {
return JSON.parse(s);
} catch {
return null;
}
}
async function main() {
const logger = createLogger();
if (!fs.existsSync(OUT_DIR)) fs.mkdirSync(OUT_DIR, { recursive: true });
const outFile = path.join(OUT_DIR, `matrix-sniff-${new Date().toISOString().replace(/[:.]/g, '-')}.json`);
logger.info('[sniff] launching browser (using persistent session)...');
const context = await chromium.launchPersistentContext(SESSION_DIR, {
headless: false,
viewport: { width: 1280, height: 800 },
});
const calls: CapturedCall[] = [];
// Map request → index so we can hydrate it when the response arrives.
const reqIndex = new WeakMap<Request, number>();
const reqStart = new WeakMap<Request, number>();
let serial = 0;
const onRequest = (request: Request) => {
try {
const u = new URL(request.url());
if (!MATRIX_HOST_RE.test(u.hostname)) return;
} catch {
return;
}
const idx = serial++;
reqIndex.set(request, idx);
reqStart.set(request, Date.now());
const body = request.postData();
const call: CapturedCall = {
index: idx,
startedAt: new Date().toISOString(),
method: request.method(),
url: request.url(),
pathLabel: labelFor(request.method(), request.url()),
requestHeaders: scrubHeaders(request.headers()),
requestBody: body ?? null,
requestBodyJson: safeParseJson(body ?? null),
responseStatus: null,
responseHeaders: null,
responseBody: null,
responseBodyJson: null,
durationMs: null,
};
calls[idx] = call;
logger.info(`[sniff] #${idx} → ${call.pathLabel}`);
if (body && body.length < 600) {
logger.info(` req body: ${body}`);
} else if (body) {
logger.info(` req body: ${body.length}B (truncated in log; full saved to file)`);
}
};
const onResponse = async (response: Response) => {
const request = response.request();
const idx = reqIndex.get(request);
if (idx === undefined) return;
const call = calls[idx];
if (!call) return;
call.responseStatus = response.status();
call.responseHeaders = scrubHeaders(response.headers());
const started = reqStart.get(request);
call.durationMs = started ? Date.now() - started : null;
// Only capture text bodies, and skip large binaries.
const ct = (response.headers()['content-type'] || '').toLowerCase();
if (ct.includes('json') || ct.includes('text')) {
try {
const text = await response.text();
if (text.length <= 100_000) {
call.responseBody = text;
call.responseBodyJson = safeParseJson(text);
} else {
call.responseBody = `[truncated: ${text.length}B ${ct}]`;
}
} catch (err) {
call.responseBody = `[read error: ${(err as Error).message}]`;
}
} else if (ct) {
call.responseBody = `[binary: ${ct}]`;
}
logger.info(`[sniff] #${idx} ← ${call.responseStatus} (${call.durationMs}ms)`);
};
context.on('request', onRequest);
context.on('response', onResponse);
const page = context.pages()[0] ?? (await context.newPage());
await page.goto('https://chat.reddit.com/', { waitUntil: 'domcontentloaded' });
logger.info('');
logger.info('═══════════════════════════════════════════════════════════');
logger.info(' Ready. In the browser:');
logger.info(' 1. Click "Start chat" (or the + icon)');
logger.info(' 2. Pick a recipient and send a short message');
logger.info(' 3. Come back here and press Ctrl-C to save & exit.');
logger.info('═══════════════════════════════════════════════════════════');
logger.info('');
const flush = async () => {
logger.info(`[sniff] writing ${calls.length} calls to ${outFile}`);
fs.writeFileSync(outFile, JSON.stringify({ capturedAt: new Date().toISOString(), calls }, null, 2));
try {
await context.close();
} catch {
/* ignore */
}
};
// Save on Ctrl-C.
process.on('SIGINT', async () => {
logger.info('[sniff] SIGINT — flushing...');
await flush();
process.exit(0);
});
// Also save if the user closes the browser window.
context.on('close', async () => {
logger.info('[sniff] browser closed — flushing...');
await flush();
process.exit(0);
});
}
/**
* Remove the bearer token before writing to disk — the captured JSON file
* is going to sit in the repo's data dir, no need to leave creds in it.
*/
function scrubHeaders(headers: Record<string, string>): Record<string, string> {
const out: Record<string, string> = {};
for (const [k, v] of Object.entries(headers)) {
if (k.toLowerCase() === 'authorization') {
out[k] = v.toLowerCase().startsWith('bearer ') ? 'Bearer <redacted>' : '<redacted>';
} else if (k.toLowerCase() === 'cookie') {
out[k] = '<redacted>';
} else {
out[k] = v;
}
}
return out;
}
main().catch((err) => {
console.error('FATAL:', err);
process.exit(1);
});
scripts/matrix-test.ts
+686
View File
@@ -0,0 +1,686 @@
/**
* Matrix-API prototype CLI.
*
* Usage:
* tsx scripts/matrix-test.ts # capture token + list rooms
* tsx scripts/matrix-test.ts --refresh # force re-extract token
* tsx scripts/matrix-test.ts --room <id> # dump message history for a room
* tsx scripts/matrix-test.ts --send <id> "hello" # send a test message
* tsx scripts/matrix-test.ts --dump # download every room's full
* # history + images to disk
*
* Reuses the bot's persistent browser session at ./data/browser-session,
* so you must already be logged into Reddit there (run `npm run login` first
* if you haven't).
*/
import fs from 'fs';
import path from 'path';
import { chromium } from 'playwright';
import { createLogger } from '../src/utils/logger.js';
import { MatrixClient, MatrixAuthError } from '../src/services/chat/matrix/client.js';
import {
extractMatrixCredentials,
loadCachedCredentials,
} from '../src/services/chat/matrix/token-extractor.js';
import type { MatrixTimelineEvent } from '../src/services/chat/matrix/types.js';
const SESSION_DIR = path.resolve(process.cwd(), './data/browser-session');
interface Args {
refresh: boolean;
roomId: string | null;
sendBody: string | null;
sendRoomId: string | null;
dump: boolean;
sync: boolean;
forceBackfill: boolean;
dumpDir: string;
maxPages: number;
}
function parseArgs(argv: string[]): Args {
const out: Args = {
refresh: false,
roomId: null,
sendBody: null,
sendRoomId: null,
dump: false,
sync: false,
forceBackfill: false,
dumpDir: 'data/matrix-dump',
maxPages: 200,
};
for (let i = 0; i < argv.length; i++) {
const a = argv[i];
if (a === '--refresh') out.refresh = true;
else if (a === '--room') out.roomId = argv[++i] ?? null;
else if (a === '--dump') out.dump = true;
else if (a === '--sync') out.sync = true;
else if (a === '--force-backfill') out.forceBackfill = true;
else if (a === '--out') out.dumpDir = argv[++i] ?? out.dumpDir;
else if (a === '--max-pages') out.maxPages = Number(argv[++i] ?? '200');
else if (a === '--send') {
out.sendRoomId = argv[++i] ?? null;
out.sendBody = argv[++i] ?? null;
}
}
return out;
}
function extOfContentType(ct: string): string {
const base = ct.split(';')[0].trim().toLowerCase();
const map: Record<string, string> = {
'image/jpeg': 'jpg',
'image/jpg': 'jpg',
'image/png': 'png',
'image/webp': 'webp',
'image/gif': 'gif',
'image/heic': 'heic',
'image/avif': 'avif',
'video/mp4': 'mp4',
'video/webm': 'webm',
'video/quicktime': 'mov',
'audio/mpeg': 'mp3',
'audio/ogg': 'ogg',
'application/octet-stream': 'bin',
};
return map[base] ?? base.split('/')[1] ?? 'bin';
}
function safeSlug(s: string): string {
return s.replace(/[^A-Za-z0-9._-]/g, '_').slice(0, 80);
}
/**
* Build a unique directory name for a room: `<displayName>__<short room-id>`.
* Many Reddit chat rooms share names like "Stunning" — without the suffix
* they collide on disk and silently overwrite each other's manifests.
*/
function roomDirSlug(roomId: string, displayName: string | null): string {
// Strip `!` and `:reddit.com` so the hash is short but stable.
const idCore = roomId.replace(/^!/, '').split(':')[0].slice(0, 8);
const base = displayName ? safeSlug(displayName) : 'room';
return `${base}__${idCore}`;
}
/** Run `worker` over `items` with at most `limit` in flight. Preserves order. */
async function pMap<T, R>(
items: T[],
limit: number,
worker: (item: T, index: number) => Promise<R>,
): Promise<R[]> {
const results: R[] = new Array(items.length);
let next = 0;
const runners = Array.from({ length: Math.min(limit, items.length) }, async () => {
while (true) {
const idx = next++;
if (idx >= items.length) return;
results[idx] = await worker(items[idx], idx);
}
});
await Promise.all(runners);
return results;
}
const ROOM_CONCURRENCY = 6;
const IMAGE_CONCURRENCY = 6;
async function paginateAllMessages(
client: MatrixClient,
roomId: string,
maxPages: number,
): Promise<MatrixTimelineEvent[]> {
const all: MatrixTimelineEvent[] = [];
let from: string | undefined = undefined;
for (let i = 0; i < maxPages; i++) {
const res = await client.getRoomMessages(roomId, { from, limit: 100, dir: 'b' });
all.push(...res.chunk);
if (!res.end || res.end === from || res.chunk.length === 0) break;
from = res.end;
}
return all;
}
interface SyncState {
nextBatch: string | null;
updatedAt: number;
rooms: Record<string, { slug: string; displayName: string | null }>;
}
function loadSyncState(dir: string): SyncState | null {
const p = path.join(dir, '_state.json');
if (!fs.existsSync(p)) return null;
try {
return JSON.parse(fs.readFileSync(p, 'utf-8')) as SyncState;
} catch {
return null;
}
}
function saveSyncState(dir: string, state: SyncState): void {
fs.writeFileSync(path.join(dir, '_state.json'), JSON.stringify(state, null, 2), 'utf-8');
}
function eventToManifestLine(e: MatrixTimelineEvent, userId: string): string {
return JSON.stringify({
event_id: e.event_id,
ts: e.origin_server_ts,
iso: new Date(e.origin_server_ts).toISOString(),
sender: e.sender,
isSelf: e.sender === userId,
msgtype: e.content.msgtype ?? null,
body: e.content.body ?? null,
mxc: typeof e.content.url === 'string' ? e.content.url : null,
});
}
function isImageEvent(e: MatrixTimelineEvent): boolean {
if (e.type !== 'm.room.message') return false;
return e.content.msgtype === 'm.image' || typeof e.content.url === 'string';
}
async function downloadEventImages(
client: MatrixClient,
events: MatrixTimelineEvent[],
roomDir: string,
logger: ReturnType<typeof createLogger>,
): Promise<{ downloaded: number; failed: number }> {
const targets = events.filter(isImageEvent);
let downloaded = 0;
let failed = 0;
await pMap(targets, IMAGE_CONCURRENCY, async (evt) => {
const mxc = evt.content.url as string;
try {
// Skip download if any file for this event already exists (delta re-runs).
const prefix = `${evt.origin_server_ts}_${safeSlug(evt.event_id)}.`;
const existing = fs.readdirSync(roomDir).find((f) => f.startsWith(prefix));
if (existing) return;
const media = await client.downloadMedia(mxc);
if (!media) {
failed++;
return;
}
const ext = extOfContentType(media.contentType);
fs.writeFileSync(path.join(roomDir, prefix + ext), media.bytes);
downloaded++;
} catch (err) {
logger.warn(`[media] failed ${mxc}: ${(err as Error).message}`);
failed++;
}
});
return { downloaded, failed };
}
async function resolveRoomDisplayName(
client: MatrixClient,
roomId: string,
logger: ReturnType<typeof createLogger>,
): Promise<string | null> {
// 1. Explicit name event
try {
const name = await client.getRoomName(roomId);
if (name) return name;
} catch (err) {
logger.debug(`[name] getRoomName failed for ${roomId}: ${(err as Error).message}`);
}
// 2. Fall back to other joined members (DM heuristic)
try {
const members = await client.getJoinedMembers(roomId);
const others = Object.keys(members).filter((m) => m !== client.userId);
if (others.length > 0) {
return others.map((m) => m.match(/^@([^:]+):/)?.[1] ?? m).join(', ');
}
} catch (err) {
logger.debug(`[name] getJoinedMembers failed for ${roomId}: ${(err as Error).message}`);
}
return null;
}
async function backfillAll(
client: MatrixClient,
args: Args,
logger: ReturnType<typeof createLogger>,
outDir: string,
): Promise<SyncState> {
const t0 = Date.now();
// One /sync up-front: gives us heroes / m.room.name for every active room
// and (crucially) the cursor for the future delta runs. This lets us avoid
// per-room name lookups for the common case.
const seed = await client.sync({ timeoutMs: 0 });
const seedJoined = seed.rooms?.join ?? {};
const seedNames = new Map<string, string | null>();
for (const [roomId, room] of Object.entries(seedJoined)) {
const nameEvent = room.state?.events?.find((e) => e.type === 'm.room.name');
if (nameEvent && typeof nameEvent.content.name === 'string') {
seedNames.set(roomId, nameEvent.content.name as string);
continue;
}
const heroes = (room.summary?.['m.heroes'] ?? []).filter((h) => h !== client.userId);
if (heroes.length > 0) {
seedNames.set(roomId, heroes.map((h) => h.match(/^@([^:]+):/)?.[1] ?? h).join(', '));
} else {
seedNames.set(roomId, null);
}
}
const joined = await client.getJoinedRooms();
logger.info(
`[backfill] /joined_rooms → ${joined.length} rooms (${seedNames.size} pre-named from /sync)`,
);
const state: SyncState = { nextBatch: null, updatedAt: Date.now(), rooms: {} };
let totalEvents = 0;
let totalImages = 0;
let totalDownloaded = 0;
let totalFailed = 0;
let done = 0;
await pMap(joined, ROOM_CONCURRENCY, async (roomId) => {
let displayName: string | null;
if (seedNames.has(roomId)) {
displayName = seedNames.get(roomId) ?? null;
} else {
displayName = await resolveRoomDisplayName(client, roomId, logger);
}
const slug = roomDirSlug(roomId, displayName);
const roomDir = path.join(outDir, slug);
fs.mkdirSync(roomDir, { recursive: true });
const events = await paginateAllMessages(client, roomId, args.maxPages);
const messages = events.filter((e) => e.type === 'm.room.message');
const lines = messages
.slice()
.reverse()
.map((e) => eventToManifestLine(e, client.userId));
fs.writeFileSync(
path.join(roomDir, 'messages.jsonl'),
lines.join('\n') + (lines.length ? '\n' : ''),
'utf-8',
);
const { downloaded, failed } = await downloadEventImages(client, events, roomDir, logger);
const imgCount = events.filter(isImageEvent).length;
totalEvents += events.length;
totalImages += imgCount;
totalDownloaded += downloaded;
totalFailed += failed;
state.rooms[roomId] = { slug, displayName };
done++;
logger.info(
`[backfill ${done}/${joined.length}] ${slug}: events=${events.length} msgs=${messages.length} imgs=${imgCount} ok=${downloaded} fail=${failed}`,
);
});
// The seed /sync's cursor is the right one to delta from — all events
// returned in that response are already on disk via the per-room walk.
state.nextBatch = seed.next_batch;
state.updatedAt = Date.now();
saveSyncState(outDir, state);
const elapsedSec = ((Date.now() - t0) / 1000).toFixed(1);
logger.info(
`[backfill] DONE in ${elapsedSec}s — rooms=${joined.length} events=${totalEvents} imgs=${totalImages} ok=${totalDownloaded} fail=${totalFailed}`,
);
logger.info(`[backfill] cursor stored: ${state.nextBatch?.slice(0, 32)}…`);
return state;
}
async function deltaSync(
client: MatrixClient,
args: Args,
logger: ReturnType<typeof createLogger>,
outDir: string,
state: SyncState,
): Promise<SyncState> {
logger.info(`[delta] resuming from cursor ${state.nextBatch?.slice(0, 32)}…`);
const sync = await client.sync({ since: state.nextBatch ?? undefined, timeoutMs: 0 });
const joined = sync.rooms?.join ?? {};
const roomIds = Object.keys(joined);
if (roomIds.length === 0) {
logger.info('[delta] no new events since last sync');
state.nextBatch = sync.next_batch;
state.updatedAt = Date.now();
saveSyncState(outDir, state);
return state;
}
let totalNew = 0;
let totalImages = 0;
let totalDownloaded = 0;
for (const roomId of roomIds) {
const room = joined[roomId];
const events = room.timeline?.events ?? [];
if (events.length === 0) continue;
// Discover (or rediscover) display name for previously-unseen rooms.
let info = state.rooms[roomId];
if (!info) {
const displayName = await resolveRoomDisplayName(client, roomId, logger);
info = { slug: roomDirSlug(roomId, displayName), displayName };
state.rooms[roomId] = info;
logger.info(`[delta] new room discovered: ${info.slug}`);
}
const roomDir = path.join(outDir, info.slug);
fs.mkdirSync(roomDir, { recursive: true });
const messageEvents = events.filter((e) => e.type === 'm.room.message');
if (messageEvents.length > 0) {
const lines = messageEvents.map((e) => eventToManifestLine(e, client.userId));
fs.appendFileSync(path.join(roomDir, 'messages.jsonl'), lines.join('\n') + '\n', 'utf-8');
}
const { downloaded } = await downloadEventImages(client, events, roomDir, logger);
const imgCount = events.filter(isImageEvent).length;
totalNew += messageEvents.length;
totalImages += imgCount;
totalDownloaded += downloaded;
logger.info(
`[delta] ${info.slug}: +${messageEvents.length} msgs +${imgCount} imgs (downloaded ${downloaded})`,
);
}
state.nextBatch = sync.next_batch;
state.updatedAt = Date.now();
saveSyncState(outDir, state);
logger.info(
`[delta] DONE — rooms_touched=${roomIds.length} new_msgs=${totalNew} new_imgs=${totalImages} downloaded=${totalDownloaded}`,
);
logger.info(`[delta] cursor updated: ${state.nextBatch?.slice(0, 32)}…`);
return state;
}
async function syncAll(
client: MatrixClient,
args: Args,
logger: ReturnType<typeof createLogger>,
) {
const outDir = path.resolve(process.cwd(), args.dumpDir);
if (!fs.existsSync(outDir)) fs.mkdirSync(outDir, { recursive: true });
logger.info(`[sync] output dir: ${outDir}`);
const existing = loadSyncState(outDir);
if (!existing?.nextBatch || args.forceBackfill) {
if (args.forceBackfill) logger.info('[sync] --force-backfill set: ignoring existing cursor');
else logger.info('[sync] no cursor on disk → running full backfill');
await backfillAll(client, args, logger, outDir);
} else {
await deltaSync(client, args, logger, outDir, existing);
}
}
async function dumpAll(client: MatrixClient, args: Args, logger: ReturnType<typeof createLogger>) {
const outDir = path.resolve(process.cwd(), args.dumpDir);
if (!fs.existsSync(outDir)) fs.mkdirSync(outDir, { recursive: true });
logger.info(`[dump] output dir: ${outDir}`);
const rooms = await client.listRooms();
logger.info(`[dump] ${rooms.length} rooms to walk`);
const overallSummary: Array<{
roomId: string;
displayName: string | null;
eventCount: number;
messageCount: number;
imageCount: number;
downloaded: number;
failed: number;
}> = [];
let totalImages = 0;
let totalDownloaded = 0;
let totalFailed = 0;
for (const room of rooms) {
const slug = safeSlug(room.displayName ?? room.roomId);
const roomDir = path.join(outDir, slug);
fs.mkdirSync(roomDir, { recursive: true });
const events = await paginateAllMessages(client, room.roomId, args.maxPages);
const messages = events.filter((e) => e.type === 'm.room.message');
const images = messages.filter(
(e) => e.content.msgtype === 'm.image' || typeof e.content.url === 'string',
);
// Write the messages.jsonl manifest.
const manifestPath = path.join(roomDir, 'messages.jsonl');
const lines = messages
.slice()
.reverse() // chronological for human reading
.map((e) =>
JSON.stringify({
event_id: e.event_id,
ts: e.origin_server_ts,
iso: new Date(e.origin_server_ts).toISOString(),
sender: e.sender,
isSelf: e.sender === client.userId,
msgtype: e.content.msgtype ?? null,
body: e.content.body ?? null,
mxc: typeof e.content.url === 'string' ? e.content.url : null,
}),
);
fs.writeFileSync(manifestPath, lines.join('\n') + '\n', 'utf-8');
let downloaded = 0;
let failed = 0;
for (const evt of images) {
const mxc = evt.content.url as string | undefined;
if (!mxc) continue;
try {
const media = await client.downloadMedia(mxc);
if (!media) {
failed++;
continue;
}
const ext = extOfContentType(media.contentType);
const fname = `${evt.origin_server_ts}_${safeSlug(evt.event_id)}.${ext}`;
fs.writeFileSync(path.join(roomDir, fname), media.bytes);
downloaded++;
} catch (err) {
logger.warn(`[dump] failed ${mxc} in ${room.roomId}: ${(err as Error).message}`);
failed++;
}
}
totalImages += images.length;
totalDownloaded += downloaded;
totalFailed += failed;
logger.info(
`[dump] ${slug}: events=${events.length} msgs=${messages.length} imgs=${images.length} ok=${downloaded} fail=${failed}`,
);
overallSummary.push({
roomId: room.roomId,
displayName: room.displayName,
eventCount: events.length,
messageCount: messages.length,
imageCount: images.length,
downloaded,
failed,
});
}
fs.writeFileSync(
path.join(outDir, '_summary.json'),
JSON.stringify(
{
capturedAt: new Date().toISOString(),
userId: client.userId,
homeserver: client.homeserver,
rooms: overallSummary,
totals: {
rooms: rooms.length,
images: totalImages,
downloaded: totalDownloaded,
failed: totalFailed,
},
},
null,
2,
),
'utf-8',
);
logger.info(
`[dump] DONE — rooms=${rooms.length} images=${totalImages} downloaded=${totalDownloaded} failed=${totalFailed}`,
);
logger.info(`[dump] manifest: ${path.join(outDir, '_summary.json')}`);
}
async function getCredentials(logger: ReturnType<typeof createLogger>, refresh: boolean) {
if (!refresh) {
const cached = await loadCachedCredentials();
if (cached) {
logger.info(`[matrix] using cached credentials (captured ${ageStr(cached.capturedAt)} ago)`);
return cached;
}
}
logger.info('[matrix] opening browser to capture access token...');
const context = await chromium.launchPersistentContext(SESSION_DIR, {
headless: false,
viewport: { width: 1280, height: 720 },
});
try {
return await extractMatrixCredentials(context, logger);
} finally {
await context.close();
}
}
function ageStr(ts: number): string {
const sec = Math.max(0, Math.floor((Date.now() - ts) / 1000));
if (sec < 60) return `${sec}s`;
if (sec < 3600) return `${Math.floor(sec / 60)}m`;
if (sec < 86400) return `${Math.floor(sec / 3600)}h`;
return `${Math.floor(sec / 86400)}d`;
}
async function main() {
const logger = createLogger();
const args = parseArgs(process.argv.slice(2));
let creds = await getCredentials(logger, args.refresh);
let client = new MatrixClient(creds, logger);
const reExtract = async () => {
logger.warn('[matrix] token rejected — re-extracting');
creds = await getCredentials(logger, true);
client = new MatrixClient(creds, logger);
};
// --- sync mode (backfill on first run, delta thereafter) -----------
if (args.sync) {
try {
await syncAll(client, args, logger);
} catch (err) {
if (err instanceof MatrixAuthError) {
await reExtract();
await syncAll(client, args, logger);
} else throw err;
}
return;
}
// --- dump mode (snapshot of /sync — recent rooms only) -------------
if (args.dump) {
try {
await dumpAll(client, args, logger);
} catch (err) {
if (err instanceof MatrixAuthError) {
await reExtract();
await dumpAll(client, args, logger);
} else throw err;
}
return;
}
// --- send mode -----------------------------------------------------
if (args.sendRoomId && args.sendBody) {
try {
const eventId = await client.sendTextMessage(args.sendRoomId, args.sendBody);
console.log(`Sent: event_id=${eventId}`);
} catch (err) {
if (err instanceof MatrixAuthError) {
await reExtract();
const eventId = await client.sendTextMessage(args.sendRoomId, args.sendBody);
console.log(`Sent (after re-auth): event_id=${eventId}`);
} else throw err;
}
return;
}
// --- single-room history mode --------------------------------------
if (args.roomId) {
let res;
try {
res = await client.getRoomMessages(args.roomId, { limit: 30 });
} catch (err) {
if (err instanceof MatrixAuthError) {
await reExtract();
res = await client.getRoomMessages(args.roomId, { limit: 30 });
} else throw err;
}
console.log(`\nRoom ${args.roomId} — ${res.chunk.length} events (most recent first):\n`);
for (const event of res.chunk) {
if (event.type !== 'm.room.message') {
console.log(` [${event.type}] from ${event.sender}`);
continue;
}
const who = event.sender === client.userId ? 'ME' : event.sender;
const when = new Date(event.origin_server_ts).toISOString();
const body = (event.content.body as string | undefined) ?? '<no body>';
const mediaTag = event.content.url ? ` [media: ${event.content.url}]` : '';
console.log(` ${when} ${who}: ${body.slice(0, 120)}${body.length > 120 ? '…' : ''}${mediaTag}`);
}
return;
}
// --- list rooms (default) ------------------------------------------
let rooms;
try {
rooms = await client.listRooms();
} catch (err) {
if (err instanceof MatrixAuthError) {
await reExtract();
rooms = await client.listRooms();
} else throw err;
}
console.log(`\nLogged in as: ${client.userId}`);
console.log(`Homeserver: ${client.homeserver}`);
console.log(`Rooms: ${rooms.length}\n`);
for (const r of rooms.slice(0, 50)) {
const unread = r.unreadCount > 0 ? ` (${r.unreadCount} unread)` : '';
const last = r.lastEvent
? `${new Date(r.lastEvent.origin_server_ts).toISOString()} ${(r.lastEvent.content.body as string | undefined)?.slice(0, 80) ?? `[${r.lastEvent.type}]`}`
: '<no events in window>';
console.log(`• ${r.displayName ?? r.roomId}${unread}`);
console.log(` roomId: ${r.roomId}`);
console.log(` last: ${last}\n`);
}
}
main().catch((err) => {
console.error('FATAL:', err);
process.exit(1);
});
src/ai/openrouter.ts
+211 -27
View File
@@ -4,9 +4,18 @@ import type { Logger } from '../utils/logger.js';
export type DetectedGender = 'male' | 'female' | 'other' | 'unknown';
export function stripDashes(text: string): string {
return text
.replace(/\s*[—–]\s*/g, ', ')
.replace(/,\s*,/g, ',')
.replace(/,\s*\./g, '.')
.trim();
}
export interface ImageAnalysisResult {
description: string;
detectedGender: DetectedGender;
attractivenessScore: number | null;
}
export class OpenRouterClient {
@@ -14,9 +23,10 @@ export class OpenRouterClient {
constructor(
private config: Config,
private logger: Logger
private logger: Logger,
client?: OpenAI,
) {
this.client = new OpenAI({
this.client = client ?? new OpenAI({
baseURL: 'https://openrouter.ai/api/v1',
apiKey: config.openrouter.apiKey,
defaultHeaders: {
@@ -70,6 +80,9 @@ export class OpenRouterClient {
...this.config.openrouter.imageAnalysisFallbackModels,
];
let lastError: unknown = null;
let sawRefusal = false;
for (let i = 0; i < models.length; i++) {
const model = models[i];
const isFallback = i > 0;
@@ -90,14 +103,22 @@ export class OpenRouterClient {
}
// Result was null = refusal, try next model
sawRefusal = true;
this.logger.warn(`Model ${model} refused to analyze image${i < models.length - 1 ? ', trying fallback...' : ''}`);
} catch (error) {
// API error (not a refusal) - still try fallback
// API error (not a refusal) — try next model but remember the error.
lastError = error;
this.logger.warn(`Model ${model} failed: ${error instanceof Error ? error.message : error}${i < models.length - 1 ? ', trying fallback...' : ''}`);
}
}
// All models refused or failed
// If any model returned a refusal, surface AI_REFUSAL so callers can branch
// on it. Otherwise propagate the most recent API error so failures aren't
// masked as refusals.
if (sawRefusal) {
throw new Error('AI_REFUSAL: All models refused to analyze image');
}
if (lastError) throw lastError;
throw new Error('AI_REFUSAL: All models refused to analyze image');
}
@@ -134,23 +155,37 @@ ${this.config.openrouter.imageAnalysisPrompt}`;
// Parse gender from response
let detectedGender: DetectedGender = 'unknown';
const genderMatch = fullResponse.match(/GENDER:(male|female|other|unknown)/i);
const genderMatch = fullResponse.match(/GENDER:\s*(male|female|other|unknown)/i);
if (genderMatch) {
detectedGender = genderMatch[1].toLowerCase() as DetectedGender;
}
// Remove gender line from description
const description = fullResponse.replace(/GENDER:(male|female|other|unknown)\n?/i, '').trim();
// Parse attractiveness rating (1-100 scale) from response.
// Tolerate markdown wrapping like **RATING:** 63 or RATING: **63**.
let attractivenessScore: number | null = null;
const ratingMatch = fullResponse.match(/RATING[:\s*]*[*_\s]*(\d{1,3})/i);
if (ratingMatch) {
const parsed = parseInt(ratingMatch[1], 10);
if (parsed >= 1 && parsed <= 100) {
attractivenessScore = parsed;
}
}
// Remove gender and rating lines from description
const description = fullResponse
.replace(/GENDER:\s*(male|female|other|unknown)\n?/i, '')
.replace(/\**\s*RATING[:\s*]*[*_\s]*\d{1,3}\**\n?/i, '')
.trim();
// Detect AI refusal - return null to signal fallback should be tried
if (this.isRefusal(description)) {
return null;
}
this.logger.debug(`Image analysis complete (${model}) - Gender: ${detectedGender}`);
this.logger.debug(`Image analysis complete (${model}) - Gender: ${detectedGender}, Rating: ${attractivenessScore ?? 'n/a'}`);
this.logger.debug(`Description: ${description}`);
return { description, detectedGender };
return { description, detectedGender, attractivenessScore };
}
private isRefusal(text: string): boolean {
@@ -189,7 +224,8 @@ ${this.config.openrouter.imageAnalysisPrompt}`;
postTitle: string,
authorName: string,
successfulExamples: string[] = [],
strategyPrompt?: string
strategyPrompt?: string,
attractivenessScore: number | null = null,
): Promise<string> {
try {
this.logger.debug('Generating message...');
@@ -197,7 +233,8 @@ ${this.config.openrouter.imageAnalysisPrompt}`;
let prompt = this.config.openrouter.messageGenerationPrompt
.replace('{analysis}', imageAnalysis)
.replace('{title}', postTitle)
.replace('{author}', authorName);
.replace('{author}', authorName)
.replace('{rating}', attractivenessScore != null ? String(attractivenessScore) : 'unknown');
// Append strategy-specific style guidance if provided
if (strategyPrompt) {
@@ -215,16 +252,11 @@ ${this.config.openrouter.imageAnalysisPrompt}`;
this.logger.debug(`Including ${successfulExamples.length} successful examples in prompt`);
}
const response = await this.client.chat.completions.create({
model: this.config.openrouter.chatModel,
messages: [{ role: 'user', content: prompt }],
const message = await this.generateTextWithFallback('generateMessage', prompt, {
max_tokens: 300,
temperature: 0.8, // Add some creativity
temperature: 0.8,
});
const message = response.choices[0]?.message?.content || '';
this.logger.debug(`Message generated: ${message}`);
return message.trim();
return message;
} catch (error) {
this.logger.error('Message generation failed:', error);
throw error;
@@ -256,22 +288,62 @@ DO NOT:
Just write the message, nothing else.`;
const response = await this.client.chat.completions.create({
model: this.config.openrouter.chatModel,
messages: [{ role: 'user', content: prompt }],
const message = await this.generateTextWithFallback('generateReply', prompt, {
max_tokens: 300,
temperature: 0.8,
});
const message = response.choices[0]?.message?.content || '';
this.logger.debug(`Suggested reply generated: ${message}`);
return message.trim();
return message;
} catch (error) {
this.logger.error('Reply generation failed:', error);
throw error;
}
}
async generateMediaAsk(
conversationHistory: { sender: string; text: string; isSelf: boolean }[],
username: string
): Promise<string> {
try {
this.logger.debug(`Generating media-ask message for conversation with ${username}...`);
const historyText = conversationHistory
.map(msg => `${msg.isSelf ? 'You' : msg.sender}: ${msg.text}`)
.join('\n');
const prompt = `You are helping continue a Reddit chat conversation. Here is the conversation so far:
${historyText}
Write the next message to send to ${username}. The goal of THIS message is to playfully nudge them into sending a pic or video OF THEMSELVES — their body, face, outfit, what they look like right now, a selfie. NOT a pic of an object, place, food, pet, or scenery. Keep the same tone and style as the previous messages from "You" — casual, flirty, direct, never robotic. Keep it to 1-3 sentences. You can reference something they just said as a hook, but the ask itself must be about seeing THEM.
Pick ONE technique that fits the current vibe — don't combine them, don't telegraph that you're using one:
- Playful dare ("bet you won't send a selfie right now")
- Direct ask for them ("send me a pic", "lemme see you", "selfie?")
- Tease about how they look / sound ("now I have to see what you look like")
- Curiosity hook about their appearance ("ok I need a face to picture here")
- Compliment + ask ("you sound hot, prove it 😏", "show me that smile")
- Outfit / "what are you wearing" angle ("send a pic of the fit")
DO NOT:
- Sound like an AI or use formal language
- Use words like "captivating", "alluring", "exquisite", "radiant"
- Beg, sound desperate, or repeat an ask they already declined
- Write long elaborate sentences
- Mention OnlyFans, paying, or anything transactional
Just write the message, nothing else.`;
const message = await this.generateTextWithFallback('generateMediaAsk', prompt, {
max_tokens: 300,
temperature: 0.9,
});
return message;
} catch (error) {
this.logger.error('Media-ask generation failed:', error);
throw error;
}
}
/**
* Check if message content contains upsell/spam keywords (OnlyFans, selling, etc.)
*/
@@ -292,6 +364,118 @@ Just write the message, nothing else.`;
return { isUpsell: false };
}
async generateChatOpener(
postTitle: string,
selftext: string,
authorName: string,
detectedGender: DetectedGender,
successfulExamples: string[] = [],
strategyPrompt?: string,
): Promise<string> {
try {
this.logger.debug('Generating chat opener...');
let prompt = this.config.openrouter.chatOpenerPrompt
.replace('{title}', postTitle)
.replace('{selftext}', selftext || '(no body)')
.replace('{author}', authorName)
.replace('{gender}', detectedGender);
if (strategyPrompt) {
prompt += `\n\n${strategyPrompt}`;
}
if (successfulExamples.length > 0) {
const examplesText = successfulExamples
.map((ex, i) => `Example ${i + 1}:\n${ex}`)
.join('\n\n');
prompt = `Here are openers that got responses:\n\n${examplesText}\n\n---\n\nNow generate a new opener in a similar style:\n\n${prompt}`;
}
const message = await this.generateTextWithFallback('generateChatOpener', prompt, {
max_tokens: 200,
temperature: 0.85,
});
return message;
} catch (error) {
this.logger.error('Chat opener generation failed:', error);
throw error;
}
}
/**
* Try the primary chatModel and then chatFallbackModels in order. A model is
* considered to have failed (and the next is tried) when it throws, returns
* empty content, or returns a refusal. Returns the trimmed message from the
* first model that produces usable content; returns '' if none do.
*/
private async generateTextWithFallback(
label: string,
prompt: string,
opts: { max_tokens: number; temperature: number },
): Promise<string> {
const models = [
this.config.openrouter.chatModel,
...this.config.openrouter.chatFallbackModels,
];
this.logger.info(`[AI] ${label} prompt (${prompt.length} chars):\n${prompt}`);
let lastError: unknown = null;
for (let i = 0; i < models.length; i++) {
const model = models[i];
const isFallback = i > 0;
try {
if (isFallback) {
this.logger.info(`[AI] ${label} trying fallback model: ${model}`);
}
const response = await this.client.chat.completions.create({
model,
messages: [{ role: 'user', content: prompt }],
max_tokens: opts.max_tokens,
temperature: opts.temperature,
});
const raw = response.choices[0]?.message?.content || '';
const choice = response.choices[0] as unknown as Record<string, unknown> | undefined;
const choiceMessage = choice?.message as Record<string, unknown> | undefined;
const respAny = response as unknown as Record<string, unknown>;
const meta = {
finish_reason: choice?.finish_reason,
native_finish_reason: choice?.native_finish_reason,
refusal: choiceMessage?.refusal,
model: response.model,
provider: respAny.provider,
usage: response.usage,
};
this.logger.info(`[AI] ${label} response from ${model} (${raw.length} chars) meta=${JSON.stringify(meta)}: ${JSON.stringify(raw)}`);
const trimmed = raw.trim();
if (!trimmed) {
this.logger.warn(`[AI] ${label}: ${model} returned empty content${i < models.length - 1 ? ', trying fallback...' : ''}`);
continue;
}
if (this.isRefusal(trimmed)) {
this.logger.warn(`[AI] ${label}: ${model} returned a refusal${i < models.length - 1 ? ', trying fallback...' : ''}`);
continue;
}
if (isFallback) {
this.logger.info(`[AI] ${label}: fallback model ${model} succeeded`);
}
return stripDashes(trimmed);
} catch (error) {
lastError = error;
this.logger.warn(`[AI] ${label}: ${model} threw ${error instanceof Error ? error.message : String(error)}${i < models.length - 1 ? ', trying fallback...' : ''}`);
}
}
if (lastError) {
throw lastError;
}
return '';
}
async analyzeAndGenerateMessage(
imageBase64: string,
mimeType: string,
@@ -301,7 +485,7 @@ Just write the message, nothing else.`;
strategyPrompt?: string
): Promise<{ analysis: ImageAnalysisResult; message: string }> {
const analysis = await this.analyzeImage(imageBase64, mimeType);
const message = await this.generateMessage(analysis.description, postTitle, authorName, successfulExamples, strategyPrompt);
const message = await this.generateMessage(analysis.description, postTitle, authorName, successfulExamples, strategyPrompt, analysis.attractivenessScore);
return { analysis, message };
}
}
src/bootstrap.ts
+297
View File
@@ -0,0 +1,297 @@
import type { Config } from './config/index.js';
import type { Logger } from './utils/logger.js';
import type Database from 'better-sqlite3';
import { createDatabase, runMigrations } from './database/client.js';
import {
PostsRepository,
MessagesRepository,
SuccessfulMessagesRepository,
ActivityLogRepository,
} from './database/repositories.js';
import { BrowserClient } from './browser/client.js';
import { RedditMessagePage } from './browser/message-page.js';
import { RedditApiClient } from './reddit/api.js';
import { OpenRouterClient } from './ai/openrouter.js';
import { MonitorService } from './services/monitor.js';
import { ProcessorService } from './services/processor.js';
import { ResponseCheckerService } from './services/response-checker.js';
import { StrategySelector } from './services/strategy-selector.js';
import {
ChatService,
MatrixClient,
MatrixMediaDownloader,
extractMatrixCredentials,
loadCachedCredentials,
type TokenRefresher,
type UsernameResolver,
type MatrixCredentials,
} from './services/chat/index.js';
import { MediaService } from './services/media.js';
import { MediaSyncService } from './services/media-sync.js';
import { RateLimiter } from './utils/rate-limiter.js';
import { fetchWithTimeout } from './utils/fetch-with-timeout.js';
import { WebServer } from './web/server.js';
export interface AppContainer {
config: Config;
logger: Logger;
db: Database.Database;
browserClient: BrowserClient;
messagePage: RedditMessagePage;
postsRepo: PostsRepository;
messagesRepo: MessagesRepository;
successfulMessagesRepo: SuccessfulMessagesRepository;
redditApi: RedditApiClient;
aiClient: OpenRouterClient;
monitorService: MonitorService;
processorService: ProcessorService;
strategySelector: StrategySelector;
responseChecker: ResponseCheckerService;
mediaService: MediaService;
chatService: ChatService;
mediaSyncService: MediaSyncService;
rateLimiter: RateLimiter;
webServer: WebServer | null;
}
/**
* Builds and wires every service required for the bot.
*
* Performs:
* 1. DB + repos
* 2. Browser + login
* 3. Three pages (compose / response-check / web-UI)
* 4. All services
* 5. Optional WebServer
*
* Returns a populated container. Caller owns lifecycle (start loop, handle
* shutdown).
*/
export async function bootstrap(config: Config, logger: Logger): Promise<AppContainer> {
// Database
const db = createDatabase(config);
runMigrations(db);
const postsRepo = new PostsRepository(db);
const messagesRepo = new MessagesRepository(db);
const successfulMessagesRepo = new SuccessfulMessagesRepository(db);
const activityLogRepo = new ActivityLogRepository(db);
logger.info('Database initialized');
// Browser
const browserClient = new BrowserClient(config, logger);
await browserClient.initialize();
const page = await browserClient.getPage();
const messagePage = new RedditMessagePage(page, logger);
messagePage.setDryRun(config.browser.dryRun);
// Login verification. When the web dashboard is enabled we treat a missing
// session as recoverable — the dashboard exposes a noVNC console so a human
// can complete the login flow in the running headed browser.
logger.info('Verifying Reddit login status...');
let loggedIn = await messagePage.isLoggedIn();
if (!loggedIn) {
logger.warn('Session invalid or not logged in.');
if (config.reddit.username && config.reddit.password) {
logger.info('Credentials found in config/env. Attempting automated login...');
loggedIn = await browserClient.login();
}
if (!loggedIn) {
if (config.web.enabled) {
logger.warn('Not logged in. Open the dashboard and click "Log in to Reddit" to authenticate via the noVNC console.');
logger.warn('The bot will stay paused until a valid session is detected.');
} else {
logger.error('Not logged in to Reddit and no recovery path (web dashboard disabled).');
logger.error('Run "npm run login" locally, OR set REDDIT_USERNAME and REDDIT_PASSWORD, OR enable the web dashboard.');
await browserClient.close();
process.exit(1);
}
}
}
if (loggedIn) {
logger.info('Reddit login verified');
}
// Reddit API client (shares browser context for auth)
const redditApi = new RedditApiClient(config, logger);
redditApi.setBrowserContext(browserClient.getContext()!);
// Dedicated page for the response checker (background scraping that
// doesn't conflict with the dashboard).
const chatPage = await browserClient.getContext()!.newPage();
// AI + services
const aiClient = new OpenRouterClient(config, logger);
const monitorService = new MonitorService(redditApi, postsRepo, messagesRepo, config, logger);
const processorService = new ProcessorService(redditApi, aiClient, config, logger);
const strategySelector = new StrategySelector(config, messagesRepo, logger);
processorService.setStrategySelector(strategySelector);
const chatStrategySelector = new StrategySelector(config, messagesRepo, logger, 'chatStrategies');
processorService.setChatStrategySelector(chatStrategySelector);
const responseChecker = new ResponseCheckerService(
chatPage,
messagesRepo,
successfulMessagesRepo,
postsRepo,
logger,
);
// ---- Matrix chat backend (replaces the Playwright scraper) ----
// 1. Get credentials: prefer cached, fall back to a fresh sniff via the
// persistent browser session that we just verified is logged in.
let matrixCreds: MatrixCredentials | null = await loadCachedCredentials();
if (!matrixCreds) {
logger.info('No cached Matrix credentials — extracting via browser session...');
matrixCreds = await extractMatrixCredentials(browserClient.getContext()!, logger);
} else {
logger.info(`Loaded cached Matrix credentials (user=${matrixCreds.userId})`);
}
const matrixClient = new MatrixClient(matrixCreds, logger);
// 2. Build the media downloader + service on top of the Matrix client so
// `mxc://` URIs resolve through the same auth + rate-limiter pipeline.
const mediaDownloader = new MatrixMediaDownloader(matrixClient, logger);
const mediaService = new MediaService(mediaDownloader, logger);
// 3. Token refresher — on 401, re-sniff via the running browser session
// and swap the new credentials into the chat service.
let chatService: ChatService;
const tokenRefresher: TokenRefresher = {
async refresh() {
const fresh = await extractMatrixCredentials(browserClient.getContext()!, logger);
const next = new MatrixClient(fresh, logger);
chatService.setMatrixClient(next);
},
};
// 4. Username resolver — turns t2 IDs into real Reddit usernames so the
// dashboard isn't full of `t2_abc123`. Uses Reddit's bulk endpoint.
const usernameResolver: UsernameResolver = {
async resolveT2Ids(ids) {
if (ids.length === 0) return {};
const out: Record<string, string> = {};
for (let i = 0; i < ids.length; i += 100) {
const chunk = ids.slice(i, i + 100).join(',');
try {
const res = await fetchWithTimeout(
`https://api.reddit.com/api/user_data_by_account_ids?ids=${encodeURIComponent(chunk)}`,
{ headers: { 'User-Agent': config.reddit.userAgent }, timeoutMs: 15_000 },
);
if (!res.ok) {
logger.debug(`user_data_by_account_ids → ${res.status}`);
continue;
}
const json = (await res.json()) as Record<string, { name?: string }>;
for (const [t2, info] of Object.entries(json)) {
if (info?.name) out[t2] = info.name;
}
} catch (err) {
logger.debug('user_data_by_account_ids failed:', err);
}
}
return out;
},
};
chatService = new ChatService(
matrixClient,
logger,
successfulMessagesRepo,
mediaService,
tokenRefresher,
usernameResolver,
);
const mediaSyncService = new MediaSyncService(chatService, logger, config.media.syncIntervalMs);
const rateLimiter = new RateLimiter(config, logger);
await chatService.loadCache();
// Lazy enrichment in the background — first sync may show t2 IDs briefly.
chatService.enrichUsernames().catch((err) => logger.debug('enrichUsernames failed:', err));
// Fire-and-forget startup sync: full if we've never backfilled, delta
// otherwise. Non-blocking so the rest of the bot can come up immediately;
// the dashboard's sync-status banner shows progress.
chatService
.runStartupSync()
.then(({ kind, chats }) => {
logger.info(`[chat] startup sync (${kind}) complete: ${chats} chats`);
})
.catch((err) => logger.error('[chat] startup sync failed:', err));
let webServer: WebServer | null = null;
if (config.web.enabled) {
webServer = new WebServer(
config.web.port,
config,
postsRepo,
messagesRepo,
successfulMessagesRepo,
activityLogRepo,
logger,
);
webServer.setChatService(chatService);
// Stream sync-progress frames over the dashboard's SSE channel so the
// full-sync UI can show live phase + per-room progress.
chatService.setProgressListener((progress) => {
webServer!.broadcastSyncProgress(progress as Record<string, unknown> | null);
});
chatService.setNewReplyListener(({ username, replySnippet }) => {
webServer!.logActivity(
'response',
`Reply from u/${username}: ${replySnippet}`,
username,
);
webServer!.pushLog('info', `💬 Reply from u/${username}`);
});
webServer.setMediaService(mediaService);
webServer.setMediaSyncService(mediaSyncService);
webServer.setResponseChecker(responseChecker);
webServer.setRateLimiter(rateLimiter);
webServer.setAiClient(aiClient);
webServer.setStrategySelector(strategySelector);
webServer.setBrowserClient(browserClient);
webServer.setOnConfigChange(() => logger.info('Runtime config updated via dashboard'));
}
// Start background services
if (config.media.syncEnabled) mediaSyncService.start();
if (webServer) webServer.start();
// Prime processor with successful examples already on disk.
const existingExamples = responseChecker.getSuccessfulExamples(3);
processorService.setSuccessfulExamples(existingExamples);
const successCount = responseChecker.getSuccessCount();
if (successCount > 0) {
logger.info(`Loaded ${successCount} successful message(s) from history`);
}
return {
config,
logger,
db,
browserClient,
messagePage,
postsRepo,
messagesRepo,
successfulMessagesRepo,
redditApi,
aiClient,
monitorService,
processorService,
strategySelector,
responseChecker,
mediaService,
chatService,
mediaSyncService,
rateLimiter,
webServer,
};
}
src/browser/client.ts
+11
View File
@@ -7,6 +7,7 @@ import fs from 'fs';
export class BrowserClient {
private context: BrowserContext | null = null;
private sessionDir: string | null = null;
private sessionValidityInFlight: Promise<boolean> | null = null;
constructor(
private config: Config,
@@ -59,6 +60,16 @@ export class BrowserClient {
}
async isSessionValid(): Promise<boolean> {
if (this.sessionValidityInFlight) {
return this.sessionValidityInFlight;
}
this.sessionValidityInFlight = this.doSessionValidityCheck().finally(() => {
this.sessionValidityInFlight = null;
});
return this.sessionValidityInFlight;
}
private async doSessionValidityCheck(): Promise<boolean> {
try {
const page = await this.getPage();
await page.goto('https://www.reddit.com', { waitUntil: 'domcontentloaded', timeout: 30000 });
src/browser/media-downloader.ts
-29
View File
@@ -1,29 +0,0 @@
import type { Page } from 'playwright';
import type { Logger } from '../utils/logger.js';
import type { MediaDownloadResult, MediaDownloader } from '../services/media.js';
export class PlaywrightMediaDownloader implements MediaDownloader {
constructor(
private page: Page,
private logger: Logger
) {}
async fetch(url: string): Promise<MediaDownloadResult | null> {
try {
const response = await this.page.request.get(url, { timeout: 15000 });
if (!response.ok()) {
this.logger.warn(`Media download failed (${response.status()}): ${url}`);
return null;
}
const headers = response.headers();
const contentType = headers['content-type'] ?? null;
const buffer = Buffer.from(await response.body());
return { buffer, contentType };
} catch (error) {
this.logger.warn(`Media download error for ${url}:`, error);
return null;
}
}
}
src/config/index.ts
+59 -18
View File
@@ -6,9 +6,21 @@ import dotenv from 'dotenv';
dotenv.config();
export const SubredditConfigSchema = z.object({
name: z.string(),
type: z.enum(['image', 'chat']).default('image'),
weight: z.number().positive().optional(),
});
export type SubredditConfig = z.infer<typeof SubredditConfigSchema>;
const SubredditEntrySchema = z
.union([z.string().transform((s) => ({ name: s })), z.object({}).passthrough()])
.pipe(SubredditConfigSchema);
export const ConfigSchema = z.object({
reddit: z.object({
subreddits: z.array(z.string()).min(1),
subreddits: z.array(SubredditEntrySchema).min(1),
pollIntervalMs: z.number().default(120000),
userAgent: z.string().default('RedditBot/1.0'),
targetGenders: z.array(z.enum(['male', 'female', 'other', 'unknown'])).default(['female']),
@@ -52,6 +64,35 @@ export const ConfigSchema = z.object({
'wishlist',
'throne.com',
]),
sellerDomains: z
.array(z.string())
.default([
'onlyfans.com',
'fansly.com',
'fanvue.com',
'loyalfans.com',
'manyvids.com',
'linktr.ee',
'allmylinks.com',
'beacons.ai',
'beacons.page',
'throne.com',
'throne.me',
'fan.ly',
'chaturbate.com',
'myfreecams.com',
'pornhub.com',
'xvideos.com',
'cash.app',
'cashapp.com',
'venmo.com',
'paypal.me',
'paypal.com',
'admireme.vip',
'fancentro.com',
'justforfans.app',
'justfor.fans',
]),
username: z.string().optional(),
password: z.string().optional(),
}),
@@ -60,39 +101,30 @@ export const ConfigSchema = z.object({
imageAnalysisModel: z.string().default('google/gemini-3-flash-preview'),
imageAnalysisFallbackModels: z.array(z.string()).default([]),
chatModel: z.string().default('google/gemini-3-flash-preview'),
chatFallbackModels: z.array(z.string()).default([]),
imageAnalysisPrompt: z.string(),
messageGenerationPrompt: z.string(),
chatOpenerPrompt: z.string().default(
'You are writing a cold opener DM to a Reddit user who just posted in a chat/sexting subreddit looking for someone to chat with.\n\nPost title: {title}\nPost body: {selftext}\nAuthor: u/{author}\nDetected gender of poster: {gender}\n\nYOUR PERSONA:\n- You are male, based in the UK, pick a random age between 30 and 35 for this message\n- Roughly half the time, mention (or imply) you\'re married/have a wife and are looking for something on the side; the other half, don\'t mention relationship status\n- Don\'t force the married angle if their post wouldn\'t vibe with it\n- Use British English spelling and phrasing where it feels natural\n\nWrite a casual Reddit DM (1-2 sentences) opening a chat.\n\nMUST follow:\n- Lead with your age/gender/location tag (e.g. "32M UK", "34M UK married") when it fits naturally\n- Reference something SPECIFIC from their title/body (their mood, what they\'re into, vibe)\n- Be confident and direct, not formal\n- Use proper capitalisation, no AI/clinical language\n- End with a hook that invites a reply (ask them something, or offer something playful)\n\nDO NOT:\n- Use words like "captivating", "alluring", "intriguing"\n- Open with "Hey there" / "How\'s your day"\n- Write more than 2 sentences\n- Claim to be a different age/gender than the persona above\n- Use em dashes (—) or en dashes (–). Use commas, full stops, or separate sentences instead.\n\nJust write the message, nothing else.',
),
}),
rateLimit: z.object({
minDelayBetweenMessagesMs: z.number().default(300000),
maxMessagesPerHour: z.number().default(8),
jitterMs: z.number().default(60000),
}),
}).default({ minDelayBetweenMessagesMs: 300000, maxMessagesPerHour: 8, jitterMs: 60000 }),
browser: z.object({
sessionDir: z.string().default('./data/browser-session'),
headless: z.boolean().default(false),
dryRun: z.boolean().default(false),
}),
}).default({ sessionDir: './data/browser-session', headless: false, dryRun: false }),
database: z.object({
path: z.string().default('./data/reddit-bot.db'),
}),
}).default({ path: './data/reddit-bot.db' }),
media: z.object({
syncIntervalMs: z.number().default(1800000), // 30 minutes
syncEnabled: z.boolean().default(true),
}).default({ syncIntervalMs: 1800000, syncEnabled: true }),
autoReply: z.object({
enabled: z.boolean().default(false),
autoSend: z.boolean().default(false), // If true, send replies automatically. If false, require manual approval
checkIntervalMs: z.number().default(300000), // 5 minutes
upsellKeywords: z.array(z.string()).default([
'onlyfans', 'only fans', 'fansly', 'manyvids', 'linktree', 'linktr.ee',
'subscribe', 'premium', 'paid content', 'custom content', 'custom vid',
'cashapp', 'cash app', 'venmo', 'paypal', 'tip me', 'spoil me',
'wishlist', 'throne.com', 'link in bio', 'check my profile',
'fanvue', 'loyalfans', 'cam show', 'buy my', 'selling',
'dm me for price', 'menu', 'rates', 'ppv',
]),
}).default({ enabled: false, autoSend: false, checkIntervalMs: 300000, upsellKeywords: [] }),
strategies: z.object({
explorationRate: z.number().min(0).max(1).default(0.2),
minSamples: z.number().default(5),
@@ -102,10 +134,19 @@ export const ConfigSchema = z.object({
promptOverride: z.string(),
})).default([]),
}).default({ explorationRate: 0.2, minSamples: 5, variants: [] }),
chatStrategies: z.object({
explorationRate: z.number().min(0).max(1).default(0.2),
minSamples: z.number().default(5),
variants: z.array(z.object({
id: z.string(),
weight: z.number().default(1),
promptOverride: z.string(),
})).default([]),
}).default({ explorationRate: 0.2, minSamples: 5, variants: [] }),
web: z.object({
enabled: z.boolean().default(true),
port: z.number().default(3000),
}),
}).default({ enabled: true, port: 3000 }),
});
export type Config = z.infer<typeof ConfigSchema>;
src/database/client.ts
+11
View File
@@ -57,12 +57,23 @@ export function runMigrations(db: Database.Database): void {
created_at INTEGER NOT NULL
);
CREATE TABLE IF NOT EXISTS activity_log (
id INTEGER PRIMARY KEY AUTOINCREMENT,
timestamp INTEGER NOT NULL,
type TEXT NOT NULL,
message TEXT NOT NULL,
recipient TEXT,
image_url TEXT
);
CREATE INDEX IF NOT EXISTS idx_posts_subreddit
ON processed_posts(subreddit);
CREATE INDEX IF NOT EXISTS idx_posts_processed_at
ON processed_posts(processed_at);
CREATE INDEX IF NOT EXISTS idx_posts_message_sent
ON processed_posts(message_sent);
CREATE INDEX IF NOT EXISTS idx_activity_log_timestamp
ON activity_log(timestamp);
`);
// Add new columns to existing tables (safe to run multiple times)
src/database/repositories.ts
+96 -2
View File
@@ -26,6 +26,7 @@ export interface SentMessageRecord {
subreddit?: string;
postTitle?: string;
postImageUrl?: string;
postImageAnalysis?: string;
}
export interface SuccessfulMessageRecord {
@@ -60,6 +61,80 @@ export interface DailyTrend {
responseRate: number;
}
export interface ActivityLogRecord {
timestamp: number;
type: string;
message: string;
recipient: string | null;
imageUrl: string | null;
}
export class ActivityLogRepository {
private insertStmt: Database.Statement;
private getRecentStmt: Database.Statement;
private pruneStmt: Database.Statement;
private countSentSkippedStmt: Database.Statement;
constructor(private db: Database.Database) {
this.insertStmt = db.prepare(`
INSERT INTO activity_log (timestamp, type, message, recipient, image_url)
VALUES (?, ?, ?, ?, ?)
`);
this.getRecentStmt = db.prepare(`
SELECT timestamp, type, message, recipient, image_url
FROM activity_log
ORDER BY id DESC
LIMIT ?
`);
this.pruneStmt = db.prepare(`
DELETE FROM activity_log
WHERE id NOT IN (SELECT id FROM activity_log ORDER BY id DESC LIMIT ?)
`);
this.countSentSkippedStmt = db.prepare(`
SELECT COUNT(*) AS count FROM activity_log WHERE type IN ('sent', 'skipped')
`);
}
insert(item: {
timestamp: number;
type: string;
message: string;
recipient?: string;
imageUrl?: string;
}): void {
this.insertStmt.run(
item.timestamp,
item.type,
item.message,
item.recipient ?? null,
item.imageUrl ?? null,
);
}
getRecent(limit: number): ActivityLogRecord[] {
const rows = this.getRecentStmt.all(limit) as Record<string, unknown>[];
return rows.map((row) => ({
timestamp: row.timestamp as number,
type: row.type as string,
message: row.message as string,
recipient: (row.recipient as string) ?? null,
imageUrl: (row.image_url as string) ?? null,
}));
}
prune(keep: number): void {
this.pruneStmt.run(keep);
}
countSentSkipped(): number {
const row = this.countSentSkippedStmt.get() as { count: number };
return row.count;
}
}
export class PostsRepository {
private insertStmt: Database.Statement;
private existsStmt: Database.Statement;
@@ -166,6 +241,7 @@ export class PostsRepository {
export class MessagesRepository {
private insertStmt: Database.Statement;
private getByPostIdStmt: Database.Statement;
private getLatestByRecipientStmt: Database.Statement;
private getRecentStmt: Database.Statement;
private getUncheckedStmt: Database.Statement;
private markResponseStmt: Database.Statement;
@@ -180,8 +256,19 @@ export class MessagesRepository {
this.getByPostIdStmt = db.prepare(`SELECT * FROM sent_messages WHERE post_id = ?`);
this.getLatestByRecipientStmt = db.prepare(`
SELECT sm.*, pp.subreddit, pp.title AS post_title, pp.image_url AS post_image_url,
pp.image_analysis AS post_image_analysis
FROM sent_messages sm
LEFT JOIN processed_posts pp ON sm.post_id = pp.id
WHERE sm.recipient = ? COLLATE NOCASE
ORDER BY sm.sent_at DESC
LIMIT 1
`);
this.getRecentStmt = db.prepare(`
SELECT sm.*, pp.subreddit, pp.title AS post_title, pp.image_url AS post_image_url
SELECT sm.*, pp.subreddit, pp.title AS post_title, pp.image_url AS post_image_url,
pp.image_analysis AS post_image_analysis
FROM sent_messages sm
LEFT JOIN processed_posts pp ON sm.post_id = pp.id
ORDER BY sm.sent_at DESC
@@ -235,6 +322,12 @@ export class MessagesRepository {
return this.mapRow(row);
}
getLatestByRecipient(recipient: string): SentMessageRecord | undefined {
const row = this.getLatestByRecipientStmt.get(recipient) as Record<string, unknown> | undefined;
if (!row) return undefined;
return this.mapRow(row);
}
getRecent(limit: number = 10): SentMessageRecord[] {
const rows = this.getRecentStmt.all(limit) as Record<string, unknown>[];
return rows.map((row) => this.mapRow(row));
@@ -340,6 +433,7 @@ export class MessagesRepository {
subreddit: (row.subreddit as string) || undefined,
postTitle: (row.post_title as string) || undefined,
postImageUrl: (row.post_image_url as string) || undefined,
postImageAnalysis: (row.post_image_analysis as string) || undefined,
};
}
}
@@ -358,7 +452,7 @@ export class SuccessfulMessagesRepository {
`);
this.getRecentStmt = db.prepare(`
SELECT * FROM successful_messages ORDER BY created_at DESC LIMIT ?
SELECT * FROM successful_messages ORDER BY created_at DESC, id DESC LIMIT ?
`);
// Get examples prioritizing same subreddit, then falling back to others
src/index.ts
+4 -366
View File
@@ -1,27 +1,7 @@
import { loadConfig } from './config/index.js';
import { createDatabase, runMigrations } from './database/client.js';
import { PostsRepository, MessagesRepository, SuccessfulMessagesRepository } from './database/repositories.js';
import { BrowserClient } from './browser/client.js';
import { RedditMessagePage } from './browser/message-page.js';
import { RedditApiClient } from './reddit/api.js';
import { OpenRouterClient } from './ai/openrouter.js';
import { MonitorService } from './services/monitor.js';
import { ProcessorService } from './services/processor.js';
import { ResponseCheckerService } from './services/response-checker.js';
import { StrategySelector } from './services/strategy-selector.js';
import { ChatService } from './services/chat.js';
import { MediaService } from './services/media.js';
import { MediaSyncService } from './services/media-sync.js';
import { AutoReplyService } from './services/auto-reply.js';
import { PlaywrightMediaDownloader } from './browser/media-downloader.js';
import { RateLimiter } from './utils/rate-limiter.js';
import { createLogger } from './utils/logger.js';
import { withRetry } from './utils/retry.js';
import { WebServer } from './web/server.js';
function sleep(ms: number): Promise<void> {
return new Promise((resolve) => setTimeout(resolve, ms));
}
import { bootstrap } from './bootstrap.js';
import { runMainLoop } from './main-loop.js';
async function main(): Promise<void> {
const logger = createLogger();
@@ -30,7 +10,6 @@ async function main(): Promise<void> {
logger.info('Reddit Bot Starting');
logger.info('='.repeat(60));
// Load configuration
let config;
try {
config = loadConfig();
@@ -40,349 +19,8 @@ async function main(): Promise<void> {
process.exit(1);
}
// Initialize database
const db = createDatabase(config);
runMigrations(db);
const postsRepo = new PostsRepository(db);
const messagesRepo = new MessagesRepository(db);
const successfulMessagesRepo = new SuccessfulMessagesRepository(db);
logger.info('Database initialized');
// Initialize browser
const browserClient = new BrowserClient(config, logger);
await browserClient.initialize();
const page = await browserClient.getPage();
const messagePage = new RedditMessagePage(page, logger);
// Set dry run mode if configured
messagePage.setDryRun(config.browser.dryRun);
// Verify login
logger.info('Verifying Reddit login status...');
const isLoggedIn = await messagePage.isLoggedIn();
if (!isLoggedIn) {
logger.warn('Session invalid or not logged in.');
// Attempt automated login if credentials are provided
if (config.reddit.username && config.reddit.password) {
logger.info('Credentials found in config/env. Attempting automated login...');
const loginSuccess = await browserClient.login();
if (!loginSuccess) {
logger.error('Automated login failed. Please check your credentials or run "npm run login" locally.');
await browserClient.close();
process.exit(1);
}
} else {
logger.error('Not logged in to Reddit and no credentials provided!');
logger.error('Please run "npm run login" first to authenticate, OR set REDDIT_USERNAME and REDDIT_PASSWORD environment variables.');
await browserClient.close();
process.exit(1);
}
}
logger.info('Reddit login verified');
// Initialize services
const redditApi = new RedditApiClient(config, logger);
redditApi.setBrowserContext(browserClient.getContext()!);
// Create separate pages for concurrent operations
// 1. chatPage: For background response checking
const chatPage = await browserClient.getContext()!.newPage();
// 2. webUIPage: For user-driven chat browsing in the web dashboard
const webUIPage = await browserClient.getContext()!.newPage();
const aiClient = new OpenRouterClient(config, logger);
const monitorService = new MonitorService(redditApi, postsRepo, messagesRepo, config, logger);
const processorService = new ProcessorService(redditApi, aiClient, config, logger);
const strategySelector = new StrategySelector(config, messagesRepo, logger);
processorService.setStrategySelector(strategySelector);
const responseChecker = new ResponseCheckerService(
chatPage,
messagesRepo,
successfulMessagesRepo,
postsRepo,
logger
);
const mediaDownloader = new PlaywrightMediaDownloader(webUIPage, logger);
const mediaService = new MediaService(mediaDownloader, logger);
const chatService = new ChatService(webUIPage, logger, successfulMessagesRepo, mediaService);
const mediaSyncService = new MediaSyncService(chatService, logger, config.media.syncIntervalMs);
const rateLimiter = new RateLimiter(config, logger);
await chatService.loadCache();
// Initialize auto-reply service
const autoReplyService = new AutoReplyService(
chatService,
aiClient,
config,
logger,
// onReplyGenerated
(chat) => {
webServer?.logActivity('info', `Auto-reply pending approval for u/${chat.username}`, chat.username);
webServer?.pushLog('info', `Reply pending approval for u/${chat.username}: "${chat.pendingReply?.slice(0, 80)}..."`);
},
// onUpsellDetected
(chat, keyword) => {
webServer?.logActivity('skipped', `Upsell detected from u/${chat.username} (keyword: "${keyword}")`, chat.username);
webServer?.pushLog('warn', `Upsell detected from u/${chat.username} — keyword: "${keyword}"`);
},
// onAutoSent
(chat) => {
webServer?.logActivity('sent', `Auto-reply sent to u/${chat.username}`, chat.username);
webServer?.pushLog('info', `Auto-reply sent to u/${chat.username}`);
},
);
// Start background media sync
if (config.media.syncEnabled) {
mediaSyncService.start();
}
// Start auto-reply service
if (config.autoReply.enabled) {
autoReplyService.start();
}
// Initialize web dashboard
let webServer: WebServer | null = null;
if (config.web.enabled) {
webServer = new WebServer(
config.web.port,
config,
postsRepo,
messagesRepo,
successfulMessagesRepo,
logger
);
webServer.setChatService(chatService);
webServer.setMediaService(mediaService);
webServer.setMediaSyncService(mediaSyncService);
webServer.setResponseChecker(responseChecker);
webServer.setRateLimiter(rateLimiter);
webServer.setAiClient(aiClient);
webServer.setStrategySelector(strategySelector);
webServer.setAutoReplyService(autoReplyService);
webServer.setOnConfigChange(() => logger.info('Runtime config updated via dashboard'));
webServer.start();
}
// Load any existing successful examples
const existingExamples = responseChecker.getSuccessfulExamples(3);
processorService.setSuccessfulExamples(existingExamples);
const successCount = responseChecker.getSuccessCount();
if (successCount > 0) {
logger.info(`Loaded ${successCount} successful message(s) from history`);
}
// Graceful shutdown handler
let isShuttingDown = false;
let responseCheckInterval: NodeJS.Timeout | null = null;
const shutdown = async (): Promise<void> => {
if (isShuttingDown) return;
isShuttingDown = true;
if (responseCheckInterval) {
clearInterval(responseCheckInterval);
}
mediaSyncService.stop();
autoReplyService.stop();
logger.info('Shutting down gracefully...');
webServer?.stop();
await browserClient.close();
db.close();
logger.info('Goodbye!');
process.exit(0);
};
process.on('SIGINT', shutdown);
process.on('SIGTERM', shutdown);
logger.info('');
logger.info('Bot is now running!');
logger.info(`Monitoring subreddits: ${config.reddit.subreddits.join(', ')}`);
logger.info(`Poll interval: ${config.reddit.pollIntervalMs / 1000} seconds`);
logger.info(`Min delay between messages: ${config.rateLimit.minDelayBetweenMessagesMs / 1000} seconds`);
logger.info(`Max messages per hour: ${config.rateLimit.maxMessagesPerHour}`);
logger.info('');
// Start background response checking
const RESPONSE_CHECK_INTERVAL_MS = Math.max(config.reddit.pollIntervalMs * 5, 60000);
const checkResponses = async () => {
if (isShuttingDown) return;
try {
logger.debug('Checking for responses to sent messages (Background Thread)...');
const responses = await responseChecker.checkForResponses();
if (responses.some((r) => r.gotResponse)) {
// Reload successful examples
const updatedExamples = responseChecker.getSuccessfulExamples(3);
processorService.setSuccessfulExamples(updatedExamples);
}
} catch (error) {
logger.error('Error in background response checker:', error);
}
};
// Run immediately then schedule
checkResponses();
responseCheckInterval = setInterval(checkResponses, RESPONSE_CHECK_INTERVAL_MS);
// Track poll states
let pollCycle = 0;
let lastPausedLog = 0;
// Main loop
while (!isShuttingDown) {
try {
// Check if bot is running (controlled via web UI)
if (webServer && !webServer.isRunning()) {
if (Date.now() - lastPausedLog > 60000) { // Log every minute
logger.info('Bot is paused. Use the web dashboard to start.');
lastPausedLog = Date.now();
}
await sleep(1000); // Check every second for quick start
continue;
}
pollCycle++;
// Get unprocessed posts
const posts = await monitorService.getUnprocessedPosts();
if (posts.length > 0) {
logger.info(`Found ${posts.length} unprocessed posts`);
webServer?.pushLog('info', `Found ${posts.length} unprocessed posts`);
} else {
logger.info('No new posts found. Monitoring...');
}
for (const post of posts) {
if (isShuttingDown) break;
try {
// Mark as processed immediately to avoid duplicates on restart
const postImageUrl = redditApi.getImageUrl(post);
postsRepo.insert({
id: post.id,
subreddit: post.subreddit,
author: post.author,
title: post.title,
imageUrl: postImageUrl,
processedAt: Date.now(),
});
// Check if user is a seller/OnlyFans profile (before spending on AI)
if (config.reddit.filterSellers && config.reddit.sellerKeywords.length > 0) {
const isSeller = await redditApi.isSellerProfile(post.author, config.reddit.sellerKeywords);
if (isSeller) {
logger.info(`Skipping u/${post.author} - seller/OnlyFans profile detected`);
webServer?.logActivity('skipped', `Skipped u/${post.author} (seller profile)`, post.author);
webServer?.pushLog('info', `🚫 Skipped u/${post.author} - seller/OnlyFans profile`);
postsRepo.setError(post.id, 'Seller profile detected');
continue;
}
}
// Refresh examples weighted toward this post's subreddit
const subredditExamples = responseChecker.getSuccessfulExamples(3, post.subreddit);
processorService.setSuccessfulExamples(subredditExamples);
// Process post (download image, analyze, generate message)
const processed = await withRetry(
() => processorService.processPost(post),
{ logger, retries: 2, operationName: 'Post processing' }
);
if (!processed) {
logger.debug(`Skipping post ${post.id} - processing returned null`);
webServer?.logActivity('skipped', `Skipped post by u/${post.author}`, post.author);
continue;
}
// Update database with analysis
postsRepo.updateAnalysis(post.id, processed.imageAnalysis, processed.generatedMessage);
// Log image analysis and strategy to realtime log
webServer?.pushLog('info', `🖼️ Image analysis for u/${post.author}: ${processed.imageAnalysis}`);
if (processed.strategyId) {
webServer?.pushLog('info', `🎯 Strategy: ${processed.strategyId}`);
}
// Create subject line
const subject = "Stunning";
// Send message with rate limiting
await rateLimiter.throttle(async () => {
logger.info(`Sending message to u/${post.author}...`);
webServer?.pushLog('info', `Sending message to u/${post.author}...`);
const success = await messagePage.sendMessage(
post.author,
subject,
processed.generatedMessage
);
if (success) {
// Message was sent successfully via /message/compose
// Note: We skip chat.reddit.com verification since /message/compose sends to inbox, not chat
// The sendMessage function already confirms success via form submission indicators
postsRepo.markMessageSent(post.id);
// Refresh top chats in background so dashboard is up-to-date
chatService.refreshTopChats(5).catch(err => logger.error('Failed to trigger chat refresh:', err));
messagesRepo.insert({
postId: post.id,
recipient: post.author,
subject,
message: processed.generatedMessage,
sentAt: Date.now(),
strategyId: processed.strategyId,
});
const stats = rateLimiter.getStats();
const currentCount = stats.messagesThisHour + 1;
logger.info(
`Message sent successfully! (${currentCount}/${config.rateLimit.maxMessagesPerHour} this hour)`
);
webServer?.logActivity('sent', `Sent message to u/${post.author}`, post.author, postImageUrl ?? undefined);
webServer?.pushLog('info', `✅ Message sent to u/${post.author} (${currentCount}/${config.rateLimit.maxMessagesPerHour} this hour)`);
webServer?.updateStats(currentCount);
} else {
postsRepo.setError(post.id, 'Failed to send message');
webServer?.logActivity('error', `Failed to send to u/${post.author}`, post.author, postImageUrl ?? undefined);
webServer?.pushLog('error', `❌ Failed to send message to u/${post.author}`);
}
return success;
}, (success) => success === true);
} catch (error) {
const errorMessage = error instanceof Error ? error.message : String(error);
logger.error(`Error processing post ${post.id}:`, error);
postsRepo.setError(post.id, errorMessage);
}
}
if (!isShuttingDown) {
// Wait before next poll
logger.debug(`Waiting ${config.reddit.pollIntervalMs / 1000} seconds before next poll...`);
await sleep(config.reddit.pollIntervalMs);
}
} catch (error) {
logger.error('Error in main loop:', error);
// Wait a minute on error before retrying
await sleep(60000);
}
}
const app = await bootstrap(config, logger);
await runMainLoop(app);
}
main().catch((error) => {
src/main-loop.ts
+276
View File
@@ -0,0 +1,276 @@
import type { AppContainer } from './bootstrap.js';
import { withRetry } from './utils/retry.js';
import { isSkip } from './services/processor.js';
function sleep(ms: number): Promise<void> {
return new Promise((resolve) => setTimeout(resolve, ms));
}
const PAUSED_LOG_INTERVAL_MS = 60_000;
const PAUSE_POLL_MS = 1_000;
const ERROR_BACKOFF_MS = 60_000;
/**
* Runs the bot's main polling loop and the background response checker until
* SIGINT/SIGTERM. Resolves once shutdown completes.
*/
export async function runMainLoop(app: AppContainer): Promise<void> {
const {
config,
logger,
browserClient,
postsRepo,
messagesRepo,
redditApi,
monitorService,
processorService,
responseChecker,
chatService,
rateLimiter,
mediaSyncService,
db,
webServer,
} = app;
let isShuttingDown = false;
let responseCheckInterval: NodeJS.Timeout | null = null;
const shutdown = async (): Promise<void> => {
if (isShuttingDown) return;
isShuttingDown = true;
if (responseCheckInterval) clearInterval(responseCheckInterval);
mediaSyncService.stop();
logger.info('Shutting down gracefully...');
webServer?.stop();
await browserClient.close();
db.close();
logger.info('Goodbye!');
process.exit(0);
};
process.on('SIGINT', shutdown);
process.on('SIGTERM', shutdown);
logger.info('');
logger.info('Bot is now running!');
logger.info(`Monitoring subreddits: ${config.reddit.subreddits.map((s) => `${s.name}[${s.type}]`).join(', ')}`);
logger.info(`Poll interval: ${config.reddit.pollIntervalMs / 1000} seconds`);
logger.info(`Min delay between messages: ${config.rateLimit.minDelayBetweenMessagesMs / 1000} seconds`);
logger.info(`Max messages per hour: ${config.rateLimit.maxMessagesPerHour}`);
logger.info('');
// Background response checker
const RESPONSE_CHECK_INTERVAL_MS = Math.max(config.reddit.pollIntervalMs * 5, 60_000);
const checkResponses = async (): Promise<void> => {
if (isShuttingDown) return;
try {
logger.debug('Checking for responses to sent messages (Background Thread)...');
const responses = await responseChecker.checkForResponses();
for (const r of responses) {
if (!r.gotResponse) continue;
const snippet = r.responseSnippet ? `: ${r.responseSnippet.slice(0, 200)}` : '';
webServer?.logActivity(
'response',
`Reply from u/${r.recipient}${snippet}`,
r.recipient,
);
webServer?.pushLog('info', `💬 Reply from u/${r.recipient}`);
}
if (responses.some((r) => r.gotResponse)) {
const updatedExamples = responseChecker.getSuccessfulExamples(3);
processorService.setSuccessfulExamples(updatedExamples);
}
} catch (error) {
logger.error('Error in background response checker:', error);
}
};
checkResponses();
responseCheckInterval = setInterval(checkResponses, RESPONSE_CHECK_INTERVAL_MS);
let lastPausedLog = 0;
let lastCooldownLog = 0;
const COOLDOWN_LOG_INTERVAL_MS = 5 * 60_000;
while (!isShuttingDown) {
try {
if (webServer && !webServer.isRunning()) {
if (Date.now() - lastPausedLog > PAUSED_LOG_INTERVAL_MS) {
logger.info('Bot is paused. Use the web dashboard to start.');
lastPausedLog = Date.now();
}
await sleep(PAUSE_POLL_MS);
continue;
}
// If Reddit told us to take a break on creating new chats, don't even
// pull new posts — they'd just queue up and skip. Wait the cooldown out.
const cooldown = chatService.getInvitationCooldown();
if (cooldown) {
const minutesLeft = Math.ceil((cooldown.until - Date.now()) / 60_000);
if (Date.now() - lastCooldownLog > COOLDOWN_LOG_INTERVAL_MS) {
logger.warn(`Chat invitation cooldown active — ~${minutesLeft}m left. ${cooldown.reason}`);
webServer?.pushLog(
'warn',
`⏸️ Chat throttle: ${cooldown.reason} (~${minutesLeft}m remaining)`,
);
lastCooldownLog = Date.now();
}
await sleep(config.reddit.pollIntervalMs);
continue;
}
lastCooldownLog = 0;
const posts = await monitorService.getUnprocessedPosts();
if (posts.length > 0) {
logger.info(`Found ${posts.length} unprocessed posts`);
webServer?.pushLog('info', `Found ${posts.length} unprocessed posts`);
} else {
logger.info('No new posts found. Monitoring...');
}
for (const post of posts) {
if (isShuttingDown) break;
try {
const subredditEntry = config.reddit.subreddits.find(
(s) => s.name.toLowerCase() === post.subreddit.toLowerCase(),
);
const isChatSub = subredditEntry?.type === 'chat';
const postImageUrl = isChatSub ? null : redditApi.getImageUrl(post);
postsRepo.insert({
id: post.id,
subreddit: post.subreddit,
author: post.author,
title: post.title,
imageUrl: postImageUrl,
processedAt: Date.now(),
});
// Skip OF/seller accounts before spending on AI.
if (
config.reddit.filterSellers &&
(config.reddit.sellerKeywords.length > 0 || config.reddit.sellerDomains.length > 0)
) {
const isSeller = await redditApi.isSellerProfile(
post.author,
config.reddit.sellerKeywords,
config.reddit.sellerDomains
);
if (isSeller) {
logger.info(`Skipping u/${post.author} - seller/OnlyFans profile detected`);
webServer?.logActivity(
'skipped',
`Skipped post by u/${post.author} (seller/OnlyFans profile)`,
post.author,
);
webServer?.pushLog('info', `🚫 Skipped u/${post.author} - seller/OnlyFans profile`);
postsRepo.setError(post.id, 'Seller profile detected');
continue;
}
}
// Refresh examples weighted toward this post's subreddit.
const subredditExamples = responseChecker.getSuccessfulExamples(3, post.subreddit);
processorService.setSuccessfulExamples(subredditExamples);
const processed = await withRetry(
() => processorService.processPost(post),
{ logger, retries: 2, operationName: 'Post processing' },
);
if (isSkip(processed)) {
logger.debug(`Skipping post ${post.id} - ${processed.skipReason}`);
webServer?.logActivity(
'skipped',
`Skipped post by u/${post.author} (${processed.skipReason})`,
post.author,
);
webServer?.pushLog('info', `⏭️ Skipped u/${post.author} — ${processed.skipReason}`);
continue;
}
postsRepo.updateAnalysis(post.id, processed.imageAnalysis, processed.generatedMessage);
webServer?.pushLog(
'info',
`${isChatSub ? '💬' : '🖼️'} ${isChatSub ? 'Chat post' : 'Image analysis'} for u/${post.author}: ${processed.imageAnalysis}`,
);
if (processed.strategyId) {
webServer?.pushLog('info', `🎯 Strategy: ${processed.strategyId}`);
}
await rateLimiter.throttle(
async () => {
logger.info(`Sending message to u/${post.author}...`);
webServer?.pushLog('info', `Sending message to u/${post.author}...`);
const chat = await chatService.startConversation(post.author, processed.generatedMessage);
const success = chat !== null;
if (success) {
postsRepo.markMessageSent(post.id);
// Refresh top chats in background so dashboard is up-to-date.
chatService
.refreshTopChats(5)
.catch((err) => logger.error('Failed to trigger chat refresh:', err));
messagesRepo.insert({
postId: post.id,
recipient: post.author,
subject: post.author,
message: processed.generatedMessage,
sentAt: Date.now(),
strategyId: processed.strategyId,
});
const stats = rateLimiter.getStats();
const currentCount = stats.messagesThisHour + 1;
logger.info(
`Message sent successfully! (${currentCount}/${config.rateLimit.maxMessagesPerHour} this hour)`,
);
webServer?.logActivity(
'sent',
`Sent message to u/${post.author} (r/${post.subreddit})`,
post.author,
postImageUrl ?? undefined,
);
webServer?.pushLog('info', `✅ Message sent to u/${post.author} (${currentCount}/${config.rateLimit.maxMessagesPerHour} this hour)`);
webServer?.updateStats(currentCount);
} else {
postsRepo.setError(post.id, 'Failed to send message');
webServer?.logActivity(
'error',
`Failed to send to u/${post.author} (r/${post.subreddit})`,
post.author,
postImageUrl ?? undefined,
);
webServer?.pushLog('error', `❌ Failed to send message to u/${post.author}`);
}
return success;
},
(success) => success === true,
);
} catch (error) {
const errorMessage = error instanceof Error ? error.message : String(error);
logger.error(`Error processing post ${post.id}:`, error);
postsRepo.setError(post.id, errorMessage);
}
}
if (!isShuttingDown) {
logger.debug(`Waiting ${config.reddit.pollIntervalMs / 1000} seconds before next poll...`);
await sleep(config.reddit.pollIntervalMs);
}
} catch (error) {
logger.error('Error in main loop:', error);
await sleep(ERROR_BACKOFF_MS);
}
}
}
src/reddit/api.ts
+49 -13
View File
@@ -13,6 +13,7 @@ export interface RedditPost {
permalink: string;
thumbnail: string;
is_self: boolean;
selftext?: string;
post_hint?: string;
preview?: {
images: Array<{
@@ -25,6 +26,25 @@ export interface RedditPost {
spoiler: boolean;
}
// Extracts URL-ish tokens (with or without scheme) and returns the first
// domain match. Matches `host` or any subdomain (`*.host`), case-insensitive.
export function findSellerDomain(text: string, domains: string[]): string | null {
if (!text || domains.length === 0) return null;
const normalized = text.toLowerCase();
const urlRegex = /(?:https?:\/\/|(?<![\w.@])(?=[a-z0-9-]+\.[a-z]))([a-z0-9.-]+\.[a-z]{2,})(?:[/?#][^\s)]*)?/gi;
const hosts: string[] = [];
for (const m of normalized.matchAll(urlRegex)) {
hosts.push(m[1].replace(/\.+$/, ''));
}
for (const host of hosts) {
for (const domain of domains) {
const d = domain.toLowerCase();
if (host === d || host.endsWith('.' + d)) return domain;
}
}
return null;
}
export class RedditApiClient {
private client: AxiosInstance;
private browserContext: BrowserContext | null = null;
@@ -62,7 +82,7 @@ export class RedditApiClient {
}
}
async getNewPosts(subreddit: string, limit = 25): Promise<RedditPost[]> {
async getNewPosts(subreddit: string, limit = 25, type: 'image' | 'chat' = 'image'): Promise<RedditPost[]> {
try {
const cookieHeader = await this.getCookieHeader();
const headers: Record<string, string> = {};
@@ -77,9 +97,9 @@ export class RedditApiClient {
const posts: RedditPost[] = response.data.data.children
.map((child: { data: RedditPost }) => child.data)
.filter((post: RedditPost) => this.isValidPost(post));
.filter((post: RedditPost) => this.isValidPost(post, type));
this.logger.debug(`Fetched ${posts.length} valid image posts from r/${subreddit}`);
this.logger.debug(`Fetched ${posts.length} valid ${type} posts from r/${subreddit}`);
return posts;
} catch (error) {
if (axios.isAxiosError(error)) {
@@ -101,17 +121,19 @@ export class RedditApiClient {
}
}
private isValidPost(post: RedditPost): boolean {
// Skip self posts (text only)
if (post.is_self) return false;
private isValidPost(post: RedditPost, type: 'image' | 'chat' = 'image'): boolean {
// Skip deleted/removed authors
if (post.author === '[deleted]' || post.author === 'AutoModerator') return false;
// Skip NSFW if we want to be safe (configurable later)
// if (post.over_18) return false;
if (type === 'chat') {
// Chat subs are text-driven. Require a title (always present) and
// ideally some self text — but allow title-only since some tags live
// in the title alone (e.g. "[25F] up late, dm me").
return true;
}
// Check if it's an image post
// image type:
if (post.is_self) return false;
return this.hasImage(post);
}
@@ -180,8 +202,12 @@ export class RedditApiClient {
return url.replace(/&amp;/g, '&');
}
async isSellerProfile(username: string, sellerKeywords: string[]): Promise<boolean> {
if (sellerKeywords.length === 0) return false;
async isSellerProfile(
username: string,
sellerKeywords: string[],
sellerDomains: string[] = []
): Promise<boolean> {
if (sellerKeywords.length === 0 && sellerDomains.length === 0) return false;
try {
const cookieHeader = await this.getCookieHeader();
@@ -210,7 +236,17 @@ export class RedditApiClient {
for (const keyword of sellerKeywords) {
if (textToCheck.includes(keyword.toLowerCase())) {
this.logger.info(
`u/${username} flagged as seller profile (matched: "${keyword}")`
`u/${username} flagged as seller profile (matched keyword: "${keyword}")`
);
return true;
}
}
if (sellerDomains.length > 0) {
const matchedDomain = findSellerDomain(textToCheck, sellerDomains);
if (matchedDomain) {
this.logger.info(
`u/${username} flagged as seller profile (matched domain: "${matchedDomain}")`
);
return true;
}
src/services/auto-reply.ts
-199
View File
@@ -1,199 +0,0 @@
import type { Logger } from '../utils/logger.js';
import type { Config } from '../config/index.js';
import type { OpenRouterClient } from '../ai/openrouter.js';
import type { ChatService, ActiveChat } from './chat.js';
export class AutoReplyService {
private interval: NodeJS.Timeout | null = null;
private processing = false;
// Track which conversations we've already processed to avoid re-generating replies
private processedConversations = new Set<string>();
constructor(
private chatService: ChatService,
private aiClient: OpenRouterClient,
private config: Config,
private logger: Logger,
private onReplyGenerated?: (chat: ActiveChat) => void,
private onUpsellDetected?: (chat: ActiveChat, keyword: string) => void,
private onAutoSent?: (chat: ActiveChat) => void,
) { }
async start(): Promise<void> {
if (!this.config.autoReply.enabled) {
this.logger.info('Auto-reply is disabled');
return;
}
// Seed processedConversations from cache so we don't re-generate on restart
const cached = await this.chatService.getCachedConversations();
for (const chat of cached) {
if (chat.replyStatus) {
const stateKey = `${chat.username}:${chat.lastMessageTime}:${chat.lastMessageSnippet}`;
this.processedConversations.add(stateKey);
}
}
this.logger.info(`Auto-reply: Seeded ${this.processedConversations.size} already-processed conversation(s) from cache`);
const intervalMs = this.config.autoReply.checkIntervalMs;
this.logger.info(`Auto-reply service started (interval: ${intervalMs / 1000}s, autoSend: ${this.config.autoReply.autoSend})`);
// Run immediately then schedule
this.processUnreadChats();
this.interval = setInterval(() => this.processUnreadChats(), intervalMs);
}
stop(): void {
if (this.interval) {
clearInterval(this.interval);
this.interval = null;
}
}
/**
* Manually trigger processing (e.g., after a chat refresh)
*/
async triggerProcess(): Promise<void> {
if (!this.config.autoReply.enabled) return;
await this.processUnreadChats();
}
private async processUnreadChats(): Promise<void> {
if (this.processing) return;
this.processing = true;
try {
const chats = await this.chatService.getCachedConversations();
// Find conversations where:
// 1. The last message is NOT from us (they sent a message)
// 2. The conversation is marked as unread OR the last message is from them
// 3. We haven't already processed this state (using message snippet + time as key)
const unreadChats = chats.filter(chat => {
if (!chat.messages || chat.messages.length === 0) return false;
if (chat.replyStatus) return false; // Already processed (pending, upsell_detected, or sent)
// The definitive check: the last message in the actual history must be from THEM, not us
const lastMessage = chat.messages[chat.messages.length - 1];
if (lastMessage.isSelf) return false;
// Create a unique key for this conversation state
const stateKey = `${chat.username}:${chat.lastMessageTime}:${chat.lastMessageSnippet}`;
if (this.processedConversations.has(stateKey)) return false;
return true;
});
if (unreadChats.length === 0) {
this.logger.debug('Auto-reply: No unread conversations to process');
this.processing = false;
return;
}
this.logger.info(`Auto-reply: Processing ${unreadChats.length} unread conversation(s)...`);
for (const chat of unreadChats) {
try {
await this.processChat(chat);
} catch (error) {
this.logger.error(`Auto-reply: Error processing chat with ${chat.username}:`, error);
}
}
} catch (error) {
this.logger.error('Auto-reply: Error in processUnreadChats:', error);
} finally {
this.processing = false;
}
}
private async processChat(chat: ActiveChat): Promise<void> {
const stateKey = `${chat.username}:${chat.lastMessageTime}:${chat.lastMessageSnippet}`;
this.logger.info(`Auto-reply: Checking conversation with ${chat.username}...`);
// Safety gate: verify the last message is truly from them, not us
const lastMsg = chat.messages![chat.messages!.length - 1];
if (lastMsg.isSelf) {
this.logger.info(`Auto-reply: Skipping ${chat.username} — we sent the last message`);
this.processedConversations.add(stateKey);
return;
}
// Step 1: Check for upsell/spam
const upsellResult = this.aiClient.detectUpsell(
chat.messages!.map(m => ({ text: m.text, isSelf: m.isSelf })),
this.config.autoReply.upsellKeywords
);
if (upsellResult.isUpsell) {
this.logger.info(`Auto-reply: Upsell detected from ${chat.username} (keyword: "${upsellResult.matchedKeyword}")`);
chat.replyStatus = 'upsell_detected';
chat.pendingReply = undefined;
this.processedConversations.add(stateKey);
this.onUpsellDetected?.(chat, upsellResult.matchedKeyword!);
// Save the updated status to cache
await this.chatService.saveCacheExternal();
return;
}
// Step 2: Generate a proposed reply
this.logger.info(`Auto-reply: Generating reply for ${chat.username}...`);
const suggestedReply = await this.aiClient.generateReply(
chat.messages!.map(m => ({ sender: m.sender, text: m.text, isSelf: m.isSelf })),
chat.username
);
if (!suggestedReply) {
this.logger.warn(`Auto-reply: Failed to generate reply for ${chat.username}`);
this.processedConversations.add(stateKey);
return;
}
// Step 3: Auto-send or store as pending
if (this.config.autoReply.autoSend) {
if (chat.url) {
this.logger.info(`Auto-reply: Auto-sending reply to ${chat.username}: "${suggestedReply.slice(0, 80)}..."`);
const sent = await this.chatService.sendChatMessage(chat.url, suggestedReply);
if (sent) {
chat.replyStatus = 'sent';
chat.pendingReply = suggestedReply;
this.onAutoSent?.(chat);
// Refresh chats in background so cache reflects the sent message
this.chatService.refreshTopChats(5).catch(err =>
this.logger.error('Failed to refresh chats after auto-send:', err)
);
} else {
this.logger.error(`Auto-reply: Failed to send message to ${chat.username}`);
}
} else {
this.logger.warn(`Auto-reply: No chat URL for ${chat.username}, cannot auto-send`);
}
} else {
// Store as pending approval
chat.pendingReply = suggestedReply;
chat.replyStatus = 'pending';
this.logger.info(`Auto-reply: Reply pending approval for ${chat.username}: "${suggestedReply.slice(0, 80)}..."`);
this.onReplyGenerated?.(chat);
}
this.processedConversations.add(stateKey);
// Save the updated status to cache
await this.chatService.saveCacheExternal();
}
/**
* Clear the processed state for a conversation so it can be re-evaluated
*/
clearProcessedState(username: string): void {
const keysToRemove: string[] = [];
for (const key of this.processedConversations) {
if (key.startsWith(`${username}:`)) {
keysToRemove.push(key);
}
}
keysToRemove.forEach(k => this.processedConversations.delete(k));
}
}
src/services/chat.ts
+1 -1178
View File
File diff suppressed because it is too large. Load diff
src/services/chat/cache.ts
+108
View File
@@ -0,0 +1,108 @@
import fs from 'fs/promises';
import path from 'path';
import type { Logger } from '../../utils/logger.js';
import type { ActiveChat, ChatCacheFile } from './types.js';
export interface SyncMarkers {
nextBatch: string | null;
lastFullSyncAt: number | null;
lastDeltaSyncAt: number | null;
}
/**
* In-memory + on-disk cache of Reddit chats plus the Matrix delta cursor
* so the bot can resume incremental sync across restarts.
*/
export class ChatCache {
private chats: ActiveChat[] = [];
private lastFetchTime = 0;
readonly cacheDurationMs = 60 * 1000;
private readonly filePath: string;
private nextBatch: string | null = null;
private lastFullSyncAt: number | null = null;
private lastDeltaSyncAt: number | null = null;
constructor(private logger: Logger, filePath?: string) {
this.filePath = filePath ?? path.join(process.cwd(), 'data', 'chat-cache.json');
}
get(): ActiveChat[] {
return this.chats;
}
set(chats: ActiveChat[]): void {
this.chats = chats;
this.lastFetchTime = Date.now();
}
isFresh(): boolean {
return Date.now() - this.lastFetchTime < this.cacheDurationMs && this.chats.length > 0;
}
getMarkers(): SyncMarkers {
return {
nextBatch: this.nextBatch,
lastFullSyncAt: this.lastFullSyncAt,
lastDeltaSyncAt: this.lastDeltaSyncAt,
};
}
setNextBatch(token: string | null): void {
this.nextBatch = token;
}
markFullSync(at: number = Date.now()): void {
this.lastFullSyncAt = at;
this.lastDeltaSyncAt = at;
}
markDeltaSync(at: number = Date.now()): void {
this.lastDeltaSyncAt = at;
}
async load(): Promise<void> {
try {
const raw = await fs.readFile(this.filePath, 'utf-8');
const parsed = JSON.parse(raw) as ChatCacheFile;
if (!parsed || !Array.isArray(parsed.chats)) {
this.logger.warn('Chat cache file is invalid, ignoring.');
return;
}
this.chats = parsed.chats;
this.lastFetchTime = Date.now();
this.nextBatch = parsed.nextBatch ?? null;
this.lastFullSyncAt = parsed.lastFullSyncAt ?? null;
this.lastDeltaSyncAt = parsed.lastDeltaSyncAt ?? null;
this.logger.info(
`Loaded ${parsed.chats.length} chats from cache` +
(this.lastFullSyncAt
? ` (last full sync ${new Date(this.lastFullSyncAt).toISOString()})`
: ' (no full sync recorded)'),
);
} catch (error) {
if ((error as NodeJS.ErrnoException).code === 'ENOENT') {
this.logger.info('No chat cache found to load');
return;
}
this.logger.warn('Failed to load chat cache:', error);
}
}
async save(chats: ActiveChat[] = this.chats): Promise<void> {
const payload: ChatCacheFile = {
version: 1,
savedAt: Date.now(),
chats,
nextBatch: this.nextBatch,
lastFullSyncAt: this.lastFullSyncAt,
lastDeltaSyncAt: this.lastDeltaSyncAt,
};
try {
await fs.mkdir(path.dirname(this.filePath), { recursive: true });
await fs.writeFile(this.filePath, JSON.stringify(payload, null, 2), 'utf-8');
} catch (error) {
this.logger.warn('Failed to save chat cache:', error);
}
}
}
src/services/chat/index.ts
+24
View File
@@ -0,0 +1,24 @@
export { ChatService } from './service.js';
export type {
TokenRefresher,
UsernameResolver,
SyncStatus,
SyncProgress,
SyncPhase,
SyncProgressListener,
} from './service.js';
export type {
ActiveChat,
ChatMessage,
ChatMedia,
ChatCacheFile,
ScrapedChatMessage,
ScrapedRoom,
} from './types.js';
export { MatrixClient, MatrixAuthError } from './matrix/client.js';
export type { MatrixCredentials } from './matrix/types.js';
export {
extractMatrixCredentials,
loadCachedCredentials,
} from './matrix/token-extractor.js';
export { MatrixMediaDownloader } from './matrix/http-media-downloader.js';
src/services/chat/matrix/client.ts
+529
View File
@@ -0,0 +1,529 @@
import type { Logger } from '../../../utils/logger.js';
import type {
MatrixCredentials,
MatrixSyncResponse,
MatrixMessagesResponse,
MatrixTimelineEvent,
MatrixJoinedRoom,
} from './types.js';
export class MatrixAuthError extends Error {
constructor(message: string) {
super(message);
this.name = 'MatrixAuthError';
}
}
/**
* Thrown when Reddit's homeserver returns a 429 with a `retry_after_ms`
* longer than we're willing to block on. Caller (typically ChatService)
* is expected to record this as a cooldown and stop trying until the
* window expires.
*/
export class MatrixRateLimitError extends Error {
constructor(
message: string,
public retryAfterMs: number,
public errcode: string | null,
public redditCode: string | null,
) {
super(message);
this.name = 'MatrixRateLimitError';
}
}
export interface SyncOptions {
/** Pagination token from a previous /sync. Omit for initial sync. */
since?: string;
/** Long-poll timeout. Default 0 = return immediately. */
timeoutMs?: number;
/** Filter blob (JSON-encoded) — omitted by default. */
filter?: string;
}
export interface RoomSummary {
roomId: string;
/** Sorted member IDs other than ourselves; useful for DMs (1 entry). */
otherMembers: string[];
/** From summary.m.heroes or m.room.name event content. */
displayName: string | null;
unreadCount: number;
highlightCount: number;
lastEvent: MatrixTimelineEvent | null;
/** Raw timeline events from this sync — initial sync gives a small window. */
recentEvents: MatrixTimelineEvent[];
}
/**
* Cap on how long we'll sit and sleep waiting out a 429. Anything longer
* is surfaced as a `MatrixRateLimitError` so the main loop can back off
* cleanly instead of silently blocking for minutes.
*/
const LONG_RETRY_THRESHOLD_MS = 5_000;
export interface MatrixClientOptions {
/**
* Minimum interval (ms) between *outbound* HTTP requests, applied across
* all callers. Defaults to 30ms (~33 req/s) which keeps us comfortably
* under Reddit's homeserver rate limit even with several parallel workers.
*/
minRequestIntervalMs?: number;
/**
* Per-request HTTP timeout. Without this a stalled connection to the
* homeserver would hang the caller forever (see incident 2026-05-20).
* An aborted fetch is treated as a transient error and retried with
* the same backoff as a 5xx.
*/
requestTimeoutMs?: number;
/** Test seam: inject a custom fetch (e.g. one that hangs). */
fetchImpl?: typeof fetch;
}
/**
* Thin Matrix Client-Server v3 wrapper for the Reddit chat homeserver.
* No SDK dependency — just `fetch`. Talks the same protocol as
* matrix-js-sdk but only the few endpoints we need.
*/
export class MatrixClient {
private nextBatch: string | null = null;
private nextRequestSlot = 0;
private readonly minRequestIntervalMs: number;
private readonly requestTimeoutMs: number;
private readonly fetchImpl: typeof fetch;
constructor(
private creds: MatrixCredentials,
private logger: Logger,
options: MatrixClientOptions = {},
) {
this.minRequestIntervalMs = options.minRequestIntervalMs ?? 30;
this.requestTimeoutMs = options.requestTimeoutMs ?? 30_000;
this.fetchImpl = options.fetchImpl ?? fetch;
}
/**
* Block until we're allowed to fire the next outbound HTTP request.
* Stamps `nextRequestSlot` atomically so concurrent acquirers get
* staggered, non-overlapping slots.
*/
private async acquireRequestSlot(): Promise<void> {
const now = Date.now();
const slot = Math.max(now, this.nextRequestSlot);
this.nextRequestSlot = slot + this.minRequestIntervalMs;
const wait = slot - now;
if (wait > 0) await new Promise((r) => setTimeout(r, wait));
}
get userId(): string {
return this.creds.userId;
}
get homeserver(): string {
return this.creds.homeserver;
}
/** Replace the bearer (e.g. after a re-extract on 401). */
updateCredentials(creds: MatrixCredentials): void {
this.creds = creds;
this.nextBatch = null;
}
/**
* `fetch` with a hard client-side timeout. Returns the Response on success,
* or null if the request aborted — caller treats null as transient and
* retries with backoff.
*/
private async timedFetch(
url: string,
init: RequestInit,
): Promise<Response | null> {
try {
return await this.fetchImpl(url, {
...init,
signal: AbortSignal.timeout(this.requestTimeoutMs),
});
} catch (err) {
const name = (err as { name?: string }).name;
if (name === 'AbortError' || name === 'TimeoutError') {
this.logger.warn(
`[matrix] request to ${url} timed out after ${this.requestTimeoutMs}ms`,
);
return null;
}
throw err;
}
}
private async request<T>(
method: 'GET' | 'POST' | 'PUT',
path: string,
options: { query?: Record<string, string | number | undefined>; body?: unknown } = {},
): Promise<T> {
const url = new URL(this.creds.homeserver + path);
if (options.query) {
for (const [k, v] of Object.entries(options.query)) {
if (v !== undefined) url.searchParams.set(k, String(v));
}
}
const maxAttempts = 10;
for (let attempt = 1; attempt <= maxAttempts; attempt++) {
await this.acquireRequestSlot();
const res = await this.timedFetch(url.toString(), {
method,
headers: {
Authorization: `Bearer ${this.creds.accessToken}`,
'Content-Type': 'application/json',
},
body: options.body ? JSON.stringify(options.body) : undefined,
});
if (res === null) {
// Aborted — retry with the same backoff as a 5xx.
if (attempt < maxAttempts) {
const backoff =
Math.min(2000, 200 * 2 ** (attempt - 1)) + Math.floor(Math.random() * 150);
await new Promise((r) => setTimeout(r, backoff));
continue;
}
throw new Error(`Matrix ${method} ${path} timed out (exceeded ${maxAttempts} attempts)`);
}
if (res.status === 401 || res.status === 403) {
throw new MatrixAuthError(`Matrix ${method} ${path} → ${res.status} (token likely expired)`);
}
// Matrix-standard rate-limit response: { retry_after_ms } per spec.
if (res.status === 429) {
const body = (await res.json().catch(() => ({}))) as {
retry_after_ms?: number;
errcode?: string;
error?: string;
['com.reddit.error.code']?: string;
['com.reddit.error.msg']?: string;
};
const baseMs = body.retry_after_ms ?? 500;
// If the homeserver tells us to wait longer than this, don't block —
// bubble out so the caller can record a cooldown and skip sends.
// 5s covers a normal Matrix flood-control burst; anything bigger
// (e.g. the ~10 min invitation_limit_mature_account) is a real
// soft throttle we shouldn't sit on.
if (baseMs > LONG_RETRY_THRESHOLD_MS) {
const msg = `Matrix ${method} ${path} → 429 ${body['com.reddit.error.code'] ?? body.errcode ?? ''} retry_after=${baseMs}ms`;
throw new MatrixRateLimitError(
msg,
baseMs,
body.errcode ?? null,
body['com.reddit.error.code'] ?? null,
);
}
// Add tiny jitter so parallel workers don't wake in lockstep.
const waitMs = baseMs + Math.floor(Math.random() * 100);
if (attempt < maxAttempts) {
await new Promise((r) => setTimeout(r, waitMs));
continue;
}
throw new Error(
`Matrix ${method} ${path} → 429 (exceeded ${maxAttempts} retries, last wait ${waitMs}ms)`,
);
}
// Transient upstream errors — Reddit's homeserver occasionally 502/504s
// under load. Retry with exponential backoff + jitter.
if (res.status >= 500 && res.status <= 599 && attempt < maxAttempts) {
const backoff = Math.min(2000, 200 * 2 ** (attempt - 1)) + Math.floor(Math.random() * 150);
await new Promise((r) => setTimeout(r, backoff));
continue;
}
if (!res.ok) {
const text = await res.text().catch(() => '');
throw new Error(`Matrix ${method} ${path} → ${res.status}: ${text.slice(0, 300)}`);
}
return (await res.json()) as T;
}
// Unreachable, but TS doesn't know.
throw new Error(`Matrix ${method} ${path} exhausted retries`);
}
async whoami(): Promise<string> {
const res = await this.request<{ user_id: string }>('GET', '/_matrix/client/v3/account/whoami');
return res.user_id;
}
/**
* All rooms the user is currently joined to. Use this for backfill —
* `/sync` only returns rooms with recent activity.
*/
async getJoinedRooms(): Promise<string[]> {
const res = await this.request<{ joined_rooms: string[] }>(
'GET',
'/_matrix/client/v3/joined_rooms',
);
return res.joined_rooms;
}
/** Member map for a room (user_id → display name/avatar). */
async getJoinedMembers(
roomId: string,
): Promise<Record<string, { display_name?: string; avatar_url?: string }>> {
const res = await this.request<{
joined: Record<string, { display_name?: string; avatar_url?: string }>;
}>(
'GET',
`/_matrix/client/v3/rooms/${encodeURIComponent(roomId)}/joined_members`,
);
return res.joined;
}
/** Returns the room's m.room.name content if set, else null. */
async getRoomName(roomId: string): Promise<string | null> {
try {
const res = await this.request<{ name?: string }>(
'GET',
`/_matrix/client/v3/rooms/${encodeURIComponent(roomId)}/state/m.room.name/`,
);
return res.name ?? null;
} catch (err) {
// 404 = no name event (normal for DMs)
if (err instanceof Error && err.message.includes('→ 404')) return null;
throw err;
}
}
async sync(options: SyncOptions = {}): Promise<MatrixSyncResponse> {
const since = options.since ?? this.nextBatch ?? undefined;
const res = await this.request<MatrixSyncResponse>('GET', '/_matrix/client/v3/sync', {
query: {
since,
timeout: options.timeoutMs ?? 0,
filter: options.filter,
},
});
this.nextBatch = res.next_batch;
return res;
}
/**
* Convenience: do a /sync and project the joined rooms into a flat list.
* For an initial sync (no `since`) Reddit returns each room with a small
* recent-event window — usually enough for "active chats" views.
*/
async listRooms(options: SyncOptions = {}): Promise<RoomSummary[]> {
const sync = await this.sync(options);
const joined = sync.rooms?.join ?? {};
const summaries: RoomSummary[] = [];
for (const [roomId, room] of Object.entries(joined)) {
summaries.push(this.summarizeRoom(roomId, room));
}
summaries.sort((a, b) => {
const at = a.lastEvent?.origin_server_ts ?? 0;
const bt = b.lastEvent?.origin_server_ts ?? 0;
return bt - at;
});
return summaries;
}
private summarizeRoom(roomId: string, room: MatrixJoinedRoom): RoomSummary {
const events = room.timeline?.events ?? [];
const lastMessage = [...events].reverse().find((e) => e.type === 'm.room.message') ?? null;
const lastEvent = events.length > 0 ? events[events.length - 1] : null;
const heroes = room.summary?.['m.heroes'] ?? [];
const otherMembers = heroes.filter((h) => h !== this.creds.userId);
let displayName: string | null = null;
const nameEvent = room.state?.events?.find((e) => e.type === 'm.room.name');
if (nameEvent && typeof nameEvent.content.name === 'string') {
displayName = nameEvent.content.name as string;
} else if (otherMembers.length > 0) {
displayName = otherMembers.map(stripMxid).join(', ');
}
return {
roomId,
otherMembers,
displayName,
unreadCount: room.unread_notifications?.notification_count ?? 0,
highlightCount: room.unread_notifications?.highlight_count ?? 0,
lastEvent: lastMessage ?? lastEvent,
recentEvents: events,
};
}
/**
* Paginate room history backwards. Pass the previous response's `end`
* token as `from` to keep walking.
*/
async getRoomMessages(
roomId: string,
options: { from?: string; limit?: number; dir?: 'b' | 'f' } = {},
): Promise<MatrixMessagesResponse> {
return this.request<MatrixMessagesResponse>(
'GET',
`/_matrix/client/v3/rooms/${encodeURIComponent(roomId)}/messages`,
{
query: {
dir: options.dir ?? 'b',
limit: options.limit ?? 50,
from: options.from,
},
},
);
}
/**
* Create a new Reddit DM room with the given recipient and return its
* room_id. Mirrors exactly what chat.reddit.com sends when you click
* "Start chat" in the UI:
*
* POST /_matrix/client/v3/createRoom
* { "preset": "reddit_dm", "invite": [mxid] }
*
* `reddit_dm` is a Reddit-specific preset that the homeserver expands
* server-side into the right power levels, history visibility, and
* `com.reddit.chat.type` state event.
*/
async createDirectRoom(recipientMxid: string): Promise<string> {
const res = await this.request<{ room_id: string }>('POST', '/_matrix/client/v3/createRoom', {
body: { preset: 'reddit_dm', invite: [recipientMxid] },
});
this.logger.debug(`[matrix] created DM room ${res.room_id} with ${recipientMxid}`);
return res.room_id;
}
/**
* One-shot: create a DM room with the recipient and send the opening
* message. Returns the new room_id and the event_id of the message.
*/
async sendInitialDirectMessage(
recipientMxid: string,
body: string,
): Promise<{ roomId: string; eventId: string }> {
const roomId = await this.createDirectRoom(recipientMxid);
const eventId = await this.sendTextMessage(roomId, body);
return { roomId, eventId };
}
/**
* Convert a Reddit t2 user id (e.g. "t2_abc123") into the Matrix user id
* Reddit's homeserver uses: "@t2_abc123:reddit.com".
*/
static t2ToMxid(t2Id: string): string {
const clean = t2Id.startsWith('t2_') ? t2Id : `t2_${t2Id}`;
return `@${clean}:reddit.com`;
}
/**
* Send a plain-text message. Returns the event_id.
* txnId must be unique per message to make sends idempotent on retry.
*/
async sendTextMessage(roomId: string, body: string, txnId?: string): Promise<string> {
const id = txnId ?? `rbot-${Date.now()}-${Math.random().toString(36).slice(2, 8)}`;
const res = await this.request<{ event_id: string }>(
'PUT',
`/_matrix/client/v3/rooms/${encodeURIComponent(roomId)}/send/m.room.message/${encodeURIComponent(id)}`,
{
body: { msgtype: 'm.text', body },
},
);
this.logger.debug(`[matrix] sent message to ${roomId}: event_id=${res.event_id}`);
return res.event_id;
}
/**
* Convert an mxc:// URI to an HTTPS download URL on this homeserver.
* Reddit's media is served from the same Matrix server.
*
* Returns the *authenticated* v1 endpoint (Matrix v1.11+); use
* `downloadMedia()` to actually fetch with the right headers.
*/
mxcToHttp(mxc: string): string | null {
const parsed = parseMxc(mxc);
if (!parsed) return null;
return `${this.creds.homeserver}/_matrix/client/v1/media/download/${parsed.serverName}/${parsed.mediaId}`;
}
/**
* Download a media object referenced by an mxc:// URI.
*
* Tries the authenticated v1 endpoint first (current spec) and falls back
* to the unauthenticated v3 endpoint (legacy) on 404 — different Matrix
* server versions deploy these at different times.
*/
async downloadMedia(mxc: string): Promise<{ bytes: Buffer; contentType: string } | null> {
const parsed = parseMxc(mxc);
if (!parsed) return null;
const { serverName, mediaId } = parsed;
// Reddit's homeserver serves the legacy unauthenticated v3 endpoint;
// the modern authed v1 endpoint 404s. Try v3 first to save a round-trip
// per image; v1 stays as a fallback for when Reddit upgrades.
const candidates = [
{
url: `${this.creds.homeserver}/_matrix/media/v3/download/${serverName}/${mediaId}`,
authed: false,
},
{
url: `${this.creds.homeserver}/_matrix/client/v1/media/download/${serverName}/${mediaId}`,
authed: true,
},
];
for (const { url, authed } of candidates) {
const headers: Record<string, string> = {};
if (authed) headers['Authorization'] = `Bearer ${this.creds.accessToken}`;
const maxAttempts = 10;
for (let attempt = 1; attempt <= maxAttempts; attempt++) {
await this.acquireRequestSlot();
const res = await this.timedFetch(url, { headers });
if (res === null) {
if (attempt < maxAttempts) {
const backoff =
Math.min(2000, 200 * 2 ** (attempt - 1)) + Math.floor(Math.random() * 150);
await new Promise((r) => setTimeout(r, backoff));
continue;
}
throw new Error(`Matrix media download ${url} timed out (exceeded ${maxAttempts} attempts)`);
}
if (res.status === 401 || res.status === 403) {
throw new MatrixAuthError(`Matrix media download → ${res.status} (token likely expired)`);
}
if (res.status === 429) {
const body = (await res.json().catch(() => ({}))) as { retry_after_ms?: number };
const waitMs = (body.retry_after_ms ?? 500) + Math.floor(Math.random() * 100);
if (attempt < maxAttempts) {
await new Promise((r) => setTimeout(r, waitMs));
continue;
}
throw new Error(`Matrix media download → 429 after ${maxAttempts} retries`);
}
if (res.status === 404 || res.status === 405) break; // try next endpoint
if (!res.ok) {
const text = await res.text().catch(() => '');
throw new Error(`Matrix media download ${url} → ${res.status}: ${text.slice(0, 200)}`);
}
const arrayBuffer = await res.arrayBuffer();
return {
bytes: Buffer.from(arrayBuffer),
contentType: res.headers.get('content-type') ?? 'application/octet-stream',
};
}
}
return null;
}
}
function parseMxc(mxc: string): { serverName: string; mediaId: string } | null {
const m = mxc.match(/^mxc:\/\/([^/]+)\/(.+)$/);
if (!m) return null;
return { serverName: m[1], mediaId: m[2] };
}
/** "@t2_abc123:reddit.com" → "t2_abc123" */
function stripMxid(mxid: string): string {
const m = mxid.match(/^@([^:]+):/);
return m ? m[1] : mxid;
}
src/services/chat/matrix/http-media-downloader.ts
+44
View File
@@ -0,0 +1,44 @@
import type { Logger } from '../../../utils/logger.js';
import type { MediaDownloader, MediaDownloadResult } from '../../media.js';
import type { MatrixClient } from './client.js';
import { fetchWithTimeout } from '../../../utils/fetch-with-timeout.js';
/**
* `MediaDownloader` for the existing `MediaService` that knows how to fetch
* both raw HTTPS URLs (legacy chat snapshots, external links) and Matrix
* `mxc://` URIs. The latter go through `MatrixClient.downloadMedia` so they
* honour the same auth + rate-limiter machinery as the rest of the chat
* code path; nothing in the chat flow needs Playwright any more.
*/
export class MatrixMediaDownloader implements MediaDownloader {
constructor(
private matrix: MatrixClient,
private logger: Logger,
) {}
async fetch(url: string): Promise<MediaDownloadResult | null> {
try {
if (url.startsWith('mxc://')) {
const result = await this.matrix.downloadMedia(url);
if (!result) return null;
return { buffer: result.bytes, contentType: result.contentType };
}
// Reddit chat sometimes embeds direct https://matrix.redditspace.com
// URLs (e.g. /thumbnail/.../blur_*) — fall through to plain fetch.
const res = await fetchWithTimeout(url, { timeoutMs: 30_000 });
if (!res.ok) {
this.logger.warn(`Media download failed (${res.status}): ${url}`);
return null;
}
const ab = await res.arrayBuffer();
return {
buffer: Buffer.from(ab),
contentType: res.headers.get('content-type') ?? null,
};
} catch (error) {
this.logger.warn(`Media download error for ${url}:`, error);
return null;
}
}
}
src/services/chat/matrix/token-extractor.ts
+131
View File
@@ -0,0 +1,131 @@
import fs from 'fs';
import path from 'path';
import type { BrowserContext } from 'playwright';
import type { Logger } from '../../../utils/logger.js';
import type { MatrixCredentials } from './types.js';
import { fetchWithTimeout } from '../../../utils/fetch-with-timeout.js';
const DEFAULT_CACHE_PATH = path.resolve(process.cwd(), 'data/matrix-credentials.json');
/** Reddit chat homeserver. The bearer token is host-scoped to this origin. */
const MATRIX_HOST_RE = /matrix\.(redditspace|reddit)\.com$/i;
export interface TokenExtractorOptions {
/** Where to persist the captured token. */
cachePath?: string;
/** Max ms to wait for a Matrix-bound request to appear. */
timeoutMs?: number;
}
export async function loadCachedCredentials(
cachePath = DEFAULT_CACHE_PATH,
): Promise<MatrixCredentials | null> {
try {
if (!fs.existsSync(cachePath)) return null;
const raw = fs.readFileSync(cachePath, 'utf-8');
const parsed = JSON.parse(raw) as MatrixCredentials;
if (!parsed.accessToken || !parsed.homeserver || !parsed.userId) return null;
return parsed;
} catch {
return null;
}
}
export function saveCredentials(
creds: MatrixCredentials,
cachePath = DEFAULT_CACHE_PATH,
): void {
const dir = path.dirname(cachePath);
if (!fs.existsSync(dir)) fs.mkdirSync(dir, { recursive: true });
fs.writeFileSync(cachePath, JSON.stringify(creds, null, 2), 'utf-8');
}
/**
* Sniffs the Matrix bearer token by opening chat.reddit.com in the supplied
* (already-logged-in) browser context and watching outbound requests for any
* call to the Matrix homeserver. The first request carrying an
* `Authorization: Bearer ...` header is captured.
*
* Requires the context to already hold a valid Reddit session — call this
* after BrowserClient.login() or with a persistent context that has cookies.
*/
export async function extractMatrixCredentials(
context: BrowserContext,
logger: Logger,
options: TokenExtractorOptions = {},
): Promise<MatrixCredentials> {
const timeoutMs = options.timeoutMs ?? 30_000;
const cachePath = options.cachePath ?? DEFAULT_CACHE_PATH;
const page = await context.newPage();
try {
let resolveCreds: (c: MatrixCredentials) => void;
let rejectCreds: (err: Error) => void;
const credsPromise = new Promise<MatrixCredentials>((resolve, reject) => {
resolveCreds = resolve;
rejectCreds = reject;
});
let done = false;
const onRequest = (request: import('playwright').Request) => {
if (done) return;
try {
const url = new URL(request.url());
if (!MATRIX_HOST_RE.test(url.hostname)) return;
const headers = request.headers();
const auth = headers['authorization'] || headers['Authorization'];
if (!auth || !auth.toLowerCase().startsWith('bearer ')) return;
const token = auth.slice('bearer '.length).trim();
if (!token) return;
const homeserver = `${url.protocol}//${url.host}`;
done = true;
logger.info(`[matrix] captured bearer token for ${homeserver}`);
resolveCreds({
homeserver,
accessToken: token,
userId: '',
capturedAt: Date.now(),
});
} catch {
// swallow malformed URLs
}
};
page.on('request', onRequest);
logger.info('[matrix] loading chat.reddit.com to capture access token...');
await page.goto('https://chat.reddit.com/', {
waitUntil: 'domcontentloaded',
timeout: 30_000,
});
const timeout = new Promise<MatrixCredentials>((_, reject) =>
setTimeout(
() => reject(new Error(`Matrix token capture timed out after ${timeoutMs}ms`)),
timeoutMs,
),
);
const captured = await Promise.race([credsPromise, timeout]);
// Resolve the user_id by calling /whoami so the caller knows who we are.
const whoamiRes = await fetchWithTimeout(
`${captured.homeserver}/_matrix/client/v3/account/whoami`,
{
headers: { Authorization: `Bearer ${captured.accessToken}` },
timeoutMs: 15_000,
},
);
if (whoamiRes.ok) {
const body = (await whoamiRes.json()) as { user_id?: string };
if (body.user_id) captured.userId = body.user_id;
} else {
logger.warn(`[matrix] whoami returned ${whoamiRes.status}; userId left blank`);
}
saveCredentials(captured, cachePath);
logger.info(`[matrix] credentials saved to ${cachePath} (user=${captured.userId || 'unknown'})`);
return captured;
} finally {
await page.close().catch(() => undefined);
}
}
src/services/chat/matrix/types.ts
+85
View File
@@ -0,0 +1,85 @@
/**
* Minimal subset of Matrix Client-Server API response shapes that the
* Reddit chat client actually uses. Reddit runs its chat on a Matrix
* homeserver (typically matrix.redditspace.com), so the same API works.
*
* Spec: https://spec.matrix.org/v1.11/client-server-api/
*/
export interface MatrixCredentials {
/** Homeserver base URL, e.g. "https://matrix.redditspace.com" */
homeserver: string;
/** Bearer access token */
accessToken: string;
/** Matrix user ID, e.g. "@t2_abc123:reddit.com" */
userId: string;
/** When the credentials were captured (ms). Used for staleness checks. */
capturedAt: number;
}
export interface MatrixEventContent {
msgtype?: string;
body?: string;
/** mxc:// URI for image/file messages */
url?: string;
info?: {
mimetype?: string;
w?: number;
h?: number;
size?: number;
};
[k: string]: unknown;
}
export interface MatrixTimelineEvent {
event_id: string;
type: string;
sender: string;
origin_server_ts: number;
content: MatrixEventContent;
unsigned?: { age?: number };
}
export interface MatrixRoomTimeline {
events: MatrixTimelineEvent[];
limited?: boolean;
prev_batch?: string;
}
export interface MatrixRoomState {
events: MatrixTimelineEvent[];
}
export interface MatrixJoinedRoom {
timeline?: MatrixRoomTimeline;
state?: MatrixRoomState;
unread_notifications?: {
notification_count?: number;
highlight_count?: number;
};
summary?: {
'm.heroes'?: string[];
'm.joined_member_count'?: number;
'm.invited_member_count'?: number;
};
}
export interface MatrixSyncResponse {
next_batch: string;
rooms?: {
join?: Record<string, MatrixJoinedRoom>;
leave?: Record<string, unknown>;
invite?: Record<string, unknown>;
};
}
export interface MatrixMessagesResponse {
chunk: MatrixTimelineEvent[];
start: string;
end?: string;
}
export interface MatrixWhoamiResponse {
user_id: string;
device_id?: string;
}
src/services/chat/service.ts
+856
View File
@@ -0,0 +1,856 @@
import type { Logger } from '../../utils/logger.js';
import type { SuccessfulMessagesRepository } from '../../database/repositories.js';
import type { MediaService } from '../media.js';
import { ChatCache } from './cache.js';
import { MatrixClient, MatrixAuthError, MatrixRateLimitError } from './matrix/client.js';
import type { MatrixJoinedRoom, MatrixTimelineEvent } from './matrix/types.js';
import type { ActiveChat, ChatMessage, ScrapedChatMessage } from './types.js';
import { fetchWithTimeout } from '../../utils/fetch-with-timeout.js';
const ACTIVE_WINDOW_MS = 96 * 60 * 60 * 1000;
const ROOM_URL_PREFIX = 'https://chat.reddit.com/room/';
export interface TokenRefresher {
/**
* Re-extract a fresh Matrix bearer token (e.g. by re-running the browser
* sniff against the persistent session) and apply it to the underlying
* client. Resolves when the client is ready to be retried.
*/
refresh(): Promise<void>;
}
/**
* Optional Reddit-API hook for resolving t2 IDs to real usernames. Without
* it, conversations show up as `t2_abc123` (stable but ugly). With it, the
* dashboard sees real names.
*/
export interface UsernameResolver {
/** Map t2 IDs (with or without `t2_` prefix) to Reddit usernames. */
resolveT2Ids(t2Ids: string[]): Promise<Record<string, string>>;
}
interface BuildChatOpts {
knownUsernames: Map<string, string>;
}
/**
* Public chat API. Internally a thin wrapper around `MatrixClient`:
*
* - `/sync` (delta cursor) drives `getActiveChats` / `refreshTopChats` —
* cheap, just-changed rooms only.
* - `/joined_rooms` + per-room `/messages` paginate every historical
* conversation when `fetchAllMessages` is called.
* - Sends use `PUT .../send/m.room.message/<txn>` directly — no UI.
*
* The public surface (`getActiveChats`, `sendChatMessage`, etc.) is
* preserved from the old Playwright implementation so callers (web routes,
* MediaSyncService) don't change.
*/
export type SyncPhase =
| 'starting'
| 'seeding'
| 'listing-rooms'
| 'resolving-usernames'
| 'loading-details'
| 'finalizing'
| 'done';
export interface SyncProgress {
kind: 'full' | 'delta';
phase: SyncPhase;
/** Number of items completed in the current phase (e.g. rooms processed). */
current: number;
/** Total items in the current phase, if known. */
total: number;
/** Optional label for the in-flight item (e.g. current room name). */
label?: string;
/** Wall-clock ms when the sync started — for elapsed-time display. */
startedAt: number;
}
export interface SyncStatus {
/** True if a sync is currently running. */
syncing: boolean;
/** Kind of sync currently running, if any. */
syncingKind: 'full' | 'delta' | null;
/** Wall-clock ms of the last successful full backfill, or null. */
lastFullSyncAt: number | null;
/** Wall-clock ms of the last delta sync, or null. */
lastDeltaSyncAt: number | null;
/** Has the bot ever completed a full sync on this machine? */
hasEverFullSynced: boolean;
/** Number of chats currently cached. */
cachedChats: number;
/** Live progress for the in-flight sync, if any. */
progress: SyncProgress | null;
}
/** `null` signals "sync ended" (clear progress UI). */
export type SyncProgressListener = (progress: SyncProgress | null) => void;
/** Fired when a chat we sent into first receives a reply from the other party. */
export type NewReplyListener = (info: {
username: string;
replySnippet: string;
}) => void;
/**
* Set when Reddit's homeserver tells us to stop creating new DMs for a while
* (e.g. the "Wow, you've created a lot of chats. Let's take a break." soft
* throttle, errcode `rate.invitation_limit_mature_account`).
*/
export interface InvitationCooldown {
/** Wall-clock ms when sends may resume. */
until: number;
/** Reddit-side error code, e.g. `rate.invitation_limit_mature_account`. */
redditCode: string | null;
/** Human-readable explanation suitable for the dashboard. */
reason: string;
/** Wall-clock ms when the cooldown was set. */
setAt: number;
}
export class ChatService {
private cache: ChatCache;
private queue: Promise<unknown> = Promise.resolve();
private readonly t2ToUsername = new Map<string, string>();
private syncingKind: 'full' | 'delta' | null = null;
private progress: SyncProgress | null = null;
private progressListener: SyncProgressListener | null = null;
private newReplyListener: NewReplyListener | null = null;
private invitationCooldown: InvitationCooldown | null = null;
constructor(
private matrix: MatrixClient,
private logger: Logger,
private successfulMessagesRepo?: SuccessfulMessagesRepository,
private mediaService?: MediaService,
private tokenRefresher?: TokenRefresher,
private usernameResolver?: UsernameResolver,
) {
this.cache = new ChatCache(logger);
}
// ------------ scheduling + auth helpers ------------
/** Serialize work so concurrent callers don't clobber cache state. */
private enqueue<T>(task: () => Promise<T>): Promise<T> {
const result = this.queue.then(() => task());
this.queue = result.catch(() => undefined);
return result;
}
/**
* Run a Matrix call; if it 401/403s, re-extract the bearer via the
* configured `TokenRefresher` and retry once. Single retry only — if the
* refresh itself fails, we propagate.
*/
private async withAuth<T>(fn: () => Promise<T>): Promise<T> {
try {
return await fn();
} catch (err) {
if (err instanceof MatrixAuthError && this.tokenRefresher) {
this.logger.warn('[chat] Matrix token rejected — refreshing via browser session');
await this.tokenRefresher.refresh();
return await fn();
}
throw err;
}
}
// ------------ public API ------------
async loadCache(): Promise<void> {
await this.cache.load();
for (const chat of this.cache.get()) {
const t2 = this.extractT2(chat);
if (t2 && chat.username && !chat.username.startsWith('t2_')) {
this.t2ToUsername.set(t2, chat.username);
}
}
}
async saveCacheExternal(): Promise<void> {
await this.cache.save();
}
async getCachedConversations(): Promise<ActiveChat[]> {
if (this.cache.get().length > 0) return this.cache.get();
await this.cache.load();
return this.cache.get();
}
/**
* Returns chats with activity in the last 96h. Uses the persisted
* `next_batch` cursor — first call is essentially the initial sync,
* subsequent calls are delta-only.
*/
async getActiveChats(): Promise<ActiveChat[]> {
return this.enqueue(async () => {
if (this.cache.isFresh()) return this.cache.get();
this.beginSync('delta');
try {
this.updateProgress({ phase: 'starting', current: 0, total: 0 });
const merged = await this.syncOnce();
const cutoff = Date.now() - ACTIVE_WINDOW_MS;
const active = merged.filter((c) => c.lastMessageTime >= cutoff);
this.cache.set(active);
await this.cache.save(merged);
this.logger.info(`[chat] sync → ${merged.length} known rooms, ${active.length} active`);
this.updateProgress({ phase: 'done', current: merged.length, total: merged.length });
return active;
} finally {
this.endSync();
}
});
}
/**
* Incremental refresh — calls /sync once and applies the deltas to the
* cache. The `limit` arg is preserved for API compatibility but Matrix
* already returns only rooms with new events, so it's a no-op.
*/
async refreshTopChats(_limit = 5): Promise<void> {
await this.enqueue(async () => {
this.beginSync('delta');
try {
this.updateProgress({ phase: 'starting', current: 0, total: 0 });
await this.syncOnce();
this.logger.info('[chat] incremental sync complete');
this.updateProgress({
phase: 'done',
current: this.cache.get().length,
total: this.cache.get().length,
});
} catch (error) {
this.logger.error('[chat] refreshTopChats failed:', error);
} finally {
this.endSync();
}
});
}
/**
* Full backfill — walks every room via /joined_rooms and paginates each
* one's full history. Slow but rare; intended for "rebuild the cache from
* scratch" or initial provisioning.
*/
async fetchAllMessages(): Promise<ActiveChat[]> {
return this.enqueue(async () => {
this.beginSync('full');
try {
this.logger.info('[chat] full backfill starting (via /joined_rooms)...');
const t0 = Date.now();
this.updateProgress({ phase: 'seeding', current: 0, total: 0 });
// Seed display-name hints from one /sync (heroes / m.room.name).
const seed = await this.withAuth(() => this.matrix.sync({ timeoutMs: 0 }));
this.cache.setNextBatch(seed.next_batch);
const seedJoined = seed.rooms?.join ?? {};
this.updateProgress({ phase: 'listing-rooms', current: 0, total: 0 });
const joinedIds = await this.withAuth(() => this.matrix.getJoinedRooms());
this.logger.info(`[chat] backfill: ${joinedIds.length} joined rooms`);
this.updateProgress({
phase: 'resolving-usernames',
current: 0,
total: joinedIds.length,
});
const knownUsernames = await this.resolveUsernamesForRooms(joinedIds);
const chats: ActiveChat[] = [];
for (const roomId of joinedIds) {
try {
const room = seedJoined[roomId];
const chat = this.roomToActiveChat(roomId, room, { knownUsernames });
chats.push(chat);
} catch (err) {
this.logger.warn(`[chat] backfill: failed to seed ${roomId}: ${(err as Error).message}`);
}
}
let done = 0;
for (const chat of chats) {
this.updateProgress({
phase: 'loading-details',
current: done,
total: chats.length,
label: chat.roomName || chat.username,
});
try {
await this.loadDetails(chat);
} catch (err) {
this.logger.error(`[chat] backfill: failed details for ${chat.roomName}`, err);
}
done++;
}
this.updateProgress({
phase: 'finalizing',
current: chats.length,
total: chats.length,
});
chats.sort((a, b) => b.lastMessageTime - a.lastMessageTime);
this.cache.set(chats);
this.cache.markFullSync();
await this.cache.save(chats);
this.logger.info(
`[chat] full backfill done in ${Math.round((Date.now() - t0) / 1000)}s — ${chats.length} chats`,
);
this.updateProgress({ phase: 'done', current: chats.length, total: chats.length });
return chats;
} catch (error) {
this.logger.error('[chat] fetchAllMessages failed:', error);
return this.cache.get();
} finally {
this.endSync();
}
});
}
/**
* Decide whether to run a full backfill or just a delta sync based on
* whether the bot has ever completed a full sync on this machine.
* Called once on bootstrap.
*/
async runStartupSync(): Promise<{ kind: 'full' | 'delta'; chats: number }> {
const { lastFullSyncAt } = this.cache.getMarkers();
if (!lastFullSyncAt || this.cache.get().length === 0) {
this.logger.info('[chat] startup: no prior full sync recorded → running full backfill');
const chats = await this.fetchAllMessages();
return { kind: 'full', chats: chats.length };
}
this.logger.info(
`[chat] startup: full sync was ${new Date(lastFullSyncAt).toISOString()} → running delta`,
);
const chats = await this.getActiveChats();
return { kind: 'delta', chats: chats.length };
}
/** Lightweight status for the dashboard's sync controls. */
getSyncStatus(): SyncStatus {
const markers = this.cache.getMarkers();
return {
syncing: this.syncingKind !== null,
syncingKind: this.syncingKind,
lastFullSyncAt: markers.lastFullSyncAt,
lastDeltaSyncAt: markers.lastDeltaSyncAt,
hasEverFullSynced: !!markers.lastFullSyncAt,
cachedChats: this.cache.get().length,
progress: this.progress,
};
}
/** Subscribe to live sync-progress updates. Single listener; replaces any prior one. */
setProgressListener(listener: SyncProgressListener | null): void {
this.progressListener = listener;
}
/** Subscribe to new-reply notifications (fires once per chat the first time it gets a reply). */
setNewReplyListener(listener: NewReplyListener | null): void {
this.newReplyListener = listener;
}
private beginSync(kind: 'full' | 'delta'): void {
this.syncingKind = kind;
this.progress = {
kind,
phase: 'starting',
current: 0,
total: 0,
startedAt: Date.now(),
};
this.emitProgress();
}
private updateProgress(patch: Partial<Omit<SyncProgress, 'kind' | 'startedAt'>>): void {
if (!this.progress) return;
this.progress = { ...this.progress, ...patch };
this.emitProgress();
}
private endSync(): void {
this.syncingKind = null;
this.progress = null;
if (this.progressListener) {
try {
this.progressListener(null);
} catch (err) {
this.logger.debug('progress listener error:', err);
}
}
}
private emitProgress(): void {
if (!this.progress || !this.progressListener) return;
try {
this.progressListener(this.progress);
} catch (err) {
this.logger.debug('progress listener error:', err);
}
}
/**
* Load full message history (+ media) for a single chat. Mutates
* `chat.messages`.
*/
async loadConversationDetails(chat: ActiveChat): Promise<void> {
return this.loadDetails(chat);
}
/**
* Send a reply into an existing room. Accepts either a full
* `https://chat.reddit.com/room/<roomId>` URL (web UI compatibility) or
* a raw Matrix room ID like `!abc:reddit.com`.
*/
async sendChatMessage(chatUrlOrRoomId: string, message: string): Promise<boolean> {
const active = this.getInvitationCooldown();
if (active) {
const secondsLeft = Math.ceil((active.until - Date.now()) / 1000);
this.logger.warn(
`[chat] skipping sendChatMessage — send cooldown active for ${secondsLeft}s (${active.redditCode ?? 'rate-limited'})`,
);
return false;
}
return this.enqueue(async () => {
try {
const roomId = ChatService.extractRoomId(chatUrlOrRoomId);
if (!roomId) {
this.logger.error(`[chat] cannot extract room ID from "${chatUrlOrRoomId}"`);
return false;
}
const eventId = await this.withAuth(() => this.matrix.sendTextMessage(roomId, message));
this.logger.info(`[chat] sent message to ${roomId} (event_id=${eventId})`);
const chat = this.cache.get().find((c) => c.roomId === roomId);
if (chat) {
chat.lastMessageSnippet = message.slice(0, 200);
chat.lastMessageTime = Date.now();
chat.isFromUs = true;
}
return true;
} catch (error) {
if (error instanceof MatrixRateLimitError) {
this.setInvitationCooldown(error);
return false;
}
this.logger.error('[chat] sendChatMessage failed:', error);
return false;
}
});
}
/**
* Start a brand-new DM with the given Reddit user and send the opening
* line. Returns the new chat (or null on failure).
*/
async startConversation(username: string, message: string): Promise<ActiveChat | null> {
const active = this.getInvitationCooldown();
if (active) {
const secondsLeft = Math.ceil((active.until - Date.now()) / 1000);
this.logger.warn(
`[chat] skipping startConversation(${username}) — invitation cooldown active for ${secondsLeft}s (${active.redditCode ?? 'rate-limited'})`,
);
return null;
}
return this.enqueue(async () => {
try {
const t2Id = await this.resolveUsernameToT2(username);
if (!t2Id) {
this.logger.error(`[chat] cannot start conversation: unknown user "${username}"`);
return null;
}
const mxid = MatrixClient.t2ToMxid(t2Id);
const { roomId, eventId } = await this.withAuth(() =>
this.matrix.sendInitialDirectMessage(mxid, message),
);
this.logger.info(
`[chat] started conversation with ${username} (${t2Id}) → room=${roomId} event=${eventId}`,
);
this.t2ToUsername.set(t2Id.replace(/^t2_/, ''), username);
const chat: ActiveChat = {
username,
roomName: username,
lastMessageTime: Date.now(),
lastMessageSnippet: message.slice(0, 200),
isFromUs: true,
unread: false,
url: ROOM_URL_PREFIX + encodeURIComponent(roomId),
roomId,
messages: [
{ sender: 'Me', text: message, timestamp: Date.now(), isSelf: true },
],
originMessage: message,
};
const existing = this.cache.get();
const merged = [chat, ...existing.filter((c) => c.roomId !== roomId)];
this.cache.set(merged);
await this.cache.save(merged);
return chat;
} catch (error) {
if (error instanceof MatrixRateLimitError) {
this.setInvitationCooldown(error);
return null;
}
this.logger.error('[chat] startConversation failed:', error);
return null;
}
});
}
/**
* Matrix sends are authoritative — the event_id in the PUT response
* confirms delivery. Kept on the public API for compatibility with the
* old Playwright sender; always returns true.
*/
async verifyMessageSent(_recipient: string, _messageText: string): Promise<boolean> {
return true;
}
/**
* Legacy debug hook. There's no DOM to inspect on the Matrix backend;
* returns a small status payload so any UI still wired to this endpoint
* stops 500-ing.
*/
async debugInspectDom(_chatUrl: string): Promise<unknown> {
return {
backend: 'matrix',
note: 'DOM inspection is not available on the Matrix backend — chat data is sourced from the Reddit Matrix homeserver API directly.',
};
}
/** Public so bootstrap can swap in a fresh client after token refresh. */
setMatrixClient(client: MatrixClient): void {
this.matrix = client;
}
/**
* Active invitation cooldown (i.e. Reddit's "you've created a lot of chats,
* take a break" throttle), or null if we're clear to send. Auto-expires
* by wall clock — callers don't have to clear it manually.
*/
getInvitationCooldown(): InvitationCooldown | null {
if (!this.invitationCooldown) return null;
if (Date.now() >= this.invitationCooldown.until) {
this.invitationCooldown = null;
return null;
}
return this.invitationCooldown;
}
/** Record a cooldown after a `MatrixRateLimitError`. Test seam + reuse. */
setInvitationCooldown(error: MatrixRateLimitError): void {
const now = Date.now();
// Add a small safety margin so we don't probe again the very second
// the homeserver thinks the window expired.
const margin = 5_000;
this.invitationCooldown = {
until: now + error.retryAfterMs + margin,
redditCode: error.redditCode,
reason:
error.redditCode === 'rate.invitation_limit_mature_account'
? "Reddit is throttling new chat invitations (\"you've created a lot of chats, take a break\")."
: `Matrix rate-limited: ${error.errcode ?? '429'}`,
setAt: now,
};
const minutes = Math.ceil(error.retryAfterMs / 60_000);
this.logger.warn(
`[chat] invitation cooldown set: ${this.invitationCooldown.reason} (~${minutes}m, code=${error.redditCode ?? error.errcode ?? '429'})`,
);
}
// ------------ internals ------------
/** Single delta /sync that updates cache + nextBatch and returns the union. */
private async syncOnce(): Promise<ActiveChat[]> {
const sync = await this.withAuth(() =>
this.matrix.sync({ since: this.cache.getMarkers().nextBatch ?? undefined, timeoutMs: 0 }),
);
this.cache.setNextBatch(sync.next_batch);
const joined = sync.rooms?.join ?? {};
const existing = new Map(this.cache.get().map((c) => [c.roomId!, c] as const));
const knownUsernames = await this.resolveUsernamesForRooms(Object.keys(joined));
for (const [roomId, room] of Object.entries(joined)) {
const prev = existing.get(roomId);
const next = this.roomToActiveChat(roomId, room, { knownUsernames }, prev);
existing.set(roomId, next);
}
const merged = Array.from(existing.values()).sort(
(a, b) => b.lastMessageTime - a.lastMessageTime,
);
this.cache.set(merged);
this.cache.markDeltaSync();
await this.cache.save(merged);
return merged;
}
private async loadDetails(chat: ActiveChat): Promise<void> {
if (!chat.roomId) {
this.logger.warn(`[chat] no roomId for ${chat.roomName}, skipping details`);
return;
}
const events: MatrixTimelineEvent[] = [];
let from: string | undefined = undefined;
for (let i = 0; i < 50; i++) {
const res = await this.withAuth(() =>
this.matrix.getRoomMessages(chat.roomId!, { from, limit: 100, dir: 'b' }),
);
events.push(...res.chunk);
if (!res.end || res.end === from || res.chunk.length === 0) break;
from = res.end;
}
// Pagination yields newest-first; flip to chronological.
const messages = events.reverse().filter((e) => e.type === 'm.room.message');
const scraped: ScrapedChatMessage[] = messages.map((e) => {
const body = (e.content.body as string | undefined) ?? '';
const mxc = typeof e.content.url === 'string' ? (e.content.url as string) : null;
return {
sender: e.sender === this.matrix.userId ? 'Me' : this.shortenSender(e.sender),
text: body || (mxc ? '[Image]' : ''),
timestamp: e.origin_server_ts,
isSelf: e.sender === this.matrix.userId,
mediaUrls: mxc ? [mxc] : [],
};
});
// Backfill the recipient from message senders BEFORE attachMedia stamps
// it into media records. /sync's heroes + state.events miss inactive
// rooms; the senders of historical messages don't.
if (!chat.username || chat.username === 'unknown' || chat.username.startsWith('@')) {
const firstOther = messages.find((e) => e.sender !== this.matrix.userId);
if (firstOther) {
const t2 = firstOther.sender.match(/^@(t2_[^:]+):/)?.[1];
if (t2) {
const resolved =
this.t2ToUsername.get(t2.replace(/^t2_/, '')) ?? t2;
chat.username = resolved;
if (!chat.roomName || chat.roomName === 'unknown') chat.roomName = resolved;
}
}
}
chat.messages = this.mediaService
? await this.mediaService.attachMedia(chat, scraped)
: scraped.map(({ mediaUrls: _ignored, ...m }) => m as ChatMessage);
if (chat.messages.length === 0) {
this.logger.debug(`[chat] no messages in ${chat.roomName}`);
return;
}
chat.lastMessageSnippet = chat.messages[chat.messages.length - 1].text;
chat.lastMessageTime = chat.messages[chat.messages.length - 1].timestamp;
const firstSelf = chat.messages.find((m) => m.isSelf);
if (firstSelf) chat.originMessage = firstSelf.text;
const hasReply = chat.messages.some((m) => !m.isSelf);
if (firstSelf && hasReply && this.successfulMessagesRepo) {
if (!this.successfulMessagesRepo.existsByMessage(firstSelf.text)) {
const firstReply = chat.messages.find((m) => !m.isSelf);
try {
this.successfulMessagesRepo.insert({
sentMessageId: null,
originalMessage: firstSelf.text,
imageAnalysis: null,
postTitle: `Chat with ${chat.username}`,
responseSnippet: firstReply?.text.slice(0, 200) || null,
subreddit: null,
});
this.logger.info(`[chat] stored successful chat example from ${chat.username}`);
try {
this.newReplyListener?.({
username: chat.username,
replySnippet: firstReply?.text.slice(0, 200) ?? '',
});
} catch (err) {
this.logger.warn('[chat] newReplyListener threw:', err);
}
} catch (err) {
this.logger.warn('[chat] failed to store successful chat example:', err);
}
}
}
}
/**
* Project a Matrix room (from /sync) into an ActiveChat. If `prev` is
* supplied, fields it carries (messages, pendingReply, originMessage)
* are preserved.
*/
private roomToActiveChat(
roomId: string,
room: MatrixJoinedRoom | undefined,
opts: BuildChatOpts,
prev?: ActiveChat,
): ActiveChat {
const timeline = room?.timeline?.events ?? [];
const lastMessage = [...timeline].reverse().find((e) => e.type === 'm.room.message');
// Prefer summary.m.heroes (cheap), but fall back to scanning the room's
// member-join state events — Reddit often omits heroes for older rooms
// even though the recipient is in state.events as an m.room.member join.
const heroes = (room?.summary?.['m.heroes'] ?? []).filter((h) => h !== this.matrix.userId);
const memberJoins: string[] = [];
for (const ev of room?.state?.events ?? []) {
if (ev.type !== 'm.room.member') continue;
const stateKey = (ev as { state_key?: string }).state_key;
const membership = (ev.content as { membership?: string }).membership;
if (membership === 'join' && stateKey && stateKey !== this.matrix.userId) {
memberJoins.push(stateKey);
}
}
const candidateMxids = [...new Set([...heroes, ...memberJoins])];
const otherT2 = candidateMxids
.map((h) => h.match(/^@(t2_[^:]+):/)?.[1])
.filter((s): s is string => !!s);
let username = prev?.username ?? null;
if (!username || username.startsWith('t2_') || username.startsWith('@')) {
const resolved = otherT2
.map((id) => opts.knownUsernames.get(id.replace(/^t2_/, '')))
.find((x): x is string => !!x);
if (resolved) username = resolved;
else if (otherT2[0]) username = otherT2[0];
}
if (!username) username = prev?.username ?? 'unknown';
const nameEvent = room?.state?.events?.find((e) => e.type === 'm.room.name');
let roomName = prev?.roomName ?? username;
if (nameEvent && typeof nameEvent.content.name === 'string' && nameEvent.content.name) {
roomName = nameEvent.content.name as string;
} else if (!prev) {
roomName = username;
}
let lastMessageTime = prev?.lastMessageTime ?? 0;
let lastMessageSnippet = prev?.lastMessageSnippet ?? '';
let isFromUs = prev?.isFromUs ?? false;
if (lastMessage) {
lastMessageTime = lastMessage.origin_server_ts;
const body = (lastMessage.content.body as string | undefined) ?? '';
lastMessageSnippet = body || (lastMessage.content.url ? '[Image]' : '');
isFromUs = lastMessage.sender === this.matrix.userId;
}
const unread = (room?.unread_notifications?.notification_count ?? 0) > 0;
return {
...(prev ?? {}),
username,
roomName,
lastMessageTime,
lastMessageSnippet,
isFromUs,
unread,
url: ROOM_URL_PREFIX + encodeURIComponent(roomId),
roomId,
};
}
/** Returns the t2 id portion of a stored `t2_xxx` username. */
private extractT2(chat: ActiveChat): string | null {
const m = chat.username?.match(/^t2_(.+)$/);
return m ? m[1] : null;
}
/** Convert `@t2_abc:reddit.com` → `t2_abc`. */
private shortenSender(mxid: string): string {
const m = mxid.match(/^@([^:]+):/);
return m ? m[1] : mxid;
}
/**
* Best-effort batch resolution of t2 IDs to usernames. Uses the injected
* `UsernameResolver` (if any); failures are logged and ignored, with the
* t2 ID staying as the fallback display.
*/
private async resolveUsernamesForRooms(_roomIds: string[]): Promise<Map<string, string>> {
// We don't have the recipient mxids until rooms are projected, so this
// method returns the in-memory cache and lets `roomToActiveChat` fall
// back to the t2 ID. Resolution happens lazily via `enrichUsernames()`
// which the caller can invoke after building the chat list.
return new Map(this.t2ToUsername);
}
/**
* After a sync, look up real usernames for any t2-only entries in the
* cache. Best-effort; safe to await or fire-and-forget.
*/
async enrichUsernames(): Promise<void> {
if (!this.usernameResolver) return;
const needed: string[] = [];
for (const chat of this.cache.get()) {
const t2 = this.extractT2(chat);
if (t2 && !this.t2ToUsername.has(t2)) needed.push(`t2_${t2}`);
}
if (needed.length === 0) return;
try {
const got = await this.usernameResolver.resolveT2Ids(needed);
let updated = 0;
for (const [t2, name] of Object.entries(got)) {
const bare = t2.replace(/^t2_/, '');
this.t2ToUsername.set(bare, name);
for (const chat of this.cache.get()) {
if (chat.username === `t2_${bare}`) {
chat.username = name;
chat.roomName = chat.roomName?.startsWith('t2_') ? name : chat.roomName;
updated++;
}
}
}
if (updated > 0) {
this.logger.info(`[chat] enriched ${updated} chats with usernames`);
await this.cache.save();
}
} catch (err) {
this.logger.debug('[chat] enrichUsernames failed:', err);
}
}
/**
* Look up the t2 ID for a Reddit username via the standard about.json
* endpoint. Returns `t2_xxx` or null.
*/
private async resolveUsernameToT2(username: string): Promise<string | null> {
try {
const res = await fetchWithTimeout(
`https://www.reddit.com/user/${encodeURIComponent(username)}/about.json`,
{ headers: { 'User-Agent': 'reddit-bot-chat/1.0' }, timeoutMs: 15_000 },
);
if (!res.ok) {
this.logger.warn(`[chat] /user/${username}/about.json → ${res.status}`);
return null;
}
const json = (await res.json()) as { data?: { id?: string } };
const id = json.data?.id;
if (!id) return null;
return id.startsWith('t2_') ? id : `t2_${id}`;
} catch (err) {
this.logger.error(`[chat] failed to resolve username "${username}":`, err);
return null;
}
}
/** Parse a chat URL like `…/room/!abc%3Areddit.com` → `!abc:reddit.com`. */
static extractRoomId(chatUrlOrRoomId: string): string | null {
if (chatUrlOrRoomId.startsWith('!')) return chatUrlOrRoomId;
const m = chatUrlOrRoomId.match(/\/room\/([^/?#]+)/);
if (!m) return null;
try {
return decodeURIComponent(m[1]);
} catch {
return null;
}
}
}
src/services/chat/types.ts
+51
View File
@@ -0,0 +1,51 @@
export interface ChatMessage {
sender: string;
text: string;
timestamp: number;
isSelf: boolean;
media?: ChatMedia[];
}
export interface ChatMedia {
originalUrl: string;
localPath: string;
fileName: string;
}
export interface ActiveChat {
username: string;
roomName: string;
lastMessageTime: number;
lastMessageSnippet: string;
isFromUs: boolean;
unread: boolean;
url?: string;
roomId?: string;
messages?: ChatMessage[];
originMessage?: string;
}
export interface ChatCacheFile {
version: number;
savedAt: number;
chats: ActiveChat[];
/** Matrix `next_batch` cursor — present after any successful sync. */
nextBatch?: string | null;
/** Wall-clock timestamp of the last successful full backfill. Null/missing
* → the bot has never done a full sync on this machine, so startup
* should run one. */
lastFullSyncAt?: number | null;
/** Timestamp of the most recent delta sync. */
lastDeltaSyncAt?: number | null;
}
export type ScrapedChatMessage = Omit<ChatMessage, 'media'> & { mediaUrls: string[] };
export interface ScrapedRoom {
href: string;
ariaLabel: string | null;
roomName: string | null;
lastMessage: string | null;
timeText: string | null;
unread: boolean;
}
src/services/media.ts
+4 -3
View File
@@ -57,9 +57,10 @@ export class MediaService {
constructor(
private downloader: MediaDownloader,
private logger: Logger
private logger: Logger,
mediaRoot?: string,
) {
this.mediaRoot = path.join(process.cwd(), 'data', 'media');
this.mediaRoot = mediaRoot ?? path.join(process.cwd(), 'data', 'media');
this.indexPath = path.join(this.mediaRoot, 'index.json');
}
@@ -84,7 +85,7 @@ export class MediaService {
async getMediaItems(): Promise<MediaRecord[]> {
await this.loadIndex();
return (this.index?.items || []).slice().sort((a, b) => b.createdAt - a.createdAt);
return (this.index?.items || []).slice().sort((a, b) => b.timestamp - a.timestamp);
}
private async downloadMessageMedia(chat: ActiveChat, message: ScrapedChatMessage): Promise<ChatMedia[]> {
src/services/monitor.ts
+75 -13
View File
@@ -2,25 +2,32 @@ import type { RedditApiClient, RedditPost } from '../reddit/api.js';
import type { PostsRepository, MessagesRepository } from '../database/repositories.js';
import type { Config } from '../config/index.js';
import type { Logger } from '../utils/logger.js';
import { parseGenderTag } from '../utils/gender-tag.js';
export class MonitorService {
// Flips on each call so that when both pools have candidates the bot
// alternates between image and chat instead of always favouring whichever
// type the random shuffle put first.
private chatGoesFirst = false;
constructor(
private redditApi: RedditApiClient,
private postsRepo: PostsRepository,
private messagesRepo: MessagesRepository,
private config: Config,
private logger: Logger
private logger: Logger,
private delayBetweenFetchesMs: number = 1000,
) { }
async getUnprocessedPosts(): Promise<RedditPost[]> {
// Create a shuffled copy of subreddits to check
const subreddits = [...this.config.reddit.subreddits];
this.shuffleArray(subreddits);
const subreddits = this.buildCheckOrder();
const targetGenders = this.config.reddit.targetGenders;
for (const subreddit of subreddits) {
try {
this.logger.debug(`Checking r/${subreddit} for new posts...`);
const posts = await this.redditApi.getNewPosts(subreddit);
this.logger.debug(`Checking r/${subreddit.name} (${subreddit.type}) for new posts...`);
const posts = await this.redditApi.getNewPosts(subreddit.name, 25, subreddit.type);
// Filter for valid, unprocessed posts
const validPosts: RedditPost[] = [];
@@ -34,11 +41,28 @@ export class MonitorService {
continue;
}
// Chat subs almost always have a self-declared gender tag in the
// title/body (e.g. [25F], M4F). Drop posts that don't match the
// configured targetGenders here so they never enter the processor.
if (subreddit.type === 'chat' && targetGenders.length > 0) {
const tag = parseGenderTag(`${post.title}\n${post.selftext ?? ''}`);
if (!tag) {
this.logger.debug(`Skipping chat post ${post.id} - no gender tag in title/body`);
continue;
}
if (!targetGenders.includes(tag)) {
this.logger.debug(
`Skipping chat post ${post.id} - gender tag "${tag}" not in target list [${targetGenders.join(', ')}]`,
);
continue;
}
}
validPosts.push(post);
}
if (validPosts.length > 0) {
this.logger.info(`Found ${validPosts.length} new posts in r/${subreddit}`);
this.logger.info(`Found ${validPosts.length} new posts in r/${subreddit.name}`);
// Sort by creation time (newest first)
validPosts.sort((a, b) => b.created_utc - a.created_utc);
@@ -49,21 +73,59 @@ export class MonitorService {
return [validPosts[0]];
}
} catch (error) {
this.logger.error(`Failed to fetch posts from r/${subreddit}:`, error);
this.logger.error(`Failed to fetch posts from r/${subreddit.name}:`, error);
}
// Small delay between subreddit fetches to be nice to Reddit
await this.sleep(1000);
if (this.delayBetweenFetchesMs > 0) {
await this.sleep(this.delayBetweenFetchesMs);
}
}
return [];
}
private shuffleArray<T>(array: T[]): void {
for (let i = array.length - 1; i > 0; i--) {
const j = Math.floor(Math.random() * (i + 1));
[array[i], array[j]] = [array[j], array[i]];
/**
* Build an interleaved check order that balances image and chat subs.
*
* Each pool is shuffled independently, then zipped together so a chat sub
* is checked roughly every other position even when there are far more
* image subs. `chatGoesFirst` flips per call so neither type is
* permanently advantaged when both pools have fresh posts.
*/
private buildCheckOrder(): Config['reddit']['subreddits'] {
const images = this.weightedShuffle(
this.config.reddit.subreddits.filter((s) => s.type === 'image'),
);
const chats = this.weightedShuffle(
this.config.reddit.subreddits.filter((s) => s.type === 'chat'),
);
const [a, b] = this.chatGoesFirst ? [chats, images] : [images, chats];
this.chatGoesFirst = !this.chatGoesFirst;
const out: Config['reddit']['subreddits'] = [];
const max = Math.max(a.length, b.length);
for (let i = 0; i < max; i++) {
if (i < a.length) out.push(a[i]);
if (i < b.length) out.push(b[i]);
}
return out;
}
// Weighted random ordering (Efraimidis–Spirakis): key = random^(1/weight),
// sorted descending. Heavier-weighted subs are more likely to land earlier
// each cycle, while every sub still has a non-zero chance of any position.
private weightedShuffle<T extends { weight?: number }>(items: T[]): T[] {
return items
.map((item) => {
const w = item.weight && item.weight > 0 ? item.weight : 1;
const r = Math.random();
const key = r === 0 ? -Infinity : Math.log(r) / w;
return { item, key };
})
.sort((a, b) => b.key - a.key)
.map(({ item }) => item);
}
private sleep(ms: number): Promise<void> {
src/services/processor.ts
+100 -13
View File
@@ -3,7 +3,8 @@ import type { OpenRouterClient, DetectedGender } from '../ai/openrouter.js';
import type { Logger } from '../utils/logger.js';
import type { Config } from '../config/index.js';
import type { StrategySelector, SelectedStrategy } from './strategy-selector.js';
import { downloadImage, bufferToBase64, isValidImageMimeType } from '../utils/image.js';
import { downloadImage, bufferToBase64, isValidImageMimeType, type DownloadedImage } from '../utils/image.js';
import { parseGenderTag } from '../utils/gender-tag.js';
export interface ProcessedPost {
post: RedditPost;
@@ -13,16 +14,33 @@ export interface ProcessedPost {
strategyId: string | null;
}
export interface SkipResult {
skipReason: string;
}
export type ProcessResult = ProcessedPost | SkipResult;
export function isSkip(result: ProcessResult): result is SkipResult {
return 'skipReason' in result;
}
export type ImageFetcher = (url: string, userAgent: string) => Promise<DownloadedImage>;
export class ProcessorService {
private successfulExamples: string[] = [];
private strategySelector: StrategySelector | null = null;
private chatStrategySelector: StrategySelector | null = null;
private fetchImage: ImageFetcher;
constructor(
private redditApi: RedditApiClient,
private aiClient: OpenRouterClient,
private config: Config,
private logger: Logger
) {}
private logger: Logger,
fetchImage?: ImageFetcher,
) {
this.fetchImage = fetchImage ?? downloadImage;
}
setStrategySelector(selector: StrategySelector): void {
this.strategySelector = selector;
@@ -31,6 +49,18 @@ export class ProcessorService {
}
}
setChatStrategySelector(selector: StrategySelector): void {
this.chatStrategySelector = selector;
if (selector.hasStrategies()) {
this.logger.info(`Chat strategy A/B testing enabled with ${this.config.chatStrategies.variants.length} variants`);
}
}
private getSubredditType(name: string): 'image' | 'chat' {
const entry = this.config.reddit.subreddits.find((s) => s.name.toLowerCase() === name.toLowerCase());
return entry?.type ?? 'image';
}
setSuccessfulExamples(examples: string[]): void {
this.successfulExamples = examples;
if (examples.length > 0) {
@@ -38,12 +68,16 @@ export class ProcessorService {
}
}
async processPost(post: RedditPost): Promise<ProcessedPost | null> {
async processPost(post: RedditPost): Promise<ProcessResult> {
if (this.getSubredditType(post.subreddit) === 'chat') {
return this.processChatPost(post);
}
const imageUrl = this.redditApi.getImageUrl(post);
if (!imageUrl) {
this.logger.warn(`No image URL found for post ${post.id}`);
return null;
return { skipReason: 'no image URL on post' };
}
this.logger.info(`Processing post: "${post.title.slice(0, 50)}..." by u/${post.author}`);
@@ -51,22 +85,23 @@ export class ProcessorService {
// Download image
let imageData;
try {
imageData = await downloadImage(imageUrl, this.config.reddit.userAgent);
imageData = await this.fetchImage(imageUrl, this.config.reddit.userAgent);
} catch (error) {
const msg = error instanceof Error ? error.message : String(error);
this.logger.error(`Failed to download image from ${imageUrl}:`, error);
return null;
return { skipReason: `image download failed: ${msg}` };
}
// Validate image type
if (!isValidImageMimeType(imageData.mimeType)) {
this.logger.warn(`Invalid image type ${imageData.mimeType} for post ${post.id}`);
return null;
return { skipReason: `unsupported image type (${imageData.mimeType})` };
}
// Check image size (skip very small images like thumbnails)
if (imageData.buffer.length < 5000) {
this.logger.warn(`Image too small (${imageData.buffer.length} bytes) for post ${post.id}`);
return null;
return { skipReason: `image too small (${imageData.buffer.length} bytes)` };
}
// Convert to base64
@@ -81,7 +116,9 @@ export class ProcessorService {
this.logger.info(
`Skipping post ${post.id} - detected gender "${detectedGender}" not in target list [${targetGenders.join(', ')}]`
);
return null;
return {
skipReason: `detected gender "${detectedGender}" not in target list [${targetGenders.join(', ')}]`,
};
}
this.logger.debug(`Gender check passed: ${detectedGender}`);
@@ -111,14 +148,16 @@ export class ProcessorService {
} catch (error) {
if (error instanceof Error && error.message.startsWith('AI_REFUSAL')) {
this.logger.warn(`AI refused to analyze post ${post.id} by u/${post.author} - skipping`);
return null;
return { skipReason: 'AI refused to analyze image' };
}
throw error;
}
if (!message || message.length < 10) {
this.logger.warn(`Generated message too short or empty for post ${post.id}`);
return null;
this.logger.warn(
`Generated message too short or empty for post ${post.id} (${message?.length ?? 0} chars): ${JSON.stringify(message?.slice(0, 200) ?? '')}`,
);
return { skipReason: `generated message too short (${message?.length ?? 0} chars)` };
}
return {
@@ -129,4 +168,52 @@ export class ProcessorService {
strategyId: selectedStrategy?.id || null,
};
}
private async processChatPost(post: RedditPost): Promise<ProcessResult> {
this.logger.info(`Processing chat post: "${post.title.slice(0, 50)}..." by u/${post.author}`);
// Gender pre-filtering for chat subs happens in MonitorService — by the
// time a chat post reaches the processor it's already passed the filter.
// We still parse the tag here so it can be threaded into the opener prompt.
const haystack = `${post.title}\n${post.selftext ?? ''}`;
const detectedGender = parseGenderTag(haystack);
let selectedStrategy: SelectedStrategy | null = null;
if (this.chatStrategySelector) {
selectedStrategy = this.chatStrategySelector.selectStrategy();
if (selectedStrategy) {
this.logger.info(`Using chat strategy "${selectedStrategy.id}" (${selectedStrategy.reason})`);
}
}
let message: string;
try {
message = await this.aiClient.generateChatOpener(
post.title,
post.selftext ?? '',
post.author,
detectedGender ?? 'unknown',
this.successfulExamples,
selectedStrategy?.promptOverride,
);
} catch (error) {
this.logger.error(`Chat opener generation failed for post ${post.id}:`, error);
throw error;
}
if (!message || message.length < 10) {
this.logger.warn(
`Generated chat opener too short or empty for post ${post.id} (${message?.length ?? 0} chars): ${JSON.stringify(message?.slice(0, 200) ?? '')}`,
);
return { skipReason: `generated chat opener too short (${message?.length ?? 0} chars)` };
}
return {
post,
imageAnalysis: `[chat] ${post.title}${post.selftext ? ` — ${post.selftext.slice(0, 200)}` : ''}`,
detectedGender: detectedGender ?? 'unknown',
generatedMessage: message,
strategyId: selectedStrategy?.id || null,
};
}
}
src/services/strategy-selector.ts
+6 -4
View File
@@ -16,11 +16,13 @@ export class StrategySelector {
constructor(
private config: Config,
private messagesRepo: MessagesRepository,
private logger: Logger
private logger: Logger,
section: 'strategies' | 'chatStrategies' = 'strategies',
) {
this.variants = config.strategies.variants;
this.explorationRate = config.strategies.explorationRate;
this.minSamples = config.strategies.minSamples;
const cfg = config[section];
this.variants = cfg.variants;
this.explorationRate = cfg.explorationRate;
this.minSamples = cfg.minSamples;
}
selectStrategy(): SelectedStrategy | null {
src/utils/fetch-with-timeout.ts
+18
View File
@@ -0,0 +1,18 @@
/**
* `fetch` with a hard client-side timeout. Node's global fetch otherwise
* inherits no timeout — a stalled connection will hang the caller forever,
* which has wedged the main loop more than once (incidents on 2026-05-20).
*
* On timeout, throws an error whose `.name === 'TimeoutError'`. Callers that
* want to differentiate timeout-vs-other failures can branch on that.
*/
export async function fetchWithTimeout(
input: string | URL | Request,
init: RequestInit & { timeoutMs?: number } = {},
): Promise<Response> {
const { timeoutMs = 30_000, ...rest } = init;
return fetch(input, {
...rest,
signal: AbortSignal.timeout(timeoutMs),
});
}
src/utils/gender-tag.ts
+39
View File
@@ -0,0 +1,39 @@
import type { DetectedGender } from '../ai/openrouter.js';
/**
* Best-effort extraction of the poster's self-declared gender from a Reddit
* chat-sub post (title + selftext). Used by chat-only subreddits like
* /r/SextingRush where posts almost always carry a tag such as `[25F]`,
* `(M, 30)`, `F4M`, `25/f`, etc.
*
* Returns null when no tag is found — callers should skip those posts rather
* than guess.
*/
export function parseGenderTag(text: string): DetectedGender | null {
if (!text) return null;
const haystack = text.toLowerCase();
// X4Y tags (e.g. "f4m", "m4f", "f4mf"). The first letter is the poster.
const x4y = haystack.match(/\b([mf])4[mfa+]+\b/);
if (x4y) return x4y[1] === 'f' ? 'female' : 'male';
// Bracketed / parenthesised / bare tags with an age + gender letter in
// either order, optionally separated by a slash, space, or comma.
// Matches: [25F], (F25), 25f, f25, 25/f, f/25, [F, 25], (m 30), 25-f.
const ageGender =
haystack.match(/\[\s*(\d{2})\s*[\s,/\-]*\s*([mf])\s*\]/) ||
haystack.match(/\[\s*([mf])\s*[\s,/\-]*\s*(\d{2})\s*\]/) ||
haystack.match(/\(\s*(\d{2})\s*[\s,/\-]*\s*([mf])\s*\)/) ||
haystack.match(/\(\s*([mf])\s*[\s,/\-]*\s*(\d{2})\s*\)/) ||
haystack.match(/(?:^|[^a-z])(\d{2})\s*[\/\-]\s*([mf])(?:[^a-z]|$)/) ||
haystack.match(/(?:^|[^a-z])([mf])\s*[\/\-]\s*(\d{2})(?:[^a-z]|$)/) ||
haystack.match(/(?:^|[^a-z])(\d{2})([mf])(?:[^a-z]|$)/) ||
haystack.match(/(?:^|[^a-z])([mf])(\d{2})(?:[^a-z]|$)/);
if (ageGender) {
const letter = /[mf]/.test(ageGender[1]) ? ageGender[1] : ageGender[2];
return letter === 'f' ? 'female' : 'male';
}
return null;
}
src/utils/logger.ts
+17 -1
View File
@@ -2,6 +2,22 @@ import winston from 'winston';
export type Logger = winston.Logger;
function safeStringify(value: unknown): string {
const seen = new WeakSet<object>();
try {
return JSON.stringify(value, (_key, val) => {
if (typeof val === 'object' && val !== null) {
if (seen.has(val)) return '[Circular]';
seen.add(val);
}
if (typeof val === 'bigint') return val.toString();
return val;
});
} catch (err) {
return `[Unserializable: ${(err as Error).message}]`;
}
}
export function createLogger(): Logger {
return winston.createLogger({
level: process.env.LOG_LEVEL || 'info',
@@ -15,7 +31,7 @@ export function createLogger(): Logger {
format: winston.format.combine(
winston.format.colorize(),
winston.format.printf(({ level, message, timestamp, ...meta }) => {
const metaStr = Object.keys(meta).length ? ` ${JSON.stringify(meta)}` : '';
const metaStr = Object.keys(meta).length ? ` ${safeStringify(meta)}` : '';
return `${timestamp} [${level}]: ${message}${metaStr}`;
})
),
src/utils/rate-limiter.ts
+9 -4
View File
@@ -2,17 +2,22 @@ import PQueue from 'p-queue';
import type { Config } from '../config/index.js';
import type { Logger } from './logger.js';
const DEFAULT_HOUR_WINDOW_MS = 3600000;
export class RateLimiter {
private queue: PQueue;
private lastMessageTime: number = 0;
private messagesThisHour: number = 0;
private hourStartTime: number = Date.now();
private readonly hourWindowMs: number;
constructor(
private config: Config,
private logger: Logger
private logger: Logger,
hourWindowMs: number = DEFAULT_HOUR_WINDOW_MS,
) {
this.queue = new PQueue({ concurrency: 1 });
this.hourWindowMs = hourWindowMs;
}
async throttle<T>(fn: () => Promise<T>, shouldCount: (result: T) => boolean = () => true): Promise<T> {
@@ -20,7 +25,7 @@ export class RateLimiter {
const now = Date.now();
// Reset hourly counter if needed
if (now - this.hourStartTime > 3600000) {
if (now - this.hourStartTime > this.hourWindowMs) {
this.messagesThisHour = 0;
this.hourStartTime = now;
this.logger.debug('Hourly rate limit counter reset');
@@ -28,7 +33,7 @@ export class RateLimiter {
// Check hourly limit
if (this.messagesThisHour >= this.config.rateLimit.maxMessagesPerHour) {
const waitTime = 3600000 - (now - this.hourStartTime);
const waitTime = this.hourWindowMs - (now - this.hourStartTime);
this.logger.info(`Hourly limit reached, waiting ${Math.round(waitTime / 60000)} minutes`);
await this.sleep(waitTime);
this.messagesThisHour = 0;
@@ -94,7 +99,7 @@ export class RateLimiter {
}
// Calculate when the hourly counter resets
const hourlyResetAt = this.hourStartTime + 3600000;
const hourlyResetAt = this.hourStartTime + this.hourWindowMs;
// Check if hourly limit is reached
const isHourlyLimitReached = this.messagesThisHour >= maxMessages;
src/utils/retry.ts
+18 -5
View File
@@ -7,16 +7,29 @@ export async function withRetry<T>(
retries?: number;
logger?: Logger;
operationName?: string;
/** Initial retry delay in ms. Set to 0 in tests to skip the backoff wait. */
minTimeout?: number;
maxTimeout?: number;
factor?: number;
randomize?: boolean;
} = {}
): Promise<T> {
const { retries = 3, logger, operationName = 'operation' } = options;
const {
retries = 3,
logger,
operationName = 'operation',
minTimeout = 5000,
maxTimeout = 60000,
factor = 2,
randomize = true,
} = options;
return pRetry(fn, {
retries,
minTimeout: 5000,
maxTimeout: 60000,
factor: 2,
randomize: true,
minTimeout,
maxTimeout,
factor,
randomize,
onFailedAttempt: (context) => {
const errorMessage = context.error instanceof Error ? context.error.message : String(context.error);
logger?.warn(
src/web/activity-log.ts
+44
View File
@@ -0,0 +1,44 @@
import type { ActivityItem } from './types.js';
import type { ActivityLogRepository } from '../database/repositories.js';
const MAX_ITEMS = 200;
/**
* Bounded log of bot actions surfaced on the dashboard.
* Hydrates from SQLite on construction and persists every push,
* so the feed survives restarts.
*/
export class ActivityLog {
private items: ActivityItem[] = [];
constructor(private repo: ActivityLogRepository) {
const rows = repo.getRecent(MAX_ITEMS);
this.items = rows
.map((r) => ({
timestamp: r.timestamp,
type: r.type as ActivityItem['type'],
message: r.message,
recipient: r.recipient ?? undefined,
imageUrl: r.imageUrl ?? undefined,
}))
.reverse();
}
push(type: ActivityItem['type'], message: string, recipient?: string, imageUrl?: string): void {
const item: ActivityItem = { timestamp: Date.now(), type, message, recipient, imageUrl };
this.items.push(item);
if (this.items.length > MAX_ITEMS) {
this.items = this.items.slice(-MAX_ITEMS);
}
this.repo.insert(item);
this.repo.prune(MAX_ITEMS);
}
recent(limit: number): ActivityItem[] {
return this.items.slice(-limit).reverse();
}
totalProcessed(): number {
return this.repo.countSentSkipped();
}
}
src/web/context.ts
+58
View File
@@ -0,0 +1,58 @@
import type { Logger } from '../utils/logger.js';
import type { Config } from '../config/index.js';
import type {
PostsRepository,
MessagesRepository,
SuccessfulMessagesRepository,
} from '../database/repositories.js';
import type { ChatService } from '../services/chat.js';
import type { MediaService } from '../services/media.js';
import type { MediaSyncService } from '../services/media-sync.js';
import type { ResponseCheckerService } from '../services/response-checker.js';
import type { RateLimiter } from '../utils/rate-limiter.js';
import type { OpenRouterClient } from '../ai/openrouter.js';
import type { StrategySelector } from '../services/strategy-selector.js';
import type { BrowserClient } from '../browser/client.js';
import type { ActivityLog } from './activity-log.js';
import type { LogBroadcaster } from './log-broadcaster.js';
import type { RuntimeConfig } from './types.js';
/** Bag of everything the route modules might need. */
export interface WebContext {
logger: Logger;
config: Config;
runtimeConfig: RuntimeConfig;
activity: ActivityLog;
broadcaster: LogBroadcaster;
repos: {
posts: PostsRepository;
messages: MessagesRepository;
successful: SuccessfulMessagesRepository;
};
services: {
chat?: ChatService;
media?: MediaService;
mediaSync?: MediaSyncService;
responseChecker?: ResponseCheckerService;
rateLimiter?: RateLimiter;
ai?: OpenRouterClient;
strategySelector?: StrategySelector;
browser?: BrowserClient;
};
/** Bot pause/resume toggle read by the main loop. */
getBotRunning(): boolean;
setBotRunning(running: boolean): void;
/** Mirrored count for the dashboard's `messagesThisHour` widget. */
getMessagesThisHour(): number;
/**
* Callback invoked when the dashboard mutates `runtimeConfig`.
* Wrapped in a getter so callers see the latest value — it can be installed
* after the WebContext is constructed.
*/
getOnConfigChange(): ((config: RuntimeConfig) => void) | null;
}
src/web/log-broadcaster.ts
+80
View File
@@ -0,0 +1,80 @@
import type { Request, Response } from 'express';
import type { LogEntry } from './types.js';
const MAX_LOGS = 500;
const INITIAL_LOGS_ON_CONNECT = 50;
/**
* Holds the recent realtime log buffer and fans out new entries to every
* connected SSE client. Owns the `/api/logs/stream` and `/api/logs` handlers.
*/
export class LogBroadcaster {
private entries: LogEntry[] = [];
private clients = new Set<Response>();
push(level: LogEntry['level'], message: string, data?: unknown): void {
const entry: LogEntry = { timestamp: Date.now(), level, message, data };
this.entries.push(entry);
if (this.entries.length > MAX_LOGS) {
this.entries = this.entries.slice(-MAX_LOGS);
}
this.fanout(JSON.stringify({ type: 'log', ...entry }));
}
/**
* Broadcast a non-log event (e.g. sync progress) to all connected clients.
* Unlike `push`, these events are not buffered or replayed on reconnect —
* they reflect transient state.
*/
broadcastEvent(type: string, payload: Record<string, unknown>): void {
this.fanout(JSON.stringify({ type, ...payload }));
}
private fanout(payload: string): void {
for (const client of this.clients) {
try {
client.write(`data: ${payload}\n\n`);
} catch {
this.clients.delete(client);
}
}
}
/** Express handler for SSE stream. */
stream = (req: Request, res: Response): void => {
res.setHeader('Content-Type', 'text/event-stream');
res.setHeader('Cache-Control', 'no-cache');
res.setHeader('Connection', 'keep-alive');
res.setHeader('X-Accel-Buffering', 'no');
res.write(`data: ${JSON.stringify({ type: 'connected', timestamp: Date.now() })}\n\n`);
this.clients.add(res);
for (const log of this.entries.slice(-INITIAL_LOGS_ON_CONNECT)) {
res.write(`data: ${JSON.stringify({ type: 'log', ...log })}\n\n`);
}
req.on('close', () => {
this.clients.delete(res);
});
};
/** Express handler for `/api/logs?limit=N`. */
recent = (req: Request, res: Response): void => {
const limit = parseInt(req.query.limit as string) || 100;
res.json(this.entries.slice(-limit));
};
closeAll(): void {
for (const client of this.clients) {
try {
client.end();
} catch {
// ignore
}
}
this.clients.clear();
}
}
src/web/routes/analytics.ts
+33
View File
@@ -0,0 +1,33 @@
import { Router } from 'express';
import type { WebContext } from '../context.js';
export function createAnalyticsRouter(ctx: WebContext): Router {
const router = Router();
router.get('/timers', (_req, res) => {
if (!ctx.services.rateLimiter) {
res.status(503).json({ error: 'Rate limiter not initialized' });
return;
}
res.json(ctx.services.rateLimiter.getTimers());
});
router.get('/analytics/by-subreddit', (_req, res) => {
res.json(ctx.repos.messages.getSubredditStats());
});
router.get('/analytics/by-strategy', (_req, res) => {
if (ctx.services.strategySelector) {
res.json(ctx.services.strategySelector.getStats());
} else {
res.json(ctx.repos.messages.getStrategyStats());
}
});
router.get('/analytics/trend', (req, res) => {
const days = parseInt(req.query.days as string) || 30;
res.json(ctx.repos.messages.getResponseRateTrend(days));
});
return router;
}
src/web/routes/chats.ts
+104
View File
@@ -0,0 +1,104 @@
import { Router } from 'express';
import type { WebContext } from '../context.js';
const ACTIVE_WINDOW_MS = 48 * 60 * 60 * 1000;
export function createChatsRouter(ctx: WebContext): Router {
const router = Router();
router.get('/chats', async (_req, res) => {
if (!ctx.services.chat) {
res.json([]);
return;
}
try {
const all = await ctx.services.chat.getCachedConversations();
const cutoff = Date.now() - ACTIVE_WINDOW_MS;
res.json(all.filter((c) => c.lastMessageTime >= cutoff));
} catch (error) {
ctx.logger.error('Failed to fetch chats:', error);
res.json([]);
}
});
/** Status snapshot for the dashboard's sync controls. */
router.get('/chats/sync', (_req, res) => {
if (!ctx.services.chat) {
res.status(503).json({ error: 'Chat service not initialized' });
return;
}
res.json(ctx.services.chat.getSyncStatus());
});
/**
* Delta sync — pulls only events since the last cursor. Fast (~seconds).
* Use this for routine refreshes.
*/
router.post('/chats/sync/delta', async (_req, res) => {
if (!ctx.services.chat) {
res.status(500).json({ error: 'Chat service not initialized' });
return;
}
const status = ctx.services.chat.getSyncStatus();
if (status.syncing) {
res.json({ success: false, reason: `Already running ${status.syncingKind} sync` });
return;
}
try {
ctx.broadcaster.push('info', 'Delta sync triggered from dashboard');
// Run asynchronously — status endpoint reports progress.
ctx.services.chat
.refreshTopChats()
.catch((err) => ctx.logger.error('Delta sync failed:', err));
res.json({ success: true, kind: 'delta' });
} catch (error) {
ctx.logger.error('Delta sync trigger failed:', error);
res.status(500).json({ error: String(error) });
}
});
/**
* Full sync — walks every joined room and rebuilds the cache from
* scratch. Slow (~1–2 min for hundreds of rooms). Use after a long
* absence, recovery from corruption, or to pick up rooms that don't
* appear in /sync.
*/
router.post('/chats/sync/full', async (_req, res) => {
if (!ctx.services.chat) {
res.status(500).json({ error: 'Chat service not initialized' });
return;
}
const status = ctx.services.chat.getSyncStatus();
if (status.syncing) {
res.json({ success: false, reason: `Already running ${status.syncingKind} sync` });
return;
}
try {
ctx.broadcaster.push('info', 'Full sync triggered from dashboard');
ctx.services.chat
.fetchAllMessages()
.catch((err) => ctx.logger.error('Full sync failed:', err));
res.json({ success: true, kind: 'full' });
} catch (error) {
ctx.logger.error('Full sync trigger failed:', error);
res.status(500).json({ error: String(error) });
}
});
/** Back-compat alias for the old "fetch-all" endpoint. */
router.post('/chats/fetch-all', async (_req, res) => {
if (!ctx.services.chat) {
res.status(500).json({ error: 'Chat service not initialized' });
return;
}
try {
const chats = await ctx.services.chat.fetchAllMessages();
res.json({ success: true, count: chats.length });
} catch (error) {
ctx.logger.error('Failed to fetch all chats:', error);
res.status(500).json({ error: String(error) });
}
});
return router;
}
src/web/routes/config.ts
+61
View File
@@ -0,0 +1,61 @@
import { Router } from 'express';
import type { WebContext } from '../context.js';
export function createConfigRouter(ctx: WebContext): Router {
const router = Router();
router.get('/config', (_req, res) => {
res.json({
rateLimit: ctx.runtimeConfig.rateLimit,
browser: { dryRun: ctx.runtimeConfig.browser.dryRun },
});
});
router.put('/config', (req, res) => {
try {
const updates = req.body;
if (updates.rateLimit) {
if (typeof updates.rateLimit.minDelayBetweenMessagesMs === 'number') {
ctx.runtimeConfig.rateLimit.minDelayBetweenMessagesMs = Math.max(
0,
updates.rateLimit.minDelayBetweenMessagesMs,
);
}
if (typeof updates.rateLimit.maxMessagesPerHour === 'number') {
ctx.runtimeConfig.rateLimit.maxMessagesPerHour = Math.max(
1,
updates.rateLimit.maxMessagesPerHour,
);
}
if (typeof updates.rateLimit.jitterMs === 'number') {
ctx.runtimeConfig.rateLimit.jitterMs = Math.max(0, updates.rateLimit.jitterMs);
}
}
if (updates.browser) {
if (typeof updates.browser.dryRun === 'boolean') {
ctx.runtimeConfig.browser.dryRun = updates.browser.dryRun;
}
}
// Mirror into the loaded config so getStats reflects new values.
ctx.config.rateLimit.minDelayBetweenMessagesMs = ctx.runtimeConfig.rateLimit.minDelayBetweenMessagesMs;
ctx.config.rateLimit.maxMessagesPerHour = ctx.runtimeConfig.rateLimit.maxMessagesPerHour;
ctx.config.rateLimit.jitterMs = ctx.runtimeConfig.rateLimit.jitterMs;
ctx.config.browser.dryRun = ctx.runtimeConfig.browser.dryRun;
ctx.getOnConfigChange()?.(ctx.runtimeConfig);
ctx.activity.push('info', 'Config updated via dashboard');
ctx.broadcaster.push('info', 'Configuration updated via dashboard', updates);
res.json({ success: true, config: ctx.runtimeConfig });
} catch (error) {
ctx.logger.error('Failed to update config:', error);
res.status(400).json({ error: String(error) });
}
});
return router;
}
src/web/routes/control.ts
+22
View File
@@ -0,0 +1,22 @@
import { Router } from 'express';
import type { WebContext } from '../context.js';
export function createControlRouter(ctx: WebContext): Router {
const router = Router();
router.post('/start', (_req, res) => {
ctx.setBotRunning(true);
ctx.activity.push('sent', 'Bot started by user');
ctx.logger.info('Bot started via web UI');
res.json({ success: true, running: true });
});
router.post('/stop', (_req, res) => {
ctx.setBotRunning(false);
ctx.activity.push('skipped', 'Bot stopped by user');
ctx.logger.info('Bot stopped via web UI');
res.json({ success: true, running: false });
});
return router;
}
src/web/routes/conversations.ts
+242
View File
@@ -0,0 +1,242 @@
import { Router } from 'express';
import type { ActiveChat } from '../../services/chat.js';
import type { WebContext } from '../context.js';
export function createConversationsRouter(ctx: WebContext): Router {
const router = Router();
async function findConversation(id: string): Promise<ActiveChat | undefined> {
if (!ctx.services.chat) return undefined;
const conversations = await ctx.services.chat.getCachedConversations();
return conversations.find(
(c) => c.roomId === id || c.username === id || c.roomName === id,
);
}
router.get('/conversations', async (_req, res) => {
if (!ctx.services.chat) {
res.json([]);
return;
}
try {
res.json(await ctx.services.chat.getCachedConversations());
} catch (error) {
ctx.logger.error('Failed to fetch conversations:', error);
res.json([]);
}
});
router.get('/conversations/:id', async (req, res) => {
if (!ctx.services.chat) {
res.status(404).json({ error: 'Chat service not initialized' });
return;
}
try {
// Pull in any new activity from the homeserver before we read the cache,
// so a stale "waiting for them to reply" view picks up the reply they
// already sent. Cheap (~one /sync call) and queued behind any in-flight sync.
try {
await ctx.services.chat.refreshTopChats();
} catch (err) {
ctx.logger.warn('Pre-open delta sync failed:', err);
}
const conversation = await findConversation(decodeURIComponent(req.params.id));
if (!conversation) {
res.status(404).json({ error: 'Conversation not found' });
return;
}
// Delta sync only populates room summaries — full message history is
// fetched on demand the first time a conversation is opened, and
// re-fetched when the summary's last-activity timestamp is newer than
// anything we have cached (i.e. they've replied since we last loaded).
const cachedLastTs = conversation.messages?.length
? conversation.messages[conversation.messages.length - 1].timestamp
: 0;
const hasNewerActivity = (conversation.lastMessageTime ?? 0) > cachedLastTs;
if (!conversation.messages || conversation.messages.length === 0 || hasNewerActivity) {
try {
await ctx.services.chat.loadConversationDetails(conversation);
await ctx.services.chat.saveCacheExternal();
} catch (err) {
ctx.logger.warn(`Lazy-load failed for conversation ${conversation.roomId}:`, err);
}
}
// Attach the post that triggered this outreach so the modal can link to it.
let triggeringPost: {
id: string;
subreddit?: string;
title?: string;
url: string;
sentAt: number;
} | null = null;
try {
const sent = ctx.repos.messages.getLatestByRecipient(conversation.username);
if (sent) {
const url = sent.subreddit
? `https://www.reddit.com/r/${sent.subreddit}/comments/${sent.postId}/`
: `https://redd.it/${sent.postId}`;
triggeringPost = {
id: sent.postId,
subreddit: sent.subreddit,
title: sent.postTitle,
url,
sentAt: sent.sentAt,
};
}
} catch (err) {
ctx.logger.warn(`Failed to look up triggering post for ${conversation.username}:`, err);
}
res.json({ ...conversation, triggeringPost });
} catch (error) {
ctx.logger.error('Failed to fetch conversation:', error);
res.status(500).json({ error: String(error) });
}
});
router.post('/conversations/:id/suggest-reply', async (req, res) => {
if (!ctx.services.chat) {
res.status(500).json({ error: 'Chat service not initialized' });
return;
}
if (!ctx.services.ai) {
res.status(500).json({ error: 'AI client not initialized' });
return;
}
try {
const conversation = await findConversation(decodeURIComponent(req.params.id));
if (!conversation) {
res.status(404).json({ error: 'Conversation not found' });
return;
}
if (!conversation.messages || conversation.messages.length === 0) {
res.status(400).json({ error: 'No messages in conversation to generate a reply from' });
return;
}
if (!conversation.messages.some((m) => !m.isSelf)) {
res.status(400).json({ error: 'No reply from user yet - nothing to respond to' });
return;
}
const suggestedReply = await ctx.services.ai.generateReply(
conversation.messages.map((m) => ({ sender: m.sender, text: m.text, isSelf: m.isSelf })),
conversation.username,
);
res.json({ suggestedReply });
} catch (error) {
ctx.logger.error('Failed to generate suggested reply:', error);
res.status(500).json({ error: String(error) });
}
});
router.post('/conversations/:id/suggest-media-ask', async (req, res) => {
if (!ctx.services.chat) {
res.status(500).json({ error: 'Chat service not initialized' });
return;
}
if (!ctx.services.ai) {
res.status(500).json({ error: 'AI client not initialized' });
return;
}
try {
const conversation = await findConversation(decodeURIComponent(req.params.id));
if (!conversation) {
res.status(404).json({ error: 'Conversation not found' });
return;
}
if (!conversation.messages || conversation.messages.length === 0) {
res.status(400).json({ error: 'No messages in conversation to generate from' });
return;
}
if (!conversation.messages.some((m) => !m.isSelf)) {
res.status(400).json({ error: 'No reply from user yet - nothing to respond to' });
return;
}
const suggestedReply = await ctx.services.ai.generateMediaAsk(
conversation.messages.map((m) => ({ sender: m.sender, text: m.text, isSelf: m.isSelf })),
conversation.username,
);
res.json({ suggestedReply });
} catch (error) {
ctx.logger.error('Failed to generate media-ask message:', error);
res.status(500).json({ error: String(error) });
}
});
router.post('/conversations/:id/send-reply', async (req, res) => {
if (!ctx.services.chat) {
res.status(500).json({ error: 'Chat service not initialized' });
return;
}
try {
const { message } = req.body;
if (!message || typeof message !== 'string' || message.trim().length === 0) {
res.status(400).json({ error: 'Message is required' });
return;
}
const conversation = await findConversation(decodeURIComponent(req.params.id));
if (!conversation) {
res.status(404).json({ error: 'Conversation not found' });
return;
}
if (!conversation.url) {
res.status(400).json({ error: 'No chat URL for this conversation' });
return;
}
const success = await ctx.services.chat.sendChatMessage(conversation.url, message.trim());
if (success) {
ctx.activity.push('sent', `Sent chat reply to u/${conversation.username}`, conversation.username);
ctx.broadcaster.push('info', `Chat reply sent to u/${conversation.username}`);
res.json({ success: true });
ctx.services.chat.refreshTopChats(5).catch((err) =>
ctx.logger.error('Failed to refresh chats after send-reply:', err),
);
} else {
res.status(500).json({ error: 'Failed to send chat message' });
}
} catch (error) {
ctx.logger.error('Failed to send chat reply:', error);
res.status(500).json({ error: String(error) });
}
});
/**
* Start a brand-new DM with a Reddit user and send the opening message.
* Body: { username: string, message: string }
*/
router.post('/conversations/start', async (req, res) => {
if (!ctx.services.chat) {
res.status(500).json({ error: 'Chat service not initialized' });
return;
}
try {
const { username, message } = req.body ?? {};
if (!username || typeof username !== 'string') {
res.status(400).json({ error: 'username is required' });
return;
}
if (!message || typeof message !== 'string' || message.trim().length === 0) {
res.status(400).json({ error: 'message is required' });
return;
}
const chat = await ctx.services.chat.startConversation(username, message.trim());
if (!chat) {
res.status(500).json({ error: 'Failed to start conversation (unknown user or send failed)' });
return;
}
ctx.activity.push('sent', `Started chat with u/${chat.username}`, chat.username);
ctx.broadcaster.push('info', `Started new chat with u/${chat.username}`);
res.json({ success: true, chat });
} catch (error) {
ctx.logger.error('Failed to start conversation:', error);
res.status(500).json({ error: String(error) });
}
});
return router;
}
src/web/routes/debug.ts
+42
View File
@@ -0,0 +1,42 @@
import { Router } from 'express';
import type { WebContext } from '../context.js';
export function createDebugRouter(ctx: WebContext): Router {
const router = Router();
router.post('/debug/inspect-dom', async (req, res) => {
if (!ctx.services.chat) {
res.status(500).json({ error: 'Chat service not initialized' });
return;
}
try {
const chatUrl = req.body?.url;
if (!chatUrl) {
res.status(400).json({ error: 'url is required' });
return;
}
const result = await ctx.services.chat.debugInspectDom(chatUrl);
res.json(result);
} catch (error) {
ctx.logger.error('DOM inspection failed:', error);
res.status(500).json({ error: String(error) });
}
});
router.post('/debug/check-responses', async (_req, res) => {
if (!ctx.services.responseChecker) {
res.status(500).json({ error: 'Response checker not initialized' });
return;
}
try {
ctx.logger.info('Manual response check triggered via API');
const results = await ctx.services.responseChecker.checkForResponses();
res.json({ success: true, results });
} catch (error) {
ctx.logger.error('Manual response check failed:', error);
res.status(500).json({ error: String(error) });
}
});
return router;
}
src/web/routes/login.ts
+54
View File
@@ -0,0 +1,54 @@
import { Router } from 'express';
import type { WebContext } from '../context.js';
export function createLoginRouter(ctx: WebContext): Router {
const router = Router();
// Info the dashboard uses to decide whether to show the VNC iframe and
// which port to point it at.
router.get('/login/info', (_req, res) => {
const novncPort = parseInt(process.env.NOVNC_PORT ?? '6080', 10);
res.json({
vncEnabled: Boolean(process.env.DISPLAY),
novncPort,
passwordRequired: Boolean(process.env.VNC_PASSWORD),
});
});
router.get('/login/status', async (_req, res) => {
const browser = ctx.services.browser;
if (!browser) {
res.status(503).json({ ready: false, error: 'browser not initialised' });
return;
}
try {
const loggedIn = await browser.isSessionValid();
res.json({ ready: true, loggedIn });
} catch (error) {
ctx.logger.error('Failed to check login status:', error);
res.status(500).json({ ready: true, loggedIn: false, error: 'check failed' });
}
});
// Navigate the headed Chromium running under Xvfb to reddit.com/login so
// the human watching the noVNC stream can complete the flow.
router.post('/login/start', async (_req, res) => {
const browser = ctx.services.browser;
if (!browser) {
res.status(503).json({ ok: false, error: 'browser not initialised' });
return;
}
try {
const page = await browser.getPage();
await page.bringToFront().catch(() => undefined);
await page.goto('https://www.reddit.com/login', { waitUntil: 'domcontentloaded' });
ctx.logger.info('Opened Reddit login page for human completion via VNC');
res.json({ ok: true });
} catch (error) {
ctx.logger.error('Failed to open login page:', error);
res.status(500).json({ ok: false, error: 'navigation failed' });
}
});
return router;
}
src/web/routes/logs.ts
+9
View File
@@ -0,0 +1,9 @@
import { Router } from 'express';
import type { WebContext } from '../context.js';
export function createLogsRouter(ctx: WebContext): Router {
const router = Router();
router.get('/logs/stream', ctx.broadcaster.stream);
router.get('/logs', ctx.broadcaster.recent);
return router;
}
src/web/routes/media.ts
+88
View File
@@ -0,0 +1,88 @@
import { Router } from 'express';
import type { WebContext } from '../context.js';
export function createMediaRouter(ctx: WebContext): Router {
const router = Router();
router.get('/media', async (_req, res) => {
if (!ctx.services.media) {
res.json([]);
return;
}
try {
const items = await ctx.services.media.getMediaItems();
// Build a roomId → current chat lookup so we can override stale
// stored usernames (e.g. items captured before username enrichment ran)
// and surface the recipient even for rooms that were originally
// ingested as `unknown`.
const liveChats = ctx.services.chat
? await ctx.services.chat.getCachedConversations()
: [];
const byRoom = new Map(liveChats.map((c) => [c.roomId ?? '', c] as const));
res.json(items.map((item) => {
const live = item.chatRoomId ? byRoom.get(item.chatRoomId) : undefined;
const liveName = live?.username && live.username !== 'unknown' ? live.username : null;
const username = liveName ?? item.chatUsername ?? null;
return {
...item,
chatUsername: username,
urlPath: `/media/${encodeURIComponent(item.relativePath)}`,
};
}));
} catch (error) {
ctx.logger.error('Failed to fetch media index:', error);
res.status(500).json({ error: String(error) });
}
});
router.get('/media/sync', (_req, res) => {
if (!ctx.services.mediaSync) {
res.status(503).json({ error: 'Media sync service not initialized' });
return;
}
res.json(ctx.services.mediaSync.getStatus());
});
router.post('/media/sync', async (req, res) => {
if (!ctx.services.mediaSync) {
res.status(503).json({ error: 'Media sync service not initialized' });
return;
}
try {
const full = req.body?.full === true;
const started = await ctx.services.mediaSync.triggerSync(full);
if (!started) {
res.json({ success: false, reason: 'Sync already in progress' });
return;
}
ctx.broadcaster.push('info', `Media sync triggered from dashboard (${full ? 'full' : 'incremental'})`);
res.json({ success: true });
} catch (error) {
ctx.logger.error('Media sync trigger failed:', error);
res.status(500).json({ error: String(error) });
}
});
router.post('/media/sync/full', async (_req, res) => {
if (!ctx.services.mediaSync) {
res.status(503).json({ error: 'Media sync service not initialized' });
return;
}
try {
const started = await ctx.services.mediaSync.triggerSync(true);
if (!started) {
res.json({ success: false, reason: 'Sync already in progress' });
return;
}
ctx.broadcaster.push('info', 'Full media sync triggered from dashboard (all conversations)');
res.json({ success: true });
} catch (error) {
ctx.logger.error('Full media sync trigger failed:', error);
res.status(500).json({ error: String(error) });
}
});
return router;
}
src/web/routes/stats.ts
+52
View File
@@ -0,0 +1,52 @@
import { Router } from 'express';
import type { WebContext } from '../context.js';
import type { BotStats } from '../types.js';
export function createStatsRouter(ctx: WebContext): Router {
const router = Router();
router.get('/stats', (_req, res) => {
const messages = ctx.repos.messages.getRecent(1000);
const totalSent = messages.length;
const totalResponses = messages.filter((m) => m.gotResponse).length;
const cooldown = ctx.services.chat?.getInvitationCooldown() ?? null;
const stats: BotStats = {
isRunning: ctx.getBotRunning(),
dryRunMode: ctx.config.browser.dryRun,
totalProcessed: ctx.activity.totalProcessed(),
totalMessagesSent: totalSent,
totalResponses,
responseRate: totalSent > 0 ? Math.round((totalResponses / totalSent) * 100) : 0,
messagesThisHour: ctx.getMessagesThisHour(),
maxMessagesPerHour: ctx.config.rateLimit.maxMessagesPerHour,
subreddits: ctx.config.reddit.subreddits.map((s) => ({ name: s.name, type: s.type })),
targetGenders: ctx.config.reddit.targetGenders,
recentActivity: ctx.activity.recent(10),
invitationCooldown: cooldown
? { until: cooldown.until, reason: cooldown.reason, redditCode: cooldown.redditCode }
: null,
};
res.json(stats);
});
router.get('/activity', (_req, res) => {
res.json(ctx.activity.recent(50));
});
router.get('/posts/by-author/:username', (req, res) => {
res.json(ctx.repos.posts.getByAuthor(req.params.username));
});
router.get('/messages', (_req, res) => {
res.json(ctx.repos.messages.getRecent(20));
});
router.get('/successful', (_req, res) => {
res.json(ctx.repos.successful.getRecent(10));
});
return router;
}
src/web/server.ts
+96 -689
View File
@@ -1,765 +1,181 @@
import express from 'express';
import path from 'path';
import type { Server } from 'http';
import type { Response } from 'express';
import type { Logger } from '../utils/logger.js';
import type { PostsRepository, MessagesRepository, SuccessfulMessagesRepository } from '../database/repositories.js';
import type { Config } from '../config/index.js';
import type { ChatService, ActiveChat } from '../services/chat.js';
import type {
PostsRepository,
MessagesRepository,
SuccessfulMessagesRepository,
ActivityLogRepository,
} from '../database/repositories.js';
import type { ChatService } from '../services/chat.js';
import type { MediaService } from '../services/media.js';
import type { MediaSyncService } from '../services/media-sync.js';
import type { ResponseCheckerService } from '../services/response-checker.js';
import type { RateLimiter } from '../utils/rate-limiter.js';
import type { OpenRouterClient } from '../ai/openrouter.js';
import type { AutoReplyService } from '../services/auto-reply.js';
import type { StrategySelector } from '../services/strategy-selector.js';
import type { BrowserClient } from '../browser/client.js';
import { ActivityLog } from './activity-log.js';
import { LogBroadcaster } from './log-broadcaster.js';
import type { WebContext } from './context.js';
import type { ActivityItem, LogEntry, RuntimeConfig } from './types.js';
import { createStatsRouter } from './routes/stats.js';
import { createChatsRouter } from './routes/chats.js';
import { createConversationsRouter } from './routes/conversations.js';
import { createMediaRouter } from './routes/media.js';
import { createControlRouter } from './routes/control.js';
import { createDebugRouter } from './routes/debug.js';
import { createLogsRouter } from './routes/logs.js';
import { createAnalyticsRouter } from './routes/analytics.js';
import { createConfigRouter } from './routes/config.js';
import { createLoginRouter } from './routes/login.js';
export interface BotStats {
isRunning: boolean;
dryRunMode: boolean;
totalProcessed: number;
totalMessagesSent: number;
totalResponses: number;
responseRate: number;
messagesThisHour: number;
maxMessagesPerHour: number;
subreddits: string[];
targetGenders: string[];
recentActivity: ActivityItem[];
}
export interface ActivityItem {
timestamp: number;
type: 'sent' | 'skipped' | 'response' | 'error' | 'info' | 'debug';
message: string;
recipient?: string;
imageUrl?: string;
}
export interface LogEntry {
timestamp: number;
level: 'info' | 'warn' | 'error' | 'debug';
message: string;
data?: unknown;
}
export interface RuntimeConfig {
rateLimit: {
minDelayBetweenMessagesMs: number;
maxMessagesPerHour: number;
jitterMs: number;
};
browser: {
dryRun: boolean;
};
}
export type { BotStats, ActivityItem, LogEntry, RuntimeConfig } from './types.js';
/**
* Thin Express shell. Owns the server lifecycle, the activity log,
* the SSE log broadcaster, and a `WebContext` that route modules read from.
*
* Service references (chat, media, etc.) are still injected via `set*`
* methods because `index.ts` builds them after the WebServer to keep
* setup ordering identical to the pre-refactor code.
*/
export class WebServer {
private app: express.Application;
private server: Server | null = null;
private activityLog: ActivityItem[] = [];
private realtimeLog: LogEntry[] = [];
private sseClients: Set<Response> = new Set();
private botRunning = false; // Start stopped
private activity: ActivityLog;
private broadcaster = new LogBroadcaster();
private botRunning = false;
private messagesThisHour = 0;
private chatService: ChatService | null = null;
private mediaService: MediaService | null = null;
private mediaSyncService: MediaSyncService | null = null;
private responseChecker: ResponseCheckerService | null = null;
private rateLimiter: RateLimiter | null = null;
private aiClient: OpenRouterClient | null = null;
private autoReplyService: AutoReplyService | null = null;
private strategySelector: StrategySelector | null = null;
private runtimeConfig: RuntimeConfig;
private onConfigChange: ((config: RuntimeConfig) => void) | null = null;
private services: WebContext['services'] = {};
constructor(
private port: number,
private config: Config,
private postsRepo: PostsRepository,
private messagesRepo: MessagesRepository,
private successfulMessagesRepo: SuccessfulMessagesRepository,
private logger: Logger
private activityLogRepo: ActivityLogRepository,
private logger: Logger,
) {
this.app = express();
// Initialize runtime config from loaded config
this.activity = new ActivityLog(activityLogRepo);
this.runtimeConfig = {
rateLimit: {
minDelayBetweenMessagesMs: config.rateLimit.minDelayBetweenMessagesMs,
maxMessagesPerHour: config.rateLimit.maxMessagesPerHour,
jitterMs: config.rateLimit.jitterMs,
},
browser: {
dryRun: config.browser.dryRun,
},
browser: { dryRun: config.browser.dryRun },
};
this.app = express();
this.setupRoutes();
}
private context(): WebContext {
return {
logger: this.logger,
config: this.config,
runtimeConfig: this.runtimeConfig,
activity: this.activity,
broadcaster: this.broadcaster,
repos: {
posts: this.postsRepo,
messages: this.messagesRepo,
successful: this.successfulMessagesRepo,
},
services: this.services,
getBotRunning: () => this.botRunning,
setBotRunning: (running) => {
this.botRunning = running;
},
getMessagesThisHour: () => this.messagesThisHour,
getOnConfigChange: () => this.onConfigChange,
};
}
private setupRoutes(): void {
this.app.use('/media', express.static(path.join(process.cwd(), 'data', 'media')));
// Serve static files
this.app.use(express.static(path.join(process.cwd(), 'public')));
this.app.use(express.json());
// API: Get stats
this.app.get('/api/stats', (req, res) => {
const stats = this.getStats();
res.json(stats);
});
const ctx = this.context();
this.app.use('/api', createStatsRouter(ctx));
this.app.use('/api', createChatsRouter(ctx));
this.app.use('/api', createConversationsRouter(ctx));
this.app.use('/api', createMediaRouter(ctx));
this.app.use('/api', createControlRouter(ctx));
this.app.use('/api', createDebugRouter(ctx));
this.app.use('/api', createLogsRouter(ctx));
this.app.use('/api', createAnalyticsRouter(ctx));
this.app.use('/api', createConfigRouter(ctx));
this.app.use('/api', createLoginRouter(ctx));
// API: Get recent activity
this.app.get('/api/activity', (req, res) => {
res.json(this.activityLog.slice(-50).reverse());
});
// API: Look up posts by author username
this.app.get('/api/posts/by-author/:username', (req, res) => {
const posts = this.postsRepo.getByAuthor(req.params.username);
res.json(posts);
});
// API: Get recent messages
this.app.get('/api/messages', (req, res) => {
const messages = this.messagesRepo.getRecent(20);
res.json(messages);
});
// API: Get successful messages (ones that got responses)
this.app.get('/api/successful', (req, res) => {
const successful = this.successfulMessagesRepo.getRecent(10);
res.json(successful);
});
// API: Get active chats (from cache - last 48 hours)
// Does NOT fetch from Reddit - use /api/chats/fetch-all to refresh from Reddit
this.app.get('/api/chats', async (req, res) => {
if (!this.chatService) {
res.json([]);
return;
}
try {
const allChats = await this.chatService.getCachedConversations();
// Filter to last 48 hours for "active" view
const cutoffTime = Date.now() - (48 * 60 * 60 * 1000);
const activeChats = allChats.filter(chat => chat.lastMessageTime >= cutoffTime);
res.json(activeChats);
} catch (error) {
this.logger.error('Failed to fetch chats:', error);
res.json([]);
}
});
// API: Fetch all chats/messages and refresh cache
this.app.post('/api/chats/fetch-all', async (req, res) => {
if (!this.chatService) {
res.status(500).json({ error: 'Chat service not initialized' });
return;
}
try {
const chats = await this.chatService.fetchAllMessages();
res.json({ success: true, count: chats.length });
} catch (error) {
this.logger.error('Failed to fetch all chats:', error);
res.status(500).json({ error: String(error) });
}
});
// API: Get cached conversations with full message history
this.app.get('/api/conversations', async (req, res) => {
if (!this.chatService) {
res.json([]);
return;
}
try {
const conversations = await this.chatService.getCachedConversations();
res.json(conversations);
} catch (error) {
this.logger.error('Failed to fetch conversations:', error);
res.json([]);
}
});
// API: Get a single cached conversation by room ID or username
this.app.get('/api/conversations/:id', async (req, res) => {
if (!this.chatService) {
res.status(404).json({ error: 'Chat service not initialized' });
return;
}
try {
const conversations = await this.chatService.getCachedConversations();
const id = decodeURIComponent(req.params.id);
const conversation = conversations.find(
(c: ActiveChat) => c.roomId === id || c.username === id || c.roomName === id
);
if (!conversation) {
res.status(404).json({ error: 'Conversation not found' });
return;
}
res.json(conversation);
} catch (error) {
this.logger.error('Failed to fetch conversation:', error);
res.status(500).json({ error: String(error) });
}
});
// API: Suggest a reply for a conversation using AI
this.app.post('/api/conversations/:id/suggest-reply', async (req, res) => {
if (!this.chatService) {
res.status(500).json({ error: 'Chat service not initialized' });
return;
}
if (!this.aiClient) {
res.status(500).json({ error: 'AI client not initialized' });
return;
}
try {
const conversations = await this.chatService.getCachedConversations();
const id = decodeURIComponent(req.params.id);
const conversation = conversations.find(
(c: ActiveChat) => c.roomId === id || c.username === id || c.roomName === id
);
if (!conversation) {
res.status(404).json({ error: 'Conversation not found' });
return;
}
if (!conversation.messages || conversation.messages.length === 0) {
res.status(400).json({ error: 'No messages in conversation to generate a reply from' });
return;
}
const hasReply = conversation.messages.some(m => !m.isSelf);
if (!hasReply) {
res.status(400).json({ error: 'No reply from user yet - nothing to respond to' });
return;
}
const suggestedReply = await this.aiClient.generateReply(
conversation.messages.map(m => ({ sender: m.sender, text: m.text, isSelf: m.isSelf })),
conversation.username
);
res.json({ suggestedReply });
} catch (error) {
this.logger.error('Failed to generate suggested reply:', error);
res.status(500).json({ error: String(error) });
}
});
// API: Send a reply in a chat conversation
this.app.post('/api/conversations/:id/send-reply', async (req, res) => {
if (!this.chatService) {
res.status(500).json({ error: 'Chat service not initialized' });
return;
}
try {
const { message } = req.body;
if (!message || typeof message !== 'string' || message.trim().length === 0) {
res.status(400).json({ error: 'Message is required' });
return;
}
const conversations = await this.chatService.getCachedConversations();
const id = decodeURIComponent(req.params.id);
const conversation = conversations.find(
(c: ActiveChat) => c.roomId === id || c.username === id || c.roomName === id
);
if (!conversation) {
res.status(404).json({ error: 'Conversation not found' });
return;
}
if (!conversation.url) {
res.status(400).json({ error: 'No chat URL for this conversation' });
return;
}
const success = await this.chatService.sendChatMessage(conversation.url, message.trim());
if (success) {
this.logActivity('sent', `Sent chat reply to u/${conversation.username}`, conversation.username);
this.pushLog('info', `Chat reply sent to u/${conversation.username}`);
res.json({ success: true });
// Refresh chats in background so cache reflects the sent message
this.chatService.refreshTopChats(5).catch(err =>
this.logger.error('Failed to refresh chats after send-reply:', err)
);
} else {
res.status(500).json({ error: 'Failed to send chat message' });
}
} catch (error) {
this.logger.error('Failed to send chat reply:', error);
res.status(500).json({ error: String(error) });
}
});
// API: Approve a pending auto-reply
this.app.post('/api/conversations/:id/approve-reply', async (req, res) => {
if (!this.chatService) {
res.status(500).json({ error: 'Chat service not initialized' });
return;
}
try {
const conversations = await this.chatService.getCachedConversations();
const id = decodeURIComponent(req.params.id);
const conversation = conversations.find(
(c: ActiveChat) => c.roomId === id || c.username === id || c.roomName === id
);
if (!conversation) {
res.status(404).json({ error: 'Conversation not found' });
return;
}
if (conversation.replyStatus !== 'pending' || !conversation.pendingReply) {
res.status(400).json({ error: 'No pending reply to approve' });
return;
}
if (!conversation.url) {
res.status(400).json({ error: 'No chat URL for this conversation' });
return;
}
const message = conversation.pendingReply;
const success = await this.chatService.sendChatMessage(conversation.url, message);
if (success) {
conversation.replyStatus = 'sent';
conversation.pendingReply = undefined;
await this.chatService.saveCacheExternal();
this.logActivity('sent', `Approved & sent auto-reply to u/${conversation.username}`, conversation.username);
this.pushLog('info', `Auto-reply approved and sent to u/${conversation.username}`);
res.json({ success: true });
// Refresh chats in background so cache reflects the sent message
this.chatService.refreshTopChats(5).catch(err =>
this.logger.error('Failed to refresh chats after approve-reply:', err)
);
} else {
res.status(500).json({ error: 'Failed to send message' });
}
} catch (error) {
this.logger.error('Failed to approve reply:', error);
res.status(500).json({ error: String(error) });
}
});
// API: Reject a pending auto-reply
this.app.post('/api/conversations/:id/reject-reply', async (req, res) => {
if (!this.chatService) {
res.status(500).json({ error: 'Chat service not initialized' });
return;
}
try {
const conversations = await this.chatService.getCachedConversations();
const id = decodeURIComponent(req.params.id);
const conversation = conversations.find(
(c: ActiveChat) => c.roomId === id || c.username === id || c.roomName === id
);
if (!conversation) {
res.status(404).json({ error: 'Conversation not found' });
return;
}
if (conversation.replyStatus !== 'pending') {
res.status(400).json({ error: 'No pending reply to reject' });
return;
}
conversation.pendingReply = undefined;
conversation.replyStatus = undefined;
// Clear processed state so it can be re-evaluated later if needed
if (this.autoReplyService) {
this.autoReplyService.clearProcessedState(conversation.username);
}
await this.chatService.saveCacheExternal();
this.logActivity('info', `Rejected auto-reply for u/${conversation.username}`, conversation.username);
this.pushLog('info', `Auto-reply rejected for u/${conversation.username}`);
res.json({ success: true });
} catch (error) {
this.logger.error('Failed to reject reply:', error);
res.status(500).json({ error: String(error) });
}
});
// API: Dismiss upsell detection flag
this.app.post('/api/conversations/:id/dismiss-upsell', async (req, res) => {
if (!this.chatService) {
res.status(500).json({ error: 'Chat service not initialized' });
return;
}
try {
const conversations = await this.chatService.getCachedConversations();
const id = decodeURIComponent(req.params.id);
const conversation = conversations.find(
(c: ActiveChat) => c.roomId === id || c.username === id || c.roomName === id
);
if (!conversation) {
res.status(404).json({ error: 'Conversation not found' });
return;
}
conversation.replyStatus = undefined;
conversation.pendingReply = undefined;
await this.chatService.saveCacheExternal();
res.json({ success: true });
} catch (error) {
this.logger.error('Failed to dismiss upsell flag:', error);
res.status(500).json({ error: String(error) });
}
});
// API: Get downloaded chat media
this.app.get('/api/media', async (req, res) => {
if (!this.mediaService) {
res.json([]);
return;
}
try {
const items = await this.mediaService.getMediaItems();
res.json(items.map((item) => ({
...item,
urlPath: `/media/${encodeURIComponent(item.relativePath)}`,
})));
} catch (error) {
this.logger.error('Failed to fetch media index:', error);
res.status(500).json({ error: String(error) });
}
});
// API: Get media sync status
this.app.get('/api/media/sync', (req, res) => {
if (!this.mediaSyncService) {
res.status(503).json({ error: 'Media sync service not initialized' });
return;
}
res.json(this.mediaSyncService.getStatus());
});
// API: Trigger media sync from dashboard
this.app.post('/api/media/sync', async (req, res) => {
if (!this.mediaSyncService) {
res.status(503).json({ error: 'Media sync service not initialized' });
return;
}
try {
const full = req.body?.full === true;
const started = await this.mediaSyncService.triggerSync(full);
if (!started) {
res.json({ success: false, reason: 'Sync already in progress' });
return;
}
this.pushLog('info', `Media sync triggered from dashboard (${full ? 'full' : 'incremental'})`);
res.json({ success: true });
} catch (error) {
this.logger.error('Media sync trigger failed:', error);
res.status(500).json({ error: String(error) });
}
});
// API: Full media sync - walks ALL conversations
this.app.post('/api/media/sync/full', async (req, res) => {
if (!this.mediaSyncService) {
res.status(503).json({ error: 'Media sync service not initialized' });
return;
}
try {
const started = await this.mediaSyncService.triggerSync(true);
if (!started) {
res.json({ success: false, reason: 'Sync already in progress' });
return;
}
this.pushLog('info', 'Full media sync triggered from dashboard (all conversations)');
res.json({ success: true });
} catch (error) {
this.logger.error('Full media sync trigger failed:', error);
res.status(500).json({ error: String(error) });
}
});
// API: Start bot
this.app.post('/api/start', (req, res) => {
this.botRunning = true;
this.logActivity('sent', 'Bot started by user');
this.logger.info('Bot started via web UI');
res.json({ success: true, running: true });
});
// API: Stop bot
this.app.post('/api/stop', (req, res) => {
this.botRunning = false;
this.logActivity('skipped', 'Bot stopped by user');
this.logger.info('Bot stopped via web UI');
res.json({ success: true, running: false });
});
// API: Debug inspect chat DOM
this.app.post('/api/debug/inspect-dom', async (req, res) => {
if (!this.chatService) {
res.status(500).json({ error: 'Chat service not initialized' });
return;
}
try {
const chatUrl = req.body?.url;
if (!chatUrl) {
res.status(400).json({ error: 'url is required' });
return;
}
const result = await this.chatService.debugInspectDom(chatUrl);
res.json(result);
} catch (error) {
this.logger.error('DOM inspection failed:', error);
res.status(500).json({ error: String(error) });
}
});
// API: Debug check responses
this.app.post('/api/debug/check-responses', async (req, res) => {
if (!this.responseChecker) {
res.status(500).json({ error: 'Response checker not initialized' });
return;
}
try {
this.logger.info('Manual response check triggered via API');
const results = await this.responseChecker.checkForResponses();
res.json({ success: true, results });
} catch (error) {
this.logger.error('Manual response check failed:', error);
res.status(500).json({ error: String(error) });
}
});
// API: SSE endpoint for realtime logs
this.app.get('/api/logs/stream', (req, res) => {
res.setHeader('Content-Type', 'text/event-stream');
res.setHeader('Cache-Control', 'no-cache');
res.setHeader('Connection', 'keep-alive');
res.setHeader('X-Accel-Buffering', 'no'); // Disable nginx buffering
// Send initial connection message
res.write(`data: ${JSON.stringify({ type: 'connected', timestamp: Date.now() })}\n\n`);
// Add client to SSE set
this.sseClients.add(res);
// Send recent logs as initial data
this.realtimeLog.slice(-50).forEach(log => {
res.write(`data: ${JSON.stringify({ type: 'log', ...log })}\n\n`);
});
// Remove client on disconnect
req.on('close', () => {
this.sseClients.delete(res);
});
});
// API: Get recent logs (for initial load)
this.app.get('/api/logs', (req, res) => {
const limit = parseInt(req.query.limit as string) || 100;
res.json(this.realtimeLog.slice(-limit));
});
// API: Get active timers
this.app.get('/api/timers', (req, res) => {
if (!this.rateLimiter) {
res.status(503).json({ error: 'Rate limiter not initialized' });
return;
}
res.json(this.rateLimiter.getTimers());
});
// API: Analytics - response rate by subreddit
this.app.get('/api/analytics/by-subreddit', (req, res) => {
res.json(this.messagesRepo.getSubredditStats());
});
// API: Analytics - response rate by strategy
this.app.get('/api/analytics/by-strategy', (req, res) => {
if (this.strategySelector) {
res.json(this.strategySelector.getStats());
} else {
res.json(this.messagesRepo.getStrategyStats());
}
});
// API: Analytics - daily response rate trend
this.app.get('/api/analytics/trend', (req, res) => {
const days = parseInt(req.query.days as string) || 30;
res.json(this.messagesRepo.getResponseRateTrend(days));
});
// API: Get runtime config
this.app.get('/api/config', (req, res) => {
res.json({
rateLimit: this.runtimeConfig.rateLimit,
browser: {
dryRun: this.runtimeConfig.browser.dryRun,
},
});
});
// API: Update runtime config
this.app.put('/api/config', (req, res) => {
try {
const updates = req.body;
// Validate and apply rate limit updates
if (updates.rateLimit) {
if (typeof updates.rateLimit.minDelayBetweenMessagesMs === 'number') {
this.runtimeConfig.rateLimit.minDelayBetweenMessagesMs = Math.max(0, updates.rateLimit.minDelayBetweenMessagesMs);
}
if (typeof updates.rateLimit.maxMessagesPerHour === 'number') {
this.runtimeConfig.rateLimit.maxMessagesPerHour = Math.max(1, updates.rateLimit.maxMessagesPerHour);
}
if (typeof updates.rateLimit.jitterMs === 'number') {
this.runtimeConfig.rateLimit.jitterMs = Math.max(0, updates.rateLimit.jitterMs);
}
}
// Validate and apply browser updates
if (updates.browser) {
if (typeof updates.browser.dryRun === 'boolean') {
this.runtimeConfig.browser.dryRun = updates.browser.dryRun;
}
}
// Also update the original config object so getStats reflects new values
this.config.rateLimit.minDelayBetweenMessagesMs = this.runtimeConfig.rateLimit.minDelayBetweenMessagesMs;
this.config.rateLimit.maxMessagesPerHour = this.runtimeConfig.rateLimit.maxMessagesPerHour;
this.config.rateLimit.jitterMs = this.runtimeConfig.rateLimit.jitterMs;
this.config.browser.dryRun = this.runtimeConfig.browser.dryRun;
// Notify listeners of config change
if (this.onConfigChange) {
this.onConfigChange(this.runtimeConfig);
}
this.logActivity('info', `Config updated via dashboard`);
this.pushLog('info', 'Configuration updated via dashboard', updates);
res.json({ success: true, config: this.runtimeConfig });
} catch (error) {
this.logger.error('Failed to update config:', error);
res.status(400).json({ error: String(error) });
}
});
// Serve dashboard for all other routes
this.app.get(/.*/, (req, res) => {
// SPA catch-all.
this.app.get(/.*/, (_req, res) => {
res.sendFile(path.join(process.cwd(), 'public', 'index.html'));
});
}
// ---------- public API used by the bootstrapper ----------
isRunning(): boolean {
return this.botRunning;
}
setChatService(chatService: ChatService): void {
this.chatService = chatService;
this.services.chat = chatService;
}
setMediaService(mediaService: MediaService): void {
this.mediaService = mediaService;
this.services.media = mediaService;
}
setMediaSyncService(mediaSyncService: MediaSyncService): void {
this.mediaSyncService = mediaSyncService;
this.services.mediaSync = mediaSyncService;
}
setResponseChecker(responseChecker: ResponseCheckerService): void {
this.responseChecker = responseChecker;
this.services.responseChecker = responseChecker;
}
setRateLimiter(rateLimiter: RateLimiter): void {
this.rateLimiter = rateLimiter;
this.services.rateLimiter = rateLimiter;
}
setAiClient(aiClient: OpenRouterClient): void {
this.aiClient = aiClient;
this.services.ai = aiClient;
}
setStrategySelector(strategySelector: StrategySelector): void {
this.strategySelector = strategySelector;
this.services.strategySelector = strategySelector;
}
setAutoReplyService(autoReplyService: AutoReplyService): void {
this.autoReplyService = autoReplyService;
}
private getStats(): BotStats {
const messages = this.messagesRepo.getRecent(1000);
const totalSent = messages.length;
const totalResponses = messages.filter((m) => m.gotResponse).length;
return {
isRunning: this.botRunning,
dryRunMode: this.config.browser.dryRun,
totalProcessed: this.getTotalProcessed(),
totalMessagesSent: totalSent,
totalResponses,
responseRate: totalSent > 0 ? Math.round((totalResponses / totalSent) * 100) : 0,
messagesThisHour: this.messagesThisHour,
maxMessagesPerHour: this.config.rateLimit.maxMessagesPerHour,
subreddits: this.config.reddit.subreddits,
targetGenders: this.config.reddit.targetGenders,
recentActivity: this.activityLog.slice(-10).reverse(),
};
}
private getTotalProcessed(): number {
// This is a simple count - in production you'd query the DB
return this.activityLog.filter((a) => a.type === 'sent' || a.type === 'skipped').length;
setBrowserClient(browserClient: BrowserClient): void {
this.services.browser = browserClient;
}
logActivity(type: ActivityItem['type'], message: string, recipient?: string, imageUrl?: string): void {
const item: ActivityItem = {
timestamp: Date.now(),
type,
message,
recipient,
imageUrl,
};
this.activityLog.push(item);
// Keep only last 200 items
if (this.activityLog.length > 200) {
this.activityLog = this.activityLog.slice(-200);
}
this.activity.push(type, message, recipient, imageUrl);
}
/**
* Push a log entry to all connected SSE clients
*/
pushLog(level: LogEntry['level'], message: string, data?: unknown): void {
const entry: LogEntry = {
timestamp: Date.now(),
level,
message,
data,
};
this.realtimeLog.push(entry);
// Keep only last 500 log entries
if (this.realtimeLog.length > 500) {
this.realtimeLog = this.realtimeLog.slice(-500);
this.broadcaster.push(level, message, data);
}
// Broadcast to all SSE clients
const eventData = JSON.stringify({ type: 'log', ...entry });
this.sseClients.forEach(client => {
try {
client.write(`data: ${eventData}\n\n`);
} catch {
// Client disconnected, will be cleaned up on close event
this.sseClients.delete(client);
}
});
/** Fan a sync-progress frame out over the existing log SSE stream. */
broadcastSyncProgress(progress: Record<string, unknown> | null): void {
this.broadcaster.broadcastEvent('sync-progress', { progress });
}
/**
* Get the current runtime configuration
*/
getRuntimeConfig(): RuntimeConfig {
return this.runtimeConfig;
}
/**
* Set a callback to be notified when config changes
*/
setOnConfigChange(callback: (config: RuntimeConfig) => void): void {
this.onConfigChange = callback;
}
@@ -771,21 +187,12 @@ export class WebServer {
start(): void {
this.server = this.app.listen(this.port, () => {
this.logger.info(`Web dashboard running at http://localhost:${this.port}`);
this.pushLog('info', `Web dashboard started on port ${this.port}`);
this.broadcaster.push('info', `Web dashboard started on port ${this.port}`);
});
}
stop(): void {
// Close all SSE connections
this.sseClients.forEach(client => {
try {
client.end();
} catch {
// Ignore errors when closing
}
});
this.sseClients.clear();
this.broadcaster.closeAll();
if (this.server) {
this.server.close();
this.server = null;
src/web/types.ts
+45
View File
@@ -0,0 +1,45 @@
export interface BotStats {
isRunning: boolean;
dryRunMode: boolean;
totalProcessed: number;
totalMessagesSent: number;
totalResponses: number;
responseRate: number;
messagesThisHour: number;
maxMessagesPerHour: number;
subreddits: { name: string; type: 'image' | 'chat' }[];
targetGenders: string[];
recentActivity: ActivityItem[];
/** Non-null when Reddit has throttled new DM invitations. */
invitationCooldown: {
until: number;
reason: string;
redditCode: string | null;
} | null;
}
export interface ActivityItem {
timestamp: number;
type: 'sent' | 'skipped' | 'response' | 'error' | 'info' | 'debug';
message: string;
recipient?: string;
imageUrl?: string;
}
export interface LogEntry {
timestamp: number;
level: 'info' | 'warn' | 'error' | 'debug';
message: string;
data?: unknown;
}
export interface RuntimeConfig {
rateLimit: {
minDelayBetweenMessagesMs: number;
maxMessagesPerHour: number;
jitterMs: number;
};
browser: {
dryRun: boolean;
};
}
tests/chat-service-cooldown.test.ts
+99
View File
@@ -0,0 +1,99 @@
import { describe, it, expect } from 'vitest';
import { ChatService } from '../src/services/chat/service.js';
import { MatrixClient, MatrixRateLimitError } from '../src/services/chat/matrix/client.js';
import { createMockLogger } from './mockLogger.js';
const creds = {
homeserver: 'https://matrix.example.test',
accessToken: 'tok',
userId: '@me:example.test',
capturedAt: Date.now(),
};
function buildService() {
const logger = createMockLogger();
// Fetch impl never matters here — we drive cooldowns directly via the
// public API, which is how MatrixClient surfaces them at runtime.
const fetchImpl = (() =>
Promise.resolve(
new Response('{}', { status: 200, headers: { 'content-type': 'application/json' } }),
)) as typeof fetch;
const matrix = new MatrixClient(creds, logger, { minRequestIntervalMs: 0, fetchImpl });
const svc = new ChatService(matrix, logger);
return { svc, logger };
}
describe('ChatService invitation cooldown', () => {
it('is null by default', () => {
const { svc } = buildService();
expect(svc.getInvitationCooldown()).toBeNull();
});
it('records a cooldown from a MatrixRateLimitError', () => {
const { svc } = buildService();
const err = new MatrixRateLimitError(
'429',
60_000,
'M_LIMIT_EXCEEDED',
'rate.invitation_limit_mature_account',
);
svc.setInvitationCooldown(err);
const active = svc.getInvitationCooldown();
expect(active).not.toBeNull();
expect(active!.redditCode).toBe('rate.invitation_limit_mature_account');
expect(active!.until).toBeGreaterThan(Date.now() + 50_000);
expect(active!.reason).toMatch(/throttling new chat invitations/);
});
it('auto-expires once the until timestamp has passed', () => {
const { svc } = buildService();
const err = new MatrixRateLimitError('429', 10, 'M_LIMIT_EXCEEDED', 'rate.foo');
svc.setInvitationCooldown(err);
expect(svc.getInvitationCooldown()).not.toBeNull();
// Force the clock past the cooldown window.
const cooldown = svc.getInvitationCooldown()!;
const realNow = Date.now;
Date.now = () => cooldown.until + 1;
try {
expect(svc.getInvitationCooldown()).toBeNull();
} finally {
Date.now = realNow;
}
});
it('startConversation short-circuits when a cooldown is active', async () => {
const { svc } = buildService();
svc.setInvitationCooldown(
new MatrixRateLimitError('429', 60_000, 'M_LIMIT_EXCEEDED', 'rate.foo'),
);
const result = await svc.startConversation('someuser', 'hi');
expect(result).toBeNull();
});
it('sendChatMessage short-circuits when a cooldown is active', async () => {
const { svc } = buildService();
svc.setInvitationCooldown(
new MatrixRateLimitError('429', 60_000, 'M_LIMIT_EXCEEDED', null),
);
const result = await svc.sendChatMessage('!room:example.test', 'hi');
expect(result).toBe(false);
});
it('sendChatMessage records a cooldown when the homeserver rate-limits a reply', async () => {
const logger = createMockLogger();
const matrix = new MatrixClient(creds, logger, { minRequestIntervalMs: 0 });
// Force sendTextMessage to throw a rate-limit error — simulates Reddit
// returning M_LIMIT_EXCEEDED on a reply into an existing room.
matrix.sendTextMessage = async () => {
throw new MatrixRateLimitError('429', 90_000, 'M_LIMIT_EXCEEDED', null);
};
const svc = new ChatService(matrix, logger);
const result = await svc.sendChatMessage('!room:example.test', 'hi');
expect(result).toBe(false);
const active = svc.getInvitationCooldown();
expect(active).not.toBeNull();
expect(active!.until).toBeGreaterThan(Date.now() + 80_000);
});
});
tests/config.test.ts
+377
View File
@@ -0,0 +1,377 @@
import { describe, it, expect, beforeEach, afterEach } from 'vitest';
import fs from 'fs';
import path from 'path';
import { ConfigSchema, loadConfig } from '../src/config/index.js';
// Helper to build a minimal valid config for partial testing
function minimalConfig(overrides: Record<string, unknown> = {}): Record<string, unknown> {
return {
reddit: { subreddits: ['test'], ...overrides.reddit },
openrouter: { apiKey: 'key', imageAnalysisPrompt: 'test', messageGenerationPrompt: 'test', ...overrides.openrouter },
rateLimit: overrides.rateLimit ?? {},
browser: overrides.browser ?? {},
database: overrides.database ?? {},
media: overrides.media ?? {},
strategies: overrides.strategies ?? {},
web: overrides.web ?? {},
};
}
describe('ConfigSchema', () => {
describe('reddit section', () => {
it('validates required subreddits array', () => {
const result = ConfigSchema.safeParse(minimalConfig({ reddit: { subreddits: ['test'] } }));
expect(result.success).toBe(true);
});
it('rejects empty subreddits array', () => {
const result = ConfigSchema.safeParse(minimalConfig({ reddit: { subreddits: [] } }));
expect(result.success).toBe(false);
if (!result.success) {
const issue = result.error.issues.find(i => i.path.includes('subreddits'));
expect(issue).toBeDefined();
}
});
it('applies default targetGenders', () => {
const result = ConfigSchema.safeParse(minimalConfig({ reddit: { subreddits: ['test'] } }));
expect(result.success).toBe(true);
if (result.success) {
expect(result.data.reddit.targetGenders).toEqual(['female']);
}
});
it('accepts valid targetGenders', () => {
const result = ConfigSchema.safeParse(minimalConfig({
reddit: { subreddits: ['test'], targetGenders: ['male', 'female', 'other', 'unknown'] },
}));
expect(result.success).toBe(true);
});
it('rejects invalid targetGender', () => {
const result = ConfigSchema.safeParse(minimalConfig({
reddit: { subreddits: ['test'], targetGenders: ['invalid'] },
}));
expect(result.success).toBe(false);
});
it('applies default sellerKeywords', () => {
const result = ConfigSchema.safeParse(minimalConfig({ reddit: { subreddits: ['test'] } }));
expect(result.success).toBe(true);
if (result.success) {
expect(result.data.reddit.sellerKeywords.length).toBeGreaterThan(0);
expect(result.data.reddit.sellerKeywords).toContain('onlyfans');
}
});
it('accepts custom sellerKeywords', () => {
const result = ConfigSchema.safeParse(minimalConfig({
reddit: { subreddits: ['test'], sellerKeywords: ['custom1', 'custom2'] },
}));
expect(result.success).toBe(true);
});
it('applies default pollIntervalMs', () => {
const result = ConfigSchema.safeParse(minimalConfig({ reddit: { subreddits: ['test'] } }));
expect(result.success).toBe(true);
if (result.success) {
expect(result.data.reddit.pollIntervalMs).toBe(120000);
}
});
it('accepts custom pollIntervalMs', () => {
const result = ConfigSchema.safeParse(minimalConfig({
reddit: { subreddits: ['test'], pollIntervalMs: 30000 },
}));
expect(result.success).toBe(true);
});
});
describe('openrouter section', () => {
// Build openrouter directly — minimalConfig's spread can only override, not delete.
function configWithOpenrouter(openrouter: Record<string, unknown>): Record<string, unknown> {
return {
reddit: { subreddits: ['test'] },
openrouter,
rateLimit: {}, browser: {}, database: {}, media: {}, strategies: {}, web: {},
};
}
it('requires apiKey', () => {
const result = ConfigSchema.safeParse(configWithOpenrouter({
imageAnalysisPrompt: 'test',
messageGenerationPrompt: 'test',
}));
expect(result.success).toBe(false);
});
it('requires imageAnalysisPrompt', () => {
const result = ConfigSchema.safeParse(configWithOpenrouter({
apiKey: 'key',
messageGenerationPrompt: 'test',
}));
expect(result.success).toBe(false);
});
it('requires messageGenerationPrompt', () => {
const result = ConfigSchema.safeParse(configWithOpenrouter({
apiKey: 'key',
imageAnalysisPrompt: 'test',
}));
expect(result.success).toBe(false);
});
it('applies default models', () => {
const result = ConfigSchema.safeParse(minimalConfig({
openrouter: { apiKey: 'key', imageAnalysisPrompt: 'test', messageGenerationPrompt: 'test' },
}));
expect(result.success).toBe(true);
if (result.success) {
expect(result.data.openrouter.imageAnalysisModel).toBeDefined();
expect(result.data.openrouter.chatModel).toBeDefined();
expect(result.data.openrouter.imageAnalysisFallbackModels).toEqual([]);
}
});
it('accepts custom models and fallbacks', () => {
const result = ConfigSchema.safeParse(minimalConfig({
openrouter: {
apiKey: 'key',
imageAnalysisModel: 'custom-model',
imageAnalysisFallbackModels: ['fallback1', 'fallback2'],
chatModel: 'chat-model',
imageAnalysisPrompt: 'test',
messageGenerationPrompt: 'test',
},
}));
expect(result.success).toBe(true);
});
});
describe('rateLimit section', () => {
it('applies defaults', () => {
const result = ConfigSchema.safeParse(minimalConfig());
expect(result.success).toBe(true);
if (result.success) {
expect(result.data.rateLimit.minDelayBetweenMessagesMs).toBe(300000);
expect(result.data.rateLimit.maxMessagesPerHour).toBe(8);
expect(result.data.rateLimit.jitterMs).toBe(60000);
}
});
it('accepts custom rate limits', () => {
const result = ConfigSchema.safeParse(minimalConfig({
rateLimit: { minDelayBetweenMessagesMs: 5000, maxMessagesPerHour: 50, jitterMs: 30000 },
}));
expect(result.success).toBe(true);
});
});
describe('browser section', () => {
it('applies defaults', () => {
const result = ConfigSchema.safeParse(minimalConfig());
expect(result.success).toBe(true);
if (result.success) {
expect(result.data.browser.sessionDir).toBe('./data/browser-session');
expect(result.data.browser.headless).toBe(false);
expect(result.data.browser.dryRun).toBe(false);
}
});
});
describe('strategies section', () => {
it('applies defaults', () => {
const result = ConfigSchema.safeParse(minimalConfig());
expect(result.success).toBe(true);
if (result.success) {
expect(result.data.strategies.explorationRate).toBe(0.2);
expect(result.data.strategies.minSamples).toBe(5);
expect(result.data.strategies.variants).toEqual([]);
}
});
it('validates explorationRate bounds', () => {
const result1 = ConfigSchema.safeParse(minimalConfig({ strategies: { explorationRate: -0.1 } }));
expect(result1.success).toBe(false);
const result2 = ConfigSchema.safeParse(minimalConfig({ strategies: { explorationRate: 1.1 } }));
expect(result2.success).toBe(false);
const result3 = ConfigSchema.safeParse(minimalConfig({ strategies: { explorationRate: 0 } }));
expect(result3.success).toBe(true);
const result4 = ConfigSchema.safeParse(minimalConfig({ strategies: { explorationRate: 1 } }));
expect(result4.success).toBe(true);
});
it('validates strategy variant structure', () => {
const result = ConfigSchema.safeParse(minimalConfig({
strategies: {
variants: [
{ id: 'v1', promptOverride: 'test' },
{ id: 'v2', weight: 3, promptOverride: 'test2' },
],
},
}));
expect(result.success).toBe(true);
});
it('rejects variant without id', () => {
const result = ConfigSchema.safeParse(minimalConfig({
strategies: {
variants: [{ promptOverride: 'test' }],
},
}));
expect(result.success).toBe(false);
});
});
describe('web section', () => {
it('applies defaults', () => {
const result = ConfigSchema.safeParse(minimalConfig());
expect(result.success).toBe(true);
if (result.success) {
expect(result.data.web.enabled).toBe(true);
expect(result.data.web.port).toBe(3000);
}
});
});
describe('full config', () => {
it('validates complete config', () => {
const result = ConfigSchema.safeParse({
reddit: { subreddits: ['test'] },
openrouter: { apiKey: 'key', imageAnalysisPrompt: 'test', messageGenerationPrompt: 'test' },
rateLimit: {},
browser: {},
database: {},
media: {},
strategies: {},
web: {},
});
expect(result.success).toBe(true);
});
});
});
describe('loadConfig', () => {
const originalCwd = process.cwd;
const savedEnv: Record<string, string | undefined> = {};
const trackedEnvKeys = ['OPENROUTER_API_KEY', 'REDDIT_USERNAME', 'REDDIT_PASSWORD', 'BROWSER_HEADLESS'];
beforeEach(() => {
// Snapshot then clear any env vars that loadConfig consults — these may be
// populated from the user's real .env via dotenv.config() at module load.
for (const key of trackedEnvKeys) {
savedEnv[key] = process.env[key];
delete process.env[key];
}
const mockDir = '/tmp/test-config-' + Date.now();
fs.mkdirSync(mockDir, { recursive: true });
process.chdir(mockDir);
});
afterEach(() => {
process.chdir(originalCwd());
try {
fs.rmSync('/tmp/test-config-*', { recursive: true, force: true });
} catch { /* ignore */ }
for (const key of trackedEnvKeys) {
if (savedEnv[key] === undefined) {
delete process.env[key];
} else {
process.env[key] = savedEnv[key];
}
}
});
it('throws when config file does not exist', () => {
expect(() => {
loadConfig();
}).toThrow('Config file not found');
});
it('loads config from config.yaml', () => {
const configPath = path.join(process.cwd(), 'config.yaml');
fs.writeFileSync(configPath, `
reddit:
subreddits:
- test-sub
pollIntervalMs: 5000
openrouter:
apiKey: yaml-key
imageAnalysisPrompt: analyze this
messageGenerationPrompt: write message
`);
const config = loadConfig();
expect(config.reddit.subreddits).toEqual([{ name: 'test-sub', type: 'image' }]);
expect(config.reddit.pollIntervalMs).toBe(5000);
expect(config.openrouter.apiKey).toBe('yaml-key');
});
it('overrides apiKey from OPENROUTER_API_KEY env var', () => {
process.env.OPENROUTER_API_KEY = 'env-key';
const configPath = path.join(process.cwd(), 'config.yaml');
fs.writeFileSync(configPath, `
reddit:
subreddits:
- test
openrouter:
apiKey: yaml-key
imageAnalysisPrompt: analyze this
messageGenerationPrompt: write message
`);
const config = loadConfig();
expect(config.openrouter.apiKey).toBe('env-key');
});
it('overrides browser headless from BROWSER_HEADLESS env var', () => {
process.env.BROWSER_HEADLESS = 'true';
const configPath = path.join(process.cwd(), 'config.yaml');
fs.writeFileSync(configPath, `
reddit:
subreddits:
- test
openrouter:
apiKey: key
imageAnalysisPrompt: analyze this
messageGenerationPrompt: write message
browser:
headless: false
`);
const config = loadConfig();
expect(config.browser.headless).toBe(true);
});
it('rejects config with invalid values', () => {
const configPath = path.join(process.cwd(), 'config.yaml');
fs.writeFileSync(configPath, `
reddit:
subreddits: []
openrouter:
apiKey: key
imageAnalysisPrompt: analyze this
messageGenerationPrompt: write message
`);
expect(() => loadConfig()).toThrow('Invalid configuration');
});
it('throws when OPENROUTER_API_KEY is missing', () => {
const configPath = path.join(process.cwd(), 'config.yaml');
fs.writeFileSync(configPath, `
reddit:
subreddits:
- test
openrouter:
imageAnalysisPrompt: analyze this
messageGenerationPrompt: write message
`);
expect(() => loadConfig()).toThrow('OPENROUTER_API_KEY environment variable is required');
});
});
tests/database.test.ts
+146
View File
@@ -0,0 +1,146 @@
import { describe, it, expect, beforeEach, afterEach } from 'vitest';
import Database from 'better-sqlite3';
import { createDatabase, runMigrations } from '../src/database/client.js';
import { getTestConfig } from '../tests/testUtils.js';
describe('Database Client', () => {
describe('createDatabase', () => {
it('creates an in-memory database', () => {
const config = getTestConfig({ database: { path: ':memory:' } });
const db = createDatabase(config);
expect(db).toBeDefined();
db.close();
});
it('enables WAL mode', () => {
const config = getTestConfig({ database: { path: ':memory:' } });
const db = createDatabase(config);
const journalMode = db.pragma('journal_mode', { simple: true }) as string;
expect(journalMode).toBe('wal');
db.close();
});
it('enables foreign keys', () => {
const config = getTestConfig({ database: { path: ':memory:' } });
const db = createDatabase(config);
const fk = db.pragma('foreign_keys', { simple: true }) as number;
expect(fk).toBe(1);
db.close();
});
});
describe('runMigrations', () => {
let db: Database.Database;
beforeEach(() => {
db = new Database(':memory:');
db.pragma('foreign_keys = ON');
});
afterEach(() => {
db.close();
});
it('creates processed_posts table', () => {
runMigrations(db);
const tables = db.prepare("SELECT name FROM sqlite_master WHERE type='table'").all() as { name: string }[];
const tableNames = tables.map(t => t.name);
expect(tableNames).toContain('processed_posts');
});
it('creates sent_messages table', () => {
runMigrations(db);
const tables = db.prepare("SELECT name FROM sqlite_master WHERE type='table'").all() as { name: string }[];
const tableNames = tables.map(t => t.name);
expect(tableNames).toContain('sent_messages');
});
it('creates successful_messages table', () => {
runMigrations(db);
const tables = db.prepare("SELECT name FROM sqlite_master WHERE type='table'").all() as { name: string }[];
const tableNames = tables.map(t => t.name);
expect(tableNames).toContain('successful_messages');
});
it('creates indexes', () => {
runMigrations(db);
const indexes = db.prepare("SELECT name FROM sqlite_master WHERE type='index'").all() as { name: string }[];
const indexNames = indexes.map(i => i.name);
expect(indexNames).toContain('idx_posts_subreddit');
expect(indexNames).toContain('idx_posts_processed_at');
expect(indexNames).toContain('idx_posts_message_sent');
expect(indexNames).toContain('idx_sent_messages_response');
expect(indexNames).toContain('idx_sent_messages_recipient');
});
it('adds got_response column', () => {
runMigrations(db);
const columns = db.prepare('PRAGMA table_info(sent_messages)').all() as { name: string }[];
const columnNames = columns.map(c => c.name);
expect(columnNames).toContain('got_response');
expect(columnNames).toContain('response_checked_at');
});
it('adds strategy_id column', () => {
runMigrations(db);
const columns = db.prepare('PRAGMA table_info(sent_messages)').all() as { name: string }[];
const columnNames = columns.map(c => c.name);
expect(columnNames).toContain('strategy_id');
});
it('adds subreddit column to successful_messages', () => {
runMigrations(db);
const columns = db.prepare('PRAGMA table_info(successful_messages)').all() as { name: string }[];
const columnNames = columns.map(c => c.name);
expect(columnNames).toContain('subreddit');
});
it('is idempotent (safe to run multiple times)', () => {
runMigrations(db);
runMigrations(db);
runMigrations(db);
const tables = db
.prepare("SELECT name FROM sqlite_master WHERE type='table' AND name NOT LIKE 'sqlite_%'")
.all() as { name: string }[];
// processed_posts, sent_messages, successful_messages, activity_log
expect(tables.length).toBe(4);
});
it('processed_posts has correct schema', () => {
runMigrations(db);
const columns = db.prepare('PRAGMA table_info(processed_posts)').all() as { name: string; notnull: number }[];
const columnMap = new Map(columns.map(c => [c.name, c]));
expect(columnMap.get('id')?.notnull).toBe(0); // TEXT is not NOT NULL by default
expect(columnMap.get('subreddit')?.notnull).toBe(1);
expect(columnMap.get('author')?.notnull).toBe(1);
expect(columnMap.get('title')?.notnull).toBe(1);
expect(columnMap.get('processed_at')?.notnull).toBe(1);
expect(columnMap.get('message_sent')).toBeDefined();
expect(columnMap.get('error')).toBeDefined();
});
it('sent_messages has correct schema', () => {
runMigrations(db);
const columns = db.prepare('PRAGMA table_info(sent_messages)').all() as { name: string; notnull: number }[];
const columnMap = new Map(columns.map(c => [c.name, c]));
expect(columnMap.get('recipient')?.notnull).toBe(1);
expect(columnMap.get('subject')?.notnull).toBe(1);
expect(columnMap.get('message')?.notnull).toBe(1);
expect(columnMap.get('sent_at')?.notnull).toBe(1);
expect(columnMap.get('post_id')?.notnull).toBe(1);
});
it('successful_messages has correct schema', () => {
runMigrations(db);
const columns = db.prepare('PRAGMA table_info(successful_messages)').all() as { name: string; notnull: number }[];
const columnMap = new Map(columns.map(c => [c.name, c]));
expect(columnMap.get('original_message')?.notnull).toBe(1);
expect(columnMap.get('created_at')?.notnull).toBe(1);
expect(columnMap.get('sent_message_id')?.notnull).toBe(0); // nullable
});
});
});
tests/gender-tag.test.ts
+59
View File
@@ -0,0 +1,59 @@
import { describe, it, expect } from 'vitest';
import { parseGenderTag } from '../src/utils/gender-tag.js';
describe('parseGenderTag', () => {
it('parses bracketed age-then-letter', () => {
expect(parseGenderTag('Bored at home [25F]')).toBe('female');
expect(parseGenderTag('looking for chat [30 M]')).toBe('male');
});
it('parses bracketed letter-then-age', () => {
expect(parseGenderTag('[F25] dtf')).toBe('female');
expect(parseGenderTag('[M, 22] up late')).toBe('male');
});
it('parses parenthesised tags', () => {
expect(parseGenderTag('Hey (25f) here')).toBe('female');
expect(parseGenderTag('(m 30) want to chat')).toBe('male');
});
it('parses X4Y tags', () => {
expect(parseGenderTag('f4m anyone awake?')).toBe('female');
expect(parseGenderTag('M4F looking')).toBe('male');
expect(parseGenderTag('f4mf bored')).toBe('female');
});
it('parses slash-separated age/gender', () => {
expect(parseGenderTag('25/f horny')).toBe('female');
expect(parseGenderTag('m/22 want a chat')).toBe('male');
});
it('parses bare age+letter', () => {
expect(parseGenderTag('hey 25f here')).toBe('female');
expect(parseGenderTag('f25 from UK')).toBe('female');
expect(parseGenderTag('30m looking')).toBe('male');
});
it('is case-insensitive', () => {
expect(parseGenderTag('[25F]')).toBe('female');
expect(parseGenderTag('[25f]')).toBe('female');
});
it('returns null for untagged posts', () => {
expect(parseGenderTag('hey looking for someone fun')).toBeNull();
expect(parseGenderTag('')).toBeNull();
expect(parseGenderTag('any chats?')).toBeNull();
});
it('does not match embedded letters in unrelated words', () => {
// "25 from" should not parse as 25f
expect(parseGenderTag('25 from the UK')).toBeNull();
// "30 minutes" should not parse as 30m
expect(parseGenderTag('30 minutes ago')).toBeNull();
});
it('prefers the first match (the poster, not the target in X4Y)', () => {
expect(parseGenderTag('M4F please dm')).toBe('male');
expect(parseGenderTag('F4M anyone?')).toBe('female');
});
});
tests/image-utils.test.ts
+74
View File
@@ -0,0 +1,74 @@
import { describe, it, expect } from 'vitest';
import { isValidImageMimeType, bufferToBase64 } from '../src/utils/image.js';
describe('isValidImageMimeType', () => {
it('accepts valid image MIME types', () => {
expect(isValidImageMimeType('image/jpeg')).toBe(true);
expect(isValidImageMimeType('image/png')).toBe(true);
expect(isValidImageMimeType('image/gif')).toBe(true);
expect(isValidImageMimeType('image/webp')).toBe(true);
});
it('rejects invalid MIME types', () => {
expect(isValidImageMimeType('text/plain')).toBe(false);
expect(isValidImageMimeType('application/pdf')).toBe(false);
expect(isValidImageMimeType('video/mp4')).toBe(false);
expect(isValidImageMimeType('application/json')).toBe(false);
expect(isValidImageMimeType('text/html')).toBe(false);
});
it('rejects empty string', () => {
expect(isValidImageMimeType('')).toBe(false);
});
it('rejects null-like values', () => {
expect(isValidImageMimeType('null' as unknown as string)).toBe(false);
expect(isValidImageMimeType('undefined' as unknown as string)).toBe(false);
});
it('is case-sensitive (lowercase required)', () => {
// The function checks exact matches against lowercase values
expect(isValidImageMimeType('IMAGE/JPEG')).toBe(false);
expect(isValidImageMimeType('Image/Png')).toBe(false);
});
it('rejects MIME types with parameters', () => {
expect(isValidImageMimeType('image/jpeg; charset=utf-8')).toBe(false);
expect(isValidImageMimeType('image/png; alpha')).toBe(false);
});
});
describe('bufferToBase64', () => {
it('converts buffer to base64 string', () => {
const buffer = Buffer.from('hello world');
const result = bufferToBase64(buffer);
expect(result).toBe(btoa('hello world'));
});
it('handles empty buffer', () => {
const buffer = Buffer.from('');
const result = bufferToBase64(buffer);
expect(result).toBe('');
});
it('handles binary data', () => {
const buffer = Buffer.from([0x00, 0x01, 0x02, 0xff, 0xfe, 0xfd]);
const result = bufferToBase64(buffer);
expect(result).toBe(btoa('\x00\x01\x02\xff\xfe\xfd'));
});
it('handles large buffer', () => {
const buffer = Buffer.alloc(1024 * 1024); // 1MB
buffer.fill(0x41); // 'A'
const result = bufferToBase64(buffer);
expect(result.length).toBeGreaterThan(0);
// Base64 encodes each 3 bytes as 4 chars, rounding up.
expect(result.length).toBe(Math.ceil((1024 * 1024) / 3) * 4);
});
it('produces valid base64 (only valid chars)', () => {
const buffer = Buffer.from(Math.random().toString(36).repeat(100));
const result = bufferToBase64(buffer);
expect(result).toMatch(/^[A-Za-z0-9+/]+=*$/);
});
});
tests/matrix-client.test.ts
+133
View File
@@ -0,0 +1,133 @@
import { describe, it, expect } from 'vitest';
import { MatrixClient, MatrixRateLimitError } from '../src/services/chat/matrix/client.js';
import { createMockLogger } from './mockLogger.js';
const creds = {
homeserver: 'https://matrix.example.test',
accessToken: 'tok',
userId: '@me:example.test',
capturedAt: Date.now(),
};
describe('MatrixClient HTTP timeout', () => {
it('aborts a hanging fetch and surfaces a timeout error rather than blocking forever', async () => {
// A fetch impl that never resolves on its own — only the AbortSignal can
// end it. If the client lacked an AbortSignal, this test would hang.
const fetchImpl = ((_url: string, init?: RequestInit) =>
new Promise<Response>((_resolve, reject) => {
init?.signal?.addEventListener('abort', () => {
const err = new Error('aborted');
(err as Error & { name: string }).name = 'AbortError';
reject(err);
});
})) as typeof fetch;
const client = new MatrixClient(creds, createMockLogger(), {
requestTimeoutMs: 25,
minRequestIntervalMs: 0,
fetchImpl,
});
await expect(client.whoami()).rejects.toThrow(/timed out/);
}, 30_000);
it('retries after a single timeout when the next attempt succeeds', async () => {
let calls = 0;
const fetchImpl = ((_url: string, init?: RequestInit) => {
calls += 1;
if (calls === 1) {
return new Promise<Response>((_resolve, reject) => {
init?.signal?.addEventListener('abort', () => {
const err = new Error('aborted');
(err as Error & { name: string }).name = 'AbortError';
reject(err);
});
});
}
return Promise.resolve(
new Response(JSON.stringify({ user_id: '@me:example.test' }), {
status: 200,
headers: { 'content-type': 'application/json' },
}),
);
}) as typeof fetch;
const client = new MatrixClient(creds, createMockLogger(), {
requestTimeoutMs: 25,
minRequestIntervalMs: 0,
fetchImpl,
});
const userId = await client.whoami();
expect(userId).toBe('@me:example.test');
expect(calls).toBe(2);
}, 10_000);
});
describe('MatrixClient 429 handling', () => {
it('throws MatrixRateLimitError on a long retry_after rather than sleeping it out', async () => {
const fetchImpl = (() =>
Promise.resolve(
new Response(
JSON.stringify({
errcode: 'M_LIMIT_EXCEEDED',
error: 'Limit exceeded for number of invites attempted',
'com.reddit.error.code': 'rate.invitation_limit_mature_account',
retry_after_ms: 591_528,
}),
{ status: 429, headers: { 'content-type': 'application/json' } },
),
)) as typeof fetch;
const client = new MatrixClient(creds, createMockLogger(), {
minRequestIntervalMs: 0,
fetchImpl,
});
const start = Date.now();
await expect(client.whoami()).rejects.toMatchObject({
name: 'MatrixRateLimitError',
retryAfterMs: 591_528,
redditCode: 'rate.invitation_limit_mature_account',
});
// Should fail near-instantly, NOT sleep for 591s.
expect(Date.now() - start).toBeLessThan(2_000);
});
it('still sleeps + retries on short 429 retry_after (normal flood control)', async () => {
let calls = 0;
const fetchImpl = (() => {
calls += 1;
if (calls === 1) {
return Promise.resolve(
new Response(JSON.stringify({ retry_after_ms: 50 }), {
status: 429,
headers: { 'content-type': 'application/json' },
}),
);
}
return Promise.resolve(
new Response(JSON.stringify({ user_id: '@me:example.test' }), {
status: 200,
headers: { 'content-type': 'application/json' },
}),
);
}) as typeof fetch;
const client = new MatrixClient(creds, createMockLogger(), {
minRequestIntervalMs: 0,
fetchImpl,
});
const userId = await client.whoami();
expect(userId).toBe('@me:example.test');
expect(calls).toBe(2);
});
it('exposes MatrixRateLimitError as a typed export', () => {
const err = new MatrixRateLimitError('test', 1000, 'M_LIMIT_EXCEEDED', 'rate.foo');
expect(err).toBeInstanceOf(Error);
expect(err.name).toBe('MatrixRateLimitError');
expect(err.retryAfterMs).toBe(1000);
});
});
tests/media.test.ts
+203
View File
@@ -0,0 +1,203 @@
import { describe, it, expect, vi, beforeEach, afterEach } from 'vitest';
import fs from 'fs/promises';
import os from 'os';
import path from 'path';
import { MediaService } from '../src/services/media.js';
import type { MediaDownloader } from '../src/services/media.js';
import type { ActiveChat, ChatMessage } from '../src/services/chat.js';
import { createMockLogger } from '../tests/mockLogger.js';
function createMockDownloader(download: Buffer | null = Buffer.from('image data')): MediaDownloader {
return {
fetch: vi.fn(async (_url: string) => {
if (!download) return null;
return { buffer: download, contentType: 'image/jpeg' };
}),
};
}
function createMockChat(username = 'user1', roomId = 'room1'): ActiveChat {
return {
username,
roomName: username,
lastMessageTime: 0,
lastMessageSnippet: '',
isFromUs: false,
unread: false,
roomId,
url: 'https://www.reddit.com/message/chat/abc',
};
}
describe('MediaService', () => {
let service: MediaService;
let logger: ReturnType<typeof createMockLogger>;
let downloader: MediaDownloader;
let tempDir: string;
beforeEach(async () => {
logger = createMockLogger();
downloader = createMockDownloader();
tempDir = await fs.mkdtemp(path.join(os.tmpdir(), 'media-test-'));
service = new MediaService(downloader, logger, tempDir);
});
afterEach(async () => {
await fs.rm(tempDir, { recursive: true, force: true });
});
describe('resolveFullMediaUrl', () => {
it('converts blurred thumbnail to full resolution', () => {
const blurred = 'https://matrix.redditspace.com/_matrix/media/v3/thumbnail/reddit.com/blur_XXXXX?height=300';
// We can't test private method directly, but we can test via attachMedia
// For now, verify the service is created
expect(service).toBeDefined();
});
it('leaves non-thumbnail URLs unchanged', () => {
// Same as above - test via public API
expect(service).toBeDefined();
});
it('handles invalid URLs gracefully', () => {
// Same as above
expect(service).toBeDefined();
});
});
describe('attachMedia', () => {
it('returns messages without media when no URLs', async () => {
const chat = createMockChat();
const messages: Array<ChatMessage & { mediaUrls?: string[] }> = [
{ sender: 'user1', text: 'Hello', timestamp: Date.now(), isSelf: false },
];
const result = await service.attachMedia(chat, messages);
expect(result.length).toBe(1);
expect(result[0].media).toBeUndefined();
});
it('downloads media from URLs', async () => {
const chat = createMockChat();
const messages: Array<ChatMessage & { mediaUrls: string[] }> = [
{ sender: 'user1', text: 'Check this out', timestamp: Date.now(), isSelf: false, mediaUrls: ['http://example.com/img.jpg'] },
];
const result = await service.attachMedia(chat, messages);
expect(result.length).toBe(1);
expect(result[0].media).toBeDefined();
expect(result[0].media!.length).toBe(1);
});
it('skips data: URLs', async () => {
const chat = createMockChat();
const messages: Array<ChatMessage & { mediaUrls: string[] }> = [
{ sender: 'user1', text: 'Test', timestamp: Date.now(), isSelf: false, mediaUrls: ['data:image/jpeg;base64,abc'] },
];
const result = await service.attachMedia(chat, messages);
expect(result[0].media).toBeUndefined();
});
it('skips null/empty URLs', async () => {
const chat = createMockChat();
const messages: Array<ChatMessage & { mediaUrls: string[] }> = [
{ sender: 'user1', text: 'Test', timestamp: Date.now(), isSelf: false, mediaUrls: ['', null as unknown as string] },
];
const result = await service.attachMedia(chat, messages);
expect(result[0].media).toBeUndefined();
});
it('handles download failures gracefully', async () => {
const failingDownloader = createMockDownloader(null);
service = new MediaService(failingDownloader, logger);
const chat = createMockChat();
const messages: Array<ChatMessage & { mediaUrls: string[] }> = [
{ sender: 'user1', text: 'Test', timestamp: Date.now(), isSelf: false, mediaUrls: ['http://example.com/img.jpg'] },
];
const result = await service.attachMedia(chat, messages);
expect(result[0].media).toBeUndefined();
});
it('caches downloaded media by URL hash', async () => {
const chat = createMockChat();
const url = 'http://example.com/img.jpg';
const messages: Array<ChatMessage & { mediaUrls: string[] }> = [
{ sender: 'user1', text: 'Test', timestamp: Date.now(), isSelf: false, mediaUrls: [url] },
];
// First download
await service.attachMedia(chat, messages);
// Reset downloader mock
vi.mocked(downloader.fetch).mockClear();
// Second download of same URL should use cache
const messages2: Array<ChatMessage & { mediaUrls: string[] }> = [
{ sender: 'user1', text: 'Test 2', timestamp: Date.now(), isSelf: false, mediaUrls: [url] },
];
await service.attachMedia(chat, messages2);
// Second download of the same URL hits the index + on-disk cache, so the
// downloader is not called again.
expect(downloader.fetch).not.toHaveBeenCalled();
});
it('processes multiple messages', async () => {
const chat = createMockChat();
const messages: Array<ChatMessage & { mediaUrls: string[] }> = [
{ sender: 'user1', text: 'Msg 1', timestamp: Date.now(), isSelf: false, mediaUrls: ['http://example.com/1.jpg'] },
{ sender: 'user1', text: 'Msg 2', timestamp: Date.now(), isSelf: false, mediaUrls: [] },
{ sender: 'user1', text: 'Msg 3', timestamp: Date.now(), isSelf: false, mediaUrls: ['http://example.com/3.jpg'] },
];
const result = await service.attachMedia(chat, messages);
expect(result.length).toBe(3);
expect(result[0].media).toBeDefined();
expect(result[1].media).toBeUndefined();
expect(result[2].media).toBeDefined();
});
});
describe('getMediaItems', () => {
it('returns empty array when no index', async () => {
const items = await service.getMediaItems();
expect(items).toEqual([]);
});
});
describe('edge cases', () => {
it('handles empty message array', async () => {
const chat = createMockChat();
const result = await service.attachMedia(chat, []);
expect(result).toEqual([]);
});
it('handles chat with no username', async () => {
const chat = createMockChat('', '');
const messages: Array<ChatMessage & { mediaUrls: string[] }> = [
{ sender: 'anon', text: 'Test', timestamp: Date.now(), isSelf: false, mediaUrls: ['http://example.com/img.jpg'] },
];
const result = await service.attachMedia(chat, messages);
expect(result.length).toBe(1);
});
it('handles messages with isSelf flag', async () => {
const chat = createMockChat();
const messages: Array<ChatMessage & { mediaUrls: string[] }> = [
{ sender: 'user1', text: 'Their message', timestamp: Date.now(), isSelf: false, mediaUrls: ['http://example.com/img.jpg'] },
{ sender: 'me', text: 'My message', timestamp: Date.now(), isSelf: true, mediaUrls: ['http://example.com/img2.jpg'] },
];
const result = await service.attachMedia(chat, messages);
expect(result.length).toBe(2);
expect(result[0].isSelf).toBe(false);
expect(result[1].isSelf).toBe(true);
});
});
});
tests/mockLogger.ts
+57
View File
@@ -0,0 +1,57 @@
import winston from 'winston';
export interface LogEntry {
level: string;
message: string;
meta: Record<string, unknown>;
timestamp: string;
}
export function createMockLogger(): winston.Logger & { logs: LogEntry[] } {
const logs: LogEntry[] = [];
const logger = winston.createLogger({
level: 'debug',
format: winston.format.combine(
winston.format.timestamp(),
winston.format.json()
),
transports: [
new winston.transports.Console({
format: winston.format.printf(({ level, message, timestamp, ...meta }) => {
logs.push({ level: String(level), message: String(message), meta, timestamp: String(timestamp) });
return '';
}),
}),
],
}) as winston.Logger & { logs: LogEntry[] };
// Override all log levels to capture
const levels = ['error', 'warn', 'info', 'http', 'debug'] as const;
for (const level of levels) {
const original = logger[level];
logger[level] = ((message: unknown, ...meta: unknown[]) => {
logs.push({
level,
message: String(message),
meta: meta[0] as Record<string, unknown> || {},
timestamp: new Date().toISOString(),
});
return logger;
}) as any;
}
(logger as any).logs = logs;
logger.logs = logs;
return logger as winston.Logger & { logs: LogEntry[] };
}
export function getLogsByLevel(logger: winston.Logger & { logs: LogEntry[] }, level: string): LogEntry[] {
return logger.logs.filter(l => l.level === level);
}
export function hasLog(logger: winston.Logger & { logs: LogEntry[] }, level: string, messageContains: string): boolean {
return logger.logs.some(l => l.level === level && l.message.includes(messageContains));
}
tests/monitor.test.ts
+430
View File
@@ -0,0 +1,430 @@
import { describe, it, expect, vi, beforeEach } from 'vitest';
import { MonitorService } from '../src/services/monitor.js';
import type { RedditApiClient, RedditPost } from '../src/reddit/api.js';
import type { PostsRepository, MessagesRepository } from '../src/database/repositories.js';
import { createMockLogger } from '../tests/mockLogger.js';
import { getTestConfig } from '../tests/testUtils.js';
function createMockRedditApi(posts: RedditPost[] = []): RedditApiClient {
return {
getNewPosts: vi.fn(async () => posts),
getImageUrl: vi.fn(() => 'http://example.com/img.jpg'),
getPostUrl: vi.fn(() => 'https://www.reddit.com/test'),
isSellerProfile: vi.fn(async () => false),
} as unknown as RedditApiClient;
}
function createMockPostsRepo(processedIds: Set<string> = new Set()): PostsRepository {
return {
isProcessed: vi.fn((id: string) => processedIds.has(id)),
insert: vi.fn(),
updateAnalysis: vi.fn(),
markMessageSent: vi.fn(),
setError: vi.fn(),
getById: vi.fn(() => undefined),
getByAuthor: vi.fn(() => []),
} as unknown as PostsRepository;
}
function createMockMessagesRepo(sentRecipients: Set<string> = new Set()): MessagesRepository {
return {
hasSentTo: vi.fn((recipient: string) => sentRecipients.has(recipient)),
insert: vi.fn(() => 1),
getByPostId: vi.fn(),
getRecent: vi.fn(),
getUncheckedMessages: vi.fn(),
markGotResponse: vi.fn(),
markChecked: vi.fn(),
getStrategyStats: vi.fn(() => []),
getSubredditStats: vi.fn(() => []),
getResponseRateTrend: vi.fn(() => []),
} as unknown as MessagesRepository;
}
describe('MonitorService', () => {
let monitor: MonitorService;
let logger: ReturnType<typeof createMockLogger>;
beforeEach(() => {
logger = createMockLogger();
});
it('returns posts from first subreddit with valid posts', async () => {
const post1: RedditPost = {
id: 't3_1', subreddit: 'sub1', author: 'user1', title: 'Post 1',
url: 'http://example.com/1.jpg', permalink: '/r/sub1/1', thumbnail: '',
is_self: false, created_utc: Date.now() / 1000, over_18: false, spoiler: false,
};
const redditApi = createMockRedditApi([post1]);
const postsRepo = createMockPostsRepo();
const messagesRepo = createMockMessagesRepo();
const config = getTestConfig({ reddit: { subreddits: [{ name: 'sub1', type: 'image' }, { name: 'sub2', type: 'image' }] } });
monitor = new MonitorService(redditApi, postsRepo, messagesRepo, config, logger, 0);
const result = await monitor.getUnprocessedPosts();
expect(result.length).toBe(1);
expect(result[0].id).toBe('t3_1');
});
it('skips processed posts', async () => {
const post1: RedditPost = {
id: 't3_1', subreddit: 'sub1', author: 'user1', title: 'Post 1',
url: 'http://example.com/1.jpg', permalink: '/r/sub1/1', thumbnail: '',
is_self: false, created_utc: Date.now() / 1000, over_18: false, spoiler: false,
};
const redditApi = createMockRedditApi([post1]);
const postsRepo = createMockPostsRepo(new Set(['t3_1']));
const messagesRepo = createMockMessagesRepo();
const config = getTestConfig({ reddit: { subreddits: [{ name: 'sub1', type: 'image' }] } });
monitor = new MonitorService(redditApi, postsRepo, messagesRepo, config, logger, 0);
const result = await monitor.getUnprocessedPosts();
expect(result.length).toBe(0);
});
it('skips posts from users already messaged', async () => {
const post1: RedditPost = {
id: 't3_1', subreddit: 'sub1', author: 'already_msgd', title: 'Post 1',
url: 'http://example.com/1.jpg', permalink: '/r/sub1/1', thumbnail: '',
is_self: false, created_utc: Date.now() / 1000, over_18: false, spoiler: false,
};
const redditApi = createMockRedditApi([post1]);
const postsRepo = createMockPostsRepo();
const messagesRepo = createMockMessagesRepo(new Set(['already_msgd']));
const config = getTestConfig({ reddit: { subreddits: [{ name: 'sub1', type: 'image' }] } });
monitor = new MonitorService(redditApi, postsRepo, messagesRepo, config, logger, 0);
const result = await monitor.getUnprocessedPosts();
expect(result.length).toBe(0);
expect(messagesRepo.hasSentTo).toHaveBeenCalledWith('already_msgd');
});
it('returns newest post when multiple valid posts exist', async () => {
const post1: RedditPost = {
id: 't3_1', subreddit: 'sub1', author: 'user1', title: 'Older',
url: 'http://example.com/1.jpg', permalink: '/r/sub1/1', thumbnail: '',
is_self: false, created_utc: 1000000, over_18: false, spoiler: false,
};
const post2: RedditPost = {
id: 't3_2', subreddit: 'sub1', author: 'user2', title: 'Newer',
url: 'http://example.com/2.jpg', permalink: '/r/sub1/2', thumbnail: '',
is_self: false, created_utc: 2000000, over_18: false, spoiler: false,
};
const redditApi = createMockRedditApi([post1, post2]);
const postsRepo = createMockPostsRepo();
const messagesRepo = createMockMessagesRepo();
const config = getTestConfig({ reddit: { subreddits: [{ name: 'sub1', type: 'image' }] } });
monitor = new MonitorService(redditApi, postsRepo, messagesRepo, config, logger, 0);
const result = await monitor.getUnprocessedPosts();
expect(result.length).toBe(1);
expect(result[0].id).toBe('t3_2');
});
it('tries next subreddit when first has no valid posts', async () => {
const post1: RedditPost = {
id: 't3_1', subreddit: 'sub2', author: 'user1', title: 'Post',
url: 'http://example.com/1.jpg', permalink: '/r/sub2/1', thumbnail: '',
is_self: false, created_utc: Date.now() / 1000, over_18: false, spoiler: false,
};
const redditApi = createMockRedditApi([post1]);
const postsRepo1 = createMockPostsRepo();
const postsRepo2 = createMockPostsRepo();
// First call returns empty (all processed), second returns valid post
let callCount = 0;
postsRepo1.isProcessed = vi.fn(() => { callCount++; return callCount <= 1; });
const messagesRepo = createMockMessagesRepo();
const config = getTestConfig({ reddit: { subreddits: [{ name: 'sub1', type: 'image' }, { name: 'sub2', type: 'image' }] } });
// Override to use postsRepo1 for first check, postsRepo2 for second
monitor = new MonitorService(redditApi, postsRepo1, messagesRepo, config, logger, 0);
// The monitor shuffles subreddits, so we can't guarantee order.
// But we can verify it tries multiple subreddits.
const result = await monitor.getUnprocessedPosts();
// Result depends on shuffle - just verify it doesn't crash
expect(Array.isArray(result)).toBe(true);
});
it('returns empty array when no subreddits have valid posts', async () => {
const redditApi = createMockRedditApi([]);
const postsRepo = createMockPostsRepo();
const messagesRepo = createMockMessagesRepo();
const config = getTestConfig({ reddit: { subreddits: [{ name: 'sub1', type: 'image' }, { name: 'sub2', type: 'image' }] } });
monitor = new MonitorService(redditApi, postsRepo, messagesRepo, config, logger, 0);
const result = await monitor.getUnprocessedPosts();
expect(result).toEqual([]);
});
it('handles API errors gracefully', async () => {
const redditApi = createMockRedditApi([]);
vi.mocked(redditApi.getNewPosts).mockRejectedValue(new Error('API error'));
const postsRepo = createMockPostsRepo();
const messagesRepo = createMockMessagesRepo();
const config = getTestConfig({ reddit: { subreddits: [{ name: 'sub1', type: 'image' }] } });
monitor = new MonitorService(redditApi, postsRepo, messagesRepo, config, logger, 0);
const result = await monitor.getUnprocessedPosts();
expect(result).toEqual([]);
expect(logger.logs.some(l => l.level === 'error' && l.message.includes('Failed to fetch'))).toBe(true);
});
it('returns only one post per call', async () => {
const posts: RedditPost[] = Array.from({ length: 10 }, (_, i) => ({
id: `t3_${i}`, subreddit: 'sub1', author: `user${i}`, title: `Post ${i}`,
url: `http://example.com/${i}.jpg`, permalink: `/r/sub1/${i}`, thumbnail: '',
is_self: false, created_utc: Date.now() / 1000 - i, over_18: false, spoiler: false,
}));
const redditApi = createMockRedditApi(posts);
const postsRepo = createMockPostsRepo();
const messagesRepo = createMockMessagesRepo();
const config = getTestConfig({ reddit: { subreddits: [{ name: 'sub1', type: 'image' }] } });
monitor = new MonitorService(redditApi, postsRepo, messagesRepo, config, logger, 0);
const result = await monitor.getUnprocessedPosts();
expect(result.length).toBe(1);
});
it('shuffles subreddit order (non-deterministic)', async () => {
const config = getTestConfig({ reddit: { subreddits: [{ name: 'a', type: 'image' }, { name: 'b', type: 'image' }, { name: 'c', type: 'image' }, { name: 'd', type: 'image' }, { name: 'e', type: 'image' }] } });
const redditApi = createMockRedditApi([]);
const postsRepo = createMockPostsRepo();
const messagesRepo = createMockMessagesRepo();
monitor = new MonitorService(redditApi, postsRepo, messagesRepo, config, logger, 0);
// Run multiple times and verify the order changes (with high probability)
const orders: string[][] = [];
for (let i = 0; i < 10; i++) {
// We can't directly access the shuffled array, but we can verify
// that the shuffle function works by checking it produces different results
// Since getNewPosts returns empty, we just verify no errors
await monitor.getUnprocessedPosts();
}
// If shuffle works, 10 runs should produce varied behavior
// (We can't directly test the shuffle, but we can verify it doesn't crash)
});
it('limits results to 10 per author via postsRepo', async () => {
const post: RedditPost = {
id: 't3_1', subreddit: 'sub1', author: 'user1', title: 'Post',
url: 'http://example.com/1.jpg', permalink: '/r/sub1/1', thumbnail: '',
is_self: false, created_utc: Date.now() / 1000, over_18: false, spoiler: false,
};
const redditApi = createMockRedditApi([post]);
const postsRepo = createMockPostsRepo();
const messagesRepo = createMockMessagesRepo();
const config = getTestConfig({ reddit: { subreddits: [{ name: 'sub1', type: 'image' }] } });
monitor = new MonitorService(redditApi, postsRepo, messagesRepo, config, logger, 0);
await monitor.getUnprocessedPosts();
// Verify isProcessed was called
expect(postsRepo.isProcessed).toHaveBeenCalledWith('t3_1');
});
describe('image/chat interleaving', () => {
it('alternates between image and chat subs across calls when both have no posts', async () => {
const redditApi = createMockRedditApi([]);
const postsRepo = createMockPostsRepo();
const messagesRepo = createMockMessagesRepo();
const config = getTestConfig({
reddit: {
subreddits: [
{ name: 'i1', type: 'image' },
{ name: 'i2', type: 'image' },
{ name: 'i3', type: 'image' },
{ name: 'c1', type: 'chat' },
],
},
});
monitor = new MonitorService(redditApi, postsRepo, messagesRepo, config, logger, 0);
// First call: should visit one chat sub among the first two positions
await monitor.getUnprocessedPosts();
const calls1 = vi.mocked(redditApi.getNewPosts).mock.calls.map((c) => c[0]);
// c1 (the only chat sub) must appear within the first 2 positions in
// each call, regardless of which type goes first.
expect(calls1.slice(0, 2)).toContain('c1');
vi.mocked(redditApi.getNewPosts).mockClear();
await monitor.getUnprocessedPosts();
const calls2 = vi.mocked(redditApi.getNewPosts).mock.calls.map((c) => c[0]);
expect(calls2.slice(0, 2)).toContain('c1');
// Across two calls, the leading subreddit should be a chat sub at
// least once (proves the flip — pure image-first would never hit c1
// at position 0).
expect([calls1[0], calls2[0]]).toContain('c1');
});
it('biases earlier check positions toward higher-weighted subs', async () => {
const redditApi = createMockRedditApi([]);
const postsRepo = createMockPostsRepo();
const messagesRepo = createMockMessagesRepo();
const config = getTestConfig({
reddit: {
subreddits: [
{ name: 'heavy', type: 'image', weight: 100 },
{ name: 'light1', type: 'image', weight: 1 },
{ name: 'light2', type: 'image', weight: 1 },
{ name: 'light3', type: 'image', weight: 1 },
],
},
});
monitor = new MonitorService(redditApi, postsRepo, messagesRepo, config, logger, 0);
let heavyFirstCount = 0;
const trials = 200;
for (let i = 0; i < trials; i++) {
vi.mocked(redditApi.getNewPosts).mockClear();
await monitor.getUnprocessedPosts();
const first = vi.mocked(redditApi.getNewPosts).mock.calls[0]?.[0];
if (first === 'heavy') heavyFirstCount++;
}
// Uniform would put 'heavy' first ~25% of the time. With weight 100 vs
// 1+1+1, expected ≈ 100/103 ≈ 97%. Allow generous slack for randomness.
expect(heavyFirstCount / trials).toBeGreaterThan(0.85);
});
it('passes subreddit type to redditApi.getNewPosts', async () => {
const redditApi = createMockRedditApi([]);
const postsRepo = createMockPostsRepo();
const messagesRepo = createMockMessagesRepo();
const config = getTestConfig({
reddit: {
subreddits: [
{ name: 'img-sub', type: 'image' },
{ name: 'chat-sub', type: 'chat' },
],
},
});
monitor = new MonitorService(redditApi, postsRepo, messagesRepo, config, logger, 0);
await monitor.getUnprocessedPosts();
const calls = vi.mocked(redditApi.getNewPosts).mock.calls;
const imgCall = calls.find((c) => c[0] === 'img-sub');
const chatCall = calls.find((c) => c[0] === 'chat-sub');
expect(imgCall?.[2]).toBe('image');
expect(chatCall?.[2]).toBe('chat');
});
});
describe('chat-sub gender pre-filter', () => {
function makeChatPost(id: string, title: string, selftext = ''): RedditPost {
return {
id, subreddit: 'chat-sub', author: `user_${id}`, title,
url: '', permalink: `/r/chat-sub/${id}`, thumbnail: '',
is_self: true, selftext, created_utc: Date.now() / 1000,
over_18: false, spoiler: false,
} as RedditPost;
}
it('drops chat posts whose gender tag does not match targetGenders', async () => {
const malePost = makeChatPost('t3_m', '[30M] looking for someone');
const femalePost = makeChatPost('t3_f', '[25F] bored at home');
const redditApi = createMockRedditApi([malePost, femalePost]);
const postsRepo = createMockPostsRepo();
const messagesRepo = createMockMessagesRepo();
const config = getTestConfig({
reddit: {
subreddits: [{ name: 'chat-sub', type: 'chat' }],
targetGenders: ['female'],
},
});
monitor = new MonitorService(redditApi, postsRepo, messagesRepo, config, logger, 0);
const result = await monitor.getUnprocessedPosts();
expect(result.length).toBe(1);
expect(result[0].id).toBe('t3_f');
});
it('drops chat posts with no gender tag when targetGenders is set', async () => {
const untagged = makeChatPost('t3_u', 'hey anyone wanna chat');
const tagged = makeChatPost('t3_f', '[22F] hi');
const redditApi = createMockRedditApi([untagged, tagged]);
const postsRepo = createMockPostsRepo();
const messagesRepo = createMockMessagesRepo();
const config = getTestConfig({
reddit: {
subreddits: [{ name: 'chat-sub', type: 'chat' }],
targetGenders: ['female'],
},
});
monitor = new MonitorService(redditApi, postsRepo, messagesRepo, config, logger, 0);
const result = await monitor.getUnprocessedPosts();
expect(result.length).toBe(1);
expect(result[0].id).toBe('t3_f');
});
it('does not filter image-sub posts by gender tag', async () => {
const imgPost: RedditPost = {
id: 't3_img', subreddit: 'img-sub', author: 'user1', title: 'just a pic',
url: 'http://example.com/x.jpg', permalink: '/r/img-sub/1', thumbnail: '',
is_self: false, created_utc: Date.now() / 1000, over_18: false, spoiler: false,
};
const redditApi = createMockRedditApi([imgPost]);
const postsRepo = createMockPostsRepo();
const messagesRepo = createMockMessagesRepo();
const config = getTestConfig({
reddit: {
subreddits: [{ name: 'img-sub', type: 'image' }],
targetGenders: ['female'],
},
});
monitor = new MonitorService(redditApi, postsRepo, messagesRepo, config, logger, 0);
const result = await monitor.getUnprocessedPosts();
expect(result.length).toBe(1);
expect(result[0].id).toBe('t3_img');
});
it('keeps all chat posts when targetGenders is empty', async () => {
const malePost = makeChatPost('t3_m', '[30M] hi');
const redditApi = createMockRedditApi([malePost]);
const postsRepo = createMockPostsRepo();
const messagesRepo = createMockMessagesRepo();
const config = getTestConfig({
reddit: {
subreddits: [{ name: 'chat-sub', type: 'chat' }],
targetGenders: [],
},
});
monitor = new MonitorService(redditApi, postsRepo, messagesRepo, config, logger, 0);
const result = await monitor.getUnprocessedPosts();
expect(result.length).toBe(1);
expect(result[0].id).toBe('t3_m');
});
});
});
tests/openrouter-client.test.ts
+704
View File
@@ -0,0 +1,704 @@
import { describe, it, expect, vi, beforeEach } from 'vitest';
import type OpenAI from 'openai';
import { OpenRouterClient, stripDashes } from '../src/ai/openrouter.js';
import { createMockLogger } from '../tests/mockLogger.js';
import { getTestConfig } from '../tests/testUtils.js';
describe('OpenRouterClient', () => {
let client: OpenRouterClient;
let logger: ReturnType<typeof createMockLogger>;
let mockCreate: ReturnType<typeof vi.fn>;
beforeEach(() => {
logger = createMockLogger();
const config = getTestConfig({
openrouter: {
apiKey: 'test-key',
imageAnalysisModel: 'google/gemini-flash',
imageAnalysisFallbackModels: ['fallback-1'],
chatModel: 'chat-model',
chatFallbackModels: [],
imageAnalysisPrompt: 'Analyze this image in detail.',
messageGenerationPrompt: 'Write a message for {author} about "{title}". Analysis: {analysis}',
},
});
mockCreate = vi.fn();
const mockOpenAI = {
chat: { completions: { create: mockCreate } },
} as unknown as OpenAI;
client = new OpenRouterClient(config, logger, mockOpenAI);
});
describe('isRefusal (internal)', () => {
// We test refusal detection by checking the analyzeImage behavior
// since isRefusal is private. We do this by mocking responses that
// contain refusal patterns.
it('detects "I cannot fulfill" as refusal', async () => {
mockCreate.mockResolvedValueOnce({
choices: [{ message: { content: "I cannot fulfill this request" } }],
});
await expect(
client.analyzeImage('base64data', 'image/jpeg')
).rejects.toThrow('AI_REFUSAL');
});
it('detects "I can\'t provide" as refusal', async () => {
mockCreate.mockResolvedValueOnce({
choices: [{ message: { content: "I can't provide that analysis" } }],
});
await expect(
client.analyzeImage('base64data', 'image/jpeg')
).rejects.toThrow('AI_REFUSAL');
});
it('detects "my safety guidelines" as refusal', async () => {
mockCreate.mockResolvedValueOnce({
choices: [{ message: { content: "This violates my safety guidelines" } }],
});
await expect(
client.analyzeImage('base64data', 'image/jpeg')
).rejects.toThrow('AI_REFUSAL');
});
it('detects "I\'m unable to" as refusal', async () => {
mockCreate.mockResolvedValueOnce({
choices: [{ message: { content: "I'm unable to analyze this" } }],
});
await expect(
client.analyzeImage('base64data', 'image/jpeg')
).rejects.toThrow('AI_REFUSAL');
});
it('detects "sexually explicit" as refusal', async () => {
mockCreate.mockResolvedValueOnce({
choices: [{ message: { content: "This is sexually explicit content" } }],
});
await expect(
client.analyzeImage('base64data', 'image/jpeg')
).rejects.toThrow('AI_REFUSAL');
});
it('detects "not appropriate for me" as refusal', async () => {
mockCreate.mockResolvedValueOnce({
choices: [{ message: { content: "This is not appropriate for me" } }],
});
await expect(
client.analyzeImage('base64data', 'image/jpeg')
).rejects.toThrow('AI_REFUSAL');
});
it('does NOT treat normal responses as refusal', async () => {
mockCreate.mockResolvedValueOnce({
choices: [{ message: { content: 'GENDER:female\n\nThe image shows a woman with blonde hair standing on a beach.' } }],
});
const result = await client.analyzeImage('base64data', 'image/jpeg');
expect(result.detectedGender).toBe('female');
expect(result.description).toBe('The image shows a woman with blonde hair standing on a beach.');
});
});
describe('analyzeImage - gender parsing', () => {
it('parses GENDER:female from response', async () => {
mockCreate.mockResolvedValueOnce({
choices: [{ message: { content: 'GENDER:female\n\nBeautiful woman with long brown hair.' } }],
});
const result = await client.analyzeImage('base64data', 'image/jpeg');
expect(result.detectedGender).toBe('female');
expect(result.description).toBe('Beautiful woman with long brown hair.');
});
it('parses GENDER:male from response', async () => {
mockCreate.mockResolvedValueOnce({
choices: [{ message: { content: 'GENDER: male\n\nA man in a gym.' } }],
});
const result = await client.analyzeImage('base64data', 'image/jpeg');
expect(result.detectedGender).toBe('male');
});
it('parses GENDER:other from response', async () => {
mockCreate.mockResolvedValueOnce({
choices: [{ message: { content: 'GENDER:other\n\nGroup of people.' } }],
});
const result = await client.analyzeImage('base64data', 'image/jpeg');
expect(result.detectedGender).toBe('other');
});
it('parses GENDER:unknown from response', async () => {
mockCreate.mockResolvedValueOnce({
choices: [{ message: { content: 'GENDER:unknown\n\nImage too dark to tell.' } }],
});
const result = await client.analyzeImage('base64data', 'image/jpeg');
expect(result.detectedGender).toBe('unknown');
});
it('defaults to unknown when no GENDER prefix', async () => {
mockCreate.mockResolvedValueOnce({
choices: [{ message: { content: 'A person in a photo.' } }],
});
const result = await client.analyzeImage('base64data', 'image/jpeg');
expect(result.detectedGender).toBe('unknown');
expect(result.description).toBe('A person in a photo.');
});
it('strips GENDER line from description', async () => {
mockCreate.mockResolvedValueOnce({
choices: [{ message: { content: 'GENDER:female\n\nTall and fit with blue eyes.\n\nAdditional details here.' } }],
});
const result = await client.analyzeImage('base64data', 'image/jpeg');
expect(result.description).toBe('Tall and fit with blue eyes.\n\nAdditional details here.');
});
it('parses RATING:<n> on the 1-100 scale', async () => {
mockCreate.mockResolvedValueOnce({
choices: [{ message: { content: 'GENDER:female\nRATING:73\n\nDescription here.' } }],
});
const result = await client.analyzeImage('base64data', 'image/jpeg');
expect(result.attractivenessScore).toBe(73);
expect(result.description).toBe('Description here.');
});
it('returns null attractivenessScore when no RATING line', async () => {
mockCreate.mockResolvedValueOnce({
choices: [{ message: { content: 'GENDER:female\n\nNo rating here.' } }],
});
const result = await client.analyzeImage('base64data', 'image/jpeg');
expect(result.attractivenessScore).toBeNull();
});
it('parses RATING when wrapped in markdown bold', async () => {
mockCreate.mockResolvedValueOnce({
choices: [{ message: { content: 'GENDER:female\n**RATING:** 63\nREASON: cute face.\n\nDescription here.' } }],
});
const result = await client.analyzeImage('base64data', 'image/jpeg');
expect(result.attractivenessScore).toBe(63);
expect(result.description).not.toMatch(/RATING/i);
});
it('parses RATING when the number itself is bolded', async () => {
mockCreate.mockResolvedValueOnce({
choices: [{ message: { content: 'GENDER:female\nRATING: **78**\n\nDescription here.' } }],
});
const result = await client.analyzeImage('base64data', 'image/jpeg');
expect(result.attractivenessScore).toBe(78);
});
it('rejects out-of-range RATING values', async () => {
mockCreate.mockResolvedValueOnce({
choices: [{ message: { content: 'GENDER:female\nRATING:250\n\nBad rating.' } }],
});
const result = await client.analyzeImage('base64data', 'image/jpeg');
expect(result.attractivenessScore).toBeNull();
});
it('handles case-insensitive GENDER prefix', async () => {
mockCreate.mockResolvedValueOnce({
choices: [{ message: { content: 'GENDER:FEMALE\n\nA woman.' } }],
});
const result = await client.analyzeImage('base64data', 'image/jpeg');
expect(result.detectedGender).toBe('female');
});
});
describe('analyzeImage - fallback models', () => {
it('tries fallback model when primary refuses', async () => {
// First call refuses, second succeeds
mockCreate
.mockResolvedValueOnce({ choices: [{ message: { content: 'I cannot fulfill this request' } }] })
.mockResolvedValueOnce({
choices: [{ message: { content: 'GENDER:female\n\nFallback succeeded.' } }],
});
const result = await client.analyzeImage('base64data', 'image/jpeg');
expect(result.detectedGender).toBe('female');
expect(mockCreate).toHaveBeenCalledTimes(2);
});
it('throws AI_REFUSAL when all models refuse', async () => {
mockCreate
.mockResolvedValueOnce({ choices: [{ message: { content: 'I cannot fulfill this request' } }] })
.mockResolvedValueOnce({ choices: [{ message: { content: 'I cannot provide that either' } }] });
await expect(client.analyzeImage('base64data', 'image/jpeg'))
.rejects.toThrow('AI_REFUSAL: All models refused to analyze image');
expect(mockCreate).toHaveBeenCalledTimes(2);
});
it('throws AI_REFUSAL when primary throws API error and fallback refuses', async () => {
mockCreate
.mockRejectedValueOnce(new Error('API error'))
.mockResolvedValueOnce({ choices: [{ message: { content: 'I cannot fulfill that' } }] });
await expect(client.analyzeImage('base64data', 'image/jpeg'))
.rejects.toThrow('AI_REFUSAL');
});
it('logs fallback usage', async () => {
mockCreate
.mockResolvedValueOnce({ choices: [{ message: { content: 'I cannot fulfill this' } }] })
.mockResolvedValueOnce({
choices: [{ message: { content: 'GENDER:unknown\n\nFallback worked.' } }],
});
await client.analyzeImage('base64data', 'image/jpeg');
expect(logger.logs.some(l => l.message.includes('Fallback model'))).toBe(true);
});
});
describe('analyzeImage - error handling', () => {
it('throws when all models throw errors', async () => {
mockCreate.mockRejectedValue(new Error('API connection failed'));
await expect(client.analyzeImage('base64data', 'image/jpeg'))
.rejects.toThrow('API connection failed');
});
it('handles empty response content', async () => {
mockCreate.mockResolvedValueOnce({
choices: [{ message: { content: '' } }],
});
const result = await client.analyzeImage('base64data', 'image/jpeg');
expect(result.detectedGender).toBe('unknown');
expect(result.description).toBe('');
});
it('handles undefined response content', async () => {
mockCreate.mockResolvedValueOnce({
choices: [{ message: { content: undefined as unknown as string } }],
});
const result = await client.analyzeImage('base64data', 'image/jpeg');
expect(result.detectedGender).toBe('unknown');
});
});
describe('detectGenderOnly', () => {
it('returns female for "female" response', async () => {
mockCreate.mockResolvedValueOnce({
choices: [{ message: { content: 'female' } }],
});
const result = await client.detectGenderOnly('base64data', 'image/jpeg');
expect(result).toBe('female');
});
it('returns male for "male" response', async () => {
mockCreate.mockResolvedValueOnce({
choices: [{ message: { content: 'male' } }],
});
const result = await client.detectGenderOnly('base64data', 'image/jpeg');
expect(result).toBe('male');
});
it('returns other for "other" response', async () => {
mockCreate.mockResolvedValueOnce({
choices: [{ message: { content: 'other' } }],
});
const result = await client.detectGenderOnly('base64data', 'image/jpeg');
expect(result).toBe('other');
});
it('returns unknown for empty response', async () => {
mockCreate.mockResolvedValueOnce({
choices: [{ message: { content: '' } }],
});
const result = await client.detectGenderOnly('base64data', 'image/jpeg');
expect(result).toBe('unknown');
});
it('handles API errors gracefully', async () => {
mockCreate.mockRejectedValue(new Error('API error'));
const result = await client.detectGenderOnly('base64data', 'image/jpeg');
expect(result).toBe('unknown');
});
it('handles malformed response', async () => {
mockCreate.mockResolvedValueOnce({
choices: [{ message: { content: null as unknown as string } }],
});
const result = await client.detectGenderOnly('base64data', 'image/jpeg');
expect(result).toBe('unknown');
});
});
describe('detectUpsell', () => {
it('detects OnlyFans keyword', () => {
const result = client.detectUpsell(
[{ text: 'Check out my OnlyFans!', isSelf: false }],
['onlyfans']
);
expect(result.isUpsell).toBe(true);
expect(result.matchedKeyword).toBe('onlyfans');
});
it('detects case-insensitive keyword', () => {
const result = client.detectUpsell(
[{ text: 'Visit my FANSLY page', isSelf: false }],
['fansly']
);
expect(result.isUpsell).toBe(true);
});
it('ignores own messages', () => {
const result = client.detectUpsell(
[{ text: 'Check out my OnlyFans!', isSelf: true }],
['onlyfans']
);
expect(result.isUpsell).toBe(false);
});
it('returns false when no keywords match', () => {
const result = client.detectUpsell(
[{ text: 'Hey, how are you?', isSelf: false }],
['onlyfans', 'fansly']
);
expect(result.isUpsell).toBe(false);
expect(result.matchedKeyword).toBeUndefined();
});
it('returns false for empty messages', () => {
const result = client.detectUpsell([], ['onlyfans']);
expect(result.isUpsell).toBe(false);
});
it('handles mixed sender messages', () => {
const result = client.detectUpsell([
{ text: 'Hey there!', isSelf: true },
{ text: 'Thanks! Check my OnlyFans', isSelf: false },
{ text: 'Oh nice', isSelf: true },
], ['onlyfans']);
expect(result.isUpsell).toBe(true);
});
it('detects multiple possible keywords (returns first match)', () => {
const result = client.detectUpsell(
[{ text: 'OnlyFans and Fansly', isSelf: false }],
['onlyfans', 'fansly']
);
expect(result.isUpsell).toBe(true);
// First matching keyword in the array wins
});
it('handles partial keyword matching', () => {
const result = client.detectUpsell(
[{ text: 'Send me cashapp', isSelf: false }],
['cashapp']
);
expect(result.isUpsell).toBe(true);
});
});
describe('generateMessage - prompt assembly', () => {
it('substitutes analysis, title, and author into prompt', async () => {
mockCreate.mockResolvedValueOnce({
choices: [{ message: { content: 'Hey gorgeous!' } }],
});
await client.generateMessage(
'Blonde woman, rating 8/10',
'My beach selfie',
'beachgirl123',
[],
undefined
);
const callArgs = mockCreate.mock.calls[0][0];
const prompt = callArgs.messages[0].content as string;
expect(prompt).toContain('Blonde woman, rating 8/10');
expect(prompt).toContain('My beach selfie');
expect(prompt).toContain('beachgirl123');
});
it('appends strategy prompt override', async () => {
mockCreate.mockResolvedValueOnce({
choices: [{ message: { content: 'Hey!' } }],
});
await client.generateMessage(
'Analysis',
'Title',
'Author',
[],
'Be direct and short.'
);
const callArgs = mockCreate.mock.calls[0][0];
const prompt = callArgs.messages[0].content as string;
expect(prompt).toContain('Be direct and short.');
});
it('includes successful examples in prompt', async () => {
mockCreate.mockResolvedValueOnce({
choices: [{ message: { content: 'Hey!' } }],
});
await client.generateMessage(
'Analysis',
'Title',
'Author',
['Example 1: Hey you look great', 'Example 2: Love your pics'],
undefined
);
const callArgs = mockCreate.mock.calls[0][0];
const prompt = callArgs.messages[0].content as string;
expect(prompt).toContain('Example 1:');
expect(prompt).toContain('Example 2:');
expect(prompt).toContain('Hey you look great');
expect(prompt).toContain('positive responses');
});
it('substitutes {rating} placeholder when score provided', async () => {
const cfg = getTestConfig({
openrouter: {
apiKey: 'test-key',
imageAnalysisModel: 'google/gemini-flash',
imageAnalysisFallbackModels: [],
chatModel: 'chat-model',
chatFallbackModels: [],
imageAnalysisPrompt: 'analyze',
messageGenerationPrompt: 'Rating: {rating}. {analysis}',
},
});
const localClient = new OpenRouterClient(cfg, logger, { chat: { completions: { create: mockCreate } } } as unknown as OpenAI);
mockCreate.mockResolvedValueOnce({
choices: [{ message: { content: 'Hey!' } }],
});
await localClient.generateMessage('desc', 't', 'a', [], undefined, 73);
const prompt = mockCreate.mock.calls[0][0].messages[0].content as string;
expect(prompt).toContain('Rating: 73');
});
it('substitutes {rating} with "unknown" when score is null', async () => {
const cfg = getTestConfig({
openrouter: {
apiKey: 'test-key',
imageAnalysisModel: 'google/gemini-flash',
imageAnalysisFallbackModels: [],
chatModel: 'chat-model',
chatFallbackModels: [],
imageAnalysisPrompt: 'analyze',
messageGenerationPrompt: 'Rating: {rating}. {analysis}',
},
});
const localClient = new OpenRouterClient(cfg, logger, { chat: { completions: { create: mockCreate } } } as unknown as OpenAI);
mockCreate.mockResolvedValueOnce({
choices: [{ message: { content: 'Hey!' } }],
});
await localClient.generateMessage('desc', 't', 'a', [], undefined, null);
const prompt = mockCreate.mock.calls[0][0].messages[0].content as string;
expect(prompt).toContain('Rating: unknown');
});
it('sets correct model and max_tokens', async () => {
mockCreate.mockResolvedValueOnce({
choices: [{ message: { content: 'Hey!' } }],
});
await client.generateMessage('Analysis', 'Title', 'Author', [], undefined);
const callArgs = mockCreate.mock.calls[0][0];
expect(callArgs.model).toBe('chat-model');
expect(callArgs.max_tokens).toBe(300);
expect(callArgs.temperature).toBe(0.8);
});
it('returns trimmed message', async () => {
mockCreate.mockResolvedValueOnce({
choices: [{ message: { content: ' Hey there! ' } }],
});
const result = await client.generateMessage('Analysis', 'Title', 'Author', [], undefined);
expect(result).toBe('Hey there!');
});
it('handles empty message response', async () => {
mockCreate.mockResolvedValueOnce({
choices: [{ message: { content: '' } }],
});
const result = await client.generateMessage('Analysis', 'Title', 'Author', [], undefined);
expect(result).toBe('');
});
it('handles undefined message response', async () => {
mockCreate.mockResolvedValueOnce({
choices: [{ message: { content: undefined as unknown as string } }],
});
const result = await client.generateMessage('Analysis', 'Title', 'Author', [], undefined);
expect(result).toBe('');
});
});
describe('generateReply', () => {
it('includes conversation history in prompt', async () => {
mockCreate.mockResolvedValueOnce({
choices: [{ message: { content: 'Thanks!' } }],
});
await client.generateReply(
[
{ sender: 'You', text: 'Hey gorgeous!', isSelf: true },
{ sender: 'user123', text: 'Aww thanks!', isSelf: false },
],
'user123'
);
const callArgs = mockCreate.mock.calls[0][0];
const prompt = callArgs.messages[0].content as string;
expect(prompt).toContain('Hey gorgeous!');
expect(prompt).toContain('Aww thanks!');
expect(prompt).toContain('user123');
});
it('uses chat model with correct params', async () => {
mockCreate.mockResolvedValueOnce({
choices: [{ message: { content: 'Reply' } }],
});
await client.generateReply([], 'user');
const callArgs = mockCreate.mock.calls[0][0];
expect(callArgs.model).toBe('chat-model');
expect(callArgs.max_tokens).toBe(300);
expect(callArgs.temperature).toBe(0.8);
});
});
describe('analyzeAndGenerateMessage', () => {
it('calls analyzeImage then generateMessage', async () => {
mockCreate
.mockResolvedValueOnce({
choices: [{ message: { content: 'GENDER:female\n\nBlonde woman.' } }],
})
.mockResolvedValueOnce({
choices: [{ message: { content: 'Hey beautiful!' } }],
});
const result = await client.analyzeAndGenerateMessage(
'base64data',
'image/jpeg',
'My selfie',
'selfiegirl'
);
expect(result.analysis.detectedGender).toBe('female');
expect(result.analysis.description).toBe('Blonde woman.');
expect(result.message).toBe('Hey beautiful!');
expect(mockCreate).toHaveBeenCalledTimes(2);
});
});
describe('generateMediaAsk', () => {
it('builds a prompt including the conversation history and recipient username', async () => {
mockCreate.mockResolvedValueOnce({
choices: [{ message: { content: 'bet you wont send a pic of that' } }],
});
const result = await client.generateMediaAsk(
[
{ sender: 'them', text: 'just got out of the shower', isSelf: false },
{ sender: 'me', text: 'lucky you', isSelf: true },
{ sender: 'them', text: 'haha yeah', isSelf: false },
],
'them',
);
expect(result).toBe('bet you wont send a pic of that');
expect(mockCreate).toHaveBeenCalledTimes(1);
const prompt = mockCreate.mock.calls[0][0].messages[0].content as string;
expect(prompt).toContain('them: just got out of the shower');
expect(prompt).toContain('You: lucky you');
expect(prompt).toContain('send to them');
expect(prompt).toMatch(/pic or video/i);
});
it('falls back to next model when primary refuses', async () => {
const config = getTestConfig({
openrouter: {
apiKey: 'test-key',
imageAnalysisModel: 'google/gemini-flash',
imageAnalysisFallbackModels: ['fallback-1'],
chatModel: 'chat-model',
chatFallbackModels: ['chat-fallback'],
imageAnalysisPrompt: 'Analyze this image in detail.',
messageGenerationPrompt: 'Write a message.',
},
});
const mockOpenAI = { chat: { completions: { create: mockCreate } } } as unknown as OpenAI;
const c = new OpenRouterClient(config, logger, mockOpenAI);
mockCreate
.mockResolvedValueOnce({ choices: [{ message: { content: 'I cannot generate that' } }] })
.mockResolvedValueOnce({ choices: [{ message: { content: 'show me 😏' } }] });
const result = await c.generateMediaAsk(
[{ sender: 'them', text: 'hi', isSelf: false }],
'them',
);
expect(result).toBe('show me 😏');
expect(mockCreate).toHaveBeenCalledTimes(2);
});
it('throws when the model errors', async () => {
mockCreate.mockRejectedValueOnce(new Error('network'));
await expect(
client.generateMediaAsk([{ sender: 'them', text: 'hi', isSelf: false }], 'them'),
).rejects.toThrow('network');
});
});
});
describe('stripDashes', () => {
it('replaces em dash with comma', () => {
expect(stripDashes('32M UK — bored tonight')).toBe('32M UK, bored tonight');
});
it('replaces en dash with comma', () => {
expect(stripDashes('hey – what are you into?')).toBe('hey, what are you into?');
});
it('collapses comma when dash sits before a full stop', () => {
expect(stripDashes('your post caught my eye — .')).toBe('your post caught my eye.');
});
it('leaves clean text untouched', () => {
expect(stripDashes('33M from the UK, your vibe is my type.')).toBe('33M from the UK, your vibe is my type.');
});
it('handles multiple dashes', () => {
expect(stripDashes('a — b — c')).toBe('a, b, c');
});
});
tests/processor.test.ts
+349
View File
@@ -0,0 +1,349 @@
import { describe, it, expect, vi, beforeEach } from 'vitest';
import { ProcessorService, isSkip, type ImageFetcher } from '../src/services/processor.js';
import type { RedditApiClient, RedditPost } from '../src/reddit/api.js';
import type { OpenRouterClient, ImageAnalysisResult, DetectedGender } from '../src/ai/openrouter.js';
// Returns a buffer large enough to pass ProcessorService's >5000-byte size check.
const fakeFetchImage: ImageFetcher = async () => ({
buffer: Buffer.alloc(6000, 0xff),
mimeType: 'image/jpeg',
});
import type { StrategySelector, SelectedStrategy } from '../src/services/strategy-selector.js';
import { createMockLogger } from '../tests/mockLogger.js';
import { getTestConfig } from '../tests/testUtils.js';
function createMockRedditApi(imageUrl: string | null = 'http://example.com/img.jpg'): RedditApiClient {
return {
getNewPosts: vi.fn(async () => []),
getImageUrl: vi.fn(() => imageUrl),
getPostUrl: vi.fn(() => 'https://www.reddit.com/test'),
isSellerProfile: vi.fn(async () => false),
} as unknown as RedditApiClient;
}
function createMockAiClient(
gender: DetectedGender = 'female',
analysis: ImageAnalysisResult = { description: 'Beautiful woman', detectedGender: 'female', attractivenessScore: null },
message: string = 'Hey gorgeous!'
): OpenRouterClient {
return {
detectGenderOnly: vi.fn(async () => gender),
analyzeImage: vi.fn(async () => analysis),
generateMessage: vi.fn(async (analysis: string) => message),
generateReply: vi.fn(async () => 'Reply'),
detectUpsell: vi.fn(() => ({ isUpsell: false })),
analyzeAndGenerateMessage: vi.fn(async (base64: string, mimeType: string, title: string, author: string, examples?: string[], strategyPrompt?: string) => {
return { analysis, message };
}),
} as unknown as OpenRouterClient;
}
function createMockStrategySelector(
selected: SelectedStrategy | null = { id: 'v1', promptOverride: 'test', reason: 'explore' },
hasStrategies: boolean = true
): StrategySelector {
return {
selectStrategy: vi.fn(() => selected),
getStats: vi.fn(() => []),
hasStrategies: vi.fn(() => hasStrategies),
} as unknown as StrategySelector;
}
describe('ProcessorService', () => {
let processor: ProcessorService;
let logger: ReturnType<typeof createMockLogger>;
let redditApi: RedditApiClient;
let aiClient: OpenRouterClient;
const mockPost: RedditPost = {
id: 't3_test',
name: 't3_test',
subreddit: 'test',
author: 'testuser',
title: 'Test Post',
url: 'http://example.com/img.jpg',
permalink: '/r/test/comments/abc/test',
thumbnail: 'http://example.com/thumb.jpg',
is_self: false,
created_utc: Date.now() / 1000,
over_18: false,
spoiler: false,
};
beforeEach(() => {
logger = createMockLogger();
redditApi = createMockRedditApi();
aiClient = createMockAiClient();
const config = getTestConfig({
reddit: { targetGenders: ['female'] },
});
processor = new ProcessorService(redditApi, aiClient, config, logger, fakeFetchImage);
});
it('processes a valid post successfully', async () => {
const result = await processor.processPost(mockPost);
if (isSkip(result)) throw new Error(`expected success, got skip: ${result.skipReason}`);
expect(result.post.id).toBe('t3_test');
expect(result.imageAnalysis).toBe('Beautiful woman');
expect(result.detectedGender).toBe('female');
expect(result.generatedMessage).toBe('Hey gorgeous!');
});
it('returns null when no image URL', async () => {
redditApi = createMockRedditApi(null);
const config = getTestConfig();
processor = new ProcessorService(redditApi, aiClient, config, logger, fakeFetchImage);
const result = await processor.processPost(mockPost);
expect(isSkip(result)).toBe(true);
expect(logger.logs.some(l => l.level === 'warn' && l.message.includes('No image URL'))).toBe(true);
});
it('skips non-matching gender', async () => {
aiClient = createMockAiClient('male');
const config = getTestConfig({ reddit: { targetGenders: ['female'] } });
processor = new ProcessorService(redditApi, aiClient, config, logger, fakeFetchImage);
const result = await processor.processPost(mockPost);
expect(isSkip(result)).toBe(true);
expect(logger.logs.some(l => l.message.includes('Skipping post'))).toBe(true);
});
it('skips when targetGenders is empty (all pass)', async () => {
aiClient = createMockAiClient('male');
const config = getTestConfig({ reddit: { targetGenders: [] } });
processor = new ProcessorService(redditApi, aiClient, config, logger, fakeFetchImage);
const result = await processor.processPost(mockPost);
expect(isSkip(result)).toBe(false);
});
it('skips when AI refuses', async () => {
const error = new Error('AI_REFUSAL: All models refused to analyze image');
vi.mocked(aiClient.analyzeAndGenerateMessage).mockRejectedValue(error);
const result = await processor.processPost(mockPost);
expect(isSkip(result)).toBe(true);
expect(logger.logs.some(l => l.message.includes('AI refused'))).toBe(true);
});
it('re-throws non-AI_REFUSAL errors', async () => {
const error = new Error('Network error');
vi.mocked(aiClient.analyzeAndGenerateMessage).mockRejectedValue(error);
await expect(processor.processPost(mockPost)).rejects.toThrow('Network error');
});
it('skips when generated message is too short', async () => {
aiClient = createMockAiClient('female',
{ description: 'Nice', detectedGender: 'female' },
'Short'
);
const config = getTestConfig();
processor = new ProcessorService(redditApi, aiClient, config, logger, fakeFetchImage);
const result = await processor.processPost(mockPost);
expect(isSkip(result)).toBe(true);
expect(logger.logs.some(l => l.message.includes('too short'))).toBe(true);
});
it('skips when generated message is empty', async () => {
aiClient = createMockAiClient('female',
{ description: 'Nice', detectedGender: 'female' },
''
);
const config = getTestConfig();
processor = new ProcessorService(redditApi, aiClient, config, logger, fakeFetchImage);
const result = await processor.processPost(mockPost);
expect(isSkip(result)).toBe(true);
});
it('includes strategy ID in result when strategy selector is set', async () => {
const strategySelector = createMockStrategySelector({
id: 'test-strategy',
promptOverride: 'Be direct',
reason: 'exploit',
});
processor.setStrategySelector(strategySelector);
const result = await processor.processPost(mockPost);
if (isSkip(result)) throw new Error(`expected success, got skip: ${result.skipReason}`);
expect(result.strategyId).toBe('test-strategy');
});
it('has null strategyId when no strategy selector', async () => {
const result = await processor.processPost(mockPost);
if (isSkip(result)) throw new Error(`expected success, got skip: ${result.skipReason}`);
expect(result.strategyId).toBeNull();
});
it('passes successful examples to message generation', async () => {
processor.setSuccessfulExamples(['Example 1: Hey', 'Example 2: Nice']);
await processor.processPost(mockPost);
expect(aiClient.analyzeAndGenerateMessage).toHaveBeenCalledWith(
expect.any(String), // base64
'image/jpeg',
'Test Post',
'testuser',
['Example 1: Hey', 'Example 2: Nice'],
undefined
);
});
it('passes strategy prompt to message generation', async () => {
const strategySelector = createMockStrategySelector({
id: 'v1',
promptOverride: 'Be teasing',
reason: 'explore',
});
processor.setStrategySelector(strategySelector);
await processor.processPost(mockPost);
expect(aiClient.analyzeAndGenerateMessage).toHaveBeenCalledWith(
expect.any(String),
'image/jpeg',
'Test Post',
'testuser',
[],
'Be teasing'
);
});
it('logs when strategy selector is enabled', async () => {
const strategySelector = createMockStrategySelector(null, true);
processor.setStrategySelector(strategySelector);
expect(logger.logs.some(l => l.message.includes('Strategy A/B testing enabled'))).toBe(true);
});
it('logs when successful examples are loaded', async () => {
processor.setSuccessfulExamples(['Example 1', 'Example 2', 'Example 3']);
expect(logger.logs.some(l => l.message.includes('Loaded 3 successful'))).toBe(true);
});
it('logs when image is too small', async () => {
// We can't easily mock the image download to return a small buffer
// without also mocking axios. Let's verify the logic path exists.
const config = getTestConfig();
const proc = new ProcessorService(redditApi, aiClient, config, logger);
// The image download mock is in axios, which is already mocked via the aiClient
// We need to mock downloadImage separately. For now, just verify the service exists.
expect(proc).toBeDefined();
});
describe('setSuccessfulExamples', () => {
it('updates examples for next processing', async () => {
processor.setSuccessfulExamples(['Example A']);
await processor.processPost(mockPost);
expect(aiClient.analyzeAndGenerateMessage).toHaveBeenCalledWith(
expect.any(String),
'image/jpeg',
'Test Post',
'testuser',
['Example A'],
undefined
);
});
it('logs when no examples provided', async () => {
processor.setSuccessfulExamples([]);
const noLog = logger.logs.some(l => l.message.includes('Loaded'));
expect(noLog).toBe(false);
});
});
describe('chat-type subreddits', () => {
const chatPost: RedditPost = {
...mockPost,
id: 't3_chat',
subreddit: 'SextingRush',
title: '[25F] bored and horny, dm me',
selftext: 'looking for someone fun tonight',
is_self: true,
};
function makeChatConfig(overrides?: { targetGenders?: ('male' | 'female' | 'other' | 'unknown')[] }) {
return getTestConfig({
reddit: {
subreddits: [{ name: 'SextingRush', type: 'chat' }],
targetGenders: overrides?.targetGenders ?? ['female'],
},
});
}
it('uses chat opener path and does not download images', async () => {
const fetchImage = vi.fn(fakeFetchImage);
const config = makeChatConfig();
const ai = {
...createMockAiClient(),
generateChatOpener: vi.fn(async () => 'Hey [25F], your post screamed bored-and-horny — what are you into?'),
} as unknown as OpenRouterClient;
processor = new ProcessorService(redditApi, ai, config, logger, fetchImage);
const result = await processor.processPost(chatPost);
if (isSkip(result)) throw new Error(`expected success, got skip: ${result.skipReason}`);
expect(result.detectedGender).toBe('female');
expect(result.generatedMessage).toContain('bored');
expect(fetchImage).not.toHaveBeenCalled();
expect(ai.analyzeAndGenerateMessage).not.toHaveBeenCalled();
expect(ai.generateChatOpener).toHaveBeenCalledWith(
chatPost.title,
chatPost.selftext,
chatPost.author,
'female',
[],
undefined,
);
});
it('uses chat strategy selector when set', async () => {
const config = makeChatConfig();
const ai = {
...createMockAiClient(),
generateChatOpener: vi.fn(async () => 'Hey, what kind of trouble are you looking for tonight?'),
} as unknown as OpenRouterClient;
processor = new ProcessorService(redditApi, ai, config, logger, fakeFetchImage);
const chatSelector = createMockStrategySelector({
id: 'chat_playful',
promptOverride: 'Be playful',
reason: 'explore',
});
processor.setChatStrategySelector(chatSelector);
const result = await processor.processPost(chatPost);
if (isSkip(result)) throw new Error(`expected success, got skip: ${result.skipReason}`);
expect(result.strategyId).toBe('chat_playful');
expect(ai.generateChatOpener).toHaveBeenCalledWith(
chatPost.title,
chatPost.selftext,
chatPost.author,
'female',
[],
'Be playful',
);
});
});
describe('setStrategySelector', () => {
it('does not log when no strategies configured', async () => {
const strategySelector = createMockStrategySelector(null, false);
processor.setStrategySelector(strategySelector);
const logExists = logger.logs.some(l => l.message.includes('Strategy A/B testing'));
expect(logExists).toBe(false);
});
});
});
tests/prompts.test.ts
+223
View File
@@ -0,0 +1,223 @@
import { describe, it, expect } from 'vitest';
import * as fs from 'fs';
import * as path from 'path';
import { ConfigSchema } from '../src/config/index.js';
describe('Prompt Testing', () => {
const configPath = path.resolve(process.cwd(), 'config.yaml');
describe('config.yaml prompts', () => {
it('loads successfully', () => {
expect(fs.existsSync(configPath)).toBe(true);
const content = fs.readFileSync(configPath, 'utf-8');
expect(content).toContain('imageAnalysisPrompt');
expect(content).toContain('messageGenerationPrompt');
});
it('imageAnalysisPrompt requests body features', () => {
const content = fs.readFileSync(configPath, 'utf-8');
const prompts = parseYamlPrompts(content);
expect(prompts.imageAnalysisPrompt).toContain('breasts');
expect(prompts.imageAnalysisPrompt).toContain('ASS');
expect(prompts.imageAnalysisPrompt).toContain('vagina');
expect(prompts.imageAnalysisPrompt).toContain('BODY');
expect(prompts.imageAnalysisPrompt).toContain('POSE');
expect(prompts.imageAnalysisPrompt).toContain('VIBE');
expect(prompts.imageAnalysisPrompt).toContain('EXTRAS');
});
it('messageGenerationPrompt has proper structure', () => {
const content = fs.readFileSync(configPath, 'utf-8');
const prompts = parseYamlPrompts(content);
expect(prompts.messageGenerationPrompt).toContain('{analysis}');
expect(prompts.messageGenerationPrompt).toContain('{title}');
expect(prompts.messageGenerationPrompt).toContain('{author}');
});
it('messageGenerationPrompt specifies DO NOT list', () => {
const content = fs.readFileSync(configPath, 'utf-8');
const prompts = parseYamlPrompts(content);
expect(prompts.messageGenerationPrompt).toContain('DO NOT');
expect(prompts.messageGenerationPrompt).toContain('captivating');
expect(prompts.messageGenerationPrompt).toContain('alluring');
expect(prompts.messageGenerationPrompt).toContain('exquisite');
expect(prompts.messageGenerationPrompt).toContain('radiant');
});
it('messageGenerationPrompt specifies GOOD HOOKS', () => {
const content = fs.readFileSync(configPath, 'utf-8');
const prompts = parseYamlPrompts(content);
expect(prompts.messageGenerationPrompt).toContain('GOOD HOOKS');
expect(prompts.messageGenerationPrompt).toContain('BAD HOOKS');
});
it('messageGenerationPrompt specifies language requirements', () => {
const content = fs.readFileSync(configPath, 'utf-8');
const prompts = parseYamlPrompts(content);
expect(prompts.messageGenerationPrompt).toContain('explicit terms');
expect(prompts.messageGenerationPrompt).toContain('tits');
expect(prompts.messageGenerationPrompt).toContain('ass');
expect(prompts.messageGenerationPrompt).toContain('pussy');
expect(prompts.messageGenerationPrompt).toContain('boobs');
});
it('messageGenerationPrompt specifies what to avoid', () => {
const content = fs.readFileSync(configPath, 'utf-8');
const prompts = parseYamlPrompts(content);
expect(prompts.messageGenerationPrompt).toContain('breasts');
expect(prompts.messageGenerationPrompt).toContain('buttocks');
expect(prompts.messageGenerationPrompt).toContain('posterior');
expect(prompts.messageGenerationPrompt).toContain('physique');
});
});
describe('strategy prompts', () => {
it('loads strategy variants', () => {
const content = fs.readFileSync(configPath, 'utf-8');
const yaml = require('yaml');
const config = yaml.parse(content);
expect(config.strategies.variants).toBeDefined();
expect(config.strategies.variants.length).toBeGreaterThan(0);
});
it('each strategy has id and promptOverride', () => {
const content = fs.readFileSync(configPath, 'utf-8');
const yaml = require('yaml');
const config = yaml.parse(content);
for (const variant of config.strategies.variants) {
expect(variant.id).toBeDefined();
expect(variant.promptOverride).toBeDefined();
expect(typeof variant.id).toBe('string');
expect(typeof variant.promptOverride).toBe('string');
expect(variant.id.length).toBeGreaterThan(0);
expect(variant.promptOverride.length).toBeGreaterThan(0);
}
});
it('strategy prompts are distinct', () => {
const content = fs.readFileSync(configPath, 'utf-8');
const yaml = require('yaml');
const config = yaml.parse(content);
const prompts = config.strategies.variants.map(v => v.promptOverride.toLowerCase());
const uniquePrompts = new Set(prompts);
expect(uniquePrompts.size).toBe(prompts.length);
});
it('strategy variants have valid IDs', () => {
const content = fs.readFileSync(configPath, 'utf-8');
const yaml = require('yaml');
const config = yaml.parse(content);
for (const variant of config.strategies.variants) {
expect(variant.id).toMatch(/^[a-z0-9_]+$/);
}
});
});
describe('prompt variable substitution', () => {
it('imageAnalysisPrompt contains all required variables', () => {
const content = fs.readFileSync(configPath, 'utf-8');
const yaml = require('yaml');
const config = yaml.parse(content);
// The image analysis prompt should not contain unsubstitutable variables
// (it receives raw image data, not text variables)
expect(config.openrouter.imageAnalysisPrompt).toBeDefined();
});
it('messageGenerationPrompt contains all substitution variables', () => {
const content = fs.readFileSync(configPath, 'utf-8');
const yaml = require('yaml');
const config = yaml.parse(content);
expect(config.openrouter.messageGenerationPrompt).toContain('{analysis}');
expect(config.openrouter.messageGenerationPrompt).toContain('{title}');
expect(config.openrouter.messageGenerationPrompt).toContain('{author}');
});
it('all substitution variables are in {variable} format', () => {
const content = fs.readFileSync(configPath, 'utf-8');
const yaml = require('yaml');
const config = yaml.parse(content);
const variables = config.openrouter.messageGenerationPrompt.match(/\{(\w+)\}/g);
expect(variables).toEqual(expect.arrayContaining(['{analysis}', '{title}', '{author}']));
});
});
describe('prompt quality checks', () => {
it('messageGenerationPrompt is not empty', () => {
const content = fs.readFileSync(configPath, 'utf-8');
const yaml = require('yaml');
const config = yaml.parse(content);
expect(config.openrouter.messageGenerationPrompt.trim().length).toBeGreaterThan(50);
});
it('imageAnalysisPrompt is not empty', () => {
const content = fs.readFileSync(configPath, 'utf-8');
const yaml = require('yaml');
const config = yaml.parse(content);
expect(config.openrouter.imageAnalysisPrompt.trim().length).toBeGreaterThan(50);
});
it('messageGenerationPrompt has examples', () => {
const content = fs.readFileSync(configPath, 'utf-8');
const yaml = require('yaml');
const config = yaml.parse(content);
expect(config.openrouter.messageGenerationPrompt).toContain('Examples of good messages');
expect(config.openrouter.messageGenerationPrompt).toContain('"Damn those tits');
});
it('imageAnalysisPrompt requests a rating', () => {
const content = fs.readFileSync(configPath, 'utf-8');
const yaml = require('yaml');
const config = yaml.parse(content);
expect(config.openrouter.imageAnalysisPrompt).toContain('rate her attractiveness');
expect(config.openrouter.imageAnalysisPrompt).toContain('1-100 scale');
expect(config.openrouter.imageAnalysisPrompt).toContain('RATING:');
});
});
describe('seller keywords', () => {
function loadParsedConfig() {
const content = fs.readFileSync(configPath, 'utf-8');
const yaml = require('yaml');
const rawConfig = yaml.parse(content);
rawConfig.openrouter = rawConfig.openrouter || {};
rawConfig.openrouter.apiKey = rawConfig.openrouter.apiKey || 'test';
return ConfigSchema.parse(rawConfig);
}
it('reddit sellerKeywords includes common platforms', () => {
const config = loadParsedConfig();
const keywords = config.reddit.sellerKeywords.map((k: string) => k.toLowerCase());
expect(keywords).toContain('onlyfans');
expect(keywords).toContain('fansly');
expect(keywords).toContain('manyvids');
expect(keywords).toContain('cash.app');
expect(keywords).toContain('venmo');
});
});
});
function parseYamlPrompts(content: string): { imageAnalysisPrompt: string; messageGenerationPrompt: string } {
const yaml = require('yaml');
const config = yaml.parse(content);
return {
imageAnalysisPrompt: config.openrouter.imageAnalysisPrompt,
messageGenerationPrompt: config.openrouter.messageGenerationPrompt,
};
}
tests/rate-limiter.test.ts
+195
View File
@@ -0,0 +1,195 @@
import { describe, it, expect, beforeEach } from 'vitest';
import { RateLimiter } from '../src/utils/rate-limiter.js';
import { createMockLogger } from '../tests/mockLogger.js';
import { getTestConfig } from '../tests/testUtils.js';
describe('RateLimiter', () => {
let limiter: RateLimiter;
let logger: ReturnType<typeof createMockLogger>;
beforeEach(() => {
logger = createMockLogger();
const config = getTestConfig({
rateLimit: { minDelayBetweenMessagesMs: 100, maxMessagesPerHour: 10, jitterMs: 50 },
});
limiter = new RateLimiter(config, logger);
});
it('allows immediate first call', async () => {
const start = Date.now();
await limiter.throttle(() => Promise.resolve('done'));
const elapsed = Date.now() - start;
expect(elapsed).toBeLessThan(50);
});
it('enforces minimum delay between messages', async () => {
await limiter.throttle(() => Promise.resolve('first'));
const start = Date.now();
await limiter.throttle(() => Promise.resolve('second'));
const elapsed = Date.now() - start;
expect(elapsed).toBeGreaterThanOrEqual(100);
expect(elapsed).toBeLessThan(200); // 100ms delay + 50ms max jitter
});
it('adds jitter to the delay', async () => {
// Run a message first to set lastMessageTime
await limiter.throttle(() => Promise.resolve('first'));
const delays: number[] = [];
for (let i = 0; i < 5; i++) {
const start = Date.now();
await limiter.throttle(() => Promise.resolve(`msg-${i}`));
delays.push(Date.now() - start);
}
// At least one delay should differ significantly from the base delay
const uniqueDelays = new Set(delays.map(d => Math.floor(d / 10)));
expect(uniqueDelays.size).toBeGreaterThan(1);
});
it('resets hourly counter', async () => {
const config = getTestConfig({
rateLimit: { minDelayBetweenMessagesMs: 10, maxMessagesPerHour: 3, jitterMs: 10 },
});
// 500ms "hour" so we can actually observe the reset wait inside the test budget.
limiter = new RateLimiter(config, logger, 500);
for (let i = 0; i < 3; i++) {
await limiter.throttle(() => Promise.resolve(`msg-${i}`));
}
// 4th should be blocked by hourly limit and wait for the reset.
const start = Date.now();
await limiter.throttle(() => Promise.resolve('blocked'));
const elapsed = Date.now() - start;
expect(elapsed).toBeGreaterThanOrEqual(350);
});
it('counts only when shouldCount returns true', async () => {
const config = getTestConfig({
rateLimit: { minDelayBetweenMessagesMs: 10, maxMessagesPerHour: 2, jitterMs: 10 },
});
limiter = new RateLimiter(config, logger, 500);
// These should NOT be counted
await limiter.throttle(() => Promise.resolve('skip'), () => false);
await limiter.throttle(() => Promise.resolve('skip'), () => false);
// These SHOULD be counted
await limiter.throttle(() => Promise.resolve('count'), () => true);
await limiter.throttle(() => Promise.resolve('count'), () => true);
// Next counted one should be delayed
const start = Date.now();
await limiter.throttle(() => Promise.resolve('blocked'), () => true);
const elapsed = Date.now() - start;
expect(elapsed).toBeGreaterThanOrEqual(10);
});
it('returns the function result', async () => {
const result = await limiter.throttle(() => Promise.resolve({ value: 42 }));
expect(result).toEqual({ value: 42 });
});
it('constrains to concurrency of 1', async () => {
const order: number[] = [];
const promises = [
limiter.throttle(() => { order.push(1); return Promise.resolve('a'); }),
limiter.throttle(() => { order.push(2); return Promise.resolve('b'); }),
limiter.throttle(() => { order.push(3); return Promise.resolve('c'); }),
];
await Promise.all(promises);
expect(order).toEqual([1, 2, 3]);
});
describe('getStats', () => {
it('returns initial stats', () => {
const stats = limiter.getStats();
expect(stats.messagesThisHour).toBe(0);
// minutesSinceLastMessage should be large initially
expect(stats.minutesSinceLastMessage).toBeGreaterThan(0);
});
it('updates stats after messages', async () => {
await limiter.throttle(() => Promise.resolve('msg'));
const stats = limiter.getStats();
expect(stats.messagesThisHour).toBe(1);
expect(stats.minutesSinceLastMessage).toBeLessThan(1);
});
});
describe('getTimers', () => {
it('returns null nextMessageAvailableAt initially', () => {
const timers = limiter.getTimers();
expect(timers.nextMessageAvailableAt).toBeNull();
expect(timers.isHourlyLimitReached).toBe(false);
expect(timers.messagesRemaining).toBe(10);
});
it('returns nextMessageAvailableAt after a message', async () => {
await limiter.throttle(() => Promise.resolve('msg'));
const timers = limiter.getTimers();
expect(timers.nextMessageAvailableAt).toBeDefined();
expect(timers.nextMessageAvailableAt!).toBeGreaterThan(Date.now());
});
it('shows hourly limit reached when max messages sent', async () => {
const config = getTestConfig({
rateLimit: { minDelayBetweenMessagesMs: 10, maxMessagesPerHour: 2, jitterMs: 10 },
});
limiter = new RateLimiter(config, logger);
await limiter.throttle(() => Promise.resolve('msg1'));
await limiter.throttle(() => Promise.resolve('msg2'));
const timers = limiter.getTimers();
expect(timers.isHourlyLimitReached).toBe(true);
expect(timers.messagesRemaining).toBe(0);
});
it('calculates messages remaining', async () => {
const config = getTestConfig({
rateLimit: { minDelayBetweenMessagesMs: 10, maxMessagesPerHour: 5, jitterMs: 10 },
});
limiter = new RateLimiter(config, logger);
await limiter.throttle(() => Promise.resolve('msg1'));
await limiter.throttle(() => Promise.resolve('msg2'));
const timers = limiter.getTimers();
expect(timers.messagesRemaining).toBe(3);
});
});
describe('edge cases', () => {
it('handles function that throws', async () => {
await expect(limiter.throttle(() => { throw new Error('fail'); })).rejects.toThrow('fail');
});
it('handles async function errors', async () => {
await expect(limiter.throttle(() => Promise.reject(new Error('async fail')))).rejects.toThrow('async fail');
});
it('handles synchronous return values', async () => {
const result = await limiter.throttle(() => 'sync value');
expect(result).toBe('sync value');
});
it('handles zero minDelay', async () => {
const config = getTestConfig({
rateLimit: { minDelayBetweenMessagesMs: 0, maxMessagesPerHour: 100, jitterMs: 0 },
});
limiter = new RateLimiter(config, logger);
const start = Date.now();
for (let i = 0; i < 5; i++) {
await limiter.throttle(() => Promise.resolve(`msg-${i}`));
}
const elapsed = Date.now() - start;
expect(elapsed).toBeLessThan(100);
});
});
});
tests/reddit-api.test.ts
+533
View File
@@ -0,0 +1,533 @@
import { describe, it, expect, vi, beforeEach } from 'vitest';
import axios from 'axios';
import { RedditApiClient } from '../src/reddit/api.js';
import type { RedditPost } from '../src/reddit/api.js';
import { createMockLogger } from '../tests/mockLogger.js';
import { getTestConfig } from '../tests/testUtils.js';
vi.mock('axios', async () => {
const actual = await vi.importActual<typeof import('axios')>('axios');
return {
default: {
...actual.default,
create: vi.fn(),
isAxiosError: (e: unknown) => Boolean(e && typeof e === 'object' && (e as { isAxiosError?: boolean }).isAxiosError),
},
};
});
describe('RedditApiClient', () => {
let client: RedditApiClient;
let logger: ReturnType<typeof createMockLogger>;
beforeEach(() => {
logger = createMockLogger();
const config = getTestConfig();
client = new RedditApiClient(config, logger);
vi.clearAllMocks();
});
describe('hasImage', () => {
const validPostFactory = (overrides: Partial<RedditPost> = {}): RedditPost => ({
id: 't3_test',
name: 't3_test',
subreddit: 'test',
author: 'testuser',
title: 'Test',
url: 'http://example.com',
permalink: '/r/test/comments/abc/test',
thumbnail: '',
is_self: false,
created_utc: Date.now() / 1000,
over_18: false,
spoiler: false,
...overrides,
});
it('accepts posts with post_hint=image', () => {
const post = validPostFactory({ post_hint: 'image' });
// We can't test private method directly, but we can test via getImageUrl
const url = client.getImageUrl(post);
// Since url doesn't match image patterns, it falls through to thumbnail
// which is empty, so returns null. The hasImage check is internal.
// Let's test via getNewPosts filtering instead.
});
it('accepts posts with preview images', () => {
const post = validPostFactory({
preview: {
images: [{
source: { url: 'http://i.redd.it/image.jpg', width: 100, height: 100 },
resolutions: [],
}],
},
});
const url = client.getImageUrl(post);
expect(url).toBe('http://i.redd.it/image.jpg');
});
it('accepts posts with i.redd.it URLs', () => {
const post = validPostFactory({ url: 'https://i.redd.it/abc123.jpg' });
const url = client.getImageUrl(post);
expect(url).toBe('https://i.redd.it/abc123.jpg');
});
it('accepts posts with i.imgur.com URLs', () => {
const post = validPostFactory({ url: 'https://i.imgur.com/abc123.jpg' });
const url = client.getImageUrl(post);
expect(url).toBe('https://i.imgur.com/abc123.jpg');
});
it('extracts imgur direct URL from imgur page URL', () => {
const post = validPostFactory({ url: 'https://imgur.com/abc123' });
const url = client.getImageUrl(post);
expect(url).toBe('https://i.imgur.com/abc123.jpg');
});
it('extracts imgur direct URL from imgur page URL with extension', () => {
const post = validPostFactory({ url: 'https://imgur.com/abc123.jpg' });
const url = client.getImageUrl(post);
expect(url).toBe('https://i.imgur.com/abc123.jpg');
});
it('accepts posts with image file extensions in URL', () => {
const post = validPostFactory({ url: 'http://example.com/photo.png' });
const url = client.getImageUrl(post);
expect(url).toBe('http://example.com/photo.png');
});
it('accepts .gif URLs', () => {
const post = validPostFactory({ url: 'http://example.com/animation.gif' });
const url = client.getImageUrl(post);
expect(url).toBe('http://example.com/animation.gif');
});
it('accepts .webp URLs', () => {
const post = validPostFactory({ url: 'http://example.com/image.webp' });
const url = client.getImageUrl(post);
expect(url).toBe('http://example.com/image.webp');
});
it('falls back to thumbnail when no other image source', () => {
const post = validPostFactory({ thumbnail: 'http://example.com/thumb.jpg' });
const url = client.getImageUrl(post);
expect(url).toBe('http://example.com/thumb.jpg');
});
it('returns null when no image source available', () => {
const post = validPostFactory({ thumbnail: '' });
const url = client.getImageUrl(post);
expect(url).toBeNull();
});
it('returns null for empty thumbnail', () => {
const post = validPostFactory({ thumbnail: '' });
const url = client.getImageUrl(post);
expect(url).toBeNull();
});
it('skips non-image thumbnail URLs', () => {
const post = validPostFactory({ thumbnail: 'about:blank' });
const url = client.getImageUrl(post);
expect(url).toBeNull();
});
});
describe('decodeRedditUrl', () => {
it('decodes HTML entities in preview URLs', () => {
const post: RedditPost = {
id: 't3_test',
name: 't3_test',
subreddit: 'test',
author: 'user',
title: 'Test',
url: 'http://example.com',
permalink: '/r/test/comments/abc/test',
thumbnail: '',
is_self: false,
created_utc: Date.now() / 1000,
over_18: false,
spoiler: false,
preview: {
images: [{
source: { url: 'http://example.com/&amp;test.jpg', width: 100, height: 100 },
resolutions: [],
}],
},
};
const url = client.getImageUrl(post);
expect(url).toBe('http://example.com/&test.jpg');
});
});
describe('getPostUrl', () => {
it('constructs correct URL from permalink', () => {
const post: RedditPost = {
id: 't3_abc',
name: 't3_abc',
subreddit: 'test',
author: 'user',
title: 'Test',
url: 'http://example.com',
permalink: '/r/test/comments/abc/test',
thumbnail: '',
is_self: false,
created_utc: Date.now() / 1000,
over_18: false,
spoiler: false,
};
expect(client.getPostUrl(post)).toBe('https://www.reddit.com/r/test/comments/abc/test');
});
});
describe('isValidPost (via getNewPosts filtering)', () => {
it('skips self posts', () => {
const textPost: RedditPost = {
id: 't3_text',
name: 't3_text',
subreddit: 'test',
author: 'user',
title: 'Text post',
url: 'https://www.reddit.com/r/test/comments/abc/',
permalink: '/r/test/comments/abc/text',
thumbnail: 'self',
is_self: true,
created_utc: Date.now() / 1000,
over_18: false,
spoiler: false,
};
// We can't easily test the private isValidPost, but we can verify
// the behavior through the public API by checking that image posts
// pass through and text posts don't.
});
it('skips deleted authors', () => {
const post: RedditPost = {
id: 't3_del',
name: 't3_del',
subreddit: 'test',
author: '[deleted]',
title: 'Deleted user',
url: 'http://i.redd.it/img.jpg',
permalink: '/r/test/comments/abc/del',
thumbnail: 'http://i.redd.it/img.jpg',
is_self: false,
created_utc: Date.now() / 1000,
over_18: false,
spoiler: false,
};
// isValidPost should return false for [deleted]
});
it('skips AutoModerator posts', () => {
const post: RedditPost = {
id: 't3_mod',
name: 't3_mod',
subreddit: 'test',
author: 'AutoModerator',
title: 'Mod message',
url: 'http://i.redd.it/img.jpg',
permalink: '/r/test/comments/abc/mod',
thumbnail: 'http://i.redd.it/img.jpg',
is_self: false,
created_utc: Date.now() / 1000,
over_18: false,
spoiler: false,
};
// isValidPost should return false for AutoModerator
});
});
describe('isSellerProfile', () => {
it('returns true when seller keyword matches', async () => {
const mockResponse = {
data: {
data: {
subreddit: {
public_description: 'Check out my OnlyFans page!',
title: 'My Profile',
display_name_prefixed: 'u/testuser',
},
},
},
};
vi.mocked(axios.create).mockReturnValue({
get: vi.fn().mockResolvedValue(mockResponse),
} as any);
const client2 = new RedditApiClient(getTestConfig(), logger);
const result = await client2.isSellerProfile('testuser', ['onlyfans']);
expect(result).toBe(true);
});
it('returns false when no keywords match', async () => {
const mockResponse = {
data: {
data: {
subreddit: {
public_description: 'Just a regular user',
title: 'My Profile',
display_name_prefixed: 'u/testuser',
},
},
},
};
vi.mocked(axios.create).mockReturnValue({
get: vi.fn().mockResolvedValue(mockResponse),
} as any);
const client2 = new RedditApiClient(getTestConfig(), logger);
const result = await client2.isSellerProfile('testuser', ['onlyfans', 'fansly']);
expect(result).toBe(false);
});
it('returns false for empty seller keywords', async () => {
const client2 = new RedditApiClient(getTestConfig(), logger);
const result = await client2.isSellerProfile('testuser', []);
expect(result).toBe(false);
});
it('returns false when user profile not found (404)', async () => {
const error = { isAxiosError: true, response: { status: 404 } };
vi.mocked(axios.create).mockReturnValue({
get: vi.fn().mockRejectedValue(error),
} as any);
const client2 = new RedditApiClient(getTestConfig(), logger);
const result = await client2.isSellerProfile('nonexistent', ['onlyfans']);
expect(result).toBe(false);
});
it('is case-insensitive for keyword matching', async () => {
const mockResponse = {
data: {
data: {
subreddit: {
public_description: 'Check out my ONLYFANS page!',
title: 'My Profile',
display_name_prefixed: 'u/testuser',
},
},
},
};
vi.mocked(axios.create).mockReturnValue({
get: vi.fn().mockResolvedValue(mockResponse),
} as any);
const client2 = new RedditApiClient(getTestConfig(), logger);
const result = await client2.isSellerProfile('testuser', ['onlyfans']);
expect(result).toBe(true);
});
it('returns false on non-404 errors', async () => {
vi.mocked(axios.create).mockReturnValue({
get: vi.fn().mockRejectedValue(new Error('Network error')),
} as any);
const client2 = new RedditApiClient(getTestConfig(), logger);
const result = await client2.isSellerProfile('testuser', ['onlyfans']);
expect(result).toBe(false);
});
it('returns true when a domain in the bio matches sellerDomains', async () => {
const mockResponse = {
data: {
data: {
subreddit: {
public_description: 'find me here https://my.linktr.ee/cutiepie',
title: 'My Profile',
display_name_prefixed: 'u/testuser',
},
},
},
};
vi.mocked(axios.create).mockReturnValue({
get: vi.fn().mockResolvedValue(mockResponse),
} as any);
const client2 = new RedditApiClient(getTestConfig(), logger);
const result = await client2.isSellerProfile('testuser', [], ['linktr.ee']);
expect(result).toBe(true);
});
it('matches bare domains without a scheme', async () => {
const mockResponse = {
data: {
data: {
subreddit: {
public_description: 'dm for fansly.com/me',
title: '',
display_name_prefixed: 'u/testuser',
},
},
},
};
vi.mocked(axios.create).mockReturnValue({
get: vi.fn().mockResolvedValue(mockResponse),
} as any);
const client2 = new RedditApiClient(getTestConfig(), logger);
const result = await client2.isSellerProfile('testuser', [], ['fansly.com']);
expect(result).toBe(true);
});
it('does not match unrelated domains that share a suffix', async () => {
const mockResponse = {
data: {
data: {
subreddit: {
public_description: 'see notlinktr.ee for my art',
title: '',
display_name_prefixed: 'u/testuser',
},
},
},
};
vi.mocked(axios.create).mockReturnValue({
get: vi.fn().mockResolvedValue(mockResponse),
} as any);
const client2 = new RedditApiClient(getTestConfig(), logger);
const result = await client2.isSellerProfile('testuser', [], ['linktr.ee']);
expect(result).toBe(false);
});
});
describe('getNewPosts', () => {
it('returns empty array for 404 subreddit', async () => {
const error = { isAxiosError: true, response: { status: 404 } };
const mockAxios = { get: vi.fn().mockRejectedValue(error) };
vi.mocked(axios.create).mockReturnValue(mockAxios as any);
const client2 = new RedditApiClient(getTestConfig(), logger);
const result = await client2.getNewPosts('nonexistent');
expect(result).toEqual([]);
});
it('returns empty array for 403 private subreddit', async () => {
const error = { isAxiosError: true, response: { status: 403 } };
const mockAxios = { get: vi.fn().mockRejectedValue(error) };
vi.mocked(axios.create).mockReturnValue(mockAxios as any);
const client2 = new RedditApiClient(getTestConfig(), logger);
const result = await client2.getNewPosts('private');
expect(result).toEqual([]);
});
it('throws on 429 rate limit', async () => {
const error = { isAxiosError: true, response: { status: 429 } };
const mockAxios = { get: vi.fn().mockRejectedValue(error) };
vi.mocked(axios.create).mockReturnValue(mockAxios as any);
const client2 = new RedditApiClient(getTestConfig(), logger);
await expect(client2.getNewPosts('test')).rejects.toThrow();
});
it('parses valid image posts from response', async () => {
const mockResponse = {
data: { data: { children: [
{
data: {
id: 't3_1',
name: 't3_1',
subreddit: 'test',
author: 'user1',
title: 'Image post 1',
url: 'https://i.redd.it/abc123.jpg',
permalink: '/r/test/comments/abc/1',
thumbnail: 'https://i.redd.it/abc123.jpg',
is_self: false,
post_hint: 'image',
created_utc: 1000000,
over_18: false,
spoiler: false,
},
},
{
data: {
id: 't3_2',
name: 't3_2',
subreddit: 'test',
author: 'user2',
title: 'Text post',
url: 'https://www.reddit.com/r/test/comments/abc/2',
permalink: '/r/test/comments/abc/2',
thumbnail: 'self',
is_self: true,
created_utc: 1000001,
over_18: false,
spoiler: false,
},
},
] } },
};
const mockAxios = { get: vi.fn().mockResolvedValue(mockResponse) };
vi.mocked(axios.create).mockReturnValue(mockAxios as any);
const client2 = new RedditApiClient(getTestConfig(), logger);
const result = await client2.getNewPosts('test');
expect(result.length).toBe(1);
expect(result[0].id).toBe('t3_1');
expect(result[0].title).toBe('Image post 1');
});
it('filters out deleted authors', async () => {
const mockResponse = {
data: { data: { children: [
{
data: {
id: 't3_del',
name: 't3_del',
subreddit: 'test',
author: '[deleted]',
title: 'Deleted',
url: 'https://i.redd.it/img.jpg',
permalink: '/r/test/comments/abc/del',
thumbnail: 'https://i.redd.it/img.jpg',
is_self: false,
created_utc: 1000000,
over_18: false,
spoiler: false,
},
},
] } },
};
const mockAxios = { get: vi.fn().mockResolvedValue(mockResponse) };
vi.mocked(axios.create).mockReturnValue(mockAxios as any);
const client2 = new RedditApiClient(getTestConfig(), logger);
const result = await client2.getNewPosts('test');
expect(result.length).toBe(0);
});
it('filters out AutoModerator posts', async () => {
const mockResponse = {
data: { data: { children: [
{
data: {
id: 't3_mod',
name: 't3_mod',
subreddit: 'test',
author: 'AutoModerator',
title: 'Mod message',
url: 'https://i.redd.it/img.jpg',
permalink: '/r/test/comments/abc/mod',
thumbnail: 'https://i.redd.it/img.jpg',
is_self: false,
created_utc: 1000000,
over_18: false,
spoiler: false,
},
},
] } },
};
const mockAxios = { get: vi.fn().mockResolvedValue(mockResponse) };
vi.mocked(axios.create).mockReturnValue(mockAxios as any);
const client2 = new RedditApiClient(getTestConfig(), logger);
const result = await client2.getNewPosts('test');
expect(result.length).toBe(0);
});
});
});
tests/repositories.test.ts
+563
View File
@@ -0,0 +1,563 @@
import { describe, it, expect, beforeEach } from 'vitest';
import type { Database } from 'better-sqlite3';
import { createTestRepositories } from '../tests/testUtils.js';
import {
PostsRepository,
MessagesRepository,
SuccessfulMessagesRepository,
} from '../src/database/repositories.js';
describe('PostsRepository', () => {
let postsRepo: PostsRepository;
let messagesRepo: MessagesRepository;
let db: Database;
beforeEach(() => {
const repos = createTestRepositories();
postsRepo = repos.postsRepo;
messagesRepo = repos.messagesRepo;
db = repos.db;
});
it('inserts and retrieves a post by id', () => {
postsRepo.insert({
id: 'post1',
subreddit: 'test',
author: 'user1',
title: 'Test Post',
imageUrl: 'http://example.com/img.jpg',
processedAt: Date.now(),
});
const post = postsRepo.getById('post1');
expect(post).toBeDefined();
expect(post!.id).toBe('post1');
expect(post!.subreddit).toBe('test');
expect(post!.author).toBe('user1');
expect(post!.title).toBe('Test Post');
expect(post!.imageUrl).toBe('http://example.com/img.jpg');
expect(post!.messageSent).toBe(false);
expect(post!.imageAnalysis).toBeNull();
expect(post!.error).toBeNull();
});
it('checks if a post is processed', () => {
expect(postsRepo.isProcessed('nonexistent')).toBe(false);
postsRepo.insert({
id: 'post2',
subreddit: 'test',
author: 'user2',
title: 'Another Post',
imageUrl: null,
processedAt: Date.now(),
});
expect(postsRepo.isProcessed('post2')).toBe(true);
});
it('marks a message as sent', () => {
postsRepo.insert({
id: 'post3',
subreddit: 'test',
author: 'user3',
title: 'Sent Post',
imageUrl: 'http://example.com/img.jpg',
processedAt: Date.now(),
});
postsRepo.markMessageSent('post3');
const post = postsRepo.getById('post3');
expect(post!.messageSent).toBe(true);
});
it('updates analysis and generated message', () => {
postsRepo.insert({
id: 'post4',
subreddit: 'test',
author: 'user4',
title: 'Analysis Post',
imageUrl: 'http://example.com/img.jpg',
processedAt: Date.now(),
});
postsRepo.updateAnalysis('post4', 'Beautiful woman with blonde hair', 'Hey gorgeous...');
const post = postsRepo.getById('post4');
expect(post!.imageAnalysis).toBe('Beautiful woman with blonde hair');
expect(post!.generatedMessage).toBe('Hey gorgeous...');
});
it('sets error on a post', () => {
postsRepo.insert({
id: 'post5',
subreddit: 'test',
author: 'user5',
title: 'Error Post',
imageUrl: 'http://example.com/img.jpg',
processedAt: Date.now(),
});
postsRepo.setError('post5', 'Failed to download image');
const post = postsRepo.getById('post5');
expect(post!.error).toBe('Failed to download image');
});
it('gets posts by author (case insensitive)', () => {
postsRepo.insert({ id: 'p1', subreddit: 'a', author: 'TestUser', title: 'T1', imageUrl: null, processedAt: Date.now() });
postsRepo.insert({ id: 'p2', subreddit: 'b', author: 'testuser', title: 'T2', imageUrl: null, processedAt: Date.now() });
postsRepo.insert({ id: 'p3', subreddit: 'c', author: 'TESTUSER', title: 'T3', imageUrl: null, processedAt: Date.now() });
const results = postsRepo.getByAuthor('testuser');
expect(results.length).toBe(3);
// Should be ordered by processed_at DESC
expect(results[0].id).toBe('p3');
expect(results[2].id).toBe('p1');
});
it('limits getByAuthor to 10 results', () => {
for (let i = 0; i < 15; i++) {
postsRepo.insert({
id: `pa${i}`,
subreddit: 'test',
author: 'bulkuser',
title: `Bulk ${i}`,
imageUrl: null,
processedAt: Date.now() + i,
});
}
const results = postsRepo.getByAuthor('bulkuser');
expect(results.length).toBe(10);
});
it('returns undefined for non-existent post', () => {
expect(postsRepo.getById('nonexistent')).toBeUndefined();
});
it('maps database columns to camelCase correctly', () => {
postsRepo.insert({
id: 'post6',
subreddit: 'camelcase',
author: 'user6',
title: 'Camel Case Test',
imageUrl: 'http://example.com/test.png',
processedAt: 1234567890,
});
const post = postsRepo.getById('post6');
expect(post).toMatchObject({
id: 'post6',
subreddit: 'camelcase',
author: 'user6',
title: 'Camel Case Test',
imageUrl: 'http://example.com/test.png',
processedAt: 1234567890,
messageSent: false,
imageAnalysis: null,
generatedMessage: null,
error: null,
});
});
});
describe('MessagesRepository', () => {
let postsRepo: PostsRepository;
let messagesRepo: MessagesRepository;
let db: Database;
beforeEach(() => {
const repos = createTestRepositories();
postsRepo = repos.postsRepo;
messagesRepo = repos.messagesRepo;
db = repos.db;
});
it('inserts and retrieves a message', () => {
const messageId = messagesRepo.insert({
postId: 'post1',
recipient: 'recipient1',
subject: 'Hello',
message: 'Hey there!',
sentAt: Date.now(),
strategyId: 'variant-a',
});
expect(messageId).toBeGreaterThan(0);
const msg = messagesRepo.getByPostId('post1');
expect(msg).toBeDefined();
expect(msg!.postId).toBe('post1');
expect(msg!.recipient).toBe('recipient1');
expect(msg!.subject).toBe('Hello');
expect(msg!.message).toBe('Hey there!');
expect(msg!.strategyId).toBe('variant-a');
expect(msg!.gotResponse).toBe(false);
});
it('retrieves recent messages', () => {
const baseTime = Date.now();
for (let i = 0; i < 5; i++) {
messagesRepo.insert({
postId: `post${i}`,
recipient: `recipient${i}`,
subject: `Subject ${i}`,
message: `Message ${i}`,
// Later i => more recent sentAt, so msg4 should come back first.
sentAt: baseTime + i * 1000,
});
}
const recent = messagesRepo.getRecent(3);
expect(recent.length).toBe(3);
// Should be ordered by sent_at DESC
expect(recent[0].recipient).toBe('recipient4');
});
it('marks a message as having gotten a response', () => {
messagesRepo.insert({
postId: 'post1',
recipient: 'recipient1',
subject: 'Hello',
message: 'Hey!',
sentAt: Date.now(),
});
const msg = messagesRepo.getByPostId('post1');
expect(msg!.gotResponse).toBe(false);
messagesRepo.markGotResponse(msg!.id);
const updated = messagesRepo.getByPostId('post1');
expect(updated!.gotResponse).toBe(true);
expect(updated!.responseCheckedAt).toBeDefined();
});
it('marks a message as checked without response', () => {
messagesRepo.insert({
postId: 'post1',
recipient: 'recipient1',
subject: 'Hello',
message: 'Hey!',
sentAt: Date.now(),
});
const msg = messagesRepo.getByPostId('post1');
messagesRepo.markChecked(msg!.id);
const updated = messagesRepo.getByPostId('post1');
expect(updated!.gotResponse).toBe(false);
expect(updated!.responseCheckedAt).toBeDefined();
});
it('gets the latest sent message by recipient with joined post info', () => {
postsRepo.insert({
id: 'post-a',
subreddit: 'gonewild',
author: 'alice',
title: 'First post',
imageUrl: null,
processedAt: Date.now(),
});
postsRepo.insert({
id: 'post-b',
subreddit: 'gonewild',
author: 'alice',
title: 'Second post',
imageUrl: null,
processedAt: Date.now(),
});
const base = Date.now();
messagesRepo.insert({ postId: 'post-a', recipient: 'alice', subject: 's', message: 'm', sentAt: base });
messagesRepo.insert({ postId: 'post-b', recipient: 'alice', subject: 's', message: 'm', sentAt: base + 1000 });
const latest = messagesRepo.getLatestByRecipient('alice');
expect(latest).toBeDefined();
expect(latest!.postId).toBe('post-b');
expect(latest!.subreddit).toBe('gonewild');
expect(latest!.postTitle).toBe('Second post');
expect(messagesRepo.getLatestByRecipient('ALICE')).toBeDefined();
expect(messagesRepo.getLatestByRecipient('nobody')).toBeUndefined();
});
it('checks if a message has been sent to a recipient', () => {
expect(messagesRepo.hasSentTo('someone')).toBe(false);
messagesRepo.insert({
postId: 'post1',
recipient: 'recipient1',
subject: 'Hello',
message: 'Hey!',
sentAt: Date.now(),
});
expect(messagesRepo.hasSentTo('recipient1')).toBe(true);
expect(messagesRepo.hasSentTo('someone')).toBe(false);
});
it('gets strategy stats from database', () => {
const now = Date.now();
messagesRepo.insert({ postId: 'p1', recipient: 'r1', subject: 's', message: 'm', sentAt: now, strategyId: 'variant-a' });
messagesRepo.insert({ postId: 'p2', recipient: 'r2', subject: 's', message: 'm', sentAt: now, strategyId: 'variant-a' });
messagesRepo.insert({ postId: 'p3', recipient: 'r3', subject: 's', message: 'm', sentAt: now, strategyId: 'variant-b' });
messagesRepo.markGotResponse(1);
messagesRepo.markGotResponse(3);
const stats = messagesRepo.getStrategyStats();
expect(stats.length).toBe(2);
const variantA = stats.find(s => s.strategyId === 'variant-a');
expect(variantA!.totalSent).toBe(2);
expect(variantA!.totalResponses).toBe(1);
expect(variantA!.responseRate).toBe(50);
const variantB = stats.find(s => s.strategyId === 'variant-b');
expect(variantB!.totalSent).toBe(1);
expect(variantB!.totalResponses).toBe(1);
expect(variantB!.responseRate).toBe(100);
});
it('excludes messages without strategy_id from stats', () => {
messagesRepo.insert({ postId: 'p1', recipient: 'r1', subject: 's', message: 'm', sentAt: Date.now() });
const stats = messagesRepo.getStrategyStats();
expect(stats.length).toBe(0);
});
it('calculates 0 response rate when no messages sent', () => {
// No messages inserted
const stats = messagesRepo.getStrategyStats();
expect(stats.length).toBe(0);
});
it('gets unchecked messages with correct time filters', () => {
const now = Date.now();
// Old message (2 hours ago, no check) - should be returned
messagesRepo.insert({ postId: 'p1', recipient: 'r1', subject: 's', message: 'm', sentAt: now - 2 * 3600000 });
// Recent message (30 min ago) - should NOT be returned (too young)
messagesRepo.insert({ postId: 'p2', recipient: 'r2', subject: 's', message: 'm', sentAt: now - 30 * 60000 });
// Already checked message - should NOT be returned
messagesRepo.insert({ postId: 'p3', recipient: 'r3', subject: 's', message: 'm', sentAt: now - 2 * 3600000 });
messagesRepo.markChecked(3);
const unchecked = messagesRepo.getUncheckedMessages(60 * 60 * 1000, 6 * 60 * 60 * 1000, 10);
expect(unchecked.length).toBe(1);
expect(unchecked[0].postId).toBe('p1');
});
it('maps sent_at as number and response_checked_at as number | null', () => {
messagesRepo.insert({
postId: 'post1',
recipient: 'r1',
subject: 's',
message: 'm',
sentAt: 1234567890,
});
const msg = messagesRepo.getByPostId('post1');
expect(typeof msg!.sentAt).toBe('number');
expect(msg!.responseCheckedAt).toBeNull();
});
it('returns optional fields as undefined when not present', () => {
messagesRepo.insert({
postId: 'post1',
recipient: 'r1',
subject: 's',
message: 'm',
sentAt: Date.now(),
});
const msg = messagesRepo.getByPostId('post1');
expect(msg!.subreddit).toBeUndefined();
expect(msg!.postTitle).toBeUndefined();
expect(msg!.postImageUrl).toBeUndefined();
});
});
describe('SuccessfulMessagesRepository', () => {
let successfulMessagesRepo: SuccessfulMessagesRepository;
let db: Database;
beforeEach(() => {
const repos = createTestRepositories();
successfulMessagesRepo = repos.successfulMessagesRepo;
db = repos.db;
});
it('inserts and retrieves a successful message', () => {
successfulMessagesRepo.insert({
sentMessageId: 1,
originalMessage: 'Hey gorgeous!',
imageAnalysis: 'Blonde woman, rating 8/10',
postTitle: 'My selfie',
responseSnippet: 'Aww thanks!',
subreddit: 'gonewild',
});
const recent = successfulMessagesRepo.getRecent(5);
expect(recent.length).toBe(1);
expect(recent[0].originalMessage).toBe('Hey gorgeous!');
expect(recent[0].imageAnalysis).toBe('Blonde woman, rating 8/10');
expect(recent[0].postTitle).toBe('My selfie');
expect(recent[0].responseSnippet).toBe('Aww thanks!');
expect(recent[0].subreddit).toBe('gonewild');
});
it('checks if a message already exists', () => {
successfulMessagesRepo.insert({
sentMessageId: 1,
originalMessage: 'Unique message',
imageAnalysis: null,
postTitle: null,
responseSnippet: null,
subreddit: null,
});
expect(successfulMessagesRepo.existsByMessage('Unique message')).toBe(true);
expect(successfulMessagesRepo.existsByMessage('Non-existent')).toBe(false);
});
it('gets weighted examples for a subreddit', () => {
successfulMessagesRepo.insert({ sentMessageId: 1, originalMessage: 'Msg 1', imageAnalysis: null, postTitle: null, responseSnippet: null, subreddit: 'gonewild' });
successfulMessagesRepo.insert({ sentMessageId: 2, originalMessage: 'Msg 2', imageAnalysis: null, postTitle: null, responseSnippet: null, subreddit: 'gonewild' });
successfulMessagesRepo.insert({ sentMessageId: 3, originalMessage: 'Msg 3', imageAnalysis: null, postTitle: null, responseSnippet: null, subreddit: 'homemadesfW' });
const examples = successfulMessagesRepo.getWeightedExamples('gonewild', 3);
// Returns up to `limit` examples weighted toward the target subreddit but
// including off-subreddit fallback rows when there's headroom.
expect(examples.length).toBe(3);
expect(examples[0].subreddit).toBe('gonewild');
expect(examples[1].subreddit).toBe('gonewild');
});
it('falls back to recent when no subreddit match', () => {
successfulMessagesRepo.insert({ sentMessageId: 1, originalMessage: 'Msg 1', imageAnalysis: null, postTitle: null, responseSnippet: null, subreddit: 'gonewild' });
successfulMessagesRepo.insert({ sentMessageId: 2, originalMessage: 'Msg 2', imageAnalysis: null, postTitle: null, responseSnippet: null, subreddit: 'other' });
const examples = successfulMessagesRepo.getWeightedExamples('nonexistent', 3);
expect(examples.length).toBe(2);
});
it('counts successful messages', () => {
expect(successfulMessagesRepo.count()).toBe(0);
successfulMessagesRepo.insert({ sentMessageId: 1, originalMessage: 'Msg 1', imageAnalysis: null, postTitle: null, responseSnippet: null, subreddit: null });
successfulMessagesRepo.insert({ sentMessageId: 2, originalMessage: 'Msg 2', imageAnalysis: null, postTitle: null, responseSnippet: null, subreddit: null });
expect(successfulMessagesRepo.count()).toBe(2);
});
it('handles null fields correctly', () => {
successfulMessagesRepo.insert({
sentMessageId: null,
originalMessage: 'No context',
imageAnalysis: null,
postTitle: null,
responseSnippet: null,
subreddit: null,
});
const recent = successfulMessagesRepo.getRecent(1);
expect(recent[0].sentMessageId).toBeNull();
expect(recent[0].imageAnalysis).toBeNull();
expect(recent[0].postTitle).toBeNull();
expect(recent[0].responseSnippet).toBeNull();
expect(recent[0].subreddit).toBeNull();
});
it('returns most recent examples first', () => {
for (let i = 0; i < 5; i++) {
successfulMessagesRepo.insert({
sentMessageId: i,
originalMessage: `Msg ${i}`,
imageAnalysis: null,
postTitle: null,
responseSnippet: null,
subreddit: null,
});
}
const recent = successfulMessagesRepo.getRecent(3);
expect(recent[0].originalMessage).toBe('Msg 4');
expect(recent[2].originalMessage).toBe('Msg 2');
});
});
describe('Repository edge cases', () => {
it('handles special characters in post titles', () => {
const repos = createTestRepositories();
repos.postsRepo.insert({
id: 'special',
subreddit: 'test',
author: "user's \"quotes\"",
title: 'Title with <html> & "special" chars',
imageUrl: null,
processedAt: Date.now(),
});
const post = repos.postsRepo.getById('special');
expect(post!.author).toBe("user's \"quotes\"");
expect(post!.title).toBe('Title with <html> & "special" chars');
});
it('handles very long messages', () => {
const repos = createTestRepositories();
const longMessage = 'x'.repeat(10000);
repos.messagesRepo.insert({
postId: 'post1',
recipient: 'r1',
subject: 's',
message: longMessage,
sentAt: Date.now(),
});
const msg = repos.messagesRepo.getByPostId('post1');
expect(msg!.message.length).toBe(10000);
});
it('handles empty string fields', () => {
const repos = createTestRepositories();
repos.postsRepo.insert({
id: 'empty',
subreddit: '',
author: '',
title: '',
imageUrl: '',
processedAt: Date.now(),
});
const post = repos.postsRepo.getById('empty');
expect(post!.subreddit).toBe('');
expect(post!.author).toBe('');
expect(post!.title).toBe('');
});
it('handles duplicate post insert (should throw due to PRIMARY KEY)', () => {
const repos = createTestRepositories();
repos.postsRepo.insert({ id: 'dup', subreddit: 't', author: 'a', title: 't', imageUrl: null, processedAt: Date.now() });
expect(() => {
repos.postsRepo.insert({ id: 'dup', subreddit: 't', author: 'a', title: 't', imageUrl: null, processedAt: Date.now() });
}).toThrow();
});
it('handles messages with strategy_id null', () => {
const repos = createTestRepositories();
const id = repos.messagesRepo.insert({
postId: 'p1',
recipient: 'r1',
subject: 's',
message: 'm',
sentAt: Date.now(),
});
const msg = repos.messagesRepo.getByPostId('p1');
expect(msg!.strategyId).toBeNull();
});
});
tests/response-checker.test.ts
+340
View File
@@ -0,0 +1,340 @@
import { describe, it, expect, vi, beforeEach } from 'vitest';
import type { Page } from 'playwright';
import { ResponseCheckerService } from '../src/services/response-checker.js';
import type { MessagesRepository, SuccessfulMessagesRepository, SentMessageRecord, ProcessedPostRecord } from '../src/database/repositories.js';
import type { PostsRepository } from '../src/database/repositories.js';
import { createMockLogger } from '../tests/mockLogger.js';
function createMockPage(): Page {
return {
goto: vi.fn(async () => {}),
waitForTimeout: vi.fn(async () => {}),
$: vi.fn(async () => null),
$$: vi.fn(async () => []),
evaluate: vi.fn(async () => false),
} as unknown as Page;
}
function createMockMessagesRepo(messages: SentMessageRecord[] = []): MessagesRepository {
return {
insert: vi.fn(() => 1),
getByPostId: vi.fn(() => undefined),
getRecent: vi.fn(() => []),
getUncheckedMessages: vi.fn(() => messages),
markGotResponse: vi.fn(),
markChecked: vi.fn(),
hasSentTo: vi.fn(() => false),
getStrategyStats: vi.fn(() => []),
getSubredditStats: vi.fn(() => []),
getResponseRateTrend: vi.fn(() => []),
} as unknown as MessagesRepository;
}
function createMockSuccessfulMessagesRepo(): SuccessfulMessagesRepository {
return {
insert: vi.fn(),
existsByMessage: vi.fn(() => false),
getRecent: vi.fn(() => []),
getWeightedExamples: vi.fn(() => []),
count: vi.fn(() => 0),
} as unknown as SuccessfulMessagesRepository;
}
function createMockPostsRepo(posts: Map<string, ProcessedPostRecord> = new Map()): PostsRepository {
return {
isProcessed: vi.fn(() => false),
insert: vi.fn(),
updateAnalysis: vi.fn(),
markMessageSent: vi.fn(),
setError: vi.fn(),
getById: vi.fn((id: string) => posts.get(id)),
getByAuthor: vi.fn(() => []),
} as unknown as PostsRepository;
}
describe('ResponseCheckerService', () => {
let checker: ResponseCheckerService;
let logger: ReturnType<typeof createMockLogger>;
let page: Page;
let messagesRepo: MessagesRepository;
let successfulMessagesRepo: SuccessfulMessagesRepository;
let postsRepo: PostsRepository;
beforeEach(() => {
logger = createMockLogger();
page = createMockPage();
messagesRepo = createMockMessagesRepo();
successfulMessagesRepo = createMockSuccessfulMessagesRepo();
postsRepo = createMockPostsRepo();
checker = new ResponseCheckerService(page, messagesRepo, successfulMessagesRepo, postsRepo, logger);
});
describe('checkForResponses', () => {
it('returns empty when no unchecked messages', async () => {
const results = await checker.checkForResponses();
expect(results).toEqual([]);
});
it('checks messages and marks responses', async () => {
const message: SentMessageRecord = {
id: 1,
postId: 'post1',
recipient: 'user1',
subject: 'Hello',
message: 'Hey!',
sentAt: Date.now() - 2 * 3600000, // 2 hours ago
gotResponse: false,
responseCheckedAt: null,
strategyId: null,
};
messagesRepo = createMockMessagesRepo([message]);
checker = new ResponseCheckerService(page, messagesRepo, successfulMessagesRepo, postsRepo, logger);
// Mock inbox to have a reply from user1
const mockItem = {
$: vi.fn(async (selector: string) => {
if (selector.includes('.author') || selector.includes('message-author') || selector.includes('.sender')) {
return { textContent: vi.fn(async () => 'user1') };
}
if (selector.includes('.md') || selector.includes('.message-body')) {
return { textContent: vi.fn(async () => 'Yes, I\'d love that!') };
}
return null;
}),
evaluate: vi.fn(async () => true),
};
vi.mocked(page.$$).mockResolvedValue([mockItem as any]);
vi.mocked(page.$).mockImplementation(async (selector: string) => {
if (selector.includes('.author') || selector.includes('message-author') || selector.includes('.sender')) {
return mockItem.$('.author') as any;
}
if (selector.includes('.md') || selector.includes('.message-body')) {
return mockItem.$('.md') as any;
}
return null;
});
const results = await checker.checkForResponses();
expect(results.length).toBeGreaterThan(0);
expect(messagesRepo.markGotResponse).toHaveBeenCalled();
});
it('marks messages as checked when no response', async () => {
const message: SentMessageRecord = {
id: 1,
postId: 'post1',
recipient: 'user1',
subject: 'Hello',
message: 'Hey!',
sentAt: Date.now() - 2 * 3600000,
gotResponse: false,
responseCheckedAt: null,
strategyId: null,
};
messagesRepo = createMockMessagesRepo([message]);
checker = new ResponseCheckerService(page, messagesRepo, successfulMessagesRepo, postsRepo, logger);
// No replies in inbox
vi.mocked(page.$$).mockResolvedValue([]);
const results = await checker.checkForResponses();
expect(messagesRepo.markChecked).toHaveBeenCalled();
});
it('stores successful messages for learning', async () => {
const post: ProcessedPostRecord = {
id: 'post1',
subreddit: 'gonewild',
author: 'poster1',
title: 'My selfie',
imageUrl: 'http://example.com/img.jpg',
imageAnalysis: 'Beautiful woman',
generatedMessage: 'Hey gorgeous!',
processedAt: Date.now(),
messageSent: true,
error: null,
};
const message: SentMessageRecord = {
id: 1,
postId: 'post1',
recipient: 'user1',
subject: 'Hello',
message: 'Hey gorgeous!',
sentAt: Date.now() - 2 * 3600000,
gotResponse: false,
responseCheckedAt: null,
strategyId: null,
};
messagesRepo = createMockMessagesRepo([message]);
postsRepo = createMockPostsRepo(new Map([['post1', post]]));
checker = new ResponseCheckerService(page, messagesRepo, successfulMessagesRepo, postsRepo, logger);
const mockItem = {
$: vi.fn(async (selector: string) => {
if (selector.includes('.author') || selector.includes('message-author') || selector.includes('.sender')) {
return { textContent: vi.fn(async () => 'user1') };
}
if (selector.includes('.md') || selector.includes('.message-body')) {
return { textContent: vi.fn(async () => 'Thanks!') };
}
return null;
}),
evaluate: vi.fn(async () => true),
};
vi.mocked(page.$$).mockResolvedValue([mockItem as any]);
vi.mocked(page.$).mockImplementation(async (selector: string) => {
if (selector.includes('.author') || selector.includes('message-author') || selector.includes('.sender')) {
return mockItem.$('.author') as any;
}
return null;
});
await checker.checkForResponses();
expect(successfulMessagesRepo.insert).toHaveBeenCalled();
const insertCall = vi.mocked(successfulMessagesRepo.insert).mock.calls[0][0];
expect(insertCall.originalMessage).toBe('Hey gorgeous!');
expect(insertCall.imageAnalysis).toBe('Beautiful woman');
expect(insertCall.postTitle).toBe('My selfie');
expect(insertCall.subreddit).toBe('gonewild');
});
it('handles inbox parsing errors gracefully', async () => {
const message: SentMessageRecord = {
id: 1,
postId: 'post1',
recipient: 'user1',
subject: 'Hello',
message: 'Hey!',
sentAt: Date.now() - 2 * 3600000,
gotResponse: false,
responseCheckedAt: null,
strategyId: null,
};
messagesRepo = createMockMessagesRepo([message]);
checker = new ResponseCheckerService(page, messagesRepo, successfulMessagesRepo, postsRepo, logger);
vi.mocked(page.$$).mockRejectedValue(new Error('DOM error'));
const results = await checker.checkForResponses();
// Should not throw
expect(Array.isArray(results)).toBe(true);
});
});
describe('getSuccessfulExamples', () => {
it('returns formatted examples', () => {
const mockSuccessful = [
{
id: 1,
sentMessageId: 1,
originalMessage: 'Hey gorgeous!',
imageAnalysis: 'Blonde woman, rating 8/10',
postTitle: 'My selfie',
responseSnippet: 'Thanks!',
subreddit: 'gonewild',
createdAt: Date.now(),
},
];
successfulMessagesRepo = {
getRecent: vi.fn(() => mockSuccessful),
getWeightedExamples: vi.fn(() => mockSuccessful),
insert: vi.fn(),
existsByMessage: vi.fn(() => false),
count: vi.fn(() => 1),
} as unknown as SuccessfulMessagesRepository;
checker = new ResponseCheckerService(page, messagesRepo, successfulMessagesRepo, postsRepo, logger);
const examples = checker.getSuccessfulExamples(3);
expect(examples.length).toBe(1);
expect(examples[0]).toContain('Hey gorgeous!');
expect(examples[0]).toContain('Blonde woman');
expect(examples[0]).toContain('My selfie');
expect(examples[0]).toContain('Thanks!');
});
it('returns empty when no successful messages', () => {
successfulMessagesRepo = {
getRecent: vi.fn(() => []),
getWeightedExamples: vi.fn(() => []),
insert: vi.fn(),
existsByMessage: vi.fn(() => false),
count: vi.fn(() => 0),
} as unknown as SuccessfulMessagesRepository;
checker = new ResponseCheckerService(page, messagesRepo, successfulMessagesRepo, postsRepo, logger);
const examples = checker.getSuccessfulExamples(3);
expect(examples).toEqual([]);
});
it('uses weighted examples for specific subreddit', () => {
const mockSuccessful = [
{
id: 1,
sentMessageId: 1,
originalMessage: 'Hey!',
imageAnalysis: null,
postTitle: null,
responseSnippet: null,
subreddit: 'gonewild',
createdAt: Date.now(),
},
];
successfulMessagesRepo = {
getRecent: vi.fn(() => []),
getWeightedExamples: vi.fn(() => mockSuccessful),
insert: vi.fn(),
existsByMessage: vi.fn(() => false),
count: vi.fn(() => 1),
} as unknown as SuccessfulMessagesRepository;
checker = new ResponseCheckerService(page, messagesRepo, successfulMessagesRepo, postsRepo, logger);
checker.getSuccessfulExamples(3, 'gonewild');
expect(successfulMessagesRepo.getWeightedExamples).toHaveBeenCalledWith('gonewild', 3);
});
it('truncates long snippets', () => {
const mockSuccessful = [
{
id: 1,
sentMessageId: 1,
originalMessage: 'x'.repeat(500),
imageAnalysis: 'x'.repeat(500),
postTitle: 'Short',
responseSnippet: 'x'.repeat(500),
subreddit: null,
createdAt: Date.now(),
},
];
successfulMessagesRepo = {
getRecent: vi.fn(() => mockSuccessful),
getWeightedExamples: vi.fn(() => mockSuccessful),
insert: vi.fn(),
existsByMessage: vi.fn(() => false),
count: vi.fn(() => 1),
} as unknown as SuccessfulMessagesRepository;
checker = new ResponseCheckerService(page, messagesRepo, successfulMessagesRepo, postsRepo, logger);
const examples = checker.getSuccessfulExamples(1);
// imageAnalysis should be truncated to 300 chars + "..."
expect(examples[0]).toContain('...');
});
});
describe('getSuccessCount', () => {
it('returns the count from repository', () => {
successfulMessagesRepo = {
getRecent: vi.fn(() => []),
getWeightedExamples: vi.fn(() => []),
insert: vi.fn(),
existsByMessage: vi.fn(() => false),
count: vi.fn(() => 42),
} as unknown as SuccessfulMessagesRepository;
checker = new ResponseCheckerService(page, messagesRepo, successfulMessagesRepo, postsRepo, logger);
expect(checker.getSuccessCount()).toBe(42);
});
});
});
tests/retry.test.ts
+100
View File
@@ -0,0 +1,100 @@
import { describe, it, expect, vi } from 'vitest';
import { withRetry } from '../src/utils/retry.js';
import { createMockLogger } from '../tests/mockLogger.js';
describe('withRetry', () => {
it('succeeds on first attempt', async () => {
const fn = vi.fn(() => Promise.resolve('success'));
const result = await withRetry(fn, { retries: 3, minTimeout: 0 });
expect(result).toBe('success');
expect(fn).toHaveBeenCalledTimes(1);
});
it('retries on failure then succeeds', async () => {
let attempts = 0;
const fn = vi.fn(() => {
attempts++;
if (attempts < 3) return Promise.reject(new Error(`Attempt ${attempts} failed`));
return Promise.resolve('success');
});
const result = await withRetry(fn, { retries: 3, minTimeout: 0 });
expect(result).toBe('success');
expect(attempts).toBe(3);
});
it('throws after all retries exhausted', async () => {
const fn = vi.fn(() => Promise.reject(new Error('permanent failure')));
await expect(withRetry(fn, { retries: 2, minTimeout: 0 })).rejects.toThrow('permanent failure');
expect(fn).toHaveBeenCalledTimes(3); // 1 initial + 2 retries
});
it('logs warnings on failed attempts', async () => {
const logger = createMockLogger();
let attempts = 0;
const fn = vi.fn(() => {
attempts++;
return Promise.reject(new Error(`fail ${attempts}`));
});
await withRetry(fn, {
retries: 2,
logger,
operationName: 'test-op',
minTimeout: 0,
}).catch(() => {});
const warnLogs = logger.logs.filter(l => l.level === 'warn');
// 1 initial + 2 retries = 3 failed attempts, each logs a warning.
expect(warnLogs.length).toBe(3);
expect(warnLogs[0].message).toContain('test-op');
expect(warnLogs[0].message).toContain('fail 1');
expect(warnLogs[1].message).toContain('fail 2');
});
it('uses default operation name', async () => {
const logger = createMockLogger();
const fn = vi.fn(() => Promise.reject(new Error('fail')));
await withRetry(fn, { logger, retries: 1, minTimeout: 0 }).catch(() => {});
const warnLogs = logger.logs.filter(l => l.level === 'warn');
expect(warnLogs[0].message).toContain('operation');
});
it('supports custom retry count', async () => {
const fn = vi.fn(() => Promise.reject(new Error('fail')));
await withRetry(fn, { retries: 5, minTimeout: 0 }).catch(() => {});
expect(fn).toHaveBeenCalledTimes(6); // 1 initial + 5 retries
});
it('handles non-Error rejections', async () => {
const fn = vi.fn(() => Promise.reject('string error'));
// p-retry wraps non-Error rejections in a TypeError. The wrap is treated
// as an abort (no retry), so only the initial attempt runs.
await expect(withRetry(fn, { retries: 1, minTimeout: 0 })).rejects.toThrow('string error');
expect(fn).toHaveBeenCalledTimes(1);
});
it('passes options to retry correctly', async () => {
const logger = createMockLogger();
let attempts = 0;
const fn = vi.fn(() => {
attempts++;
return Promise.reject(new Error('fail'));
});
await withRetry(fn, {
retries: 1,
logger,
operationName: 'custom-op',
minTimeout: 0,
}).catch(() => {});
expect(attempts).toBe(2);
expect(logger.logs.some(l => l.message.includes('custom-op'))).toBe(true);
});
});
tests/strategy-selector.test.ts
+351
View File
@@ -0,0 +1,351 @@
import { describe, it, expect, beforeEach, vi } from 'vitest';
import { StrategySelector } from '../src/services/strategy-selector.js';
import { MessagesRepository } from '../src/database/repositories.js';
import type { StrategyStats } from '../src/database/repositories.js';
import { createMockLogger, getLogsByLevel } from '../tests/mockLogger.js';
import { getTestConfig } from '../tests/testUtils.js';
function createMockMessagesRepo(stats: StrategyStats[] = []): MessagesRepository {
return {
getStrategyStats: vi.fn(() => stats),
getSubredditStats: vi.fn(() => []),
getResponseRateTrend: vi.fn(() => []),
insert: vi.fn(() => 1),
getByPostId: vi.fn(),
getRecent: vi.fn(),
getUncheckedMessages: vi.fn(),
markGotResponse: vi.fn(),
markChecked: vi.fn(),
hasSentTo: vi.fn(),
} as unknown as MessagesRepository;
}
describe('StrategySelector', () => {
let selector: StrategySelector;
let logger: ReturnType<typeof createMockLogger>;
beforeEach(() => {
logger = createMockLogger();
});
it('returns null when no variants configured', () => {
const config = getTestConfig({ strategies: { variants: [] } });
const repo = createMockMessagesRepo();
selector = new StrategySelector(config, repo, logger);
const result = selector.selectStrategy();
expect(result).toBeNull();
});
it('hasStrategies returns false when no variants', () => {
const config = getTestConfig({ strategies: { variants: [] } });
const repo = createMockMessagesRepo();
selector = new StrategySelector(config, repo, logger);
expect(selector.hasStrategies()).toBe(false);
});
it('hasStrategies returns true when variants exist', () => {
const config = getTestConfig({ strategies: { variants: [
{ id: 'v1', weight: 1, promptOverride: 'test' },
] } });
const repo = createMockMessagesRepo();
selector = new StrategySelector(config, repo, logger);
expect(selector.hasStrategies()).toBe(true);
});
describe('explore mode', () => {
it('selects random variant when exploration rate is 1.0', () => {
const config = getTestConfig({ strategies: { explorationRate: 1.0, variants: [
{ id: 'v1', weight: 1, promptOverride: 'prompt1' },
{ id: 'v2', weight: 1, promptOverride: 'prompt2' },
] } });
const repo = createMockMessagesRepo();
selector = new StrategySelector(config, repo, logger);
const result = selector.selectStrategy();
expect(result).not.toBeNull();
expect(['v1', 'v2']).toContain(result!.id);
expect(result!.reason).toBe('explore');
});
it('selects random variant when exploration rate is 0.5', () => {
const config = getTestConfig({ strategies: { explorationRate: 0.5, variants: [
{ id: 'v1', weight: 1, promptOverride: 'prompt1' },
] } });
const repo = createMockMessagesRepo();
selector = new StrategySelector(config, repo, logger);
// With 50% exploration rate, we can't guarantee the outcome,
// but we can verify it returns a valid strategy
const results = new Set<string>();
for (let i = 0; i < 20; i++) {
const result = selector.selectStrategy();
if (result) results.add(result.id);
}
expect(results.size).toBeGreaterThan(0);
});
it('logs explore selection', () => {
const config = getTestConfig({ strategies: { explorationRate: 1.0, variants: [
{ id: 'v1', weight: 1, promptOverride: 'prompt1' },
] } });
const repo = createMockMessagesRepo();
selector = new StrategySelector(config, repo, logger);
selector.selectStrategy();
const debugLogs = getLogsByLevel(logger, 'debug');
expect(debugLogs.some(l => l.message.includes('explore'))).toBe(true);
});
});
describe('exploitation mode', () => {
it('selects best performing strategy when all have enough samples', () => {
const stats: StrategyStats[] = [
{ strategyId: 'v1', totalSent: 10, totalResponses: 2, responseRate: 20 },
{ strategyId: 'v2', totalSent: 10, totalResponses: 5, responseRate: 50 },
{ strategyId: 'v3', totalSent: 10, totalResponses: 1, responseRate: 10 },
];
const repo = createMockMessagesRepo(stats);
const config = getTestConfig({ strategies: { explorationRate: 0.0, minSamples: 5, variants: [
{ id: 'v1', weight: 1, promptOverride: 'p1' },
{ id: 'v2', weight: 1, promptOverride: 'p2' },
{ id: 'v3', weight: 1, promptOverride: 'p3' },
] } });
selector = new StrategySelector(config, repo, logger);
const result = selector.selectStrategy();
expect(result).not.toBeNull();
expect(result!.id).toBe('v2');
expect(result!.reason).toBe('exploit');
});
it('selects first variant when all have equal response rates', () => {
const stats: StrategyStats[] = [
{ strategyId: 'v1', totalSent: 10, totalResponses: 5, responseRate: 50 },
{ strategyId: 'v2', totalSent: 10, totalResponses: 5, responseRate: 50 },
];
const repo = createMockMessagesRepo(stats);
const config = getTestConfig({ strategies: { explorationRate: 0.0, minSamples: 5, variants: [
{ id: 'v1', weight: 1, promptOverride: 'p1' },
{ id: 'v2', weight: 1, promptOverride: 'p2' },
] } });
selector = new StrategySelector(config, repo, logger);
const result = selector.selectStrategy();
expect(result).not.toBeNull();
// When rates are equal, the first one with the best rate wins
// Since v1 has rate 50 and v2 has rate 50, v1 wins (first encountered)
expect(result!.id).toBe('v1');
});
it('selects best strategy when some have 0 responses', () => {
const stats: StrategyStats[] = [
{ strategyId: 'v1', totalSent: 10, totalResponses: 0, responseRate: 0 },
{ strategyId: 'v2', totalSent: 10, totalResponses: 3, responseRate: 30 },
];
const repo = createMockMessagesRepo(stats);
const config = getTestConfig({ strategies: { explorationRate: 0.0, minSamples: 5, variants: [
{ id: 'v1', weight: 1, promptOverride: 'p1' },
{ id: 'v2', weight: 1, promptOverride: 'p2' },
] } });
selector = new StrategySelector(config, repo, logger);
const result = selector.selectStrategy();
expect(result!.id).toBe('v2');
});
it('logs exploit selection with response rate', () => {
const stats: StrategyStats[] = [
{ strategyId: 'v1', totalSent: 10, totalResponses: 5, responseRate: 50 },
];
const repo = createMockMessagesRepo(stats);
const config = getTestConfig({ strategies: { explorationRate: 0.0, minSamples: 5, variants: [
{ id: 'v1', weight: 1, promptOverride: 'p1' },
] } });
selector = new StrategySelector(config, repo, logger);
selector.selectStrategy();
const debugLogs = getLogsByLevel(logger, 'debug');
expect(debugLogs.some(l => l.message.includes('exploit'))).toBe(true);
expect(debugLogs.some(l => l.message.includes('50'))).toBe(true);
});
});
describe('under-sampled strategies', () => {
it('selects under-sampled variant even in exploit mode', () => {
const stats: StrategyStats[] = [
{ strategyId: 'v1', totalSent: 10, totalResponses: 2, responseRate: 20 },
// v2 has no stats (never used)
];
const repo = createMockMessagesRepo(stats);
const config = getTestConfig({ strategies: { explorationRate: 0.0, minSamples: 5, variants: [
{ id: 'v1', weight: 1, promptOverride: 'p1' },
{ id: 'v2', weight: 1, promptOverride: 'p2' },
] } });
selector = new StrategySelector(config, repo, logger);
// v2 has 0 samples which is < minSamples(5), so it should be selected
const result = selector.selectStrategy();
expect(result!.id).toBe('v2');
expect(result!.reason).toBe('explore');
});
it('selects under-sampled variant with insufficient samples', () => {
const stats: StrategyStats[] = [
{ strategyId: 'v1', totalSent: 3, totalResponses: 1, responseRate: 33 },
// v2 has no stats
];
const repo = createMockMessagesRepo(stats);
const config = getTestConfig({ strategies: { explorationRate: 0.0, minSamples: 5, variants: [
{ id: 'v1', weight: 1, promptOverride: 'p1' },
{ id: 'v2', weight: 1, promptOverride: 'p2' },
] } });
selector = new StrategySelector(config, repo, logger);
// Both variants are under-sampled (v1: 3 < 5, v2: 0 < 5) so selection
// is random between them. Just assert the explore branch is taken.
const result = selector.selectStrategy();
expect(['v1', 'v2']).toContain(result!.id);
expect(result!.reason).toBe('explore');
});
it('skips under-sampled check when all have enough samples', () => {
const stats: StrategyStats[] = [
{ strategyId: 'v1', totalSent: 10, totalResponses: 2, responseRate: 20 },
{ strategyId: 'v2', totalSent: 10, totalResponses: 8, responseRate: 80 },
];
const repo = createMockMessagesRepo(stats);
const config = getTestConfig({ strategies: { explorationRate: 0.0, minSamples: 5, variants: [
{ id: 'v1', weight: 1, promptOverride: 'p1' },
{ id: 'v2', weight: 1, promptOverride: 'p2' },
] } });
selector = new StrategySelector(config, repo, logger);
const result = selector.selectStrategy();
expect(result!.id).toBe('v2');
expect(result!.reason).toBe('exploit');
});
it('prioritizes unexplored over under-sampled', () => {
// v1 has 10 samples (above minSamples), v2 has none — only v2 is unexplored.
const stats: StrategyStats[] = [
{ strategyId: 'v1', totalSent: 10, totalResponses: 5, responseRate: 50 },
];
const repo = createMockMessagesRepo(stats);
const config = getTestConfig({ strategies: { explorationRate: 0.0, minSamples: 5, variants: [
{ id: 'v1', weight: 1, promptOverride: 'p1' },
{ id: 'v2', weight: 1, promptOverride: 'p2' },
] } });
selector = new StrategySelector(config, repo, logger);
const result = selector.selectStrategy();
expect(result!.id).toBe('v2');
});
});
describe('getStats', () => {
it('returns all configured variants', () => {
const stats: StrategyStats[] = [
{ strategyId: 'v1', totalSent: 5, totalResponses: 2, responseRate: 40 },
];
const repo = createMockMessagesRepo(stats);
const config = getTestConfig({ strategies: { variants: [
{ id: 'v1', weight: 1, promptOverride: 'p1' },
{ id: 'v2', weight: 1, promptOverride: 'p2' },
] } });
selector = new StrategySelector(config, repo, logger);
const result = selector.getStats();
expect(result.length).toBe(2);
expect(result[0].configured).toBe(true);
expect(result[1].configured).toBe(true);
});
it('includes zero counts for variants with no data', () => {
const repo = createMockMessagesRepo([]);
const config = getTestConfig({ strategies: { variants: [
{ id: 'v1', weight: 1, promptOverride: 'p1' },
] } });
selector = new StrategySelector(config, repo, logger);
const result = selector.getStats();
expect(result[0].totalSent).toBe(0);
expect(result[0].totalResponses).toBe(0);
expect(result[0].responseRate).toBe(0);
});
it('includes historical strategies not in config', () => {
const stats: StrategyStats[] = [
{ strategyId: 'old-strategy', totalSent: 10, totalResponses: 3, responseRate: 30 },
];
const repo = createMockMessagesRepo(stats);
const config = getTestConfig({ strategies: { variants: [
{ id: 'v1', weight: 1, promptOverride: 'p1' },
] } });
selector = new StrategySelector(config, repo, logger);
const result = selector.getStats();
expect(result.length).toBe(2);
const historical = result.find(r => r.strategyId === 'old-strategy');
expect(historical).toBeDefined();
expect(historical!.configured).toBe(false);
});
it('maps response rates correctly', () => {
const stats: StrategyStats[] = [
{ strategyId: 'v1', totalSent: 10, totalResponses: 3, responseRate: 30 },
];
const repo = createMockMessagesRepo(stats);
const config = getTestConfig({ strategies: { variants: [
{ id: 'v1', weight: 1, promptOverride: 'p1' },
] } });
selector = new StrategySelector(config, repo, logger);
const result = selector.getStats();
expect(result[0].totalSent).toBe(10);
expect(result[0].totalResponses).toBe(3);
expect(result[0].responseRate).toBe(30);
});
});
describe('edge cases', () => {
it('handles single variant', () => {
const config = getTestConfig({ strategies: { explorationRate: 0.0, variants: [
{ id: 'only', weight: 1, promptOverride: 'only one' },
] } });
const repo = createMockMessagesRepo();
selector = new StrategySelector(config, repo, logger);
const result = selector.selectStrategy();
expect(result!.id).toBe('only');
});
it('handles many variants', () => {
const variants = Array.from({ length: 10 }, (_, i) => ({
id: `v${i}`,
weight: 1,
promptOverride: `prompt ${i}`,
}));
const config = getTestConfig({ strategies: { explorationRate: 1.0, variants } });
const repo = createMockMessagesRepo();
selector = new StrategySelector(config, repo, logger);
const results = new Set<string>();
for (let i = 0; i < 50; i++) {
const result = selector.selectStrategy();
if (result) results.add(result.id);
}
expect(results.size).toBeGreaterThan(1);
});
it('returns promptOverride from the selected variant', () => {
const config = getTestConfig({ strategies: { explorationRate: 1.0, variants: [
{ id: 'v1', weight: 1, promptOverride: 'My custom prompt' },
] } });
const repo = createMockMessagesRepo();
selector = new StrategySelector(config, repo, logger);
const result = selector.selectStrategy();
expect(result!.promptOverride).toBe('My custom prompt');
});
});
});
tests/testUtils.ts
+91
View File
@@ -0,0 +1,91 @@
import Database from 'better-sqlite3';
import type { Config } from '../src/config/index.js';
import { runMigrations } from '../src/database/client.js';
import { PostsRepository, MessagesRepository, SuccessfulMessagesRepository } from '../src/database/repositories.js';
export function createTestDb(): Database.Database {
const db = new Database(':memory:');
db.pragma('journal_mode = WAL');
// Disable FK for tests - we manage referential integrity manually
db.pragma('foreign_keys = OFF');
runMigrations(db);
return db;
}
export function createTestRepositories(db?: Database.Database) {
const database = db || createTestDb();
return {
postsRepo: new PostsRepository(database),
messagesRepo: new MessagesRepository(database),
successfulMessagesRepo: new SuccessfulMessagesRepository(database),
db,
};
}
export function getTestConfig(overrides?: Partial<Config>): Config {
return {
reddit: {
subreddits: [
{ name: 'test1', type: 'image' as const },
{ name: 'test2', type: 'image' as const },
],
pollIntervalMs: 1000,
userAgent: 'TestBot/1.0',
targetGenders: ['female'],
filterSellers: true,
sellerKeywords: ['onlyfans', 'fansly'],
...overrides?.reddit,
},
openrouter: {
apiKey: 'test-key',
imageAnalysisModel: 'google/gemini-flash',
imageAnalysisFallbackModels: ['fallback-model'],
chatModel: 'chat-model',
chatFallbackModels: [],
imageAnalysisPrompt: 'Analyze this image',
messageGenerationPrompt: 'Generate a message for {analysis}',
chatOpenerPrompt: 'Chat opener for {title} {selftext} {author} {gender}',
...overrides?.openrouter,
},
rateLimit: {
minDelayBetweenMessagesMs: 1000,
maxMessagesPerHour: 10,
jitterMs: 500,
...overrides?.rateLimit,
},
browser: {
sessionDir: './data/browser-session',
headless: true,
dryRun: false,
...overrides?.browser,
},
database: {
path: './data/test.db',
...overrides?.database,
},
media: {
syncIntervalMs: 1800000,
syncEnabled: true,
},
strategies: {
explorationRate: 0.2,
minSamples: 5,
variants: [
{ id: 'variant-a', weight: 1, promptOverride: 'Be direct' },
{ id: 'variant-b', weight: 1, promptOverride: 'Be teasing' },
],
...overrides?.strategies,
},
chatStrategies: {
explorationRate: 0.2,
minSamples: 5,
variants: [],
...overrides?.chatStrategies,
},
web: {
enabled: true,
port: 3000,
...overrides?.web,
},
};
}
vitest.config.ts
+15
View File
@@ -0,0 +1,15 @@
import { defineConfig } from 'vitest/config';
export default defineConfig({
test: {
globals: false,
environment: 'node',
include: ['tests/**/*.test.ts'],
coverage: {
provider: 'v8',
reporter: ['text', 'json', 'html'],
exclude: ['node_modules/', 'tests/', 'dist/'],
},
testTimeout: 30000,
},
});