RedditBot/docs/data-flow.md

Runtime flows

Candidate funnel

The production multi-account path is three independent loops backed by candidate_queue.

stateDiagram-v2
    [*] --> discovered: listing intake
    discovered --> screening: leased by screening worker
    screening --> ready: eligibility and preparation pass
    screening --> retry_wait: transient failure
    screening --> rejected: terminal eligibility or AI failure
    ready --> sending: leased to available account
    sending --> sent: Matrix confirms conversation
    sending --> retry_wait: account cooldown or transient failure
    sending --> rejected: terminal delivery result
    retry_wait --> discovered: screening retry due
    retry_wait --> ready: delivery retry due
    discovered --> expired: outside configured recent window
    ready --> expired: outside configured recent window

1. Discovery

The discovery loop rotates through enabled subreddits and authenticated runtimes. A runtime fetches Reddit /new data using the process-shared RedditRequestCoordinator. In the default post-author targeting mode those posts enter intake directly. An image subreddit configured for comment-author instead reads top-level comments from at most five newly listed posts, batches previously unseen eligible comment text through the reaction classifier, and creates candidates only for role-alignment verdicts at or above 0.8 confidence. A bounded in-process cache prevents unchanged comments from repeatedly consuming model calls. The candidate ID, author, timestamp, and permalink come from the comment while the original post image and title remain attached as preparation context; the usual image screen later verifies the depicted gender against reddit.targetGenders. Intake then deduplicates candidate IDs and authors, excludes processed/contacted history, stores the full snapshot, source type, weight, and timestamps, and logs exact intake counts.

The queue grows to a demand-based target rather than a fixed backlog. The target is the smaller of configured reddit.maxReadyCandidates and the work that authenticated running accounts can still send within the recent-post window, accounting for rolling hourly capacity and minimum message spacing. Intake admits only the current deficit, newest Reddit posts first, pauses at zero outbound capacity, and reports fetched posts left unevaluated because the funnel is full. Reddit request pacing, not a tight application poll, remains the primary request limiter.

2. Screening and preparation

A screening worker leases one discovered candidate. MonitorService.screenCandidate() applies cheap rules before expensive/networked rules:

1. Global processed-post and recipient-history checks.
2. For chat sources, self-declared gender and adult age parsing.
3. Seller evidence from the post, profile/about (including structured social links when exposed and narrow NSFW link-funnel language), and recent activity.
4. Source-specific preparation in ProcessorService.

For image posts, preparation downloads and validates the image, detects target gender, and performs model analysis. The analysis includes a separate 1–10 visible-physique score only when enough of the body is visible. Its strict anchors place a broadly average visible physique at 5, reserve 8–10 for rare athletic or model-like physiques, and explicitly forbid using 6 as a safe default; nudity, breast size, face, pose, and photographic quality do not affect it. If physique targeting is enabled, an unknown score or a score outside the configured inclusive range becomes an explicit rejection before strategy selection and opener generation. Passing image posts then select an image strategy and generate an opener. Chat posts do not use physique targeting; they select a chat strategy and generate from the post text. Both paths apply outbound safety checks and return either a prepared immutable result or an explicit skip reason.

Terminal results become rejected candidates. Rate limits and transient failures become retryable candidates. Prepared candidates become ready; the persisted snapshot includes the opener, grounded image analysis, attractiveness and physique scores, the physique range used, source-image URL, and experiment/model/tone metadata. This lets a different account deliver without repeating AI work and lets the authenticated operator inspect the exact preparation context. Saving a physique-range change transactionally re-evaluates prepared image candidates in ready and delivery-retry stages, rejects those outside the new range, and refreshes the displayed range snapshot for retained candidates. A candidate already leased for sending is left unchanged until that send resolves.

3. Dispatch

An account is eligible for new outreach only when its runtime is running, both identities verify, it has no actionable reply backlog, there is no invitation cooldown, and its rate limiter permits a send. Available accounts are ordered by their current hourly send count.

Matrix cooldowns retain the operation that triggered them. A room-creation or invitation limit pauses new outreach but does not suppress replies in existing rooms; a limit returned while sending into an existing room pauses those sends as well.

The operator can manually clear the unsent funnel from the authenticated dashboard. The repository atomically moves discovered, screening, ready, and retryable candidates to a terminal rejection with the reason Cleared manually by operator, releases their leases, and retains the audit rows so those candidates are not rediscovered. A sending lease is never changed because its remote Matrix outcome may already be in flight. Clearing does not change bot mode; running discovery may refill the funnel immediately.

The same surface supports individual cancellation and prepared-message editing. Individual cancellation uses the same terminal audit transition and refuses sending or completed rows. Editing is limited to ready and retryable candidates with a preparation snapshot; it atomically replaces only the opener, clears generated strategy/experiment selection metadata, labels the model attribution operator-edited, and re-runs the deterministic outbound location/meeting safeguard. Both operations return HTTP 409 when a stale dashboard loses a race to dispatch.

Screening, retry promotion, dispatch, and the operator's active-queue view are ordered by the Reddit post creation time, newest first; source priority is only a tie-breaker. The dispatch worker leases the newest ready candidate for an account, materializes the processed post in that account's repository, applies the account rate limiter, checks dry-run, and atomically claims the recipient globally. ChatService.startConversationDetailed() returns one of:

• sent: persist the message and outcome metadata, commit recipient history, update activity, and mark the candidate sent;
• terminal: release the recipient reservation and reject the candidate;
• retryable: release/defer so another healthy account can claim it soon.

Only Matrix confirmation makes a send successful. A healthy HTTP process, room identifier lookup, or attempted request does not.

Reddit request and cooldown flow

Every runtime shares one RedditRequestCoordinator, because Reddit can enforce limits by origin as well as account. JSON requests are serialized with a minimum interval; failed requests consume spacing too. A 429 extends the global cooldown using the server delay plus a buffer. Broad 403 behavior can also result in coordinated backoff. A listing 404 is treated as terminal for that subreddit and persisted as enabled=false.

Do not add local retries around this layer without preserving the shared schedule. Per-account bursts defeat the coordinator.

Browsing session flow

After reply handling, response checks, and comment-growth work, an enabled account may run one due browsing session. The durable next_due_at gate survives restarts; each completed attempt schedules the next one at the operator's minimum gap plus bounded random jitter, moved past quiet hours when necessary. The session picks one configured growth community by weight, reads its hot listing through the shared Reddit request coordinator, optionally reads typed top-level comments, applies deterministic exclusions, and only then selects up to the configured session size without replacement. Scheduled sessions cannot select beyond the remaining deterministic daily target.

Each terminal selection is recorded as pending in SQLite before its remote write, so an interrupted or uncertain vote is never retried. With the separate automatic-upvote setting enabled, running mode sends authenticated Reddit api/vote requests sequentially through the same browser transport and shared coordinator as growth comments, then marks each audit record submitted only after Reddit confirms it; safe failures become failed. Scan mode and browser dry-run record simulated selections; stopped mode performs no discovery or decision work. Reddit 429 responses extend the shared cooldown, while ordinary read failures remain isolated from replies and comment growth. An operator can also run one immediate session for an enabled, running account: it bypasses the due, quiet-hours, and normal daily-target gates, uses the configured session size, preserves content exclusions, and follows the same automatic-upvote and dry-run rules.

Reply-first conversation automation

Explicit bot clients can send to a username through POST /api/v1/messages or the MCP send_message_to_user tool without selecting an account. The selector compares cached conversations case-insensitively and reuses the account that most recently interacted with that recipient. If no conversation exists, it chooses an authenticated running account with no invitation cooldown, preferring the lowest hourly send count. The selected account's normal rate limiter still owns the send. An API reply to an existing conversation marks automation as user-overridden so autonomous replies do not race the external caller.

Each account runtime refreshes recent chats and processes actionable inbound conversations oldest-first before it can accept new outreach. After the initial conservative startup scan, the persisted latest Matrix message event plus the worker's prior-scan watermark is authoritative: a newly observed inbound event newer than the confirmed opener remains actionable even if Matrix's presentation-only unread notification count has already returned to zero. On startup, read-marked rooms without a pending reply claim are not revived. Durable event claims, conversation policy state, and a later outbound event prevent duplicate replies.

Every running-mode reply scan also applies the shared deterministic seller/off-platform rules to the complete cached inbound transcript before actionability is calculated. This retrospective pass catches Telegram, Snapchat, other contact-platform pushes, payment requests, verification scams, paid-content funnels, and configured seller evidence that arrived before a restart or after an operator took ownership. It uses an in-process transcript watermark, spends no classifier tokens, persists the seller/terminal state, and moves killed conversations to Done. Scan mode remains non-mutating. The operator-forced seller scan uses the same deterministic decision before its optional batched AI fallback.

Outgoing Matrix events are attributed against the recorded opener, durable automated-reply fingerprints, and confirmed exchange-library event records. If synchronization observes an unmatched outgoing message, the conversation becomes operator-owned before another reply can be generated. An unmatched outgoing message also supersedes, rather than confirms, a pending automated claim; only the claimed reply text can reconcile that claim as sent.

flowchart TD
    IN["Unread inbound Matrix event"] --> VERIFIED{"Confirmed after a recorded opener?"}
    VERIFIED -- no --> IGNORE["Ignore as unverified"]
    VERIFIED -- yes --> CLAIM{"Durable event claim exists?"}
    CLAIM -- sent --> IGNORE
    CLAIM -- pending --> RECON["Reconcile against Matrix history"]
    CLAIM -- no --> STATE["Load or snapshot conversation policy"]
    STATE --> STOP{"Manual, seller, image, boundary, unsafe, or turn limit?"}
    STOP -- yes --> HANDOFF["Persist terminal/human-required outcome"]
    STOP -- no --> GEN["Generate policy-specific reply"]
    GEN --> SAFE{"Outbound checks pass?"}
    SAFE -- no --> HANDOFF
    SAFE -- yes --> PERSIST["Persist intended reply claim"]
    PERSIST --> WAIT["Wait for the account rate limiter"]
    WAIT --> RECHECK{"Refresh Matrix; claimed event still latest and safe?"}
    RECHECK -- no --> SUPERSEDE["Mark claim superseded; persist boundary/handoff if present"]
    RECHECK -- yes --> SNAPSHOT["Persist intended conversation state"]
    SNAPSHOT --> SEND["Matrix send"]
    SEND --> MARK["Mark claim sent and record latency"]

The inbound event key, intended reply, claim status, and conversation state make retries idempotent. Immediately after any limiter wait, automation refreshes Matrix and revalidates the latest inbound event, boundaries, images, the shared seller/off-platform decision, policy status, and the one-media-request limit. Newer state makes the old claim terminally superseded; it is never sent. If the replacement event is a seller or off-platform push, the same revalidation ends and classifies the conversation instead of leaving it for a later pass. If the process dies after Matrix accepted a reply, reconciliation finds the outbound event and marks the old claim sent instead of sending it again.

General R4R generation explicitly permits consensual adult NSFW conversation. A model response that softly refuses, moralizes, or diverts solely because the topic is sexual is rejected before send and falls through to the next configured chat model. General replies may offer a generic approved saved picture without that offer being miscounted as a request for recipient media.

Legacy conversations may make at most one optional autonomous media request, including a request already present in the opener. The hybrid policy retains its single rapport-gated media ask. Deterministic, sequence-aware refusal checks run before generation and again at send time; generated media asks must pass the same non-explicit, non-coercive validator.

Chat and media synchronization

ChatService communicates with Reddit's Matrix homeserver, maintains an account-scoped SQLite-backed cache, resolves t2_ identities to Reddit usernames, and emits sync progress/new-reply callbacks. Startup chooses full or delta sync from persisted cursor state. Operators can explicitly run tracked sync jobs through the versioned API.

Each authenticated account runtime also checks the Reddit profiles represented in its cached conversations every four hours. Profile results are shared process-wide by case-insensitive username, so a recently checked recipient is not looked up again by another account runtime. A valid profile is recorded as active, Reddit's explicit suspension marker is recorded separately, and only an HTTP 404 is recorded as deleted; authentication, rate-limit, network, server, and malformed-response failures do not change the previous state. The dashboard removes deleted recipients from actionable conversation queues and exposes them through the dedicated Account deleted bucket.

MediaService resolves Matrix media, downloads it into the persistent data directory, and writes an account-scoped media index. Each Matrix message occurrence gets its own gallery record, while repeated occurrences of the same remote asset reuse one private file. Generic download content types are verified from the image bytes before an item is rejected as unsupported.

MediaSyncService schedules incremental work only while the account is authenticated. Incremental sync consumes the rooms changed since the durable Matrix cursor; the reported count is the number of rooms Matrix actually returned, not an assumed inbox limit. The cursor advances only after media attachment and cache persistence succeed, so a transient media failure is retried. When Matrix marks a room timeline as limited, chat sync paginates that room's history before advancing the cursor to close the omitted-event gap.

Full media sync walks every joined room and paginates until Matrix reports the end of history. Per-room history or media failures leave the full-sync marker unset and make an operator-triggered full media sync fail visibly; a later full sync can retry the incomplete rooms. Startup may retain the successfully loaded partial cache for availability, but it does not claim that a full backfill completed.

The media listing reads persisted indexes for every stored account, including accounts whose runtimes are stopped or require login, and exposes received items only: records attributed to the owning Reddit/Matrix account are excluded, including historical self-sender forms. Private media is served behind the same authorization boundary as the dashboard and APIs.

The process-global exchange library is separate. Dashboard uploads are decoded and re-encoded with sharp, which validates the actual JPEG/PNG/WebP contents, applies orientation, and removes EXIF/location/camera metadata. OpenRouter then assigns overlapping anatomy tags plus SFW/NSFW classification. Failed or low-confidence classifications remain disabled until the operator corrects them.

For each latest inbound text event, MediaLibraryExchangeService detects direct or indirect image requests before normal reply generation. A clear affirmative response to the immediately preceding generic bot media offer is also a generic request; unrelated agreement is not. Exact requests require the matching tag. Generic requests use the conversation's completed library-send count to broaden the eligible safety tier and randomly choose an unused asset. A unique durable reservation on account + conversation + asset prevents reuse, while account + conversation + inbound event makes restart handling idempotent.

After the normal reply limiter waits, automation refreshes Matrix and rechecks that the same inbound request is still latest and that no image, boundary, seller signal, or other terminal state has appeared. It then uploads the private file to Matrix, sends an m.image event with a stable transaction ID, and only after confirmation sends the optional reciprocal text. If the image is confirmed but the text is not, the asset remains consumed and the conversation is handed to the operator rather than risking a duplicate.

The conversation UI also supports an explicitly confirmed manual choice from this same eligible set. Its read endpoint reports request-matched options without exposing filesystem paths. The send endpoint refreshes the conversation inside the owning account's rate limiter, rechecks the latest inbound request and safety boundaries, reserves the selected asset in the same durable ledger, and uses the same Matrix image-plus-reciprocal delivery path. A successful manual share marks automation as user-overridden.

Conversation performance is derived from the complete attributed inbound transcript. Engagement cannot become a prompt-learning example while automation is active, and a later decline, boundary, or upsell removes an earlier example before future prompts load it.

Dashboard data and events

Routes assemble snapshots from repositories and injected services. LogBroadcaster/the realtime event hub publishes typed events for bot state, sends, failures, conversations, replies, sync, sessions, rate limits, and logs. The React client validates incoming events, invalidates matching TanStack Query keys, and reconnects with replay support. Cached screens remain visible but stale during a disconnect.

Failure classification

Use three categories consistently:

• Terminal: the candidate/action is unsuitable or impossible; persist a reason and do not retry automatically.
• Retryable: service limits, temporary auth/network errors, or account cooldown; preserve the candidate and retry time.
• Human-required: policy or safety cannot continue autonomously; stop automation and surface an attention item.

Avoid converting errors into generic skips. Operational diagnosis depends on preserving which stage failed and why.

Human learning loop

After a room has an authoritative Matrix timeline, ChatService records the canonical response outcome and invokes the account-scoped learning observer. The observer:

1. reconciles opener, automation-claim, exchange-library, manual-dashboard, recipient, and legacy events;
2. stores text-minimising event provenance with an attribution confidence;
3. finds received-image outcomes and the manual decisions that preceded them;
4. assigns human-created, human-improved, human-completed, autonomous, recipient-initiated, or uncertain credit;
5. stores bounded, identifier-redacted decision context; and
6. leaves new trajectories provisional until the maturity window passes.

Full chat synchronization naturally backfills older rooms. Upserts are keyed by account, room, and outcome event, so the pass is repeatable. A later negative canonical outcome invalidates both the trajectory and every derived decision.

Retrieval is account-scoped and excludes the current conversation. It uses explicit human and autonomous quotas, same-policy/stage preference, confidence filtering, and a generation audit ledger. With the default configuration it runs in shadow mode: selected IDs are recorded but examples are not put into the prompt.