Harden conversation replies and prioritize newest candidates

docs/data-flow.md

@@ -25,7 +25,7 @@ stateDiagram-v2
 
 The discovery loop rotates through enabled subreddits and authenticated runtimes. A runtime fetches Reddit `/new` data using the process-shared `RedditRequestCoordinator`. Intake deduplicates post IDs and authors, excludes processed/contacted history, stores the full post snapshot, source type, weight, and timestamps, and logs exact intake counts.
 
-The queue grows to a demand-based target rather than a fixed backlog. The target is the smaller of configured `reddit.maxReadyCandidates` and the work that authenticated running accounts can still send within the recent-post window, accounting for rolling hourly capacity and minimum message spacing. Intake admits only the current deficit, pauses at zero outbound capacity, and reports fetched posts left unevaluated because the funnel is full. Reddit request pacing, not a tight application poll, remains the primary request limiter.
+The queue grows to a demand-based target rather than a fixed backlog. The target is the smaller of configured `reddit.maxReadyCandidates` and the work that authenticated running accounts can still send within the recent-post window, accounting for rolling hourly capacity and minimum message spacing. Intake admits only the current deficit, newest Reddit posts first, pauses at zero outbound capacity, and reports fetched posts left unevaluated because the funnel is full. Reddit request pacing, not a tight application poll, remains the primary request limiter.
 
 ### 2. Screening and preparation
 
@@ -46,7 +46,7 @@ An account is eligible for new outreach only when its runtime is running, both i
 
 Matrix cooldowns retain the operation that triggered them. A room-creation or invitation limit pauses new outreach but does not suppress replies in existing rooms; a limit returned while sending into an existing room pauses those sends as well.
 
-The dispatch worker leases a ready candidate for an account, materializes the processed post in that account's repository, applies the account rate limiter, checks dry-run, and atomically claims the recipient globally. `ChatService.startConversationDetailed()` returns one of:
+Screening, retry promotion, dispatch, and the operator's active-queue view are ordered by the Reddit post creation time, newest first; source priority is only a tie-breaker. The dispatch worker leases the newest ready candidate for an account, materializes the processed post in that account's repository, applies the account rate limiter, checks dry-run, and atomically claims the recipient globally. `ChatService.startConversationDetailed()` returns one of:
 
 - `sent`: persist the message and outcome metadata, commit recipient history, update activity, and mark the candidate sent;
 - `terminal`: release the recipient reservation and reject the candidate;
@@ -77,12 +77,18 @@ flowchart TD
     STOP -- no --> GEN["Generate policy-specific reply"]
     GEN --> SAFE{"Outbound checks pass?"}
     SAFE -- no --> HANDOFF
-    SAFE -- yes --> PERSIST["Claim event and persist intended state"]
-    PERSIST --> SEND["Rate-limited Matrix send"]
+    SAFE -- yes --> PERSIST["Persist intended reply claim"]
+    PERSIST --> WAIT["Wait for the account rate limiter"]
+    WAIT --> RECHECK{"Refresh Matrix; claimed event still latest and safe?"}
+    RECHECK -- no --> SUPERSEDE["Mark claim superseded; persist boundary/handoff if present"]
+    RECHECK -- yes --> SNAPSHOT["Persist intended conversation state"]
+    SNAPSHOT --> SEND["Matrix send"]
     SEND --> MARK["Mark claim sent and record latency"]
 ```
 
-The inbound event key, intended reply, claim status, and conversation state make retries idempotent. If the process dies after Matrix accepted a reply, reconciliation finds the outbound event and marks the old claim sent instead of sending it again.
+The inbound event key, intended reply, claim status, and conversation state make retries idempotent. Immediately after any limiter wait, automation refreshes Matrix and revalidates the latest inbound event, boundaries, images, seller evidence, policy status, and the one-media-request limit. Newer state makes the old claim terminally `superseded`; it is never sent. If the process dies after Matrix accepted a reply, reconciliation finds the outbound event and marks the old claim sent instead of sending it again.
+
+Legacy conversations may make at most one optional autonomous media request, including a request already present in the opener. The hybrid policy retains its single rapport-gated media ask. Deterministic, sequence-aware refusal checks run before generation and again at send time; generated media asks must pass the same non-explicit, non-coercive validator.
 
 ## Chat and media synchronization
 
@@ -90,6 +96,8 @@ The inbound event key, intended reply, claim status, and conversation state make
 
 `MediaService` resolves Matrix media, downloads it into the persistent data directory, and writes an account-scoped media index. `MediaSyncService` schedules incremental work only while the account is authenticated. Private media is served behind the same authorization boundary as the dashboard and APIs.
 
+Conversation performance is derived from the complete attributed inbound transcript. Engagement cannot become a prompt-learning example while automation is active, and a later decline, boundary, or upsell removes an earlier example before future prompts load it.
+
 ## Dashboard data and events
 
 Routes assemble snapshots from repositories and injected services. `LogBroadcaster`/the realtime event hub publishes typed events for bot state, sends, failures, conversations, replies, sync, sessions, rate limits, and logs. The React client validates incoming events, invalidates matching TanStack Query keys, and reconnects with replay support. Cached screens remain visible but stale during a disconnect.

docs/database.md

@@ -21,7 +21,7 @@ The special legacy-unassigned account exists only to preserve rows created befor
 
 - `processed_posts`: account/post key, source metadata, analysis, generated opener, send marker, and terminal error/rejection reason.
 - `sent_messages`: account-owned opener/reply attribution, strategy/experiment/model metadata, Matrix room, response/image flags, and source type.
-- `successful_messages`: examples derived from positive outcomes for prompt context.
+- `successful_messages`: mature positive examples for prompt context; terminal negative reclassification removes the linked example.
 - `conversation_outcomes`: normalized outcome for a sent message (`engaged`, `declined`, `boundary`, `upsell`, or `unknown`).
 - `activity_log`: operator-facing events with optional recipient/post context.
 - `reddit_users`: global mapping between Reddit fullname/username and Matrix identity.
@@ -32,7 +32,7 @@ The special legacy-unassigned account exists only to preserve rows created befor
 - `candidate_queue`: global candidate snapshot, source/priority, state, prepared payload, worker lease, retry timing, selected account, and rejection/error details.
 - `delivery_outbox`: account-scoped prepared delivery and idempotency status retained for legacy/recovery paths.
 - `conversation_states`: account/conversation policy and derived state JSON.
-- `conversation_automation_claims`: account/conversation/inbound-event claim with intended reply and pending/sent state.
+- `conversation_automation_claims`: account/conversation/inbound-event claim with intended reply and `pending`, `sent`, or terminal `superseded` state plus audit timestamps.
 - `runtime_state`: small process controls such as whether bot work is running.
 
 ### Configuration and telemetry
@@ -71,10 +71,12 @@ Any new candidate state needs:
 
 ## Conversation reply claims
 
-The unique identity is account + conversation + inbound message key. The claim persists the intended reply before the remote send. A pending claim is not blindly retried: Matrix history is checked first. This closes the process-crash window between remote acceptance and local completion.
+The unique identity is account + conversation + inbound message key. The claim persists the intended reply before the remote send. A pending claim is not blindly retried: Matrix history is checked first. After rate-limit waiting, Matrix is refreshed again; a newer event, boundary, received image, seller signal, operator-owned state, or repeated media request changes the claim to `superseded` and records `superseded_at`. This closes both the process-crash window between remote acceptance and local completion and the stale-send window while a reply is queued.
 
 Changes to this flow must prove at-most-once observable replies across retry and restart tests.
 
+Outcome rows are re-evaluated from the complete attributed recipient transcript. Negative outcomes take precedence over reciprocal activity. `successful_messages` rows are created only after the conversation becomes terminal/human-owned and are deleted by `sent_message_id` when later evidence changes the outcome.
+
 ## Configuration persistence
 
 On an empty database, `config.yaml` is parsed, environment-owned values are injected, the result is validated, and the application-owned subset is saved to `app_config`. Later starts load SQLite first. Dashboard changes update that record.

docs/product-and-safety.md

@@ -19,7 +19,7 @@ The product has five connected goals:
 - Candidate inventory follows near-term outbound capacity and the recent-post window; discovery pauses instead of accumulating an ever-older backlog when accounts cannot keep pace.
 - Discovery is global while authentication, rate limits, conversations, and content remain account-owned.
 - Recipient deduplication is case-insensitive, durable across restarts, and protected by a live reservation during sends.
-- Remote side effects are fail-closed: a send is recorded only after the chat service confirms it; interrupted reply claims reconcile against Matrix history before retrying.
+- Remote side effects are fail-closed: a send is recorded only after the chat service confirms it; interrupted reply claims reconcile against Matrix history before retrying. A queued reply refreshes Matrix immediately before sending and becomes an auditable superseded claim if a newer inbound event or terminal state appeared while it waited.
 - Dry-run prepares and records decisions but never sends.
 - Operators can see funnel stages, account state, rate limits, failures, handoffs, and recent activity rather than inferring success from a process-health check.
 
@@ -41,11 +41,11 @@ Each subreddit snapshots a policy into a conversation when automation first hand
 | Mode | Meaning |
 | --- | --- |
 | `manual` | Never auto-reply; surface a human handoff. |
-| `legacy` | A short, bounded media-request flow retained for existing behavior. |
+| `legacy` | At most one optional autonomous media request, counting a request already present in the opener, followed by human handoff. |
 | `r4r` | Bounded rapport-style replies with autonomous safety checks. |
 | `r4r-media` | Rapport first, one bounded media ask, then a handoff/terminal boundary. |
 
-The snapshot matters: changing configuration later must not silently rewrite the rules for an active conversation. Automation stops for seller/upsell evidence, an image received, a detected boundary, unsafe content, an exhausted turn budget, or a manual policy.
+The snapshot matters: changing configuration later must not silently rewrite the rules for an active conversation. Automation stops for seller/upsell evidence, an image received, a detected boundary, unsafe content, an exhausted turn budget, a repeated media request, or a manual policy. Short refusals are interpreted in sequence: phrases such as “I’m good” become media boundaries only when they answer a media request.
 
 ## Success and analytics
 
@@ -56,6 +56,8 @@ Do not collapse all progress into “messages sent.” Useful states include:
 - replies, declined/boundary/upsell outcomes, and images received;
 - model usage, fallback behavior, strategy/sample size, and account ownership.
 
+Learning examples mature only after the conversation policy is no longer active. Full recipient history is reclassified on sync; a later decline or boundary overrides earlier reciprocal activity and removes any previously promoted example.
+
 Fetched listing size is not an eligibility count. Logs and UI should quantify rejection and deferral reasons so a dry funnel can be diagnosed without guessing.
 
 ## Operational authority

docs/services.md

@@ -36,8 +36,9 @@ This is an ownership map, not an API dump. Read the owning implementation and te
 | `services/chat/cache.ts` | Account-scoped cached conversations and sync state | SQLite is durable; do not share caches across accounts. |
 | `services/chat/matrix/username-resolver.ts` | Matrix/Reddit identity mapping | Preserve unresolved internal IDs without leaking them in operator text. |
 | `services/chat/matrix/token-refresher.ts` | Explicit refresh-unavailable behavior | New credentials come from verified extension dumps, not token sniffing. |
-| `services/conversation-automation.ts` | Reply-first policy, claims, retries, handoffs, turn budgets | Snapshot policy; claim before send; reconcile pending claims; fail closed. |
+| `services/conversation-automation.ts` | Reply-first policy, claims, retries, send-time refresh, handoffs, turn budgets | Snapshot policy; claim before send; supersede stale queued work; reconcile pending claims; fail closed. |
 | `services/conversation-state.ts` | Pure conversation parsing/state/safety helpers | Conversation text is untrusted data; keep prompt boundaries explicit. |
+| `services/conversation-boundaries.ts` | Sequence-aware media requests, refusals, and full-stop detection | Short ambiguous replies become boundaries only in the matching request context. |
 | `services/outbound-message-safety.ts` | Final no-proximity/no-meeting rule | This is defense in depth beyond prompts. |
 | `services/conversation-outcome*.ts` | Response outcome classification/recording | Keep analytics attribution account/message scoped. |
 | `services/response-checker.ts` | Inbox/performance response checks and successful examples | Non-overlapping; do not confuse chat sync with outcome maturity. |