0 Fork 0

Files

T

Jamie BruntonandClaude Opus 5 142378df33 feat: show the conversation evidence behind each learning trajectory

Reviewers were asked to include or exclude learning trajectories while
only seeing metadata — credit class, confidence, status. The decisions
and their redacted context turns were already captured and stored, but
no endpoint returned them, so the evidence was unreachable from the UI.

Add GET /api/learning/trajectories/:id and an expandable row on the
learning page that fetches on demand.

Also widen SentMessageSchema to accept matrixEventId, imageOutcomeCredit
and imageOutcomeAt. The learning loop added these to SentMessageRecord
and they flow out of /api/messages, but the strict contract rejected
them, breaking the dashboard.

Enable generation with shadow mode off so eligible examples reach reply
prompts.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

2026-08-12 08:23:22 +01:00

4.4 KiB

Raw Blame History

Human-led conversation learning

Purpose

The loop learns from the decision that caused a useful outcome, not merely from the last AI message before it. Manual interventions are first-class evidence because an operator may repair tone, timing, boundaries, or a media invitation before a recipient shares an image.

Pipeline and gates

authoritative Matrix timeline
  -> event attribution (exact > fingerprint > inferred)
  -> outcome trajectory and causal credit
  -> identifier-redacted decision examples
  -> maturity / review / invalidation gates
  -> account-scoped, quota-controlled retrieval
  -> shadow audit or injected prompt context
  -> offline evaluation record

Capture and use are independent. The default configuration captures and audits retrieval in shadow mode, but cannot affect generated replies. Injection requires both generationEnabled: true and shadowMode: false.

Attribution and credit

Authoritative Matrix event IDs link openers, automated claims, dashboard manual sends, and exchange-library sends. Fingerprints support legacy backfill only. An unmatched outbound event is human-owned for the automation safety boundary, but inferred learning is held back when exact attribution is required.

Credit classes are:

human-created-success: the human introduced the successful decision.
human-improved-success: AI attempted the step and the human replaced or improved it.
human-completed-ai-trajectory: the human intervention helped complete an AI-started exchange without directly making the media ask.
autonomous-success: the bounded automated policy made the attributed ask.
recipient-initiated-media: media arrived without an attributed ask.
uncertain: provenance is insufficient for causal use.

Image receipt does not override a decline, boundary, or upsell outcome. Those outcomes invalidate the trajectory and all derived decisions. Operator exclusion is sticky across re-sync; explicit inclusion can reverse review/exclusion, but cannot resurrect an invalidated safety outcome.

Retrieval and evaluation

Retrieval stays within the owning account, excludes the current room, filters confidence, prefers the same policy/stage/community, and fills separate human and autonomous quotas. Stored context is bounded and redacted. The generation ledger stores selected decision IDs, mode, policy version, and an optional prompt hash—not prompt text.

npm run learning:eval -- --db <path> --policy <version> versions the current eligible dataset and enforces hard gates for leaked identifiers, invalid decision states, and empty outputs. Evaluation is deliberately local and deterministic. More subjective pairwise or model-graded evaluation can be added behind the same saved-run contract.

Infrastructure and framework boundary

No new Dokploy service is required. SQLite, the existing backend, Matrix synchronization, OpenRouter client, and React dashboard are sufficient. Deployment needs only the normal application image plus a database backup before the migration; it does not require a vector database, queue, object store, or hosted tracing system.

The repository/provider/evaluation interfaces are the framework seams. A hosted platform such as Langfuse or an experiment tool can later receive generation and evaluation records, and DSPy-style optimization or model fine-tuning can consume exported reviewed examples. None is placed in the online reply path until volume, evaluation quality, and operational need justify the extra infrastructure.

Rollout

Deploy with capture enabled, generation disabled, and shadow mode enabled.
Run a full chat sync to backfill authoritative histories.
Review inferred and fingerprint-attributed trajectories in /learning. Expanding a row calls GET /api/learning/trajectories/{id} and shows the redacted context turns and chosen response behind each decision, so include/exclude is a judgement on visible evidence rather than on metadata alone.
Let examples mature and run the offline evaluation against a backup/non-production copy.
Compare human, autonomous, and recipient-initiated cohorts; do not blend them into the opener strategy metric.
Enable generation while retaining shadow mode to exercise selection/auditing.
Only after review and evaluation, turn shadow mode off for a small account/policy cohort; rollback is setting it true again.

Starting, restarting, deploying, or enabling the live bot remains a separate explicit operational action.

4.4 KiB Raw Blame History