SLOPSHOPPER

engram-memory

Automatic memory for Claude Code: puts the notes of the vault that bear on a prompt in the agent's context

newguardpromptmodelprocesstimer
★ 9v0.1.0Apache-2.0updated 2026-10-21toejough/engram/agent-instructions/mods/engram-memory
A shopper browsing a rack in a slop shop
README

Engram

⚠️ Breaking change. The pre-vault TOML memory-record storage layer (~/.local/share/engram/memory/) was removed. Engram now writes only to an agent-memory Obsidian vault. Migration from the old layout is not automated. An LLM should be able to migrate easily.

Overview

Engram gives Claude Code and Pi agents persistent memory via a zettelkasten-style vault. All six skills — recall, learn, please, route, curate, and write-memory — ship as SKILL.md installed by engram update. Each skill may additionally carry a companion skill-claude-<name> vault runbook note (key claude:<name>) offered by engram update/engram register-skills and curated like any other pending offer. Registration also offers runbook notes for every other skill, command and Pi prompt template the harnesses load by default: user, synced, plugin, Pi-configured, and project-local sources, namespaced by source (e.g. skill-superpowers-brainstorming, skill-project-github-com-toejough-engram-cmd-opsx-apply). please orchestrates end-to-end work by sequencing recall, learn, and other skills around a user's <ask> (matched by the literal triggers /please, take this end-to-end, please), route encodes the delegate-everything doctrine please draws on to staff its subagents, curate judges engram serve's pending-offer notes against the host vault (matched by the literal triggers curate, /curate, pending offers, pending offer), and write-memory executes the vault-write commands recall/learn invoke natively at their write sites (parents judge, the worker writes). The please and route skills are pure meta-orchestration.

engram update installs the skills into Claude Code and Pi, and the vault is harness-agnostic — so recall and learn work the same on each. Automatic sweeping of raw session transcripts into the chunk index reads Claude Code JSONL and Pi session files.

After a few months of use, the vault's wikilink graph looks like this in Obsidian — each dot is a note, each line a [[wikilink]]; dense clusters are groups of related notes, and the connective tissue reflects thematic proximity:

Obsidian graph view of an engram vault

Screenshot pre-dates the 2026-07-10 vocab→tags migration (#678): the ~25 vocab term-note hubs visible here no longer exist — vocab membership now rides tags: [vocab/<term>], not wikilinks, so a fresh graph view no longer shows these hubs.

Installing

Requires Go 1.25+, git and git-lfs on PATH: engram update builds the binary from the engram checkout it runs in, or from a fresh clone when there is none, and refuses a clone whose embedded model is a git-lfs pointer.

  1. Install the binary:
   go install github.com/toejough/engram/cmd/engram@latest

Make sure $GOBIN (or the first $GOPATH entry's bin, default ~/go/bin; go env GOBIN GOPATH shows them, go env -w settings included, and engram update installs there too) is on your PATH. This proxy-built binary is only a bootstrap: its embedded model is a git-lfs pointer (#645), and step 2 replaces it with a build from a clone.

  1. Deploy the skills to every detected harness (Claude Code, Pi):
   engram update                 # install / refresh
   engram update --with-guidance # also sync guidance docs (recall.md, delegate.md, learn.md, shim.md) to engram-owned roots; --with-guidance is a one-time opt-in: once any guidance file is imported, plain `engram update` keeps all of them current
   engram update --dry-run       # show what would change, without writing
   engram update --allow-downgrade # override the local-mode refusal to install an older revision than what's installed

engram update syncs engram artifacts to engram-owned roots per harness (~/.claude/engram/, ~/.pi/agent/engram/) and materializes them as symlinks in each harness's discovery paths (~/.claude/skills/, ~/.pi/agent/skills/, etc.). Removals from the source propagate on every update. First update performs a dark migration of pre-existing copies to symlinks. Run it again any time to upgrade — it also reinstalls the binary via go install, then re-execs the fresh binary so the sync itself runs with the new logic (792a11e The rest of the update runs in the new binary). --with-guidance additionally syncs guidance docs to the root's guidance/ subtree (canonical paths) and materializes symlinks; compat symlinks at flat paths (~/.claude/engram/*.md) keep existing @import lines in CLAUDE.md and AGENTS.md resolving (Claude Code + Pi; opt-in). It's a one-time opt-in per file — once your CLAUDE.md (or AGENTS.md) imports a guidance file, plain engram update keeps it current. Until then, plain engram update prints a one-line hint. See install-and-update for the deployment-as-sync behavior.

On Claude Code, engram update also turns on automatic memory, the engram-memory mod: from the next session on, each prompt arrives with the vault notes that bear on it, with nothing else to run. After each turn the lessons it holds become notes and the conversation is indexed for engram query (a headless claude -p run is judged and indexed only with ENGRAM_AUTO_CAPTURE_SDK=1). At a prompt Claude Code shows you at most four lines in the session: the notes delivered, the notes written, the lessons left out, and what could not be done (a delivery, write or sweep, with the reason); the agent reads none of them (a headless run receives them as ui_log events of its JSON stream, and prints none as plain text). A conversation too long to index within the time of a sweep is left alone, and you are told once to run engram ingest --auto. engram update --dry-run lists it without installing it, and Pi is not given it.

Skills

SkillWhat it does
recallSurfaces relevant notes and raw chunks via a single engram query call: a clustered relevance channel (recency-biased per-phrase cosine over notes+chunks → bounded matched set → one unified chunk+note clustering that builds candidate_l2s from within-cluster top-5 only) plus an explore channel (notes sampled from vocab-term centroids by proximity to the query, budgeted to the matched-note count, delivered as top-level items tagged provenance: explore) plus an un-clustered recency channel (the newest chunks, tagged recent). For each cluster (and the explore items) it judges coverage inline (covered/near/absent) and crystallizes via engram amend (update an existing note) or a handoff to the write-memory skill (create one) in its default deep mode (a glance pass is read-only), activates only the notes it actually used, then reports whether the surfaced memory changed the agent's plan.
learnCaptures the session's explicit lessons — corrections, explicit save-requests, self-discovered reversals, and confirmed approaches (positive reinforcement, user-praised or self-validated) — as permanent vault notes via write-memory handoffs. Along the way it mechanically sweeps every conversation and doc into the searchable chunk index (engram ingest --auto) and checks vocab liveness (engram vocab stats, auto-refitting when due), so raw event memory stays current even when no explicit lesson exists.

See agent-instructions/skills/{recall,learn,please,route,curate,write-memory}/SKILL.md for the full skill definitions.

All six ship with a SKILL.md (see skill-runbook-registration for the mechanism). please drives an ask end-to-end through a fixed seven-step workflow (capture, orient, plan, execute (TDD), document, complete, capture), tracked on the task list, with four adversarial review gates dispatching fresh per-angle reviewer subagents over the plan, each refactor, touched docs, and outward prose, plus a step-7 lessons audit; it is one skill and runbook note (its former three sub-runbooks merged into the single note/skill body), triggered by /please <ask> and the literal phrases take this end-to-end and please. route guides subagent selection (agent type, model, effort): easy work goes to a cheap model, complex work is decomposed before dispatch, and every dispatched subagent recalls first; please consults it when staffing gate reviewers. curate judges pending-offer notes (a child's served learn, a note pulled down from the parent) against the host vault using the same covered/near/absent reasoning recall documents, authors the runbook fields of pending skill notes instead of judging them, and composes and executes engram amend directly (an offer's content already exists as a note file, so it does not hand off to write-memory); it runs host-local only and is triggered by the literal words curate, /curate, pending offers, and pending offer. write-memory is invoked natively as a skill by recall/learn at their write sites, not reached via basename/wikilink.

Vault location

Engram reads and writes a zettelkasten vault. Resolution order:

  1. --vault <path> flag
  2. ENGRAM_VAULT_PATH environment variable
  3. $XDG_DATA_HOME/engram/vault (fallback: ~/.local/share/engram/vault)

Every vault-resolving command (not only engram learn) creates a missing vault on first use: the directory is bootstrapped with a minimal .obsidian/ config so Obsidian recognizes it, a .gitignore, a .engram/ state directory and a vault ID, and one stderr line (engram: created new vault at <path>) announces it so a mistyped --vault path is visible. An existing vault is left untouched by read-only use. See 792a2f Existing vaults are left alone.

Vault layout (flat since the 2026-06-12 flat-vault migration — notes live at the vault root; the Permanent/ and MOCs/ subdirectories are retired and ignored by the scanner):

<vault>/
  <luhmann-id>.<YYYY-MM-DD>.<slug>.md   atomic notes at the root
  <luhmann-id>.<YYYY-MM-DD>.<slug>.vec.json   sibling embedding sidecar

Binary commands

engram learn feedback --slug ... --source ... --situation ... --behavior ... --impact ... --action ... [--tag <family>[/<value>] ...] [--project <slug>] [--issue <id>]
engram learn fact     --slug ... --source ... --situation ... --subject ... --predicate ... --object ... [--tag <family>[/<value>] ...] [--project <slug>] [--issue <id>]
engram learn qa       --slug ... --source ... [--question <text>] [--answer <text>|--answer-file <path>] [--contributors <basename>...] [--certainty high|medium|low]   Write a QA pair (Q+A notes) to the vault. --slug and --source required; --answer and --answer-file are mutually exclusive; --contributors repeatable, validated against the vault; --certainty defaults to medium.
engram learn runbook  --slug ... --source ... --situation ... --done-when ... [--body <numbered steps>] [--red-flag <text> ...] [--trigger <text> ...] [--tag <family>[/<value>] ...] [--project <slug>] [--issue <id>]   Write a runbook note. `--trigger` (repeatable, each ≥ 3 chars) declares literal cues — a slash form or multi-word phrase, never a lone common word — that surface the runbook first when they appear in `engram query --text`; `engram amend --trigger` replaces the list.
engram embed apply [--all|--missing|--stale|--force|--dry-run]   (Re-)embed notes per selection (default: missing)
engram embed status                    Report counts per state (total / with-embeddings / without / stale / incompatible / broken)
engram query --phrase <p> [--phrase <p>...] [--text <user message>] [--limit N] [--project <slug>] [--chunks-dir <dir>] [--content-budget N] [--recent-fill N] [--lazy-chunks]   Semantic search over vault notes + chunk index; YAML output. `--limit` (default 20) caps the returned `items[]` count — a real, enforced cap, not report-only metadata — except runbook trigger hits, which do not count against it. (A runbook can only be a trigger hit if it has a compatible embedding sidecar.) `--text` takes the user's message verbatim (never embedded; at least one of `--phrase`/`--text` is required): a runbook whose `triggers:` list has an entry appearing in it (case-insensitive whole-word match, whitespace collapsed: `curate` hits "curate!" and "/curate" but not "accurate") is a trigger hit and leads `items[]` with provenance `trigger`, regardless of similarity. With `ENGRAM_PARENT` set, merges local + parent results into one payload — see [Parent sync](#parent-sync-engram_parent). Recency-weights chunks AND notes. Builds a bounded matched set (per-phrase top-30, union/dedup, relevance floor 0.25, cap ~300), clusters it in one pass (AutoK k-means), emits `candidate_l2s: [{path, cosine, content}]` per cluster — within-cluster top-5 notes only — plus superseded-note ride-alongs at the next rank. Separately samples an **explore** half from vocab-term centroids by proximity to the query (softmax allocation, budgeted to the matched-note count, δ=0.05 match-evidence bonus for terms with an exploit-half member, core-first within-term selection, dedupe+backfill), delivered as top-level `items[]` entries (`provenance: explore`, `source_term`; budget reported in `explore_allocated`, a term → delivered-count map, always present — `{}` on missing/unreadable centroid data). Appends the newest chunks un-clustered (tagged `recent`; default 25, controlled by `--recent-fill`). `--content-budget` caps how many chunk items render with full content (default 15; later chunks get a snippet). `--lazy-chunks` renders matched chunk items path/score only — the agent fetches evidence on demand via `engram show-chunk`. Activation is agent-driven — the binary emits no `activated` flag. --project restricts items to notes whose frontmatter `project:` matches.
engram query-chunks --phrase <p> [--phrase <p>...] [--limit N] [--chunks-dir <dir>]   Semantic search over the chunk index only (YAML output). Scores chunks by max cosine across phrases; clusters results with AutoK k-means. No vault notes, no recency channel — chunk-space search only.
engram resituate --note <ref> --situation <text>   Rewrite a note's situation field in sync: frontmatter, body opener, and sidecar situation_vector (D4/INV-S2). Both flags required; no --dry-run.
engram check   Run vault-invariant checks (G0 and G3 links, M5 situation, S1 sidecars); a PASS/WARN/FAIL line each, exit non-zero only on FAIL
engram ingest [--auto]   Merge-append session transcripts + markdown into the per-source chunk index — re-chunks/re-embeds only changed content; within one source this is append-only (a re-chunk never drops a prior record). Across sources, byte-identical content is deduplicated by content hash: only one canonical copy per hash group is indexed, and a duplicate's index file is removed only once its retained twin's index file is verified to cover every one of its own records — never on hash-match alone. `--auto` sweeps all known sources, skips session-log directories whose slugified project path starts with a non-persistent-workspace prefix (slugified forms of `/private/tmp`, `/tmp`, and macOS `$TMPDIR`), and additionally drops any resolved sweep root whose own path sits under `/tmp`, `/private/tmp`, `/var/folders`, or `/private/var/folders` — preventing eval/test runs from bloating the main index. An ancestor `.claude` dir's sweep also now excludes its `jobs/` subdirectory (agent-harness scratch, including whole snapshot copies of the vault) — matching the exclude the `.pi` ancestor sweep already had; `.claude/jobs` was previously swept and indexed. Configurable via `.engram/sweep.json` (`non_persistent_prefixes` / `non_persistent_path_prefixes` keys); bypassed by explicit `--sweep`/`--transcript`/`--markdown`/`--pi-sessions` or an isolated index via `ENGRAM_CHUNKS_DIR`. Used by /learn and /recall.
engram prune [--empty [--dry-run]] [--duplicates [--dry-run]]   Detach chunk index entries whose source file no longer exists (GC). Operator-run; reads the manifest and drops the stale per-source manifest entry, keeping the embedded chunks on disk (still searchable). `--empty` instead removes existing 0-byte `.jsonl` chunk-index files left by zero-record sources (ranking-neutral — empties hold zero records); `--duplicates` retroactively collapses every exact-content-hash group down to one canonical member, removing the rest's index files + manifest entries — safe by construction (a duplicate is removed only once its retained twin's index file is verified to cover its records; otherwise refused, not removed) and convergent (a second run removes nothing); `--dry-run` previews the count without deleting or removing. Not part of the recall/learn flows.
engram count [--group-by <attr> [--filter <attr=value>...]] [--backlinks-of <basename>]   Read-only vault aggregation, off the query/similarity path. `--group-by` counts distinct note membership per frontmatter attribute value (list attrs count one per distinct element; a value listed twice in one note still counts once), optionally restricted by repeatable AND-ed `--filter attr=value` predicates (scalar equality or list-contains); output is `value<TAB>count` lines sorted count-desc then value-asc, an `(attr absent): N` bucket when any in-set note lacks the attribute, then `total: N` (empty result prints nothing). `--backlinks-of <basename>` prints a vault-graph node's wikilink in-degree plus its sorted linkers. The two modes are mutually exclusive and each independently Obsidian-verifiable (group-by against a property/tag filter, backlinks-of against the backlinks panel) — they are NOT equal to each other: backlinks-of counts every linker (e.g. an index/MOC page) while group-by counts only frontmatter members, so the two diverge by the count of non-member linkers.
engram show <ref> [--parent]   Print a note (frontmatter + body) and its outbound wikilink targets, read-only. One required positional; no --ref flag. (candidate_l2s carry content inline, so /recall no longer shows candidates.) `--parent` resolves the ref against `ENGRAM_PARENT` instead of the local vault — errors if unset. On a local miss, with `--parent` not passed, `engram show` falls back to `ENGRAM_PARENT` automatically if configured, prefixing output with `# from_parent: true`; a local hit never contacts the parent, and with no `ENGRAM_PARENT` configured the same not-found error surfaces as before.
engram show-chunk <source#anchor> [--chunks-dir <dir>]   Print a chunk's text by its source#anchor id (read-only). Used by /recall with `--lazy-chunks` to fetch a specific chunk's evidence on demand. Local only — chunks never cross vaults, so there is no `--parent` flag and no parent fallback.
engram amend --target <ref> [--activate] [--clear-pending] [--supersedes "<basename>|<type>|<claim>"] [--chunk-source <source#anchor>...] [--situation/--subject/--predicate/--object | --behavior/--impact/--action | --done-when/--body ...] [--red-flag <text>...] [--trigger <text>...] [--discard [--into <existing>]] [--expect-hash <exchange hash>]   Amend a note in place: merge chunk-source provenance (idempotent), overwrite only supplied content fields, re-embed only on a content change; `--activate` bumps `LastUsed`; `--clear-pending` clears the pending-offer marker a served write or a pull-down set, accepting it as a normal live note; `--supersedes` writes typed supersession frontmatter + inverse + body line. `--discard` deletes the target note and its sidecar; with `--into <existing>` it folds the target's basename, aliases and parent links into `<existing>` first (curation's covered/near fold), instead of a bare delete. `--expect-hash` is required for `--clear-pending`, `--discard` and `--discard --into` on a note carrying `offer.origin`: the command fails without changing anything unless the note's current exchange hash equals the expected one (any hash that is not equal to the current one — including one from another hash version, which compares as unknown — fails), so a curator's judgment can never be silently overtaken by an in-place offer update. The /recall update path: covered link-enriches, near re-synthesizes content.
engram activate --note <ref> [--note <ref>...] [--parent]   Mark note(s) as recently used — bumps `LastUsed` in the sidecar so usefulness keeps useful notes fresh (called by /recall on only the notes the agent actually used). `--note` paths are vault-relative (resolved against the vault root / `ENGRAM_VAULT_PATH`); absolute paths are used as-is. On a local miss, a basename or `<basename>.md` ref is a parent candidate when `ENGRAM_PARENT` is set: it is fetched and written as a new local pending offer instead of failing (a bare Luhmann id never goes to the parent — ids are minted per vault). `--parent` forces parent resolution even when a local file of that name exists, and errors if `ENGRAM_PARENT` is unset; a path ref with `--parent` is refused as "not sent to the parent" rather than looked up locally. Either way, activating a parent note also bumps its use on the parent, best-effort (a failure there is not fatal). With `ENGRAM_PARENT` set, a local hit on a note linked to a parent note (a previously pulled or offered copy) also re-checks that parent note: when its exchange hash has changed since the link was recorded, and that version was not declined, the new version is pulled down as another local pending offer; an unchanged note, a declined version or an unreachable parent writes nothing and never fails the local activate. The re-check does not bump the parent note's use. A served `POST /activate` resolves refs only against the vault's note names and refuses an absolute path, a path separator or `..` with a 400.
engram vocab bootstrap --seed <yaml> [--floor <f>]     Seed vocab definition fact notes (bare `vocab` tag plus their own `vocab/<term>` self-tag per term — one family note stays bare-only) from the validated term set (--seed, required); embed them; tag all existing notes with `vocab/<term>` entries in the shared `tags:` list (no separate index file — the index is emergent via `engram count`). --floor sets the minimum cosine similarity for vocab assignment (default 0.35). Idempotent.
engram vocab tag-definitions [--vault <dir>]  Idempotent backfill: adds the missing `vocab/<term>` self-tag to every existing per-term definition note (skips the family note). Tags are not content-hash inputs so no re-embed occurs. `--vault` overrides the vault root.
engram vocab propose --term <t> --description <d>  LLM-gated: create a new vocab definition note if no existing term covers it and projected attachment ≤ 20% of vault (~$0.05/proposal). Both flags required.
engram vocab stats                     Per-term member counts, vault untagged-rate, hub terms (> 25% of vault), orphan terms (< 2 members), version staleness.
engram vocab refit [--dry-run] [--names <file>]  Derivational: derives the vocabulary from the vault's note embeddings; matches clusters to existing terms, retires unmatched derived terms (proposed terms are never auto-retired), and emits naming requests for new clusters — answer via --names <file>. --dry-run previews the matched/new/retired diff. Rewrites member `tags:` entries in the `vocab/<term>` namespace; major version bump on the family definition note (no index to regenerate — the index is emergent). Triggered growth-only (≥40 new notes AND ≥14 days).
engram auto deliver | judge | write | sweep   The commands the `engram-memory` mod runs for automatic memory, not meant to be run by hand: each reads one JSON object on standard input, creates no vault, and acts only for the prompts the user sends (a headless `claude -p` prompt only with `ENGRAM_AUTO_CAPTURE_SDK=1`). What became of each lesson is recorded under `$XDG_DATA_HOME/engram/auto`; `engram auto <command> --help
Source 1 files
hooks/register.js 654 lines
1// Automatic memory: the notes and passages of the vault that apply to a prompt arrive beside it (793a2b), and after the
2// turn the lessons it holds become notes (793a2c) and its conversation is indexed (793a2f). The logic is in the engram
3// binary (`engram auto moment`, `phrases`, `recall`, `deliver`, `judge`, `place`, `write` and `sweep`); this module
4// routes to it, asks the model for the requests the steps print, puts the block the delivery prints beside the prompt,
5// and shows the user the lines it prints for the user, through the host's log (793d2a).
6
7// Each wait of the flow of a prompt, a step or a model call, has 2 minutes of its own (793a2d): a wait that is only
8// slow is waited for, and one that has not answered by then has stalled, and the flow steps aside.
9const STEP_BOUND_MS = 120000
10
11// The steps of the flow of a prompt that print a request, in order; the delivery follows them (793d2a).
12const CHAIN = ['moment', 'phrases', 'recall']
13
14// Every request a step of the flow of a prompt prints begins with this; anything else is not a request.
15const STEP = '{"request":'
16
17// Every block the command prints opens with this (793d4c); anything else is not a block.
18const MARKER = '[engram memory]'
19
20// The bounds of the stages of the capture of a turn (793d2b): the call for the session's ID, the steps `engram auto
21// moment`, `judge` and `place`, `engram auto write` and the run of it that reports a failure. The three model calls
22// hold the limits of their requests, 10, 30 and 15 seconds; together they are 115 seconds, under the 2 minutes of the
23// product (793a2c).
24const ID_BOUND_MS = 5000
25const MOMENT_BOUND_MS = 5000
26const JUDGE_BOUND_MS = 10000
27const PLACE_BOUND_MS = 5000
28const WRITE_BOUND_MS = 20000
29const REPORT_BOUND_MS = 15000
30
31// The write that follows a deep delivery has a bound of its own (793d2a): it holds neither the prompt nor the agent.
32const DEEP_WRITE_BOUND_MS = 60000
33
34// The sweep of the conversation has a bound of its own (793d2c): the call for the ID, the sweep and the report of its
35// failure are 110 seconds, under the 2 minutes of the product (793a2f), and the capture's bounds are not extended.
36const SWEEP_BOUND_MS = 90000
37
38// A sweep that is ended is run once more, as ended, so that the command gives up for the session (793d2c): the call
39// for the ID (5), the sweep (90), this run (5) and the report of its failure (15) are 115 seconds, under the 2 minutes.
40const GIVE_UP_BOUND_MS = 5000
41
42// The prompts the session raised that have started no turn, oldest first, each with its text and its origin; and the
43// text and the origin of each turn the main loop has started and not completed, by the turn's ID (793d2b): the prompt
44// names its origin, the start of the turn names its text, the end of the turn names only the turn.
45const raised = []
46const turns = new Map()
47
48// The IDs of the notes and passages the module delivered in each session, by the session's ID (855d2c): the file name
49// of a note and the chunk ID of a passage, each given to the delivery as held, so that a note is delivered once.
50const delivered = new Map()
51
52// The last delivery of each session, as a promise that settles when it is over: the deliveries of one session run one
53// after another, so that each reads the set after the one before added to it, and a note is marked as used once
54// (793d2a).
55const deliveries = new Map()
56
57// The writes after the deliveries of a session, each chained after the one before it, for the same reason.
58const writes = new Map()
59
60// The text and the origin of the turn the main loop is in, which the events of its tools carry none of (855d2c): the
61// last turn started.
62let current = { text: '', origin: '' }
63
64// The IDs of the calls whose failure was judged and whose batch has not come (855d2c): a batch of calls all judged as
65// failures is not judged again.
66const failed = new Set()
67
68// The keys a tool.call event carries beside the arguments of the tool (855d2c).
69const RESERVED = ['tool', 'tool_use_id', 'agentId', 'requestMeta', 'consent']
70
71// The Bash commands that are the agent's own use of the memory, which the binary holds the text and result of as well
72// (855d2c1o), and the skills that are: a call that is one is not looked up for.
73const MEMORY_COMMANDS = ['engram query', 'engram show', 'engram show-chunk', 'engram recall', 'engram learn',
74  'engram amend']
75const MEMORY_SKILLS = ['recall', 'learn', 'curate', 'write-memory']
76
77// A Bash command is one of the memory commands only at the position of a command: at the start, after `;`,
78// `&`, `|` or `(`, or after `-c` and a quote, with a path before `engram` allowed, so that a commit message or a grep
79// that names a command is not one.
80const MEMORY_COMMAND = new RegExp(String.raw`(?:^|[;&|(]|-c\s+['"])\s*(?:\S*/)?engram\s+(?:${
81  MEMORY_COMMANDS.map((memory) => memory.replace('engram ', '')).join('|')})\b`)
82
83// So many prompts are kept at most: one that is dropped, or delivered into a running turn, starts no turn.
84const RAISED_MAX = 20
85
86export function register(on) {
87  on('prompt.submit', async ($, e, next) => {
88    raised.push({ text: e.text, origin: originOf(e) })
89
90    if (raised.length > RAISED_MAX) {
91      raised.shift()
92    }
93
94    const delivery = await deliveryFor($, 'a prompt', (state) => promptChain($, e, state))
95
96    // the lines go to the user and the model reads none of them: they are logged, not added to the context (793d2a)
97    for (const line of delivery.shown) {
98      $.ui.log(line)
99    }
100
101    return next(delivery.block ? { ...e, context: [...(e.context ?? []), delivery.block] } : e)
102  })
103
104  on('turn.start', async (_$, e, next) => {
105    if (e.agentId === undefined) {
106      current = { text: e.text, origin: originOfStart(e.text) }
107      turns.set(e.turnId, current)
108    }
109
110    return next(e)
111  })
112
113  // the three cues of a turn (855d2c): the chain of the prompt runs for the event, and the block rides the answer
114  on('tool.call', async ($, e, next) => {
115    if (e.agentId !== undefined || isMemoryCall(e.tool, e)) {
116      return next(e)
117    }
118
119    const input = Object.fromEntries(Object.entries(e).filter(([key]) => !RESERVED.includes(key)))
120    const event = { tool: e.tool, input }
121    // the call is let run while its chain runs; the block rides the result once both are over
122    const [delivery, result] = await Promise.all([
123      cueDelivery($, 'before-tool', () => bounded($.session.id(), ID_BOUND_MS, "the call for the session's ID"), event),
124      next(e),
125    ])
126
127    // a call that was denied has no context to add to; the notes of the block are held no longer
128    if (result.deny !== undefined) {
129      release(delivery)
130
131      return result
132    }
133
134    return withBlock(result, 'context', delivery.block)
135  })
136
137  on('classic.PostToolUseFailure', async ($, e, next) => {
138    if (e.agent_id !== undefined || e.is_interrupt) {
139      return next(e)
140    }
141
142    failed.add(e.tool_use_id)
143
144    if (isMemoryCall(e.tool_name, e.tool_input)) {
145      return next(e)
146    }
147
148    const event = { tool: e.tool_name, input: e.tool_input, error: e.error }
149    const delivery = await cueDelivery($, 'failure', async () => e.session_id, event)
150
151    return withBlock(await next(e), 'additionalContext', delivery.block)
152  })
153
154  on('classic.PostToolBatch', async ($, e, next) => {
155    const calls = e.tool_calls ?? []
156    // a batch is left alone when each call was judged as a failure or is the agent's own use of the memory
157    const skipped = calls.every((call) => failed.has(call.tool_use_id) || isMemoryCall(call.tool_name, call.tool_input))
158
159    for (const call of calls) {
160      failed.delete(call.tool_use_id)
161    }
162
163    if (e.agent_id !== undefined || skipped) {
164      return next(e)
165    }
166
167    const event = {
168      calls: calls.map((call) => ({ tool: call.tool_name, input: call.tool_input, response: call.tool_response })),
169    }
170    const delivery = await cueDelivery($, 'batch-done', async () => e.session_id, event)
171
172    return withBlock(await next(e), 'additionalContext', delivery.block)
173  })
174
175  on('turn.complete', async ($, e, next) => {
176    const result = await next(e)
177    const turn = turns.get(e.turnId) ?? { text: '', origin: '' }
178
179    turns.delete(e.turnId)
180
181    // the lessons are captured and the conversation is swept (793d2c) apart from the turn, and apart from each other,
182    // from a timer: the host holds the next event of the mod, a prompt among them, until the calls an event left in
183    // flight settle, and a timer's calls are no event's; the turn of a subagent is left out
184    if (e.agentId === undefined) {
185      $.clock.after(0, () => {
186        void afterTurn($, e, (session) => captureLessons($, e, turn, session))
187        void afterTurn($, e, (session) => sweep($, session, turn.origin))
188      })
189    }
190
191    return result
192  })
193}
194
195// isMemoryCall reports whether a call is the agent's own use of the memory (855d2c1o): a Bash command that runs one of
196// the memory commands, or the Skill tool with one of the memory skills. The input holds the command or the skill.
197function isMemoryCall(tool, input) {
198  if (tool === 'Bash') {
199    return typeof input?.command === 'string' && MEMORY_COMMAND.test(input.command)
200  }
201
202  return tool === 'Skill' && MEMORY_SKILLS.includes(input?.skill)
203}
204
205// ask is the text of the model's answer to a request, and fails when the model gives none (793d2b). The call ends at
206// the time limit that the request holds.
207async function ask($, request) {
208  let reply
209
210  try {
211    reply = await $.model.complete(request)
212  } catch (error) {
213    throw new Error(`the model call failed: ${describe(error)}`)
214  }
215
216  if (!reply.isAnswered) {
217    throw new Error(`the model call was not answered: ${reply.reason}`)
218  }
219
220  return reply.text
221}
222
223// askModel is the text of the model's answer to a request of the flow of a prompt, and fails when the model gives none.
224// The request is asked exactly as the step printed it; the call is aborted through its signal when it has not answered
225// within 2 minutes (793d2a).
226async function askModel($, request) {
227  const stop = new AbortController()
228  const timer = setTimeout(() => stop.abort(), STEP_BOUND_MS)
229  let reply
230
231  try {
232    reply = await $.model.complete(request, { signal: stop.signal })
233  } catch (error) {
234    // a call that is aborted may reject, and it was the bound that ended it
235    throw new Error(stop.signal.aborted ? `the model call gave nothing within ${limitText(STEP_BOUND_MS)}`
236      : `the model call failed: ${describe(error)}`)
237  } finally {
238    clearTimeout(timer)
239  }
240
241  if (!reply.isAnswered) {
242    const why = stop.signal.aborted
243      ? `gave nothing within ${limitText(STEP_BOUND_MS)}`
244      : `was not answered: ${reply.reason}`
245
246    throw new Error(`the model call ${why}`)
247  }
248
249  return reply.text
250}
251
252// bounded is what a promise gives, or a failure that names it when it gives nothing within the bound: the host gives
253// some calls no time limit of their own (793d2b).
254async function bounded(promise, bound, what) {
255  let timer
256  const late = new Promise((_resolve, reject) => {
257    timer = setTimeout(() => reject(new Error(`${what} gave nothing within ${bound} ms`)), bound)
258  })
259
260  try {
261    return await Promise.race([promise, late])
262  } finally {
263    clearTimeout(timer)
264  }
265}
266
267// afterTurn does a job of the capture of a turn that ended, with the session's ID, and has the writing record the
268// failure of a step that fails: the turn went on as if automatic memory were not installed. A session whose ID does not
269// come is left alone, since the writing cannot record anything without it (793d2b, 793d2c).
270async function afterTurn($, e, job) {
271  let session = ''
272
273  try {
274    session = await bounded($.session.id(), ID_BOUND_MS, "the call for the session's ID")
275
276    await job(session)
277  } catch (error) {
278    await report($, session, e.turnId, error)
279  }
280}
281
282// captureLessons judges a turn that ended and has the lessons written (793d2b): the moment, the judgement and the
283// placement each print the request for the model, which is asked exactly as printed, and the writing is given the
284// replies and the carry of the placement. It ends quietly when the moment or the judgement prints nothing, and writes
285// the lessons at the top when the placement prints nothing. Each stage has its bound.
286async function captureLessons($, e, turn, session) {
287  const fields = { session, turn: e.turnId, text: turn.text, answer: e.answer, origin: turn.origin }
288  const moment = await asked($, 'moment', { ...fields, cue: 'turn-end' }, MOMENT_BOUND_MS)
289
290  if (moment === null) {
291    return
292  }
293
294  const judged = await asked($, 'judge', { ...fields, reply: moment.reply, carry: moment.carry }, JUDGE_BOUND_MS)
295
296  if (judged === null) {
297    return
298  }
299
300  const written = { session, turn: e.turnId, reply: judged.reply }
301
302  try {
303    const placed = await asked($, 'place', { session, turn: e.turnId, reply: judged.reply, carry: judged.carry },
304      PLACE_BOUND_MS)
305
306    if (placed !== null) {
307      written.carry = placed.carry
308      written.placement = placed.reply
309    }
310  } catch (error) {
311    // a placement that fails does not fail the capture: the lessons are written at the top and the writing records why
312    written.placementFailure = describe(error)
313  }
314
315  await run($, 'write', written, WRITE_BOUND_MS)
316}
317
318// asked runs a step of the capture and asks the model for the request it prints (793d2b): the reply and the carry, or
319// nothing when the step prints nothing. A request that is not an object is a failure of the step.
320async function asked($, step, input, bound) {
321  const { stdout } = await run($, step, input, bound)
322
323  if (stdout === '') {
324    return null
325  }
326
327  const printed = stepOf(stdout, step)
328
329  try {
330    return { reply: await ask($, printed.request), carry: printed.carry }
331  } catch (error) {
332    throw new Error(`${step}: ${describe(error)}`)
333  }
334}
335
336// sweep indexes the conversation of the session (793d2c).
337// A sweep that is ended at its bound is run once more, as ended:
338// the command gives up for the session and fails once, naming what to run (793d4n), and that failure is reported.
339async function sweep($, session, origin) {
340  try {
341    await run($, 'sweep', { session, origin }, SWEEP_BOUND_MS)
342  } catch (error) {
343    if (!error.ended) {
344      throw error
345    }
346
347    await run($, 'sweep', { session, origin, ended: true }, GIVE_UP_BOUND_MS)
348  }
349}
350
351// limitText is a time limit written for a person: whole minutes, else milliseconds.
352function limitText(ms) {
353  return ms >= 60000 && ms % 60000 === 0 ? `${ms / 60000} minutes` : `${ms} ms`
354}
355
356// describe is the text of a failure.
357function describe(error) {
358  return error instanceof Error ? error.message : String(error)
359}
360
361// deliveryFor is the block and the lines for the user for a prompt or a tool event, or nothing when the flow does not
362// finish or fails: the work goes on as if automatic memory were not installed, and the failure is reported (793d2a,
363// 855d2c). The chain is given the state, in which the session's ID is left as soon as it is known, for the report.
364async function deliveryFor($, what, chainOf) {
365  const state = { session: '', write: null }
366
367  try {
368    const delivery = await chainOf(state)
369
370    writeAfter($, state)
371
372    return delivery
373  } catch (error) {
374    reportDelivery($, state.session, what, error)
375
376    return { block: '', shown: [] }
377  }
378}
379
380// cueDelivery is deliveryFor for the event of a tool: the chain of a prompt with the event and the text the turn began
381// with in place of the prompt's text (855d2c). The lines for the user are logged here.
382async function cueDelivery($, cue, sessionOf, event) {
383  const delivery = await deliveryFor($, 'a tool call', async (state) => {
384    state.session = await sessionOf()
385
386    return chain($, state, { origin: current.origin, cue }, { text: current.text, event })
387  })
388
389  for (const line of delivery.shown) {
390    $.ui.log(line)
391  }
392
393  return delivery
394}
395
396// withBlock is the answer of the hooks beneath, with the block added as one more entry of the list the key names, or
397// the answer as it is when there is no block (855d2c).
398function withBlock(result, key, block) {
399  return block ? { ...result, [key]: [...(result[key] ?? []), block] } : result
400}
401
402// promptChain runs the chain of a prompt (793d2a).
403async function promptChain($, e, state) {
404  state.session = await bounded($.session.id(), ID_BOUND_MS, "the call for the session's ID")
405
406  return chain($, state, { origin: originOf(e), cue: 'prompt' }, { text: e.text })
407}
408
409// chain runs the steps of the flow in order (793d2a): the moment, the phrases and the recall, each given the reply to
410// the request before it and its carry and asking the model for the request it prints, and then the delivery with the
411// last reply and its carry. A step that prints nothing ends the chain there, and the delivery is run with the fields
412// of the moment alone: the prompt's text, or for a tool event the event and the text the turn began with.
413async function chain($, state, who, fields) {
414  const base = { session: state.session, ...who, held: [...(delivered.get(state.session) ?? [])] }
415  let answer = {}
416
417  for (const step of CHAIN) {
418    const { stdout } = await run($, step, { ...base, ...(step === 'moment' ? fields : answer) }, STEP_BOUND_MS)
419
420    if (stdout === '') {
421      answer = {}
422
423      break
424    }
425
426    const printed = stepOf(stdout, step)
427
428    answer = { reply: await askModel($, printed.request), carry: printed.carry }
429
430    // a deep recall says that a write follows the delivery (793d2a)
431    state.write = printed.write === true ? { cue: base.cue, ...answer } : null
432  }
433
434  return inOrder(deliveries, state.session, async () => {
435    const held = [...(delivered.get(state.session) ?? [])]
436    const { stdout, stderr } = await run($, 'deliver', { ...base, held, ...fields, ...answer }, STEP_BOUND_MS)
437    const lines = stderr.split('\n').map((text) => text.trim()).filter((text) => text !== '')
438
439    // the lines go to the user and the model reads none of them
440    return stdout.startsWith(MARKER) ? keepNew(state.session, stdout, lines) : { block: '', shown: lines }
441  })
442}
443
444// writeAfter has the lessons and the notes of a deep delivery written from a timer once the block is given, so that the
445// writing holds neither the prompt nor the agent and no call is in flight when the hook settles; a write that fails is
446// reported (793d2a). The writes of a session are run one after the other, so that two deep cues in a turn do
447// not crystallize the same lesson twice.
448function writeAfter($, state) {
449  if (state.write === null) {
450    return
451  }
452
453  const { session, write } = state
454
455  $.clock.after(0, () => {
456    void (async () => {
457      await inOrder(writes, session, async () => {
458        try {
459          await run($, 'write', { session, ...write }, DEEP_WRITE_BOUND_MS)
460        } catch (error) {
461          await report($, session, '', new Error(`the write after a deep delivery failed: ${describe(error)}`))
462        }
463      })
464    })()
465  })
466}
467
468// inOrder runs a job of a session after the one before it in the chains is over, whatever became of that one (793d2a).
469async function inOrder(chains, session, job) {
470  const before = chains.get(session) ?? Promise.resolve()
471  let over
472  const mine = new Promise((resolve) => { over = resolve })
473
474  chains.set(session, before.then(() => mine))
475  await before
476
477  try {
478    return await job()
479  } finally {
480    over()
481  }
482}
483
484// reportDelivery has the writing record that the delivery for a prompt or a tool call could not be done, from a timer,
485// so that the prompt is held no longer and no call is left in flight when the hook settles (793d2a). A delivery whose
486// session's ID did not come is left alone, since the writing cannot record anything without it.
487function reportDelivery($, session, what, error) {
488  if (session === '') {
489    return
490  }
491
492  $.clock.after(0, () => {
493    void report($, session, '', new Error(`the delivery for ${what} could not be done: ${describe(error)}`))
494  })
495}
496
497// stepOf is what a step printed: the request for the model and the carry, or a failure that names the step when what it
498// printed is not one (793d2a, 793d2b).
499function stepOf(printed, step) {
500  if (!printed.startsWith(STEP)) {
501    throw new Error(`engram auto ${step} printed what is not a request`)
502  }
503
504  try {
505    return JSON.parse(printed)
506  } catch (error) {
507    throw new Error(`engram auto ${step} printed a request that is not JSON: ${describe(error)}`)
508  }
509}
510
511// keepNew is the block less each line whose ID the session's set holds, and the lines for the user, less the notes and
512// passages that line of theirs names which the block left out, with the IDs of the lines left added to the set in the
513// same step, so that a note or a passage is delivered once however many chains find it; no block when no line is left
514// (855d2c, 793d2a).
515function keepNew(session, block, shown) {
516  const ids = delivered.get(session) ?? new Set()
517  const lines = block.split('\n')
518  const left = []
519  const kept = lines.filter((line) => {
520    if (!line.startsWith('- ')) {
521      return true
522    }
523
524    const entry = entryOf(line)
525
526    if (ids.has(entry.id)) {
527      return false
528    }
529
530    left.push(entry)
531
532    return true
533  })
534
535  for (const entry of left) {
536    ids.add(entry.id)
537  }
538
539  delivered.set(session, ids)
540
541  const all = lines.filter((line) => line.startsWith('- ')).length
542
543  return {
544    block: left.length > 0 ? kept.join('\n') : '',
545    shown: left.length === all ? shown : shown.flatMap((line) => deliveredLineOf(line, left)),
546    session,
547    added: left.map((entry) => entry.id),
548  }
549}
550
551// release takes the IDs of a block that was not given off the session's set, so that its notes are delivered later
552// (855d2c).
553function release(delivery) {
554  for (const id of delivery.added ?? []) {
555    delivered.get(delivery.session)?.delete(id)
556  }
557}
558
559// entryOf is the ID of a line of a block, whole, and whether the line is a passage's (855d2c): the line of a note is a
560// dash, the file name, which holds no space, and the situation in parentheses; that of a passage is a dash, the chunk
561// ID, which may hold spaces, and `(passage)`.
562function entryOf(line) {
563  const rest = line.slice(2)
564  const note = /^(\S+\.md) \(/.exec(rest)
565
566  if (note) {
567    return { id: note[1], passage: rest.startsWith(`${note[1]} (passage): `) }
568  }
569
570  const at = rest.indexOf(' (passage): ')
571
572  if (at >= 0) {
573    return { id: rest.slice(0, at), passage: true }
574  }
575
576  const open = rest.indexOf(' (')
577
578  return { id: open >= 0 ? rest.slice(0, open) : rest, passage: false }
579}
580
581// deliveredLineOf is the line for the user that names what was delivered, written again for the entries that are left,
582// or nothing when none is left; any other line goes through (793d2a).
583function deliveredLineOf(line, left) {
584  if (!line.startsWith('delivered ')) {
585    return [line]
586  }
587
588  const parts = []
589
590  for (const [passage, word] of [[false, 'note'], [true, 'passage']]) {
591    const ids = left.filter((entry) => entry.passage === passage).map((entry) => entry.id)
592
593    if (ids.length > 0) {
594      parts.push(`the ${word}${ids.length === 1 ? '' : 's'} ${ids.join(', ')}`)
595    }
596  }
597
598  return parts.length > 0 ? [`delivered ${parts.join(' and ')}`] : []
599}
600
601// originOf is the kind of origin that Claude Code stamped on a prompt, or nothing when it stamped none (793d2a).
602function originOf(e) {
603  return e.origin?.kind ?? ''
604}
605
606// originOfStart is the origin of the prompt that a turn which starts with a text began with, and takes that prompt off
607// the ones kept, with the prompts before it, which began no turn of their own. The prompt is the oldest kept with the
608// text; when none has it, the newest one; a turn that starts with no text has no origin (793d2b).
609function originOfStart(text) {
610  const at = raised.findIndex((prompt) => prompt.text === text)
611  const taken = at >= 0 ? raised.splice(0, at + 1) : text === '' ? [] : raised.splice(0)
612
613  return taken.length > 0 ? taken[taken.length - 1].origin : ''
614}
615
616// report has the writing record the failure of a step of the capture, or of a delivery, when the session is known:
617// the failure goes whole, for the writing redacts it and cuts it (793d4k). A report that fails is the end of it.
618async function report($, session, turn, error) {
619  if (session === '') {
620    return
621  }
622
623  try {
624    await run($, 'write', { session, turn, failure: describe(error) }, REPORT_BOUND_MS)
625  } catch {
626    // nothing more can be done
627  }
628}
629
630// run is what a step of `engram auto` printed for an input, on standard output and on standard error, and fails, naming
631// the command, when it exits with other than 0 or cannot be run. The command is ended after timeoutMs, and after the
632// default of the process call when there is none.
633async function run($, step, input, timeoutMs) {
634  const command = `engram auto ${step}`
635  let result
636
637  try {
638    result = await $.process.run(['engram', 'auto', step], { stdin: JSON.stringify(input), timeoutMs })
639  } catch (error) {
640    throw Object.assign(
641      new Error(
642        `${command} was ended at its limit of ${limitText(timeoutMs ?? 30000)} or could not run: ${describe(error)}`,
643      ),
644      { ended: true },
645    )
646  }
647
648  if (result.exitCode !== 0) {
649    throw new Error(`${command} exited with ${result.exitCode}: ${String(result.stderr ?? '').trim().split('\n')[0]}`)
650  }
651
652  return { stdout: result.stdout.trim(), stderr: String(result.stderr ?? '') }
653}
654