SLOPSHOPPER

ruflo-market-data

Market data ingestion — feed normalization, OHLCV vectorization, and HNSW-indexed pattern matching. As a mod (ADR-445): keeps credentials and credentialed feed…

newguardcommand
A shopper browsing a rack in a slop shop
Preview · a replayed session in a sandbox
claude · ~/work/app · ruflo-market-data
› fix the failing auth test and add an audit log call ⏺ Read(src/auth.ts) ⎿ Read 6 lines ⏺ Update(src/auth.ts) ⎿ Added 2 lines, removed 1 line ⏺ Bash(bun test) ⎿ 3 pass, 1 fail ● Done. refresh now rejects expired claims and logs an audit event. ✻ Worked for 42s · done 4:20 PM › /market-mod ⎿ ruflo-market-data: /market-mod status ⎿ ruflo-market-data: /market-mod scan <text> ⎿ ruflo-market-data: /market-mod recent ⎿ ruflo-market-data: /market-mod tools ────────────────────────────────────────────────────────────────────────────────────────────────────────────────────── › ? for shortcuts
README

ruflo-market-data

Market data ingestion -- feed normalization, OHLCV vectorization, and candlestick pattern detection (read-only analysis, no order placement).

Overview

Ingests market data from REST APIs and WebSocket feeds, normalizes to OHLCV vectors with consistent scaling, and vectorizes candlestick patterns for similarity search (use ruflo-ruvector for nearest-neighbour search over history; the built-in ruvllm_hnsw_* router holds only about 11 patterns). Detects single-candle (doji, hammer) and multi-candle (engulfing, morning star, head & shoulders) formations with reliability scoring.

Installation

claude --plugin-dir plugins/ruflo-market-data

Agents

AgentModelRole
data-engineersonnetIngest market feeds, normalize OHLCV, vectorize candlestick patterns, pattern matching

Skills

SkillUsageDescription
market-ingest/market-ingest <symbol> [--source api]Ingest and normalize market data into OHLCV vectors stored by symbol and date
market-pattern/market-pattern <symbol> [--period 1D]Detect and classify candlestick patterns from ingested data

Commands (5 subcommands)

market ingest <symbol> [--period 1D]    # Ingest and normalize OHLCV data
market patterns <symbol>                 # Detect candlestick patterns
market search <pattern-name>             # Search historical pattern occurrences
market history <symbol>                  # Show ingestion history and data coverage
market compare <sym1> <sym2>             # Compare pattern profiles between symbols

OHLCV Normalization

FieldNormalization
OpenRelative to previous close: (open - prev_close) / prev_close
HighRelative to open: (high - open) / open
LowRelative to open: (low - open) / open
CloseRelative to open: (close - open) / open
VolumeZ-score: (vol - mean_vol) / std_vol

Pattern Library

PatternTypeCandlesReliability
DojiReversal1Medium
HammerReversal1Medium-High
EngulfingReversal2High
Morning/Evening StarReversal3High
Three White SoldiersContinuation3High
Head & ShouldersReversal5-7Very High
Double Top/BottomReversalVariableHigh

Each pattern is encoded as a 64-dimension padded vector.

Compatibility

  • CLI: pinned to @claude-flow/cli v3.6 major+minor.
  • Verification: bash plugins/ruflo-market-data/scripts/smoke.sh is the contract.

Namespace coordination

This plugin owns two AgentDB namespaces (kebab-case, follows the convention from ruflo-agentdb ADR-0001 §"Namespace convention"):

  • market-data — normalized OHLCV vectors per symbol+date
  • market-patterns — detected candlestick patterns with reliability scores

Both accessed via memory_* (namespace-routed). Reserved namespaces (pattern, claude-memories, default) MUST NOT be shadowed.

Routing note: Earlier versions of these skills used agentdb_hierarchical-* and agentdb_pattern-* with namespace arguments — those tool families route by tier/ReasoningBank and ignore namespace strings. ADR-0001 fixed the skills to use memory_* for namespaced reads/writes.

Verification

bash plugins/ruflo-market-data/scripts/smoke.sh
# Expected: "11 passed, 0 failed"

Architecture Decisions

Related Plugins

  • ruflo-agentdb — namespace convention owner; defines the routing rules ADR-0001 fixes violations of
  • ruflo-neural-trader -- Consumes market patterns as strategy signals
  • ruflo-ruvector -- HNSW indexing engine for nearest-neighbour search over candle windows
  • ruflo-observability -- Data feed health and ingestion latency dashboards

License

MIT

As a mod

This plugin ships a function-hook mod (ADR-445, hooks/register.ts). It makes no network call, spawns no process and makes no model call.

  • Guard (tighten-only, on by default): Refuses writes to a market* namespace (memory_store, agentdb_pattern-store, agentdb_hierarchical-store, ruvllm_hnsw_add) that hold a key or token, or a feed URL with a credential in its query string or as user:pass@host. Other namespaces are untouched. A refusal never echoes the secret.
  • /market-mod: status, recent, tools (which of this plugin's tools are connected) and scan <text> (would the guard refuse it). Answered locally.
  • Status file: .claude-flow/market-mod/status.json ({version: 1, updatedMs, guard, calls, total, blocked, recent}, counters only, never tool input), written at session start and after every call to this plugin's tools.

Options (userConfig): guard (on).

Test it: claude plugin validate plugins/ruflo-market-data, claude plugin test plugins/ruflo-market-data, bash plugins/ruflo-market-data/scripts/smoke.sh.

Source 4 files
hooks/register.ts 109 lines
1import type { Hook, Register } from 'claude-code'
2
3import { isOwn, shortName, verdict } from './guard'
4import { readOptions, type ModOptions } from './options'
5import { scan } from './screen'
6
7const COMMAND = 'market-mod'
8const STATUS_PATH = '.claude-flow/market-mod/status.json'
9const RECENT = 5
10
11type Dollar = Parameters<Hook<'session.start'>>[0]
12
13/** Everything one session of the mod keeps: its settings, per-tool counters, what was refused and the project root. */
14type Session = {
15  readonly opts: ModOptions
16  readonly calls: Record<string, number>
17  blocked: number
18  readonly recent: string[]
19  root?: string
20}
21
22const total = (s: Session) => Object.values(s.calls).reduce((a, b) => a + b, 0)
23
24function note(s: Session, line: string): void {
25  s.recent.push(line)
26  if (s.recent.length > RECENT) s.recent.splice(0, s.recent.length - RECENT)
27}
28
29/** The status file the console reads; `version` lets it refuse a shape it does not know. Counters only, never tool input. */
30async function flush($: Dollar, s: Session): Promise<void> {
31  if (s.root === undefined) return
32  const body = { version: 1, updatedMs: await $.clock.now(), guard: s.opts.guard, ...s.opts.extra, calls: s.calls, total: total(s), blocked: s.blocked, recent: s.recent }
33  try {
34    await $.fs.write(`${s.root}/${STATUS_PATH}`, `${JSON.stringify(body, null, 2)}\n`)
35  } catch {
36    /* the status file is a courtesy */
37  }
38}
39
40/** The connected tools this plugin owns, by short name. */
41async function connected($: Dollar): Promise<string[]> {
42  const tools = await $.tool.list()
43  return tools.filter(t => t.mcp && isOwn(shortName(t.name))).map(t => shortName(t.name))
44}
45
46const HELP = ['/market-mod status', '/market-mod scan <text>', '/market-mod recent', '/market-mod tools'].join('\n')
47
48async function answer($: Dollar, s: Session, args: string): Promise<string> {
49  const [verb = '', ...rest] = args.trim().split(/\s+/)
50  const arg = rest.join(' ')
51  if (verb === '' || verb === 'help') return HELP
52  if (verb === 'status') {
53    const per = Object.entries(s.calls).map(([k, v]) => `${k} ${v}`).join(' · ')
54    return [`guard ${s.opts.guard ? 'on' : 'off'}${Object.entries(s.opts.extra).map(([k, v]) => ` · ${k} ${String(v)}`).join('')}`, `${total(s)} call${total(s) === 1 ? '' : 's'}${per ? ` (${per})` : ''} · refused ${s.blocked}`].join('\n')
55  }
56  if (verb === 'recent') return s.recent.length ? s.recent.map(r => `- ${r}`).join('\n') : 'Nothing has gone through this session.'
57  if (verb === 'tools') {
58    const names = await connected($)
59    return names.length ? names.join('\n') : 'None of this plugin\'s tools are connected. Connect the ruflo MCP server and try again.'
60  }
61  if (verb === 'scan') {
62    if (arg === '') return 'usage: /market-mod scan <text>'
63    const found = scan(arg)
64    return found.secrets.length ? `The guard would refuse that (${found.secrets.join(', ')}).` : 'Nothing found: the guard would let that through.'
65  }
66  return `Unknown: ${verb}\n${HELP}`
67}
68
69/**
70 * Market data as a mod (ADR-445): a tighten-only guard on this plugin's own tools, a status file the console reads, and `/market-mod`.
71 * No network, no process, no model call.
72 */
73export const register: Register = (on, options) => {
74  const s: Session = { opts: readOptions(options), calls: {}, blocked: 0, recent: [] }
75
76  on('session.start', async ($, e, next) => {
77    const result = await next(e)
78    s.root = (await $.session.root()) as string | undefined
79    try {
80      await $.command.register({ name: COMMAND, description: 'Market data mod: status, scan <text>, recent, tools' })
81    } catch {
82      /* a name taken by another plugin must not stop the mod */
83    }
84    await flush($, s)
85    return result
86  })
87
88  on('tool.call', async ($, e, next) => {
89    const name = shortName(e.tool)
90    if (!isOwn(name, e)) return next(e)
91    if (s.opts.guard) {
92      const reason = verdict(name, e, s.opts, s.calls)
93      if (reason !== undefined) {
94        s.blocked++
95        note(s, `refused ${name}`)
96        await flush($, s)
97        return { deny: reason }
98      }
99    }
100    s.calls[name] = (s.calls[name] ?? 0) + 1
101    note(s, name)
102    await flush($, s)
103    return next(e)
104  })
105
106  /** `/market-mod`: the plugin's own commands are prompt commands, which no hook can answer. */
107  on('command.run', { command: COMMAND }, async ($, e) => ({ text: await answer($, s, typeof e.args === 'string' ? e.args : '') }))
108}
109
hooks/guard.ts 36 lines
1import { hasSecret } from './screen'
2import type { ModOptions } from './options'
3import { textsOf } from './screen'
4export { textsOf }
5
6/** `mcp__<server>__<tool>` into its tool half; tool names never hold a double underscore. */
7export function shortName(name: string): string {
8  const at = name.lastIndexOf('__')
9  return name.startsWith('mcp__') && at > 5 ? name.slice(at + 2) : name
10}
11
12const field = (input: unknown, key: string): unknown => (typeof input === 'object' && input !== null ? (input as Record<string, unknown>)[key] : undefined)
13
14
15// A feed URL that carries its credential in the query string, or in user:pass@host form. Built from strings so no literal is easy to mistake for a key.
16const FEED_CREDENTIAL = new RegExp('[?&](?:api_?key|apikey|token|access_token|secret)=[A-Za-z0-9._~-]{12,}|://[^/\\s:@]+:[^/\\s@]{6,}@', 'i')
17
18const WRITERS = new Set(['memory_store', 'agentdb_pattern-store', 'agentdb_hierarchical-store', 'ruvllm_hnsw_add'])
19const READERS = new Set(['memory_search', 'memory_list', 'agentdb_pattern-search', 'agentdb_hierarchical-recall', 'agentdb_semantic-route', 'embeddings_generate', 'ruvllm_hnsw_route', 'ruvllm_hnsw_create'])
20
21const inMarketNamespace = (input: unknown): boolean => /^market/.test(String(field(input, 'namespace') ?? ''))
22
23/** The tools this plugin owns: the HNSW router, and the memory and pattern tools when they address a `market-*` namespace. */
24export const isOwn = (name: string, input?: unknown): boolean =>
25  name.startsWith('ruvllm_hnsw_') || ((WRITERS.has(name) || READERS.has(name)) && (input === undefined || inMarketNamespace(input)))
26
27/** The reason a call is refused, or undefined when it may go. Never names or echoes a secret. */
28export function verdict(name: string, input: unknown, _opts: ModOptions, _calls: Record<string, number>): string | undefined {
29  if (!WRITERS.has(name)) return undefined
30  const texts = textsOf(input)
31  if (texts.some(hasSecret) || texts.some(t => FEED_CREDENTIAL.test(t))) {
32    return 'ruflo-market-data: this market write holds what looks like a credential (a key, a token, or a feed URL with one in it). Store the feed name, not the credentialed URL.'
33  }
34  return undefined
35}
36
hooks/options.ts 15 lines
1import type { PluginOptions } from 'claude-code'
2
3/** The plugin's `userConfig`, validated: a bad value is the default (guard on). */
4export type ModOptions = { readonly guard: boolean; readonly extra: Readonly<Record<string, unknown>> }
5
6// BEGIN SHARED FLAG (generated by scripts/sync-mod-screen.mjs; edit plugins/ruflo-agentdb/hooks/options.ts)
7export const flag = (value: unknown, fallback: boolean) =>
8  value === true || value === 'true' || value === 'on' ? true : value === false || value === 'false' || value === 'off' ? false : fallback
9// END SHARED FLAG
10
11export function readOptions(options: PluginOptions | undefined): ModOptions {
12  const o = options ?? {}
13  return { guard: flag(o.guard, true), extra: {} }
14}
15
hooks/screen.ts 268 lines
1/**
2 * Pure text screening for the AgentDB mod (ADR-445). Two jobs: find secrets (so none is stored) and find prompt-injection phrasing (so
3 * retrieved memory cannot instruct the model). Findings are NAMES only: the matched text is never returned, logged or counted by value.
4 */
5
6// BEGIN SHARED SCREEN (generated from plugins/ruflo-agentdb/hooks/screen.ts by scripts/sync-mod-screen.mjs; do not edit in a copy)
7export type Rules = readonly (readonly [string, RegExp])[]
8
9/** The secret shapes every mod screens for. A plugin adds its own after these, outside the markers. */
10export const COMMON_SECRETS: Rules = [
11  ['private key', /-----BEGIN [A-Z ]*PRIVATE KEY-----/],
12  ['aws access key', /\b(?:AKIA|ASIA)[0-9A-Z]{16}\b/],
13  ['github token', /\b(?:gh[pousr]_[A-Za-z0-9]{30,}|github_pat_[A-Za-z0-9_]{40,})\b/],
14  ['slack token', /\bxox[abprs]-[A-Za-z0-9-]{10,}/],
15  ['slack webhook', /\bhooks\.slack\.com\/services\/T[A-Z0-9]{6,}\/B[A-Z0-9]{6,}\/[A-Za-z0-9]{16,}/],
16  ['google api key', /\bAIza[0-9A-Za-z_-]{35}\b/],
17  ['anthropic or openai key', /\bsk-(?:(?:ant|proj|svcacct|admin)-[A-Za-z0-9_-]{20,}|(?=[A-Za-z]{0,40}\d)[A-Za-z0-9]{32,})/],
18  ['stripe key', /\b[rs]k_live_[A-Za-z0-9]{16,}/],
19  ['npm token', /\bnpm_[A-Za-z0-9]{36}\b/],
20  ['huggingface token', /\bhf_[A-Za-z0-9]{30,}\b/],
21  ['sendgrid key', /\bSG\.[A-Za-z0-9_-]{16,}\.[A-Za-z0-9_-]{16,}/],
22  ['twilio key', /\bSK[0-9a-f]{32}\b/],
23  ['jwt', /\beyJ[A-Za-z0-9_-]{8,}\.eyJ[A-Za-z0-9_-]{8,}\.[A-Za-z0-9_-]{8,}/],
24  ['bearer token', /\bBearer\s+([A-Za-z0-9._~+/=-]{24,})/],
25  ['database url with credentials', /\b[a-z][a-z0-9+.-]{1,20}:\/\/[^\s:@/]+:[^\s@/]{3,}@[^\s/]+/i],
26]
27
28export const INJECTION: Rules = [
29  ['override instructions', /\b(?:ignore|disregard|forget|override)\b[^.\n]{0,40}\b(?:previous|prior|above|earlier|all|any|system)\b[^.\n]{0,30}\b(?:instructions?|rules?|prompts?|guidelines?)\b/i],
30  ['role reassignment', /\byou are (?:now|no longer)\b|\bact as (?:an? )?(?:unrestricted|jailbroken)\b/i],
31  ['new instructions', /\b(?:new|updated|real) (?:system )?instructions?\s*:/i],
32  ['fake role tags', /<\/?\s*(?:system|assistant|developer|instructions?)\s*>|^\s*(?:system|assistant)\s*:/im],
33  ['concealment', /\bdo not (?:tell|inform|mention|reveal)[^.\n]{0,30}\b(?:user|human|operator)\b/i],
34  // `.env` has no word start before its dot, so it is matched on its own ("upload the .env file", "send ./config.env",
35  // ".envrc"); a dot inside a word does not end the phrase. A model's token counts are not credentials: --max-tokens,
36  // /oauth/token and "token usage/count/limit/budget/window" are left out (a file named token, /run/token, is not).
37  ['exfiltration', /\b(?:exfiltrate|send|post|upload|e-?mail)\b(?:[^.\n]|\.(?=\S)){0,50}(?:\b(?:secrets?|credentials?|api keys?)\b|(?:(?<!max[_-]?)\btokens\b|(?<!oauth\/)\btoken\b)(?![\s_-]*(?:usage|counts?|limits?|budgets?|window)\b)|\.env(?:rc)?\b)/i],
38  ['shell pipe', /\b(?:curl|wget)\b[^|\n]{0,200}\|\s*(?:sudo\s+)?(?:ba|z)?sh\b/i],
39]
40
41// C0/C1 controls (keeping tab and newline), DEL, soft hyphen, combining grapheme joiner, Arabic letter mark, Hangul and Mongolian fillers/separators,
42// zero-width, bidi (overrides and isolates) and invisible-format characters, variation selectors; built with escapes, never raw.
43const INVISIBLE = new RegExp(
44  '[\\u0000-\\u0008\\u000b-\\u001f\\u007f-\\u009f\\u00ad\\u034f\\u061c\\u115f\\u1160\\u17b4\\u17b5\\u180b-\\u180e\\u200b-\\u200f\\u2028-\\u202e\\u2060-\\u206f\\u3164\\ufe00-\\ufe0f\\ufeff\\uffa0\\ufff9-\\ufffb]',
45  'g',
46)
47
48/** Longest input scanned in one pass; a longer one keeps its head and tail halves. One regex pass per rule, so cost stays linear. */
49const MAX_SCAN = 200_000
50
51/** Input bounded to MAX_SCAN characters with invisible characters removed, so none can hide a secret or a phrase. */
52export const bare = (text: string) =>
53  (text.length > MAX_SCAN ? text.slice(0, MAX_SCAN / 2) + '\n' + text.slice(-MAX_SCAN / 2) : text).replace(INVISIBLE, '')
54
55// A value is a secret CANDIDATE only when it is not a reference (env var, call, identifier path, placeholder, secret-manager path) and its
56// shape is random enough: at least two character classes, one of them a digit or symbol, and Shannon entropy of at least 2.5 bits per character.
57const PLACEHOLDER = /placeholder|your[-_ ]|example|changeme|change[-_]?me|redacted|dummy|replace[-_]?me|insert[-_]|\*{3,}|x{5,}|\.{3}|^(?:none|null|undefined|true|false)$/i
58const REFERENCE =
59  /^(?:\$(?:\{[^}]*\}|\(|[A-Za-z_]\w*$)|%[^%]*%$|<[^>]*>$|\{\{|process\.env|os\.environ|env[.[]|import\.meta|System\.getenv|secrets?\.|vault:|op:\/\/|ref\+|arn:|projects\/[^/]+\/secrets\/|gcp:|kms:|aws:|file:)/i
60const CALL = /^[A-Za-z_$][\w$]*(?:\.[A-Za-z_$][\w$]*)*\(|^[A-Za-z_$][\w$]*\[/
61const IDENT_PATH = /^[A-Za-z_$][\w$]*(?:\.[A-Za-z_$][\w$]*)+$/
62const UUID = /^[0-9a-f]{8}-(?:[0-9a-f]{4}-){3}[0-9a-f]{12}$/i
63const NAME_LIKE = /^[a-z][a-z0-9]*(?:[-_./][a-z0-9]+){2,}$/
64/** A ./ ../ or ~/ path to a file with an extension names where a secret lives; it is not one. */
65const RELATIVE_FILE = /^(?:\.{1,2}|~)\/[A-Za-z0-9_./-]*\.[A-Za-z0-9]{1,8}$/
66/** A secret-manager reference: NAME:latest, or name-with-separators:3, optionally as NAME=ref (--set-secrets=NAME=ref:latest). A bare literal:3 is not excused. */
67const SECRET_REF = /^(?:[A-Za-z][A-Za-z0-9_-]*=)?(?:[A-Za-z][A-Za-z0-9]*(?:[_-][A-Za-z0-9]+)*:latest|[A-Za-z][A-Za-z0-9]*(?:[_-][A-Za-z0-9]+)+:\d{1,4})$/
68
69function entropy(v: string): number {
70  const counts = new Map<string, number>()
71  for (const ch of v) counts.set(ch, (counts.get(ch) ?? 0) + 1)
72  let h = 0
73  for (const n of counts.values()) h -= (n / v.length) * Math.log2(n / v.length)
74  return h
75}
76
77/** True when `v` is a name, call, path or placeholder rather than a literal credential. */
78function isReference(v: string): boolean {
79  if (PLACEHOLDER.test(v) || REFERENCE.test(v) || CALL.test(v) || IDENT_PATH.test(v) || URL_NO_CREDS.test(v) || RELATIVE_FILE.test(v) || SECRET_REF.test(v)) return true
80  return NAME_LIKE.test(v) && v.replace(/\D/g, '').length / v.length < 0.15
81}
82
83export function plausibleSecret(v: string): boolean {
84  if (v.length < 8 || v.length > 256 || /\s/.test(v) || UUID.test(v) || isReference(v)) return false
85  const symbol = /[^A-Za-z0-9]/.test(v)
86  const digit = /\d/.test(v)
87  const classes = [/[a-z]/.test(v), /[A-Z]/.test(v), digit, symbol].filter(Boolean).length
88  return classes >= 2 && (digit || symbol) && entropy(v) >= 2.5
89}
90
91/** Under a secret-named key a literal this long is a secret even with one character class or a UUID shape, unless it is a clear reference. */
92const KEYED_MIN = 20
93const PLACEHOLDER_WORD = /(?:^|[^a-z])(?:your|placeholder|changeme|change[-_]?me|example|redacted|dummy|replace[-_]?me|insert)(?:[^a-z]|$)|\*{3,}|x{5,}|\.{3}|^(?:none|null|undefined|true|false)$/i
94const URL_NO_CREDS = /^[a-z][a-z0-9+.-]{1,20}:\/\/[^\s@]*$/i
95
96/** Three or more lowercase hyphen-separated words (no hex or digit-only run of 8+, few digits), such as my-k8s-secret-name-for-database. */
97function hyphenName(v: string): boolean {
98  const parts = v.split('-')
99  return parts.length >= 3 && parts.every(p => /^[a-z0-9]{2,}$/.test(p) && !/^[0-9a-f]{8,}$/.test(p)) && v.replace(/\D/g, '').length / v.length < 0.15
100}
101
102function keyedSecret(v: string): boolean {
103  if (v.length < KEYED_MIN || v.length > 256 || /\s/.test(v)) return false
104  return !(PLACEHOLDER_WORD.test(v) || REFERENCE.test(v) || CALL.test(v) || IDENT_PATH.test(v) || URL_NO_CREDS.test(v) || RELATIVE_FILE.test(v) || SECRET_REF.test(v) || (!UUID.test(v) && hyphenName(v)))
105}
106
107const KEY_NAME = /(?:api[_-]?key|secret|token|passw(?:or)?d|passwd|pwd|credential|private[_-]?key|auth(?!or))s?[A-Za-z0-9_-]{0,40}["']?\s*[:=]\s*/gi
108const QUOTED = /(["'\x60])((?:(?!\1)[^\n]){1,256})\1/y
109const BARE_VALUE = /[^\s"'\x60,;]{1,256}/y
110const QUERY_VALUE = /[^\s"'\x60,;&]{1,256}/y
111
112/** A secret-named key assigned a literal value: env style, JSON, YAML, code. Values that are calls, references or placeholders do not count. */
113function assignmentSecret(text: string): boolean {
114  let valueEnd = 0
115  for (const m of text.matchAll(KEY_NAME)) {
116    if (m.index < valueEnd) continue // a key-looking word inside the previous value, such as secretsmanager in an ARN
117    const at = m.index + m[0].length
118    let back = m.index
119    while (back > 0 && m.index - back < 64 && /[A-Za-z0-9_.-]/.test(text.charAt(back - 1))) back--
120    const re = /["'\x60]/.test(text.charAt(at)) ? QUOTED : /[?&]/.test(text.charAt(back - 1)) ? QUERY_VALUE : BARE_VALUE
121    re.lastIndex = at
122    const hit = re.exec(text)
123    const v = hit && (hit[2] ?? hit[0])
124    valueEnd = hit ? at + hit[0].length : at
125    if (v && (plausibleSecret(v) || keyedSecret(v))) return true
126  }
127  return false
128}
129
130/** A password in a URL's userinfo that is not a placeholder such as user:password or ${DB_PASSWORD}. */
131function urlCredential(url: string): boolean {
132  const pass = /^[^:]+:\/\/[^\s:@/]+:([^\s@/]+)@/.exec(url)?.[1]
133  if (!pass || /\$\{|\{\{|%\(|%s/.test(pass)) return false
134  return !/^(?:password|passwd|pass|pwd|secret|changeme|dbpassword|db_password|\$\w*|<.*>|\{.*\}|\*+|x+)$/i.test(pass) && !PLACEHOLDER.test(pass)
135}
136
137const CHECKS: Readonly<Record<string, (m: RegExpMatchArray) => boolean>> = {
138  'bearer token': m => !isReference(m[1] ?? ''),
139  'database url with credentials': m => urlCredential(m[0]),
140  'database url with password': m => urlCredential(m[0]),
141}
142const globals = new WeakMap<RegExp, RegExp>()
143
144function matches(name: string, re: RegExp, text: string): boolean {
145  if (name === 'key assignment') return assignmentSecret(text)
146  const check = CHECKS[name]
147  if (!check) return re.test(text)
148  let g = globals.get(re)
149  if (!g) globals.set(re, (g = new RegExp(re.source, re.flags.includes('g') ? re.flags : re.flags + 'g')))
150  for (const m of text.matchAll(g)) if (check(m)) return true
151  return false
152}
153
154/**
155 * The text textsOf appends when it had to drop input (a node, character or per-string budget ran out). It is never matched against a rule:
156 * `names` reports it as a finding of its own, so every guard that asks "is there a secret in these texts" refuses what it could not read in full.
157 */
158export const TRUNCATED = 'ruflo-screen: input exceeded the screening budget'
159export const TRUNCATED_NAME = 'input too large to screen'
160
161/** Names of the rules that match `text` (already bare'd). A rule named 'key assignment' is judged by assignmentSecret, whatever its regex. */
162export const names = (rules: Rules, text: string) => text === TRUNCATED ? [TRUNCATED_NAME] : rules.filter(([name, re]) => matches(name, re, text)).map(([name]) => name)
163
164export type Findings = { readonly secrets: readonly string[]; readonly injection: readonly string[] }
165
166/** Names of every secret shape in `secrets` and every injection phrase found in `text`. Cost is linear in the capped input. */
167export function screenWith(secrets: Rules, text: string): Findings {
168  const bounded = bare(text)
169  return { secrets: names(secrets, bounded), injection: names(INJECTION, bounded) }
170}
171
172export const hasSecretIn = (secrets: Rules, text: string) => names(secrets, bare(text)).length > 0
173
174/** Makes stored text safe to show: no control or bidi characters, whitespace collapsed, at most `max` characters. */
175export function tidy(text: string, max: number): string {
176  const flat = text.replace(INVISIBLE, '').replace(/\s+/g, ' ').trim()
177  return flat.length > max ? `${flat.slice(0, Math.max(0, max - 1))}…` : flat
178}
179/** Bounds for textsOf: nodes visited, characters returned, the longest string read in full, the size of one returned chunk, chunk overlap. */
180export type TextLimits = { readonly nodes?: number; readonly chars?: number; readonly perString?: number }
181const NODES = 20_000
182const CHARS = 2_000_000
183const PER_STRING = 1_500_000
184const OVERLAP = 2_048
185const BARE_KEY_MIN = 8
186
187/** A string as texts the screen can read whole: one text up to MAX_SCAN, else overlapping MAX_SCAN windows so a secret anywhere is inside one. */
188function windows(text: string, out: string[]): void {
189  if (text.length <= MAX_SCAN) {
190    out.push(text)
191    return
192  }
193  for (let at = 0; ; at += MAX_SCAN - OVERLAP) {
194    out.push(text.slice(at, at + MAX_SCAN))
195    if (at + MAX_SCAN >= text.length) return
196  }
197}
198
199/**
200 * Every string in a tool input, for the screen to read: iterative (no recursion, so nesting 5000 deep cannot overflow the stack) and
201 * breadth-first (siblings before depth, so a long list cannot hide a nested value). A string under an object key comes back as `key=value`,
202 * so a secret-named key is judged with its value; a key whose value is not a string is returned bare. Strings longer than the screen window
203 * come back as overlapping windows; one over `perString` keeps its head and tail. Work is bounded by `nodes` slots and `chars` characters.
204 * Anything dropped (slots or characters ran out, or a string lost its middle) is reported by a final TRUNCATED text, which `names` and
205 * `hasSecretIn` count as a finding, so the screen fails closed instead of passing what it did not read.
206 */
207export function textsOf(input: unknown, limits: TextLimits = {}): string[] {
208  const out: string[] = []
209  let slots = limits.nodes ?? NODES
210  let chars = limits.chars ?? CHARS
211  const perString = limits.perString ?? PER_STRING
212  let truncated = false
213  const take = (text: string): void => {
214    if (chars <= 0) {
215      truncated = true
216      return
217    }
218    if (text.length <= Math.min(perString, chars)) {
219      chars -= text.length
220      windows(text, out)
221      return
222    }
223    truncated = true
224    const half = Math.floor(Math.min(perString, chars) / 2)
225    chars -= 2 * half
226    windows(text.slice(0, half), out)
227    windows(text.slice(-half), out)
228  }
229  const queue: unknown[] = [input]
230  let head = 0
231  for (; head < queue.length && chars > 0; head++) {
232    const node = queue[head]
233    if (typeof node === 'string') take(node)
234    else if (Array.isArray(node)) {
235      let i = 0
236      for (; i < node.length && slots > 0; i++, slots--) if (i in node) queue.push(node[i])
237      if (i < node.length) truncated = true
238    } else if (typeof node === 'object' && node !== null) {
239      for (const k in node) {
240        if (slots-- <= 0) {
241          truncated = true
242          break
243        }
244        if (!Object.prototype.hasOwnProperty.call(node, k)) continue
245        const v = (node as Record<string, unknown>)[k]
246        if (typeof v === 'string') queue.push(k + '=' + v)
247        else {
248          if (k.length >= BARE_KEY_MIN) take(k)
249          queue.push(v)
250        }
251      }
252    }
253  }
254  if (queue.length > head) truncated = true
255  if (truncated) out.push(TRUNCATED)
256  return out
257}
258// END SHARED SCREEN
259
260const SECRETS: Rules = [
261  ...COMMON_SECRETS,
262  ['key assignment', /\b(?:api[_-]?key|secret|token|passw(?:or)?d|credential)s?["']?\s*[:=]\s*["']?[A-Za-z0-9/+=_.-]{16,}/i],
263]
264
265export const scan = (text: string): Findings => screenWith(SECRETS, text)
266
267export const hasSecret = (text: string) => hasSecretIn(SECRETS, text)
268