model-fit
After each ask you type, says what it looked like it needed and what it cost; after a simple or routine ask on a bigger model or effort than it needed, what…
newbandcommandtoastpromptmodel
After each ask you type, says what it looked like it needed and what it cost; after a simple or routine ask on a bigger model or effort than it needed, what…
It is a mod rather than a hook because its advice is for you, not the model: a hook's output goes into Claude's context, costs tokens and can steer the answer, while the band and its buttons reach you directly and keep the session's cache and cost state between asks.
Haiku labels each prompt you type as trivial, normal or complex, reading it together with the end of the last reply, so "yes please" is judged by the plan it approves. A concept or arithmetic question that needs no code read ("what is a hashmap") counts as trivial.
◇ Looked routine · Opus/high · ≈ $0.04 · judged by Haiku. A silent band no longer leaves you guessing whether the mod ran./model haiku or /model sonnet mid-conversation makes the new model re-cache the whole conversation once, at 2x the input price on the one-hour cache or 1.25x on the five-minute one (and on Haiku, every price is five times higher above 100k tokens). Routine-ask advice shows once a session, since most asks are routine. A second Haiku call judges whether the ask needed the conversation:/model haiku in the prompt box./effort low (simple) or /effort medium (routine), with a button. On Opus 5.5, Sonnet 5.5, Haiku 5.5 and Fable 5.1, with an API key or a subscription, an effort change keeps the cache, so this is offered at any context size and beside "stay". Elsewhere (Bedrock, Agent Platform, Foundry, Claude Platform on AWS, a custom ANTHROPIC_BASE_URL, CLAUDE_CODE_DISABLE_EXPERIMENTAL_BETAS) it rebuilds the cache, so it is offered only below 50k tokens. The saving is not priced: Anthropic publishes no output per effort level./model opus, with what moving the conversation to Opus costs once; a routine ask on Haiku gets the same for /model sonnet, once a session. A complex ask at low effort gets /effort high, and says whether that keeps the cache./clear for a new task, /compact or /model-fit:baton for the same task, /rewind after a wrong turn. Buttons put /compact keep: or /model-fit:baton in the prompt box. After a long idle spell it adds what the next action costs while the cache is rebuilt, and says to shrink first./model-fit:baton writes a handoff note to the OS temp folder (or a path you give): next action first, verified versus assumed state, decisions with rejected options, gotchas, pointers, and the skills to invoke next. It writes in plain ASD-STE100 style and gives the line to paste after /clear.All dollar figures are list-price estimates; /cost is the authority. /fit shows the typical steps and output per label, learned from your turns, and how many turns it suspects were mislabelled: those that ran over three times the steps their label predicts. It also totals the session: what the judged asks cost, how many were simple and how many of those did not need the conversation, and an estimate of the Haiku labelling itself (two small calls per ask; the mod API does not report their usage, so it is guessed from their length).
Assumptions behind the figures: list prices from Anthropic's pricing page (2026-10-12): cache reads at 0.1x input on Haiku, 0.05x on Sonnet and Opus, 0.025x on Fable; older Opus and Sonnet versions are priced as current ones. Another model's price assumes it writes the same output. The cache lifetime follows Claude Code's rules: FORCE_PROMPT_CACHING_5M, then CLAUDE_CODE_PROMPT_CACHE_TTL, the promptCacheTtl setting, ENABLE_PROMPT_CACHING_1H, then one hour on a subscription and five minutes with ANTHROPIC_API_KEY or a cloud provider. A subscription that has run into usage credits drops to five minutes, which a mod cannot see: set the cacheTtl option to 5m then (claude plugin configure model-fit@my-mods). A new session's size is taken as the smallest context seen this session. On a subscription plan the dollars are notional, but the same tokens count against your usage limits. /fit reset forgets the history and the shadow log.
The advice is a suggestion for the next ask like it: model-fit never switches the model, mid-turn or otherwise. The band gives way while the app shows a survey there, and comes back when the survey closes.
claude plugin configure model-fit@my-modsSet jevShadow to on, and typesafeApiKey (or TYPESAFE_API_KEY). Set it back to off to stop. Each typed ask then also goes to Jev, with the same text Haiku sees: the ask and the end of the last reply.
/fit shows the comparison: asks compared, agreements among Jev's sure answers, how often Jev was unsure or failed, its median time, its total cost, and the last five disagreements. The last 200 asks are kept.How to read it: neither model is ground truth. Read the disagreements, and spot-check some agreements too: two models can agree and both be wrong. The deciding evidence is the turn itself: how many steps it took and whether a smaller model would have done the job. Replace Haiku with Jev only after enough of your own asks, and only if the disagreements tend to go Jev's way. These mods make no claim about savings: the figures behind posts about routing with Jev are modelled, not measured, and your mix of asks decides what you would save.
Limits: the log holds the first 4,000 characters of each ask in the mod's store on this machine. The costs are list-price estimates. Prompts sent by other plugins, task notifications and continuations are neither judged nor logged.
hooks/register.tsx 601 lines1import { atom, read, update } from 'claude-code'
2import type { EngineInterface, PluginOptions, Register, Timer, TurnCompleteInput } from 'claude-code'
3
4import type { Advice, Billing, BillingEnv, Effort, Need, Scope, ShadowJev, ShadowRow } from '../types'
5import {
6 CACHE_LIFETIME_MS,
7 CLASSIFY_BRIEF,
8 NEEDS,
9 SCOPES,
10 NEED_WORDS,
11 REPLY_CHARS,
12 advise,
13 billingOf,
14 paysNow,
15 classifyCostOf,
16 classifyText,
17 scopeText,
18 contextAlert,
19 contextLevel,
20 describeModel,
21 dollars,
22 emptyHistory,
23 record,
24 tokens,
25 typicalTurn,
26 usageCost,
27 warn,
28} from './fit'
29import type { History } from './fit'
30import { JEV_TIMEOUT_MS, JEV_URL, jevCostOf, keep, parseJev, requestBody, shadowReport } from './jev'
31
32const advice = atom({ plugin: 'model-fit', key: 'advice' } as const, null)
33const context = atom({ plugin: 'model-fit', key: 'context' } as const, null)
34const verdict = atom({ plugin: 'model-fit', key: 'verdict' } as const, null)
35
36const HISTORY_KEY = 'history'
37const SHADOW_KEY = 'shadow'
38const JUDGED_BY = 'Haiku'
39const PROMPT_CHARS = 4000
40/** The person's own prompts: typed at the terminal, or sent by Remote Control. */
41const HUMAN = new Set(['composer', 'bridge'])
42const EFFORTS: readonly string[] = ['low', 'medium', 'high', 'xhigh', 'max']
43
44const asEffort = (value: unknown): Effort | undefined =>
45 typeof value === 'string' && EFFORTS.includes(value) ? (value as Effort) : undefined
46
47const asNeed = (value: unknown): Need | undefined => NEEDS.find(need => need === value)
48
49function asHistory(value: unknown): History {
50 const history = emptyHistory()
51
52 if (typeof value !== 'object' || value === null) return history
53
54 for (const need of NEEDS) {
55 const kept = (value as Record<string, unknown>)[need]
56
57 if (typeof kept === 'object' && kept !== null) {
58 const { turns, steps, output, misses } = kept as Record<string, unknown>
59
60 if (typeof turns === 'number' && typeof steps === 'number' && typeof output === 'number') {
61 history[need] = { turns, steps, output, misses: typeof misses === 'number' ? misses : 0 }
62 }
63 }
64 }
65
66 return history
67}
68
69type Usage = NonNullable<TurnCompleteInput['usage']>
70type Ended = { usage: Usage; context: number; steps: number; effort: Effort | undefined }
71
72/** This session's judged turns: what they cost, the simple ones and those that did not need the conversation. */
73type Tally = { turns: number; usd: number; simple: number; simpleUsd: number; standalone: number; labelUsd: number }
74
75type Judged = { need: Need; scope?: Scope }
76
77type Shadowed = { ask: string; jev: Promise<ShadowJev>; jevUsd: number }
78
79type Session = {
80 tally: Tally
81 /** Jev's shadow labels, by turn, while the turn runs. */
82 shadowed: Map<string, Shadowed>
83 /** A failing Jev is reported once a session, not on every ask. */
84 isJevReported: boolean
85 model: string | undefined
86 effort: Effort | undefined
87 history: History
88 needs: Map<string, Judged>
89 /** The smallest context seen this session: roughly what a new session starts from. */
90 baseline: number | undefined
91 steps: Map<string, number>
92 running: Set<string>
93 /** Turns that ended before their label came back: concluded when it does. */
94 ended: Map<string, Ended>
95 typed: string | undefined
96 level: number
97 isCold: boolean
98 cooling: Timer | undefined
99 billing: Billing
100 /** Routine-ask advice shows once a session: most asks are routine, and it would show after nearly every one. */
101 isRoutineAdvised: boolean
102}
103
104const setCold = ($: EngineInterface, isCold: boolean) =>
105 update($, context, alert => (alert === null || alert.isCold === isCold ? alert : { ...alert, isCold }))
106
107/** The variables that decide the cache lifetime and whether an effort change keeps the cache. */
108async function billingEnvOf($: EngineInterface): Promise<BillingEnv> {
109 const [bedrock, mantle, vertex, foundry, anthropicAws, apiKey, baseUrl, noBetas, ttl, force5m, enable1h] = await Promise.all([
110 $.env.get('CLAUDE_CODE_USE_BEDROCK'),
111 $.env.get('CLAUDE_CODE_USE_MANTLE'),
112 $.env.get('CLAUDE_CODE_USE_VERTEX'),
113 $.env.get('CLAUDE_CODE_USE_FOUNDRY'),
114 $.env.get('CLAUDE_CODE_USE_ANTHROPIC_AWS'),
115 $.env.get('ANTHROPIC_API_KEY'),
116 $.env.get('ANTHROPIC_BASE_URL'),
117 $.env.get('CLAUDE_CODE_DISABLE_EXPERIMENTAL_BETAS'),
118 $.env.get('CLAUDE_CODE_PROMPT_CACHE_TTL'),
119 $.env.get('FORCE_PROMPT_CACHING_5M'),
120 $.env.get('ENABLE_PROMPT_CACHING_1H'),
121 ])
122
123 return {
124 providers: [bedrock, mantle, vertex, foundry, anthropicAws],
125 apiKey,
126 baseUrl,
127 noBetas,
128 ttl,
129 force5m,
130 enable1h,
131 }
132}
133
134function isFirstOfItsKind(session: Session, found: Advice) {
135 if (found.need !== 'normal') return true
136 if (session.isRoutineAdvised) return false
137
138 session.isRoutineAdvised = true
139
140 return true
141}
142
143/** The text of the assistant's last reply, so a short go-ahead is judged by the plan it approves. */
144async function lastReply($: EngineInterface) {
145 const rows = await $.session.messages().catch(() => [])
146
147 for (let at = rows.length - 1; at >= 0; at -= 1) {
148 const row = rows[at]
149
150 if (row?.role === 'assistant' && row.text.trim() !== '') return row.text
151 }
152
153 return ''
154}
155
156async function jevKeyOf($: EngineInterface, options: PluginOptions) {
157 if (options.jevShadow !== 'on') return undefined
158
159 const configured = options.typesafeApiKey
160 const key = typeof configured === 'string' && configured !== '' ? configured : await $.env.get('TYPESAFE_API_KEY')
161
162 return key === undefined || key === '' ? undefined : key
163}
164
165async function askJev($: EngineInterface, key: string, body: string): Promise<ShadowJev> {
166 const startedAt = Date.now()
167 const timedOut = $.clock.sleep(JEV_TIMEOUT_MS).then(
168 () => undefined,
169 () => new Promise<never>(() => undefined),
170 )
171
172 try {
173 const answered = await Promise.race([
174 $.http.fetch(JEV_URL, {
175 method: 'POST',
176 headers: { Authorization: `Bearer ${key}`, 'Content-Type': 'application/json' },
177 body,
178 }),
179 timedOut,
180 ])
181
182 if (answered === undefined) {
183 return { status: 'failed', reason: `Jev took over ${JEV_TIMEOUT_MS / 1000}s`, tookMs: Date.now() - startedAt }
184 }
185
186 return parseJev(answered.status, answered.text, Date.now() - startedAt)
187 } catch {
188 return { status: 'failed', reason: 'Jev could not be reached', tookMs: Date.now() - startedAt }
189 }
190}
191
192async function shadowRowsOf($: EngineInterface): Promise<ShadowRow[]> {
193 const rows = await $.store.get(SHADOW_KEY).catch(() => undefined)
194
195 return Array.isArray(rows) ? (rows as ShadowRow[]) : []
196}
197
198/** Puts Jev's label beside Haiku's for later review; it never changes the advice. */
199async function logShadow($: EngineInterface, session: Session, turnId: string, need: Need, ended: Ended) {
200 const shadowed = session.shadowed.get(turnId)
201 session.shadowed.delete(turnId)
202
203 if (shadowed === undefined) return
204
205 const jev = await shadowed.jev
206 const row: ShadowRow = {
207 at: Date.now(),
208 ask: shadowed.ask,
209 haiku: need,
210 jev,
211 model: ended.usage.model,
212 usd: usageCost(ended.usage, ended.usage.model, ended.context, session.billing.ttl).usd,
213 jevUsd: shadowed.jevUsd,
214 }
215 await $.store.set(SHADOW_KEY, keep(await shadowRowsOf($), row))
216
217 if (jev.status === 'failed' && !session.isJevReported) {
218 session.isJevReported = true
219 $.ui.toast(`model-fit: Jev shadow failed (${jev.reason}); Haiku labels carry on. /fit has the details.`)
220 }
221}
222
223/** Learns the finished turn, shows what it was judged to need and cost, and leaves advice when it ran on more. */
224async function conclude($: EngineInterface, session: Session, turnId: string, judged: Judged, ended: Ended) {
225 const { need, scope } = judged
226 const found = advise({
227 turnId,
228 need,
229 ...(scope === undefined ? {} : { scope }),
230 model: ended.usage.model,
231 ...ended,
232 ...(session.baseline === undefined ? {} : { baseline: session.baseline }),
233 history: session.history,
234 billing: session.billing,
235 })
236 const actual = usageCost(ended.usage, ended.usage.model, ended.context, session.billing.ttl)
237 session.history = record(session.history, need, ended.steps, ended.usage.output_tokens)
238 await $.store.set(HISTORY_KEY, session.history)
239 const { tally } = session
240 tally.turns += 1
241 tally.usd += actual.usd
242
243 if (need === 'trivial') {
244 tally.simple += 1
245 tally.simpleUsd += actual.usd
246 }
247
248 if (need === 'trivial' && scope === 'standalone') tally.standalone += 1
249
250 await update($, verdict, () => ({
251 turnId,
252 need,
253 ...(scope === undefined ? {} : { scope }),
254 on: describeModel(ended.usage.model, ended.effort),
255 actual,
256 judgedBy: JUDGED_BY,
257 }))
258
259 if (found !== undefined && isFirstOfItsKind(session, found)) await update($, advice, () => found)
260
261 $.clock.after(0, () => void logShadow($, session, turnId, need, ended))
262}
263
264async function judge($: EngineInterface, options: PluginOptions, session: Session, turnId: string, text: string) {
265 const ask = text.slice(0, PROMPT_CHARS)
266 const reply = await lastReply($)
267 const asked = classifyText(ask, reply)
268 const key = await jevKeyOf($, options)
269
270 if (key !== undefined) {
271 const body = requestBody(CLASSIFY_BRIEF, ask, reply.slice(-REPLY_CHARS))
272 session.shadowed.set(turnId, { ask, jev: askJev($, key, body), jevUsd: jevCostOf(body) })
273 }
274
275 const scoped = scopeText(ask, reply)
276 session.tally.labelUsd += classifyCostOf(asked) + classifyCostOf(scoped)
277 const [label, scopeLabel] = await Promise.all([
278 $.model.classify(asked, NEEDS, { model: 'haiku' }).catch(() => undefined),
279 $.model.classify(scoped, SCOPES, { model: 'haiku' }).catch(() => undefined),
280 ])
281 const need = asNeed(label)
282 const scope = SCOPES.find(one => one === scopeLabel)
283 const ended = session.ended.get(turnId)
284 session.ended.delete(turnId)
285
286 if (need === undefined) {
287 session.shadowed.delete(turnId)
288
289 return
290 }
291
292 const judged: Judged = scope === undefined ? { need } : { need, scope }
293
294 if (ended !== undefined) return conclude($, session, turnId, judged, ended)
295 if (!session.running.has(turnId) || session.model === undefined) return
296
297 session.needs.set(turnId, judged)
298 const context = (await $.session.usage().catch(() => undefined))?.context.tokens ?? 0
299 const warning = warn(turnId, need, session.model, session.effort, context, session.billing)
300
301 if (warning !== undefined && isFirstOfItsKind(session, warning)) await update($, advice, () => warning)
302}
303
304const asks = (count: number) => `${count} ${count === 1 ? 'ask' : 'asks'}`
305
306const ASK_WORDS: Record<Need, string> = { trivial: 'Simple', normal: 'Routine', complex: 'Complex' }
307const LOOKS: Record<Need, string> = { trivial: 'simple', normal: 'routine', complex: 'complex' }
308
309const keepsCacheNote = (shown: Advice) => (shown.keepsCache === true ? "; it keeps this conversation's cache" : '')
310
311/**
312 * The advice: the first line the finding, the second what to do. Caches are per model, so any change of model is
313 * priced with the one-time re-cache it costs.
314 */
315const WORDS: Record<Advice['kind'], (shown: Advice) => [string, string]> = {
316 smaller: shown => [
317 `${ASK_WORDS[shown.need]} ask on ${shown.on}: ≈ ${dollars(shown.actual?.usd ?? 0)}. On ${shown.target.label} ≈ ${dollars(shown.target.usd ?? 0)} per ask like it.`,
318 paysNow(shown)
319 ? `Switching re-caches this conversation once (≈ ${dollars(shown.target.switchUsd ?? 0)}) and pays back on the next ask like it.`
320 : `Switching re-caches this conversation once (≈ ${dollars(shown.target.switchUsd ?? 0)}); worth it only if ${asks(shown.breakEven ?? 0)} like this follow.`,
321 ],
322 stay: shown => [
323 `${ASK_WORDS[shown.need]} ask on ${shown.on}: ≈ ${dollars(shown.actual?.usd ?? 0)}, mostly rereading ${tokens(shown.context ?? 0)} tokens of conversation.`,
324 [
325 `Stay on ${shown.on}: moving ${tokens(shown.context ?? 0)} tokens to ${shown.target.label} costs ≈ ${dollars(shown.target.switchUsd ?? 0)} once, ${asks(shown.breakEven ?? 0)} to earn back. Start your next new task on ${shown.target.label}.`,
326 ...(shown.lower === undefined ? [] : [`Or /effort ${shown.lower} for asks like it: it keeps this conversation's cache.`]),
327 ].join(' '),
328 ],
329 fresh: shown => [
330 `That ask didn't need this conversation, but rereading its ${tokens(shown.context ?? 0)} tokens cost ≈ ${dollars(shown.actual?.usd ?? 0)}.`,
331 `Ask ones like it in a new session on ${shown.target.label}: ≈ ${dollars(shown.target.usd ?? 0)}. Keep this one for the work it's about.`,
332 ],
333 lighter: shown => [
334 `${ASK_WORDS[shown.need]} ask on ${shown.on}: ≈ ${dollars(shown.actual?.usd ?? 0)}.`,
335 `For asks like it, ${shown.target.command ?? ''} would do and uses fewer tokens${keepsCacheNote(shown)}.`,
336 ],
337 bigger: shown => [
338 `This ask looks ${LOOKS[shown.need]} and runs on ${shown.on}.`,
339 `If the answer looks thin, run it again on ${shown.target.label}; moving this conversation there costs ≈ ${dollars(shown.target.switchUsd ?? 0)} once.`,
340 ],
341 harder: shown => [
342 `This ask looks ${LOOKS[shown.need]} and runs on ${shown.on}.`,
343 `If the answer looks thin, run ${shown.target.command ?? ''} and ask again${keepsCacheNote(shown)}.`,
344 ],
345}
346const COMPACT = '/compact keep:'
347/** The handoff skill this plugin ships; plugin skills are typed with the plugin's name in front. */
348const BATON = '/model-fit:baton'
349
350async function putCommand($: EngineInterface, command: string) {
351 if ((await $.prompt.read()).text.trim() !== '') {
352 $.ui.toast(`model-fit: clear the prompt box first; ${command} goes there`)
353
354 return
355 }
356
357 await $.prompt.fill({ text: command, mode: 'replace' })
358}
359
360export const register: Register = (on, options) => {
361 const session: Session = {
362 tally: { turns: 0, usd: 0, simple: 0, simpleUsd: 0, standalone: 0, labelUsd: 0 },
363 baseline: undefined,
364 shadowed: new Map(),
365 isJevReported: false,
366 model: undefined,
367 effort: undefined,
368 history: emptyHistory(),
369 needs: new Map(),
370 steps: new Map(),
371 running: new Set(),
372 ended: new Map(),
373 typed: undefined,
374 level: 0,
375 isCold: true,
376 cooling: undefined,
377 billing: { ttl: '1h', isEffortCacheSafe: false },
378 isRoutineAdvised: false,
379 }
380
381 on('session.start', async ($, e, next) => {
382 const settings: Record<string, unknown> = await $.settings.read()
383 session.model = typeof settings.model === 'string' ? settings.model : undefined
384 session.effort = asEffort(settings.effortLevel)
385 session.history = asHistory(await $.store.get(HISTORY_KEY))
386 session.billing = billingOf(await billingEnvOf($), settings.promptCacheTtl, options.cacheTtl)
387 // Advice belongs to the turn just run; a reload or resume must not draw the last one again.
388 await update($, advice, () => null)
389 await update($, verdict, () => null)
390 await $.command.register({
391 name: 'fit',
392 description: 'Model fit: what tasks cost so far, and the Jev shadow comparison; "/fit reset" forgets it',
393 })
394
395 return next(e)
396 })
397
398 on('prompt.submit', async ($, e, next) => {
399 const entered = await next(e)
400 const text = entered.text?.trim() ?? ''
401
402 if (HUMAN.has(e.origin.kind) && text !== '' && !text.startsWith('/')) session.typed = text
403
404 return entered
405 })
406
407 // Only a turn the person started is judged: a task notification, a continuation or a plugin's prompt keeps the advice.
408 on('turn.start', async ($, e, next) => {
409 session.isCold = false
410 session.cooling?.cancel()
411 session.cooling = undefined
412 await setCold($, false)
413 const started = await next(e)
414 const text = session.typed
415 session.typed = undefined
416
417 if (text !== undefined) {
418 await update($, advice, () => null)
419 await update($, verdict, () => null)
420 session.running.add(e.turnId)
421 $.clock.after(0, () => void judge($, options, session, e.turnId, text))
422 }
423
424 return started
425 })
426
427 on('turn.step', async function* ($, e, next) {
428 if (e.agentId === undefined) {
429 session.model = e.model
430 session.effort = asEffort(e.effort) ?? session.effort
431 session.steps.set(e.turnId, (session.steps.get(e.turnId) ?? 0) + 1)
432 }
433
434 return yield* next(e)
435 })
436
437 on('turn.complete', async ($, e, next) => {
438 const done = await next(e)
439
440 if (e.agentId !== undefined) return done
441
442 session.cooling?.cancel()
443 session.cooling = $.clock.after(CACHE_LIFETIME_MS[session.billing.ttl], () => {
444 session.isCold = true
445 void setCold($, true)
446 })
447 const wasTyped = session.running.delete(e.turnId)
448 const judged = session.needs.get(e.turnId)
449 const steps = session.steps.get(e.turnId) ?? 0
450 session.needs.delete(e.turnId)
451 session.steps.delete(e.turnId)
452
453 if (!wasTyped || e.usage === undefined || e.isAborted) {
454 session.shadowed.delete(e.turnId)
455
456 return done
457 }
458
459 const usage = await $.session.usage()
460 const ended: Ended = { usage: e.usage, context: usage.context.tokens ?? 0, steps, effort: session.effort }
461
462 if (judged === undefined) session.ended.set(e.turnId, ended)
463 else await conclude($, session, e.turnId, judged, ended)
464
465 return done
466 })
467
468 on('session.measure', async ($, e, next) => {
469 const count = e.context.tokens
470
471 if (count === undefined) return next(e)
472
473 const level = contextLevel(count)
474
475 if (level > session.level && session.model !== undefined) {
476 const alert = contextAlert(count, session.model, session.effort, session.billing.ttl)
477 await update($, context, () => alert)
478 } else if (level < session.level) {
479 await update($, context, () => null)
480 }
481
482 session.level = level
483
484 if (count > 0) session.baseline = Math.min(session.baseline ?? count, count)
485
486 return next(e)
487 })
488
489 on('command.run', { command: 'fit' }, async ($, e) => {
490 if (e.args.trim() === 'reset') {
491 session.history = emptyHistory()
492 await $.store.delete(HISTORY_KEY)
493 await $.store.delete(SHADOW_KEY)
494
495 return { text: 'Forgot the cost history and the Jev shadow log.' }
496 }
497
498 const current = session.model === undefined ? 'unknown' : describeModel(session.model, session.effort)
499 const rows = NEEDS.map(need => {
500 const typical = typicalTurn(session.history, need)
501 const { turns, misses } = session.history[need]
502 const learned = turns === 0 ? 'a guess so far' : `from ${turns} turns`
503 const missed = misses === 0 ? '' : `; ${misses} more ran over 3x that, so their label was wrong`
504
505 return `${need}: ~${typical.steps} steps, ~${tokens(typical.output)} output tok (${learned}${missed})`
506 })
507 const { tally } = session
508 const spent =
509 tally.turns === 0
510 ? 'This session: no ask judged yet.'
511 : `This session: ${tally.turns} ${tally.turns === 1 ? 'ask' : 'asks'} judged, ≈ ${dollars(tally.usd)}. ${tally.simple} simple, ≈ ${dollars(tally.simpleUsd)}; ${tally.standalone} of those did not need the conversation. Labelling with Haiku ≈ ${dollars(tally.labelUsd)}.`
512 const shadow =
513 (await jevKeyOf($, options)) === undefined
514 ? ['Jev shadow: off (opt in with the jevShadow option and a TypeSafe key).']
515 : shadowReport(await shadowRowsOf($))
516
517 return {
518 text: [`On ${current}.`, ...rows, '', spent, 'List-price estimates; /cost has the real figure.', '', ...shadow].join('\n'),
519 }
520 })
521
522 on('ui.render', { component: 'AbovePrompt' }, async ($, e, next) => {
523 const shown = await read($, advice)
524 const heavy = await read($, context)
525 const judged = await read($, verdict)
526
527 const below = await next(e)
528
529 if (e.props.hasSurvey || (shown === null && heavy === null && judged === null)) return below
530
531 const { Box, Button, Text } = $.ui.resolve(e)
532 const isWarning = shown?.kind === 'bigger' || shown?.kind === 'harder'
533 const command = shown?.kind === 'stay' || shown?.kind === 'fresh' ? undefined : shown?.target.command
534 const lower = shown?.kind === 'stay' ? shown.lower : undefined
535
536 return (
537 <Box flexDirection="column">
538 {judged === null || shown !== null ? null : (
539 <Box key="verdict">
540 <Text dimColor wrap="wrap">
541 ◇ {NEED_WORDS[judged.need]} · {judged.on} · ≈ {dollars(judged.actual.usd)} · judged by {judged.judgedBy}{' '}
542 </Text>
543 <Button key="dismiss-verdict" label="Dismiss" plain dimColor onPress={() => update($, verdict, () => null)} />
544 </Box>
545 )}
546 {shown === null ? null : (
547 <Box key="advice" flexDirection="column">
548 <Box>
549 <Text color={isWarning ? 'warning' : 'suggestion'}>{isWarning ? '▲ ' : '◆ '}</Text>
550 <Text wrap="wrap">
551 {WORDS[shown.kind](shown)[0]} <Text dimColor>Judged by {JUDGED_BY}.</Text>
552 </Text>
553 </Box>
554 <Box>
555 <Text wrap="wrap">
556 {' '}
557 {WORDS[shown.kind](shown)[1]}{' '}
558 </Text>
559 {command === undefined ? null : (
560 <Button key="use" label={`Put ${command} in prompt`} onPress={() => putCommand($, command)} />
561 )}
562 {lower === undefined ? null : (
563 <Button key="effort" label={`Put /effort ${lower} in prompt`} onPress={() => putCommand($, `/effort ${lower}`)} />
564 )}
565 <Button key="dismiss-advice" label="Dismiss" onPress={() => update($, advice, () => null)} />
566 </Box>
567 </Box>
568 )}
569 {heavy === null ? null : (
570 <Box key="context" flexDirection="column">
571 <Box>
572 <Text color="warning">▲ </Text>
573 <Text wrap="wrap">
574 Long conversation ({tokens(heavy.tokens)} tokens): each step Claude takes rereads it, ≈{' '}
575 {dollars(heavy.cachedUsd)} a step at list prices.{' '}
576 </Text>
577 </Box>
578 <Text wrap="wrap">
579 {' '}New task: /clear (this one stays in /resume). Same task: /compact what to keep, or {BATON} to pick it up later. Wrong turn: /rewind.
580 </Text>
581 {heavy.isCold ? (
582 <Text key="cold" dimColor wrap="wrap">
583 {' '}After this pause the cache has expired: the next action rereads all of it for ≈ {dollars(heavy.coldUsd)} on {heavy.on}. Shrink it first: {BATON} to pick up later, /compact to carry on now. This is also the cheapest moment to change model.
584 </Text>
585 ) : null}
586 <Box>
587 <Text>{' '}</Text>
588 <Button key="compact" label="Put /compact in prompt" onPress={() => putCommand($, COMPACT)} />
589 <Text> </Text>
590 <Button key="baton" label={`Put ${BATON} in prompt`} onPress={() => putCommand($, BATON)} />
591 <Text> </Text>
592 <Button key="dismiss-context" label="Dismiss" onPress={() => update($, context, () => null)} />
593 </Box>
594 </Box>
595 )}
596 {below.type === 'engine' ? null : below}
597 </Box>
598 )
599 })
600}
601hooks/fit.ts 371 lines1import type { Advice, Billing, BillingEnv, CacheTtl, ContextAlert, Cost, Effort, Need, Scope, Tier } from '../types'
2
3type Price = { input: number; output: number; cacheRead: number }
4export type Usage = {
5 input_tokens: number
6 output_tokens: number
7 cache_read_input_tokens: number
8 cache_creation_input_tokens: number
9}
10
11/** `misses`: turns given this label that ran far past it, so the label was wrong. */
12export type Totals = { turns: number; steps: number; output: number; misses: number }
13export type History = Record<Need, Totals>
14
15// List prices in USD per million tokens for the current models (Haiku 5.5, Sonnet 5.5, Opus 5.5,
16// Fable 5.1), from platform.claude.com/docs/en/about-claude/pricing on 2026-10-12; /cost stays the authority on
17// what was actually spent.
18const PRICE: Record<Tier, Price> = {
19 small: { input: 0.1, output: 0.5, cacheRead: 0.01 },
20 mid: { input: 2, output: 10, cacheRead: 0.1 },
21 large: { input: 4, output: 20, cacheRead: 0.2 },
22 top: { input: 10, output: 50, cacheRead: 0.25 },
23}
24const SMALL_LONG_PROMPT = 100_000
25const SMALL_LONG_PRICE: Price = { input: 0.5, output: 2.5, cacheRead: 0.05 }
26/** Cache writes cost this many times the input price, by how long the entry lives. */
27const CACHE_WRITE: Record<CacheTtl, number> = { '5m': 1.25, '1h': 2 }
28export const CACHE_LIFETIME_MS: Record<CacheTtl, number> = { '5m': 5 * 60_000, '1h': 60 * 60_000 }
29const MILLION = 1_000_000
30
31const priceOf = (tier: Tier, context: number) =>
32 tier === 'small' && context > SMALL_LONG_PROMPT ? SMALL_LONG_PRICE : PRICE[tier]
33
34const LABEL: Record<Tier, string> = { small: 'Haiku', mid: 'Sonnet', large: 'Opus', top: 'Fable' }
35const ALIAS: Record<Tier, string> = { small: 'haiku', mid: 'sonnet', large: 'opus', top: 'fable' }
36const TIERS: readonly Tier[] = ['small', 'mid', 'large', 'top']
37const EFFORTS: readonly Effort[] = ['low', 'medium', 'high', 'xhigh', 'max']
38
39/** The model and effort each label needs, from Claude Code's guidance on choosing an effort level. */
40const FIT: Record<Need, { tier: Tier; effort: Effort }> = {
41 trivial: { tier: 'small', effort: 'low' },
42 normal: { tier: 'mid', effort: 'medium' },
43 complex: { tier: 'large', effort: 'high' },
44}
45
46/**
47 * The models on which an effort change keeps the cache (Opus 5.5, Sonnet 5.5, Haiku 5.5, Fable 5.1), and those of
48 * them whose default effort is medium; a bare alias names the current model.
49 */
50const CURRENT = /^(opus|sonnet|haiku)(\[1m\])?$|(opus|sonnet|haiku)-5-5|fable-5-1/i
51const MEDIUM_DEFAULT = /^(opus|sonnet|haiku)(\[1m\])?$|(opus|sonnet|haiku)-5-5/i
52
53const defaultEffort = (model: string): Effort => (MEDIUM_DEFAULT.test(model) ? 'medium' : 'high')
54const above = (effort: Effort, than: Effort) => EFFORTS.indexOf(effort) > EFFORTS.indexOf(than)
55
56/** Whether changing effort keeps this conversation's cache on `model`, rather than rebuilding it. */
57export const keepsCacheOnEffort = (model: string, billing: Billing) => billing.isEffortCacheSafe && CURRENT.test(model)
58
59const isOn = (value: string | undefined) => value !== undefined && value !== '' && value !== '0'
60const asTtl = (value: unknown): CacheTtl | undefined => (value === '5m' || value === '1h' ? value : undefined)
61
62/**
63 * The main conversation's cache lifetime, in Claude Code's own order: FORCE_PROMPT_CACHING_5M, the TTL variable,
64 * the promptCacheTtl setting, ENABLE_PROMPT_CACHING_1H, then the default, which is an hour only on a subscription.
65 * A subscription that has run into usage credits drops to five minutes, which a mod cannot see: `override` fixes it.
66 * An effort change keeps the cache only with an API key or a subscription straight to Anthropic.
67 */
68export function billingOf(env: BillingEnv, setting: unknown, override: unknown): Billing {
69 const viaProvider = env.providers.some(isOn)
70 const isEffortCacheSafe = !viaProvider && !isOn(env.noBetas) && !isOn(env.baseUrl)
71 const detected = isOn(env.force5m)
72 ? '5m'
73 : (asTtl(env.ttl) ?? asTtl(setting) ?? (isOn(env.enable1h) ? '1h' : viaProvider || isOn(env.apiKey) ? '5m' : '1h'))
74
75 return { ttl: asTtl(override) ?? detected, isEffortCacheSafe }
76}
77
78const SEED: Record<Need, Omit<Totals, 'misses'>> = {
79 trivial: { turns: 1, steps: 2, output: 800 },
80 normal: { turns: 1, steps: 8, output: 5_000 },
81 complex: { turns: 1, steps: 25, output: 20_000 },
82}
83
84const WORTH_SWITCHING = 0.75
85/** Switching models pays back on the next ask like this one: recommend it. */
86const PAYS_NOW = 1
87/** Pays back only if a run of this many simple asks follows; past it, staying is the right call. */
88const PAYS_SOON = 4
89/** A standalone ask that reread at least this many tokens of conversation it did not need belongs in a new session. */
90const UNNEEDED_CONTEXT = 30_000
91/** Below this, re-caching on a change of effort costs too little to weigh, where the change does not keep the cache. */
92const SMALL_CONTEXT = 50_000
93/** A turn this many times its label's typical steps was labelled wrong: counted as a miss, not learned from. */
94const MISLABEL_FACTOR = 3
95export const REPLY_CHARS = 1_200
96export const CONTEXT_LEVELS = [100_000, 200_000, 400_000, 700_000]
97
98export const NEEDS: readonly Need[] = ['trivial', 'normal', 'complex']
99
100export const CLASSIFY_BRIEF = [
101 'A request to a coding agent working in a repository. Label how much model it needs.',
102 'trivial: a lookup, a one-line or one-file edit, a rename within one file, a git or shell chore, a question one file answers, a general concept or arithmetic question that needs no code read ("what is a hashmap", "2 plus 2").',
103 'complex: designing something, a feature across several modules, debugging a cause nobody knows yet, a wide refactor, a security or concurrency question.',
104 'normal: everything else, including a rename or find-and-replace across many files ("rename X everywhere").',
105 'A short go-ahead ("yes", "do it", "go on") approves the plan in the previous reply: label the work that plan needs.',
106].join('\n')
107
108export const classifyText = (request: string, previousReply: string) =>
109 [CLASSIFY_BRIEF, '', 'Previous assistant reply (context only):', previousReply.slice(-REPLY_CHARS), '', 'Request:', request].join('\n')
110
111export const SCOPES: readonly Scope[] = ['standalone', 'conversation']
112
113export const SCOPE_BRIEF = [
114 'A request to a coding agent, in the middle of a conversation. Does answering it need that conversation?',
115 'conversation: it refers to or depends on earlier turns: files or changes discussed, a plan, "it", "that", "the changes", a go-ahead.',
116 'standalone: a brand-new session with no history would answer it just as well: a general question, arithmetic, a self-contained lookup.',
117].join('\n')
118
119export const scopeText = (request: string, previousReply: string) =>
120 [SCOPE_BRIEF, '', 'Previous assistant reply (context only):', previousReply.slice(-REPLY_CHARS), '', 'Request:', request].join('\n')
121
122/**
123 * The one-time cost of moving this conversation onto `tier`: caches are per model, so its first request writes the
124 * whole context instead of reading it.
125 */
126export function switchCost(tier: Tier, context: number, ttl: CacheTtl) {
127 const price = priceOf(tier, context)
128
129 return (context * (CACHE_WRITE[ttl] * price.input - price.cacheRead)) / MILLION
130}
131
132export function tierOf(model: string | undefined): Tier | undefined {
133 if (model === undefined) return undefined
134 if (/haiku/i.test(model)) return 'small'
135 if (/sonnet/i.test(model)) return 'mid'
136 if (/opus/i.test(model)) return 'large'
137 if (/fable/i.test(model)) return 'top'
138
139 return undefined
140}
141
142export const describeModel = (model: string, effort: Effort | undefined) => {
143 const tier = tierOf(model)
144 const name = tier === undefined ? model : LABEL[tier]
145
146 return effort === undefined ? name : `${name}/${effort}`
147}
148
149export const emptyHistory = (): History => ({
150 trivial: { turns: 0, steps: 0, output: 0, misses: 0 },
151 normal: { turns: 0, steps: 0, output: 0, misses: 0 },
152 complex: { turns: 0, steps: 0, output: 0, misses: 0 },
153})
154
155/** Learns a finished turn, or counts it as a miss when it ran far past its label. */
156export function record(history: History, need: Need, steps: number, output: number): History {
157 const was = history[need]
158
159 if (isMislabelled(history, need, steps)) return { ...history, [need]: { ...was, misses: was.misses + 1 } }
160
161 return { ...history, [need]: { ...was, turns: was.turns + 1, steps: was.steps + steps, output: was.output + output } }
162}
163
164export function typicalTurn(history: History, need: Need) {
165 const seen = history[need]
166 const seed = SEED[need]
167 const turns = seen.turns + seed.turns
168
169 return {
170 steps: Math.max(1, Math.round((seen.steps + seed.steps) / turns)),
171 output: Math.round((seen.output + seed.output) / turns),
172 isLearned: seen.turns > 0,
173 }
174}
175
176export const isMislabelled = (history: History, need: Need, steps: number) =>
177 steps > typicalTurn(history, need).steps * MISLABEL_FACTOR
178
179/** `context` is the conversation's size, which sets Haiku's long-prompt price. */
180export function usageCost(usage: Usage, model: string, context: number, ttl: CacheTtl): Cost {
181 const price = priceOf(tierOf(model) ?? 'large', context)
182 const input =
183 (usage.input_tokens + usage.cache_creation_input_tokens * CACHE_WRITE[ttl]) * price.input +
184 usage.cache_read_input_tokens * price.cacheRead
185 const tokens =
186 usage.input_tokens + usage.cache_creation_input_tokens + usage.cache_read_input_tokens + usage.output_tokens
187
188 return { tokens, usd: (input + usage.output_tokens * price.output) / MILLION }
189}
190
191/**
192 * Before the work: an ask on a smaller model than its label needs, or a complex ask at low effort, may give a thin
193 * answer.
194 */
195export function warn(
196 turnId: string,
197 need: Need,
198 model: string,
199 effort: Effort | undefined,
200 context: number,
201 billing: Billing,
202): Advice | undefined {
203 const on = describeModel(model, effort)
204 const fit = FIT[need]
205
206 if (need !== 'trivial' && tierOf(model) === 'small') {
207 return {
208 turnId,
209 need,
210 kind: 'bigger',
211 on,
212 context,
213 target: {
214 label: LABEL[fit.tier],
215 command: `/model ${ALIAS[fit.tier]}`,
216 switchUsd: switchCost(fit.tier, context, billing.ttl),
217 },
218 }
219 }
220
221 if (need === 'complex' && effort === 'low') {
222 return {
223 turnId,
224 need,
225 kind: 'harder',
226 on,
227 keepsCache: keepsCacheOnEffort(model, billing),
228 target: { label: `${fit.effort} effort`, command: `/effort ${fit.effort}` },
229 }
230 }
231
232 return undefined
233}
234
235type Finished = {
236 turnId: string
237 need: Need
238 /** Whether the ask needed this conversation; undefined when that could not be judged. */
239 scope?: Scope
240 model: string
241 effort: Effort | undefined
242 usage: Usage
243 context: number
244 /** The smallest context seen this session: what a new session starts from. */
245 baseline?: number
246 steps: number
247 history: History
248 billing: Billing
249}
250
251/** What the turn would have cost on `tier` in a new session: the baseline written once and reread per step. */
252function freshCost(turn: Finished, baseline: number, tier: Tier) {
253 const fresh: Usage = {
254 input_tokens: turn.usage.input_tokens,
255 output_tokens: turn.usage.output_tokens,
256 cache_creation_input_tokens: baseline,
257 cache_read_input_tokens: baseline * Math.max(0, turn.steps - 1),
258 }
259
260 return usageCost(fresh, ALIAS[tier], baseline, turn.billing.ttl)
261}
262
263/**
264 * The turn as the next ask like it would run on a warm cache. A turn that wrote at least half the context to the
265 * cache rebuilt it, a one-time cost; the next ask rereads those tokens instead, so the write becomes a read.
266 */
267function warmed(usage: Usage, context: number): Usage {
268 if (usage.cache_creation_input_tokens * 2 < context) return usage
269
270 return {
271 ...usage,
272 cache_read_input_tokens: usage.cache_read_input_tokens + usage.cache_creation_input_tokens,
273 cache_creation_input_tokens: 0,
274 }
275}
276
277/**
278 * After the work, for the next ask like it, when a simple or routine ask ran on a bigger model or a higher effort
279 * than its label needs. Caches are per model, so a model switch is weighed against the re-cache it costs: an ask
280 * that did not need the conversation is pointed at a new session; one that did is told to switch only when the
281 * saving per ask earns the re-cache back quickly, and to stay otherwise. A lower effort is offered where changing
282 * effort keeps the cache, or where the context is too small for the re-cache to matter. Its saving is not priced:
283 * the docs publish no output per effort level. The other model is priced as if it wrote the same output. A turn
284 * that ran far past its label gets no advice.
285 */
286export function advise(turn: Finished): Advice | undefined {
287 const tier = tierOf(turn.model)
288
289 if (tier === undefined || turn.need === 'complex') return undefined
290 if (isMislabelled(turn.history, turn.need, turn.steps)) return undefined
291
292 const fit = FIT[turn.need]
293 const { ttl } = turn.billing
294 const actual = usageCost(turn.usage, turn.model, turn.context, ttl)
295 const base = {
296 turnId: turn.turnId,
297 need: turn.need,
298 on: describeModel(turn.model, turn.effort),
299 actual,
300 context: turn.context,
301 }
302 const keepsCache = keepsCacheOnEffort(turn.model, turn.billing)
303 const isEffortHigh = above(turn.effort ?? defaultEffort(turn.model), fit.effort)
304 const lower = isEffortHigh && (keepsCache || turn.context < SMALL_CONTEXT) ? fit.effort : undefined
305
306 if (TIERS.indexOf(tier) > TIERS.indexOf(fit.tier)) {
307 const unneeded = turn.context - (turn.baseline ?? 0)
308
309 if (turn.scope === 'standalone' && turn.baseline !== undefined && unneeded >= UNNEEDED_CONTEXT) {
310 return { ...base, kind: 'fresh', target: { label: LABEL[fit.tier], usd: freshCost(turn, turn.baseline, fit.tier).usd } }
311 }
312
313 const recurring = warmed(turn.usage, turn.context)
314 const onFit = usageCost(recurring, ALIAS[fit.tier], turn.context, ttl)
315 const saving = usageCost(recurring, turn.model, turn.context, ttl).usd - onFit.usd
316 const switchUsd = switchCost(fit.tier, turn.context, ttl)
317 const target = { label: LABEL[fit.tier], command: `/model ${ALIAS[fit.tier]}`, usd: onFit.usd, switchUsd }
318
319 if (onFit.usd <= (onFit.usd + saving) * WORTH_SWITCHING && saving > 0) {
320 const breakEven = Math.max(1, Math.ceil(switchUsd / saving))
321
322 if (breakEven <= PAYS_SOON) return { ...base, kind: 'smaller', breakEven, target }
323
324 return { ...base, kind: 'stay', breakEven, target, ...(lower !== undefined && keepsCache ? { lower, keepsCache } : {}) }
325 }
326 }
327
328 if (lower === undefined) return undefined
329
330 return { ...base, kind: 'lighter', keepsCache, target: { label: `${lower} effort`, command: `/effort ${lower}` } }
331}
332
333export const paysNow = (shown: Advice) => (shown.breakEven ?? Infinity) <= PAYS_NOW
334
335export const contextLevel = (count: number) => CONTEXT_LEVELS.filter(level => count >= level).length
336
337export function contextAlert(count: number, model: string, effort: Effort | undefined, ttl: CacheTtl): ContextAlert {
338 const price = priceOf(tierOf(model) ?? 'large', count)
339
340 return {
341 tokens: count,
342 on: describeModel(model, effort),
343 cachedUsd: (count * price.cacheRead) / MILLION,
344 coldUsd: (count * CACHE_WRITE[ttl] * price.input) / MILLION,
345 isCold: false,
346 }
347}
348
349export function tokens(count: number) {
350 if (count >= MILLION) return `${(count / MILLION).toFixed(1)}M`
351 if (count >= 1000) return `${Math.round(count / 1000)}k`
352
353 return String(count)
354}
355
356/** At list prices, an estimate: /cost and the bill are the authority. */
357export const dollars = (usd: number) => (usd < 0.01 ? '<$0.01' : `$${usd.toFixed(2)}`)
358
359export const NEED_WORDS: Record<Need, string> = {
360 trivial: 'Looked simple',
361 normal: 'Looked routine',
362 complex: 'Looked complex',
363}
364
365const CHARS_PER_TOKEN = 4
366const LABEL_TOKENS = 5
367
368/** The Haiku labelling call's own cost, guessed from its text: the API does not report it to a mod. */
369export const classifyCostOf = (text: string) =>
370 ((text.length / CHARS_PER_TOKEN) * PRICE.small.input + LABEL_TOKENS * PRICE.small.output) / MILLION
371hooks/jev.ts 104 lines1import type { Need, ShadowJev, ShadowRow } from '../types'
2
3export const JEV_URL = 'https://api.typesafe.ai/v1/systemone'
4export const JEV_TIMEOUT_MS = 8000
5/** Below this, Jev's top label is no call at all: an uncertain answer is not read as "complex". */
6export const SURE = 0.8
7export const SHADOW_KEEP = 200
8const SHOWN_DISAGREEMENTS = 5
9const ASK_CHARS = 60
10
11// TypeSafe's list price for Jev: $0.042 per million input tokens, output free. Tokens are guessed at four
12// characters each, so this is an estimate.
13const JEV_INPUT_PER_MILLION = 0.042
14const CHARS_PER_TOKEN = 4
15
16export const jevCostOf = (body: string) => (body.length / CHARS_PER_TOKEN) * (JEV_INPUT_PER_MILLION / 1_000_000)
17
18const ORDER: readonly Need[] = ['trivial', 'normal', 'complex']
19
20export function requestBody(brief: string, request: string, previousReply: string) {
21 return JSON.stringify({
22 model: 'jev-latest',
23 state: { request, previous_assistant_reply: previousReply },
24 questions: {
25 need: {
26 type: 'score',
27 instructions: brief,
28 criteria: [
29 'trivial: a lookup, a concept question, a one-line or one-file edit, a rename within one file, a git or shell chore.',
30 'normal: everything between trivial and complex, including a rename or find-and-replace across many files.',
31 'complex: design, a feature across modules, debugging an unknown cause, a wide refactor, security or concurrency.',
32 ],
33 },
34 },
35 })
36}
37
38type RawAnswer = { probabilities?: Record<string, number> }
39
40export function parseJev(status: number, text: string, tookMs: number): ShadowJev {
41 if (status === 401) return { status: 'failed', reason: 'Jev refused the API key (401)', tookMs }
42 if (status < 200 || status >= 300) return { status: 'failed', reason: `Jev answered ${status}`, tookMs }
43
44 let answer: RawAnswer | undefined
45
46 try {
47 const parsed: unknown = JSON.parse(text)
48 answer =
49 typeof parsed === 'object' && parsed !== null && 'answers' in parsed
50 ? (parsed.answers as Record<string, RawAnswer>).need
51 : undefined
52 } catch {
53 answer = undefined
54 }
55
56 const shares = Object.entries(answer?.probabilities ?? {})
57 .sort(([a], [b]) => Number(a) - Number(b))
58 .map(([, share]) => share)
59
60 if (shares.length !== ORDER.length) return { status: 'failed', reason: 'Jev sent an answer model-fit could not read', tookMs }
61
62 const top = shares.indexOf(Math.max(...shares))
63 const confidence = shares[top] ?? 0
64 const label = confidence >= SURE ? ORDER[top] : undefined
65
66 return { status: 'judged', ...(label === undefined ? {} : { label }), confidence, shares, tookMs }
67}
68
69export const keep = (rows: ShadowRow[], row: ShadowRow) => [...rows, row].slice(-SHADOW_KEEP)
70
71const median = (values: number[]) => {
72 const sorted = [...values].sort((a, b) => a - b)
73
74 return sorted.length === 0 ? 0 : (sorted[Math.floor(sorted.length / 2)] ?? 0)
75}
76
77const askOf = (ask: string) => (ask.length > ASK_CHARS ? `${ask.slice(0, ASK_CHARS - 1)}…` : ask)
78
79/** What the shadow comparison has found so far: agreement, Jev's doubts and failures, its speed and its cost. */
80export function shadowReport(rows: ShadowRow[]) {
81 if (rows.length === 0) return ['Jev shadow: on, nothing compared yet.']
82
83 const judged = rows.filter(row => row.jev.status === 'judged')
84 const sure = judged.filter(row => row.jev.status === 'judged' && row.jev.label !== undefined)
85 const agreed = sure.filter(row => row.jev.status === 'judged' && row.jev.label === row.haiku)
86 const failures = rows.flatMap(row => (row.jev.status === 'failed' ? [row.jev.reason] : []))
87 const disagreements = sure.flatMap(row =>
88 row.jev.status === 'judged' && row.jev.label !== row.haiku
89 ? [` "${askOf(row.ask)}": Haiku ${row.haiku}, Jev ${row.jev.label} (${Math.round(row.jev.confidence * 100)}%)`]
90 : [],
91 )
92 const spent = rows.reduce((sum, row) => sum + row.jevUsd, 0)
93 const agreement = sure.length === 0 ? 'no sure answers yet' : `agree on ${agreed.length} of ${sure.length} sure answers`
94 const cost = spent < 0.01 ? 'under $0.01' : `≈ $${spent.toFixed(2)}`
95
96 return [
97 `Jev shadow (Haiku still drives the advice): ${rows.length} ${rows.length === 1 ? 'ask' : 'asks'} compared, ${agreement}.`,
98 ` Jev unsure on ${judged.length - sure.length}, failed on ${failures.length}; median ${(median(rows.map(row => row.jev.tookMs)) / 1000).toFixed(1)}s; ${cost} in all.`,
99 ...(failures.length > 0 ? [` Last failure: ${failures.at(-1)}.`] : []),
100 ...(disagreements.length > 0 ? [' Recent disagreements (neither is ground truth):'] : []),
101 ...disagreements.slice(-SHOWN_DISAGREEMENTS),
102 ]
103}
104types/index.d.ts 82 lines1export type Need = 'trivial' | 'normal' | 'complex'
2export type Tier = 'small' | 'mid' | 'large' | 'top'
3export type Effort = 'low' | 'medium' | 'high' | 'xhigh' | 'max'
4export type Cost = { tokens: number; usd: number }
5
6/** How long the main conversation's cache lives; it also sets the write price, 1.25x input for 5m, 2x for 1h. */
7export type CacheTtl = '5m' | '1h'
8
9/** How this session is billed, as far as it moves the advice: `isEffortCacheSafe` when an effort change can keep the cache. */
10export type Billing = { ttl: CacheTtl; isEffortCacheSafe: boolean }
11
12/**
13 * The environment variables behind Billing: `providers` the CLAUDE_CODE_USE_* switches for Bedrock, Mantle, Agent
14 * Platform, Foundry and Claude Platform on AWS; `noBetas` CLAUDE_CODE_DISABLE_EXPERIMENTAL_BETAS; `ttl`
15 * CLAUDE_CODE_PROMPT_CACHE_TTL; `force5m` and `enable1h` FORCE_PROMPT_CACHING_5M and ENABLE_PROMPT_CACHING_1H.
16 */
17export type BillingEnv = {
18 providers: (string | undefined)[]
19 apiKey?: string
20 baseUrl?: string
21 noBetas?: string
22 ttl?: string
23 force5m?: string
24 enable1h?: string
25}
26
27/** Whether an ask needed the conversation it was asked in, or could have been asked in a new session. */
28export type Scope = 'standalone' | 'conversation'
29
30/**
31 * What to use next time. `command` is what the user types to switch; `usd` what the turn would have cost on it
32 * with its cache warm; `switchUsd` the one-time cost of caching this conversation on it, since caches are per model.
33 */
34export type Target = { label: string; command?: string; usd?: number; switchUsd?: number }
35
36/**
37 * `bigger` warns while an ask runs on a smaller model than its label needs, `harder` while a complex one runs at low
38 * effort. After a simple or routine ask ran on too much, for the next ask like it: `smaller` when switching pays
39 * back within `breakEven` asks, `stay` when it would not, `fresh` when the ask did not need the conversation and a
40 * new session is the cheaper place for it, `lighter` when only the effort was too high. `actual` is what the turn
41 * cost; `context` the conversation's size. `lower` is an effort to offer beside `stay`; `keepsCache` whether an
42 * effort change keeps this conversation's cache.
43 */
44export type Advice = {
45 turnId: string
46 need: Need
47 kind: 'smaller' | 'stay' | 'fresh' | 'lighter' | 'bigger' | 'harder'
48 on: string
49 target: Target
50 actual?: Cost
51 context?: number
52 breakEven?: number
53 lower?: Effort
54 keepsCache?: boolean
55}
56
57export type ContextAlert = { tokens: number; on: string; cachedUsd: number; coldUsd: number; isCold: boolean }
58
59/** What the finished turn was judged to need and what it cost: shown even when there is no advice. */
60export type TurnVerdict = { turnId: string; need: Need; scope?: Scope; on: string; actual: Cost; judgedBy: string }
61
62/** Jev's label in shadow mode: no `label` when its top share is under the bar, so no call was made. */
63export type ShadowJev =
64 | { status: 'judged'; label?: Need; confidence: number; shares: number[]; tookMs: number }
65 | { status: 'failed'; reason: string; tookMs: number }
66
67export type ShadowRow = {
68 at: number
69 ask: string
70 haiku: Need
71 jev: ShadowJev
72 model: string
73 usd: number
74 jevUsd: number
75}
76
77declare module 'claude-code' {
78 interface PluginState {
79 'model-fit': { advice: Advice | null; context: ContextAlert | null; verdict: TurnVerdict | null }
80 }
81}
82