silica-core
Retrieval tools for coding agents, no model in the loop: files, search, read, code_pack and write_note over the folder the session is opened in, served over…
newbandcommandtoastpromptprocess
Retrieval tools for coding agents, no model in the loop: files, search, read, code_pack and write_note over the folder the session is opened in, served over…
<img src="https://raw.githubusercontent.com/kiycoh/silica-core/main/assets/banner-light.svg" alt="Silica Core" width="590" />
Lightweight, local evidence retrieval tools for code and research
Silica locates the source, symbol, page, or passage you and your agents need. Maximum signal, minimum machinery.
<a href="https://pypi.org/project/silica-core/"><img src="https://img.shields.io/pypi/v/silica-core?style=flat&labelColor=000000&color=000000" alt="PyPI" /></a> <a href="https://github.com/kiycoh/silica-core/blob/main/LICENSE"><img src="https://img.shields.io/github/license/kiycoh/silica-core?style=flat&labelColor=000000&color=000000" alt="License" /></a>
<!-- The MCP registry proves ownership of the PyPI package by finding this string in the description PyPI renders, which is this file. --> <!-- mcp-name: io.github.kiycoh/silica-core -->
Silica indexes the markdown, code, PDFs and office files under one root and serves them to Claude Code, Cursor, Hermes, Codex, OpenCode and every other popular harness, as MCP tools or as shell commands that print the same JSON. A hit is a path, a section, a line, a window of text and the numbers to judge it by. The harness owns the loop.
<img src="https://raw.githubusercontent.com/kiycoh/silica-core/main/assets/quickstart.gif" alt="the quickstart recorded end to end: uv tool install, silica-core init reporting nine indexed documents, silica-core setup claude registering the MCP server, then Claude Code answering a question about the LSM compaction design space by calling silica-core and citing the PDF with its page and its line" width="900" />
Three commands, then a question asked the way you would ask any other. The harness calls silica_search, and the answer carries the file, the page and the line it came from.
uv tool install 'silica-core[mcp]' # BM25 over documents and their sections (faster, lighter)
uv tool install 'silica-core[mcp,dense]' # in addition the dense leg: numpy, a static model (still fast, more precise)The second line adds the dense leg: numpy and a static model2vec model, no torch and no GPU. It stays inert until the model is named and the sections are embedded — the last stanza of the Quickstart. Take it when the questions are paraphrases that share no words with the text; an exact term or an identifier is answered by the lexical leg either way, and only that leg reports terms_absent.
pipx works the same. The package and the command are silica-core, the tools are silica_*.
Upgrading from 0.9 or older: the silica command is silica-core since 0.10.0 (silica belongs to silica-harness). A client registered by an older silica setup still launches silica mcp, which stops starting once uvx fetches 0.10.0. Run silica-core setup <client> once and it rewrites that entry.
In any folder of markdown, code, PDFs or office files:
silica-core init # adopt the folder: ignore file, vault.yaml, first index; the MCP server serves adopted folders only
silica-core search "leveled compaction" -k 5 # the best located passages
silica-core setup claude # register the MCP server, write the guidance block into ~/.claude/CLAUDE.md
# optional, with the [dense] extra: the dense leg, a static model, nothing leaves the machine
export SILICA_EMBEDDING_MODEL=model2vec/minishlab/potion-retrieval-32M
silica-core index --embed<img src="https://raw.githubusercontent.com/kiycoh/silica-core/main/assets/search-hit.png" alt="silica-core search "leveled compaction" over nine LSM papers: the hit carries the file, p. 5 and the passage; beside it the PDF is open on that page with the same passage highlighted" width="900" />
Nine arXiv papers, indexed in 2.1 s. The hit names the file, the page and the passage; the page beside it is the check.
| Tool | Shell | Returns |
|---|---|---|
silica_files | silica-core files | the inventory and what the index did with each file: indexed, changed, excluded, failed, unconverted |
silica_search | silica-core search | ranked passages: path, section, line, BM25, matched terms, coverage, and the query terms absent from the corpus |
silica_read | silica-core read | a slice by lines or by heading (a page, in a PDF), the outline, and a version to carry forward |
silica_code_pack | silica-core code-pack | an AST context pack for one source file inside a character budget |
silica_write_note | silica-core write-note | one atomic write, linted for structure and unresolved wikilinks |
In a source tree every function, method, class and constant is its own unit: a hit's section is the symbol, span its lines, and silica_read(path, section=…) serves the body. For a symbol whose name is known, grep wins; for a question that names none, the search comes first: the tool description and the block silica-core setup claude writes say so. The contract, the reply shapes and the acceptance checks are in TOOLS.md. Nothing needs an API key or a network.
A ranked list always has a top, even when the corpus does not answer. Three fields say how much the result is worth:
coverage: the share of the query's idf mass the hit's matched terms carry. Near 1, every rare term matched; near 0, only common words did.terms_absent: query terms that occur nowhere in the corpus.matched_terms: the words this hit actually contains.<img src="https://raw.githubusercontent.com/kiycoh/silica-core/main/assets/search-says-no.png" alt="silica-core search "raft consensus log replication" over the same nine papers: terms_absent lists raft and consensus, coverage falls to 0.19, and the top hit is a passage about data replication rather than Raft" width="900" />
raft consensus log replication over the same nine papers: raft and consensus occur in none of them, coverage falls to 0.19, and the top hit is about data replication. On 254 papers the top hit of an answered question carries 0.69 to 1.00; a question the corpus does not cover, 0.44. Silica exposes the signals; the harness decides whether to stop, read or rephrase.
nDCG@10 on BEIR SciFact · NFCorpus, the same documents and queries for every arm. BEIR's published BM25 baselines are 0.665 · 0.325. Silica's lexical index needs no model; the others serve lexical search from an index that also holds embeddings.
| Mode | Silica | zvec-grep 0.2.2 | ck 0.7.11 |
|---|---|---|---|
| Lexical | 0.662 · 0.311 | 0.649 · 0.297 | 0.630 · 0.289 |
Hybrid, same potion-retrieval-32M embedder | 0.675 · 0.328 | 0.672 · 0.330 |
Code, on the twenty SWE-QA questions zvec-grep publishes for its own benchmark, same embedder, k = 10, scored on the files and symbols the reference answer rests on.
| Arm | file hit@5 · @10 | file MRR | symbol hit@10 | symbol recall | chars returned |
|---|---|---|---|---|---|
| Silica, hybrid | 0.85 · 0.90 | 0.68 | 0.75 | 0.24 | 8,266 |
| Silica, vectors | 0.80 · 0.85 | 0.67 | 0.65 | 0.21 | 6,225 |
| Silica, lexical | 0.65 · 0.75 | 0.47 | 0.45 | 0.14 | 8,194 |
| zvec-grep 0.2.2, hybrid | 0.65 · 0.75 | 0.54 | 0.55 | 0.17 | 7,326 |
| zvec-grep 0.2.2, vector | 0.60 · 0.80 | 0.61 | 0.55 | 0.18 | 6,821 |
| zvec-grep 0.2.2, FTS | 0.45 · 0.60 | 0.36 | 0.45 | 0.11 | 6,690 |
On BEIR the two hybrids tie at the 95% interval: the same vectors rank the same, with no daemon and no vector store. On code, Silica's hybrid file MRR is +0.135 over zvec-grep's hybrid (95% interval +0.01 to +0.27), paired per question. The fusion also gains +0.21 MRR and +0.30 symbol hit over Silica's lexical arm; no reranker or graph expansion is involved.
On this measured scope, Silica is a compact, local, SOTA-competitive retriever: it matches zvec-grep on BEIR and leads the paired SWE-QA code-localization replay with the same embedder.
Retrieval matters only if the agent does less work without losing the answer. These are separate experiments and are not pooled:
| Workload and arm | Runs | Quality | Search used | Turns | Tool calls | Seconds | Warm cost |
|---|---|---|---|---|---|---|---|
| Repository, search-first contract | 20 | Judge 59.7 | 17/20 | 4.7 | — | 24 | $0.197 |
| Repository, same plugin without contract | 20 | Judge 50.6 | 0/20 | 6.5 | — | 28 | $0.180 |
| Documents, resident Silica tools | 12 tasks | 12/12 correct | 12/12 | 3.9 | 2.9 | — | $0.20 |
| Documents, no plugin | 12 tasks | 12/12 correct | — | 5.0 | 4.0 | — | $0.22 |
The repository result is one repetition: turns improve by 1.75 (95% interval 0.55 to 3.05 fewer), while Judge and cost remain inconclusive. The document rows belong to a 144-run study over twelve questions and a 5.5M-token corpus. They establish less work on that workload, not a universal agent claim.
Corpora, intervals, per-task exceptions and reproduction commands are in benchmarks.
silica-core setup <client> writes the registration into the client's own config and backs up what was there; for claude it also puts a guidance block, when to search before grep, into ~/.claude/CLAUDE.md. silica-core setup --list names the clients: claude, codex, cursor, windsurf, zed, cline, roo, continue, goose, opencode, openhands, gemini, antigravity, dsh, hermes, openclaw, agent-zero, claude-desktop, lmstudio, anythingllm and librechat; shell, python and generic print recipes for anything else. The server serves the folder the client opens in; --vault DIR or SILICA_VAULT fixes the root.
Every written block, and the Claude Code plugin, run silica-core mcp --retrieval local-hybrid: potion-retrieval-32M in the server process, index and vectors built in the background at start, the search lexical and dense: warming until they land. Nothing leaves the machine; the one download is the model, once. npx skills add kiycoh/silica-core installs the skills (the one that tells an agent when to reach for the tools, and five that explain, summarize, compare, schematize and diagram from the indexed documents), and nothing else. Shell recipes, Docker and the REPL are in public/harnesses.md.
.txt, .rst and PDFs with a text layer directly, one PDF page per section; DOCX, EPUB, FB2, RTF, XLS and ODF converted with no extra; scanned PDFs, images, PPTX and XLSX through silica-core import with MinerU; audio and video with ffmpeg plus SILICA_STT_BASE_URL; CSV readable by line, excluded from search. In a source tree the code lane adds source files and their json, yaml, toml, cfg and ini. silica-core doctor says which lanes this machine has.uv tool install stays lexical until silica-core index --embed, with the [dense] extra's local model or any OpenAI-compatible /v1/embeddings endpoint. Text leaves the machine only for a remote endpoint, and only after silica-core index --embed --allow-remote grants that host once. Variables and reply states in TOOLS.md, the ollama recipe in public/harnesses.md.silica-core mcp --extended adds the wikilink tools; silica-core connect (extra [connect]) hosts the bridge the Obsidian plugin dials into, so writes land through the vault API while the app is open; silica-core repl runs a small reference agent over the same tools, the one surface that needs a model (SILICA_MODEL).MIT. See LICENSE.
hooks/register.ts 94 lines1import { atom, read, update } from 'claude-code'
2import type { EngineInterface, Register } from 'claude-code'
3import { GUIDANCE, START } from './guidance.ts'
4
5// The ask of docs/adr/0001, lever 1: the UserPromptSubmit hook removed in
6// 0573f2e, back in-process and only where `silica-core init` left vault.yaml.
7// QUESTION and IDENTIFIER are its two regexes as measured on 2026-09-10.
8const QUESTION = /\?|^\s*(what|where|how|why|which|when|who|does|do|is|are|can|list|explain|describe|find)\b/i
9// a backtick, snake_case, camelCase, a path or an extension, `::`, `#L`: the
10// prompt names something grep can look for
11const IDENTIFIER = /`|\b[A-Za-z]\w*_\w+\b|\b[A-Za-z]*[a-z][A-Z]\w*\b|\/\w|\.\w{1,4}\b|::|#L\d/
12
13export const NUDGE = 'silica-core: this question names no identifier. Before any grep (a Grep tool, or grep or rg in a shell), Glob or Read, state whether silica_search applies (yes or no); if yes, call it first with the question as `query`.'
14
15// ponytail: the folder the session started in; SILICA_VAULT pointing elsewhere is not seen (ADR-0001)
16const adopted = async ($: EngineInterface) => $.fs.exists(`${await $.session.root()}/vault.yaml`)
17
18// Where `silica-core setup claude` writes its block: guidance.claude_md_path().
19async function claudeMdHoldsTheBlock($: EngineInterface): Promise<boolean> {
20 const dir = (await $.env.get('CLAUDE_CONFIG_DIR'))
21 || `${(await $.env.get('HOME')) ?? (await $.env.get('USERPROFILE'))}/.claude`
22 const text = await $.fs.read(`${dir}/CLAUDE.md`).catch(() => '')
23 return String(text).includes(START)
24}
25
26// Lever 5: the folder that holds hooks/ is the plugin root (the module does not
27// see CLAUDE_PLUGIN_ROOT), and init runs through mcp.json's own launch, extras
28// included, so uv never re-syncs the environment the server shares
29const PLUGIN_ROOT = decodeURIComponent(new URL('..', import.meta.url).pathname).replace(/\/$/, '').replace(/^\/([A-Za-z]:)/, '$1')
30const INIT = ['uv', 'run', '--project', PLUGIN_ROOT, '--extra', 'mcp', '--extra', 'dense', 'silica-core', 'init']
31const offer = atom({ plugin: 'silica-core', key: 'offer' } as const, 'hidden')
32
33async function adopt($: EngineInterface): Promise<{ isIndexed: boolean; text: string }> {
34 const root = await $.session.root()
35 const r = await $.process.run(INIT, { cwd: root, timeoutMs: 600_000 })
36 .catch(err => ({ exitCode: 1, stderr: String(err) }))
37 return r.exitCode === 0
38 ? { isIndexed: true, text: `${root} is indexed; silica_search serves it from its next call.` }
39 : { isIndexed: false, text: `silica-core init failed in ${root}: ${r.stderr.trim().split('\n').at(-1) ?? ''}` }
40}
41
42export const register: Register = (on) => {
43 on('session.start', async ($, e, next) => {
44 await $.command.register({ name: 'silica-init', description: 'Index this folder for silica-core (runs silica-core init)' })
45 const root = await $.session.root()
46 const home = (await $.env.get('HOME')) ?? (await $.env.get('USERPROFILE'))
47 const isAsked = e.isInteractive && root !== '/' && root !== home
48 && !(await adopted($)) && !(await $.store.get(`dismissed:${root}`))
49 await update($, offer, () => (isAsked ? 'offer' : 'hidden'))
50 return next(e)
51 })
52
53 on('command.run', { command: 'silica-init' }, async ($) => {
54 const r = await adopt($)
55 if (r.isIndexed) await update($, offer, () => 'hidden')
56 return { text: r.text }
57 })
58
59 on('ui.render', { component: 'AbovePrompt' }, async ($, e, next) => {
60 const phase = await read($, offer)
61 if (phase === 'hidden' || e.props.hasSurvey) return next(e)
62 const { Box, Button, Text } = $.ui.resolve(e)
63 if (phase === 'indexing') return h(Box, {}, h(Text, { dimColor: true }, 'silica-core: indexing this folder…'))
64 return h(Box, {},
65 h(Text, {}, 'silica-core: this folder is not indexed. '),
66 h(Button, { key: 'index', label: 'Index', variant: 'primary', onPress: async () => {
67 await update($, offer, () => 'indexing')
68 const r = await adopt($)
69 $.ui.toast(r.text) // the engine names the plugin in front of it
70 await update($, offer, () => (r.isIndexed ? 'hidden' : 'offer'))
71 } }),
72 h(Button, { key: 'dismiss', label: 'Not here', onPress: async () => {
73 await $.store.set(`dismissed:${await $.session.root()}`, true)
74 await update($, offer, () => 'hidden')
75 } }))
76 })
77
78 on('prompt.submit', async ($, e, next) => {
79 const text = e.text.trim()
80 if (!text || !QUESTION.test(text) || IDENTIFIER.test(text)) return next(e)
81 if (!(await adopted($))) return next(e)
82 return next({ ...e, context: [...(e.context ?? []), NUDGE] })
83 })
84
85 // Lever 3: the block the bench measured through --append-system-prompt-file,
86 // last in the system prompt, where the request offers silica_search
87 on('prompt.compose', async ($, e, next) => {
88 const composed = await next(e)
89 if (!e.tools.some(t => t.endsWith('__silica_search'))) return composed
90 if (!(await adopted($)) || (await claudeMdHoldsTheBlock($))) return composed
91 return { sections: [...composed.sections, { id: 'silica-core:guidance', text: GUIDANCE, scope: 'session' }] }
92 })
93}
94hooks/guidance.ts 4 lines1// Generated from silica_core/onboarding/guidance.py; tests/test_contract_nudges.py fails when they differ.
2export const START = "<!-- SILICA_START -->"
3export const GUIDANCE = "## silica-core\n\n- For a question about what the files in the current folder say, or how a behaviour is implemented, when no identifier is known: call `silica_search` with the question before any grep (a Grep tool, or grep or rg in a shell), Glob or Read. It ranks passages and, with the code index on, the functions, methods, classes and constants themselves, by the question's words and their vectors; a hit's `section` is the symbol, `span` its lines.\n- Grep for an exact string or a symbol name you already know; read a file directly when its path and section are already known.\n- Cite a hit's passage by path and line when it answers; read with `silica_read(path, section=\u2026)` only for missing context, carrying the hit's `version` as `expect_version`.\n- `terms_absent` and `coverage` are lexical diagnostics, not verdicts: a dense hit can answer with coverage 0. If the passages give no evidence, rephrase once keeping names and identifiers; then report what remains unfound."
4hooks/types.d.ts 9 lines1// The values the module keeps in $.state (claude plugin validate holds it to this).
2export type SilicaCoreOffer = 'hidden' | 'offer' | 'indexing'
3
4declare module 'claude-code' {
5 interface PluginState {
6 'silica-core': { offer: SilicaCoreOffer }
7 }
8}
9