[[wikilinks]]. When you query with --save, the answer becomes a new page and future queries use it as context. Embeddings, BM25 reranking, and wikilink-graph expansion then run over this compiled artifact, narrowing hundreds of pages to a tight, citation-traceable evidence pack.
The Compile Pipeline
Two-Phase Compile
llmwiki splits compilation into two distinct phases rather than processing each source end-to-end.1
Phase 1 - Concept Extraction
Every changed source is sent to the LLM, which identifies and extracts the key concepts each source contains. All extractions complete before any page is written. This means the compiler knows the full concept universe - including which concepts appear in multiple sources - before it commits to writing a single file.
2
Phase 2 - Page Generation
For each extracted concept, the LLM generates a structured wiki page with YAML frontmatter, prose body, and
[[wikilinks]] to related pages. Concepts claimed by more than one source are merged into a single page at this stage instead of producing duplicate files.orphaned rather than silently disappearing.
Incremental Compilation
llmwiki avoids re-processing unchanged work at every layer of the pipeline.Source hashing
Every file in
sources/ is SHA-256 hashed and compared against .llmwiki/state.json. Only sources whose hash changed - or that are brand new - flow through the LLM pipeline.Embedding updates
Chunk embeddings in the active JSON or binary index are content-hash-aware. Re-running on an unchanged corpus skips all embedding work.
Cached citations
Citation judgements from
llmwiki eval --suite full are cached in .llmwiki/eval/citation-cache.jsonl. Subsequent eval runs only re-judge new pairs.Hybrid Retrieval
Retrieval runs over the compiled wiki, not raw source chunks.1
Cosine similarity
The v3 embedding index carries page and chunk vectors under qualified page ids, in either JSON or binary storage. A query narrows hundreds of pages to a small top-K via cosine similarity over chunk embeddings.
2
BM25 rerank
The candidate set is reranked using BM25 to boost lexically relevant pages that dense retrieval may have under-ranked.
3
Wikilink-graph expansion
Pages directly linked from the top-K results are pulled in as graph neighbors, broadening the evidence pack with contextually related content the embedding alone might not surface.
When no embedding store is present - for example, when using the GitHub Copilot provider, which has no embeddings endpoint - llmwiki automatically falls back to lexical (index-based) ranking and surfaces a stable warning code rather than hard-failing.
Source Freshness
Every compiled page records the sources - and their SHA-256 content hashes - that produced it. On any later command, llmwiki compares those recorded hashes againstsources/ on disk:
- Stale - a page whose recorded source hashes no longer match the files on disk. The source still exists, but its content changed since the last compile.
- Orphaned - a page whose recorded sources were all deleted from
sources/.
llmwiki lint, llmwiki status, the local viewer, the JSON export, and the MCP wiki_status tool all surface stale and orphaned pages without triggering a recompile. To repair them, run llmwiki refresh --stale: it recompiles only the changed sources that own stale pages and cleans up orphaned pages, deliberately leaving unrelated new sources for a full llmwiki compile. Use --dry-run to preview the repair plan with no LLM calls or writes.