Research wiki 3 — Search over the same files

Notes from a research desk I built to teach myself audio ML — get signal, find relevant data, and answer questions from a compounding LLM wiki kept as plain files under OKF.

Part 3 — after the files

Part 1 is the job graph: scheduled ingest into a compounding wiki.

Part 2 is the file contract: typed pages, plain links, per-directory indexes, git — so the corpus stays readable and portable.

Because that contract is files on disk, a retrieval layer is additive and reversible. You can put one on top, and take it off, without rewriting the corpus.

This post explains when I use that search layer and when I do not.

The question I keep asking

The wiki already answers a lot of “what do we know about X?”

Concept pages hold synthesis revised in place. Entity pages hold named models and datasets that showed up more than once. Source pages hold the short, structured note for each ingest. Indexes route you there.

So why add search-by-meaning at all?

Because I do not always know the exact terms used in a paper, and I do not always need a broad summary.

My rule so far: I use search by meaning when there are many documents and I do not know which file contains the answer. I may not know the paper’s phrasing. I only know what I am trying to find.

That is a different job from “open the multimodal machine listening concept and read the synthesis,” and a different job from “list every page that touches X.”

Sometimes I ask about a theme (“anything on performer identity bias”). Sometimes I need a specific number, ablation result, or caveat that never made it into the short source page, and I cannot remember which paper contained it. Search by meaning helps in both cases. It does not replace exact keyword search or produce a complete list of every relevant page.

Two jobs, same corpus

The overnight agents that maintain this wiki are deliberately narrow. They navigate by index, grep, and explicit paths. They are not meant to wander the whole corpus on every run — that is how token cost and sloppy promotion creep back in.

The chat UI serves a different purpose: searching across hundreds of pages and full-text extracts without knowing the filename first. Related wording can match even when my keywords miss — “performer identity bias” ≈ “artist-specific confounds” ≈ “recording-identity shortcut.”

The files are the same. The way I search them depends on the question.

I want…	Best first move	Why
An overview of a theme	Open the concepts/ page	The summary and contradictions already live there
A named model / dataset across papers	Open the entities/ page (or its mentions)	The page exists because the name crossed 2+ sources
“Anything about X,” but I do not know the file or exact wording	Ask the synced Knowledge Base	It combines keyword and meaning-based search, then sorts the results by relevance
A detail I cannot place — hyperparameter, ablation, footnote caveat	Ask the synced Knowledge Base	Search the full paper text without loading every paper into chat
A complete list — “which pages touch X,” what recurs on Y	Grep, indexes, or a harness that can iterate	Top-k search returns likely matches, not every match
Daily / weekly maintenance	Leave it to the agents	Targeted, batched, cheap by design — they should not “search everything” live

These are defaults, not strict rules. I also ask the Knowledge Base about themes and named models when I forget the page name, do not know whether a page exists, or want to search the full paper text. If the concept or entity page already exists, I usually open it first because it already contains the summary and connections.

If I make the overnight agents search the whole wiki for every question, each query uses more tokens and takes longer. The search index avoids repeating that work.

If I ask the search layer to summarize a concept that already has a maintained page, I get a new answer assembled from retrieved passages instead of using the summary that already exists.

When search by meaning is useful

I use similarity search when I do not know the file or the exact wording. It helps me find likely matches. It does not tell me every page that mentions a topic.

Meaning-based search does not replace keyword search. This wiki contains model names, arXiv ids, acronyms, and other exact terms. A vector search can miss a page even when its title contains the term I typed.

The Research Wiki combines keyword search (BM25, with its weight left at the default 0.5) and vector search. It then uses a reranking model to reorder the retrieved passages by relevance. This works better for both exact terms and paraphrased questions than vector search alone.

The full paper text in fulltext/ is what makes this search useful.

Primary-topic arXiv papers get a GROBID extract on disk. The source page stays short on purpose — its Key Insight, Methodology, and Open Questions are based on the paper body, but the page is not a copy of the PDF. Most details are left out.

Vector search over those extracts can find details that the short page left out. Full text is essential here. Without it, the query layer would mostly find summaries I already wrote.

Does it find details deep in a paper?

Finding a claim from the abstract is easy. I wanted to know whether it could find a result from an ablation table on page nine.

I asked a question using none of the paper’s own vocabulary: “does it matter more which speech-to-text model I use or which search model?” instead of “ASR” and “retrieval backend.”

Six chunks came back. The useful one wasn’t from the abstract or conclusion. It was from the Ablation Study: a small ASR model paired with a strong retriever beats a huge ASR model paired with a weak one. The result appears once, deep in the paper. This is the job I wanted fulltext indexing and reranking to do.

Query:    "if my audio is noisy, does it matter more which
           speech-to-text model I use or which search model?"
Sources:  6 chunks retrieved
Hit:      Ablation Study subsection — not the abstract
Outcome:  specific finding surfaced from a paraphrased query

Does it keep claims from different papers separate?

The corpus has more than one paper that mentions BM25. The second test was whether retrieval would quietly stitch claims from different papers into one confident answer.

I asked “how did BM25 do?” — as bare as that, cold, first message in a fresh chat, no prior context to lean on. Four chunks came back, pulled from four different papers. The answer stayed anchored to one paper across every claim it made, and rather than blend in the other paper’s unrelated mention of BM25, it named the second paper and offered to cover it separately if that’s what I meant.

Query:    "how did BM25 do?" (cold — no history, no memory context)
Sources:  4 chunks, from 4 distinct papers
Outcome:  scoped to one paper; no cross-paper blending;
          alternate paper flagged rather than merged in

These are fixed tests I can repeat, not a sample of normal usage. I chose them because I already knew what a correct answer looked like, so a bad result would be obvious. Further tuning should wait until I have enough traces from real chat searches, not more hand-picked examples.

I open concept pages when I want an overview. I use the Knowledge Base when I need a specific detail but do not know the file or exact wording. I use grep or a harness when I need a complete list of matching pages.

Why vectors, not a graph

Part 1 used Herk’s levels to compare different knowledge-system designs. For this project, the curated wiki remains the main source, and semantic search is an optional way to query the same files. I do not need a typed knowledge graph yet.

This search layer solves a specific problem: I know what I need, but not the filename or the words used in the source. A knowledge graph solves a different problem: following defined relationship types across many pages.

The wiki already provides most of the structure I need. Concept and entity pages contain the summaries. Plain links and backlinks connect related pages. Those relationships are written in prose rather than stored as formal edge types.

A graph would require defined relationship types, rules for keeping them consistent, and another system to synchronize. I have not encountered a problem that only GraphRAG or LightRAG would solve. The current wiki plus optional vector search is enough.

Why Open WebUI + vectors, not a chat harness

The other option is to point a coding or research harness at the wiki files — Claude Code, a Claude Project, or an OpenClaw chat agent — and let it grep, read, and summarize on demand.

I still use that for known file paths, exact terms, and questions that require a complete list (“which pages touch X,” what recurs across notes on Y). Those questions favor grep, indexes, or a harness that can keep searching. Top-k retrieval is useful for finding likely passages, but not for listing every match.

Karpathy’s LLM Wiki gist (April 2026) is the clearest published version of this alternative: compile sources into a persistent wiki instead of relying on retrieval-augmented generation (RAG). Parts 1–2 of this series already use that pattern — compiled markdown, typed pages, links to raw sources, cross-links, and agent maintenance — although I arrived at it independently. The only difference is how I search the wiki: I add RAG as an optional search method rather than using it as the only way to access the corpus. The compiled wiki is still better for summaries and complete lists.

Open WebUI + Qdrant provide three practical differences:

• The search index and answer model are separate. I can attach one Knowledge Base to a chat and switch between local or API models. Today Open WebUI is the only client reading the index, so this works inside one UI; it is not yet a general retrieval service.
• Indexing does not use an LLM. A daily timer compares files and syncs only the changes. The same input produces the same result, costs little to run, and is easier to debug than agent-driven indexing.
• Each document version is indexed once. Using a harness to explore the whole corpus would spend model tokens and time on every query.

I agree with Karpathy on this: if a concept or entity page already has the summary, I should open that page rather than ask chat to rebuild it from retrieved passages. That does not mean every use of Claude Code or a similar tool is wrong — opening a known file, searching for an exact term, or listing matching pages is still useful.

Retrieval quality depends on how documents are split into passages, or “chunks,” and on the model that converts those passages into vectors for comparison. Chunks that include their section headers and document titles usually carry more context than simple fixed-size splits; Open WebUI’s default splitting is still basic. Changing the embedding model requires re-indexing. The file sync must also remove deleted or renamed files, or stale entries remain in the vector store. Combining keyword and vector search, then reranking the results, improves their order but does not remove these limits.

I still open files directly and use agents for maintenance. Open WebUI is one access method, not the only way to use the corpus.

How it was wired

Short version — method, not a product tour.

The wiki directory syncs once a day into an Open WebUI Knowledge Base. Qdrant backs the vectors instead of the UI’s default ChromaDB.

I did not compare several vector databases for this post. Qdrant was already in the lab compose from the base-infrastructure setup — VECTOR_DB=qdrant pointed Open WebUI at a store with its own volume and dashboard, rather than storing the vectors inside the UI’s application data. The wiki sync reused that configuration.

Sync uses Open WebUI’s official KB CLI (oikb) on a systemd user timer — deliberately not an agent cron. There is no LLM in the sync path. It is file-diffing and indexing. The standing rule: systemd for deterministic file work; agent cron only for jobs that need an LLM.

I attach the Knowledge Base per chat rather than baking it into a fixed “research assistant” model. Base model stays swappable (local or API) while the same corpus is available when I want it.

The current Open WebUI settings retrieve 20 candidate passages, rerank them with BAAI/bge-reranker-v2-m3, and send the best 4–5 to the answer model. BM25, the keyword-search part, stays at its default weight of 0.5. The relevance threshold stays at 0. The two tests above check that this configuration works, but they do not represent normal usage. I will wait for enough real chat traces before tuning it again.

Problems I encountered:

• The KB is a daily snapshot. Something ingested mid-morning is not queryable until the next sync. Files on disk are always newer than the vectors.
• After an Open WebUI upgrade or container recreate, the sync CLI’s API key can 401 even when the KB volume is fine — regenerate the key; do not assume the old one survived.
• Empty or fresh KBs can make status helpers crash; listing by KB id was the reliable check here.
• “Reranking enabled” with no model and Top K equal to Top K Reranker does nothing. My prior settings were 3 → 3 with the model unset, so the system returned the top three vector-search results without reranking them. Reranking needs a model and a larger candidate set to sort and reduce.

Close

Parts 1 and 2 built a compounding research wiki as plain files, then made those files portable. This part adds optional search through Open WebUI over the same corpus without changing the wiki itself.

I open the written pages when a summary already exists. I use search by meaning when I need a specific detail but do not know the file or exact wording. Hybrid search and reranking improve matching and result order; they do not replace concept pages, grep, or complete enumeration. The agents remain focused on specific maintenance tasks, and the full paper text stays in the search index. I will add a typed graph only if I encounter a problem that requires formal relationship queries.

The query layer works. It did not replace the wiki. It did not force a graph. It sits on top of files I can still cat, diff, and git.

Pointers

• Prequel: Research wiki 1 — Agents as jobs
• Prequel: Research wiki 2 — OKF for a machine-maintained wiki
• Open WebUI: open-webui/open-webui
• Vector store: Qdrant
• Reranker: BAAI/bge-reranker-v2-m3
• PDF structure extract: GROBID
• Compiled-wiki approach: Karpathy — llm-wiki
• Knowledge-system levels: Nate Herk — Every Level of a Claude Second Brain Explained