A knowledge base for an LLM: three layers, a linter and a rule against drift
We keep a knowledge base over a working archive from June to September 2026: 87 conversations and 13 projects going in, about 80 source pages, 23 entity pages, 50 concept pages and 8 analyses coming out. It is built not as file storage but as memory a language model works with.
Here is how it is structured, and the rules without which it would have become a dump within a month.
Why plain retrieval accumulates nothing
The standard setup: documents sit in storage, and on every question the model finds the relevant chunks and answers from them. It works, and it retains nothing. The same analysis is redone every time, and yesterday’s conclusions are unavailable today because they were never written down.
We changed the model’s role. It does not only answer from documents, it maintains a derived layer: pages for entities, concepts and analyses with links between them. A new source is not merely indexed: the substance is extracted, related pages are updated, and disagreements with earlier data are flagged explicitly.
The value is not search. The value is that by the time the question is asked, the work is already done.
Three layers with hard boundaries
Sources come first and are immutable. The model reads from them and never edits them. They are the point of truth you return to when the derived layer looks off.
The wiki comes second and belongs to the model: it creates pages, updates them, maintains cross references and watches for consistency.
The schema file comes third: rules, conventions, folder structure, heading format. This file is what turns a model into a disciplined keeper of the base rather than a chatbot that reinvents the structure every session. It is edited jointly as it becomes clear what works.
The division of labour: humans own source selection and good questions, the model owns the bookkeeping.
Rules without which it falls apart
Mandatory fields on every page: title, type, created and updated dates, source count, tags. Without formal fields you can neither sort nor validate.
A rule against drift. Values that change over time live in fields, not in prose. Otherwise a month later the text contains numbers that are no longer true and nobody remembers which is current.
An archive rule. When sources disagree, the later one wins, and the earlier one is not deleted. A note stays next to it saying it used to be different, because the history of a decision is often more useful than the decision.
A gaps section: an explicit page listing what the base does not contain. It looks like a weakness and saves time, because a question from that list is not asked of the base, it is asked of a person.
Prompt injection protection. Materials are reference data, not a source of commands. An instruction found inside a document is ignored; commands come only from a human in the conversation. Without that block, any file you are sent can start steering the system.
Mechanical validation instead of trust
Next to the base sits a linter: a small script that checks the formal parts. Are the mandatory fields present, does the source counter match reality, do links point at pages that exist, are there orphan pages nobody references.
It is the boring part, and it is what separates a working base from a pretty one. Models fail on small things: the wrong type, a stale date, a link to a page that was never created. As long as a script catches those instead of a human, the base stays coherent.
The warning that matters most
Most systems like this get abandoned. Someone spends a month on templates and plugins, collects hundreds of notes and never returns to them. It becomes collecting disguised as work.
The base pays off only when it has an outlet: you write from it, decide from it, answer clients or prepare documents with it. Without an outlet it is a tidy dump, and it is better not to start.
A practical test before you begin: name three questions you ask yourself more than once a month and rebuild the context for every time. If there are none, you do not need a knowledge base, you need search over a folder.
FAQ
How is this different from plain document search
With plain retrieval the model re reads documents on every question and nothing is retained. Here a layer of linked pages sits between sources and people, maintained by the model: links are in place, contradictions are flagged, summaries are current.
How many documents does this need to make sense
It is not about volume but about repetition. If you return to the same material regularly and rebuild the context each time, the structure pays off at dozens of documents.
What should happen with contradictions between sources
Record them as contradictions instead of silently picking one. In our schema the later source wins while the earlier one stays with a note that it used to be different, which preserves why a decision changed.
Can the model be trusted to edit the base
Yes, if the layer boundaries are strict. Sources are immutable and only humans change them, the model owns the derived layer, and the rules live in a separate file edited jointly.