A knowledge base for an LLM: three layers, a linter and a rule against drift

The standard way of working with documents through a language model goes like this: files sit in storage, and on every question the model finds the relevant chunks and answers from them. It works. And it retains nothing.

The same analysis gets redone every time, while yesterday’s conclusion is unavailable today because nobody wrote it down. We changed the model’s role, and here is what came out of it.

The model maintains a wiki instead of only searching files

It does not just answer from documents, it keeps a derived layer: pages for entities, concepts and analyses with links between them. A new source does not merely enter an index. The substance is extracted, related pages are updated, and disagreements with earlier data are flagged explicitly. The value of such a base is not search. It is that by the time the question arrives, the work is already done, and that changes the feel of the work more than the description suggests. You are not searching. You are reading something finished.

Keep the three layers strictly apart

Sources come first and they are immutable. The model reads from them and never edits, so there is a point of truth to return to when the derived layer looks off. The wiki comes second and belongs to the model: it creates pages, updates them, maintains cross references and watches consistency. The schema file comes third. Rules, conventions, folder structure, heading format. This file is what turns a model into a disciplined keeper of the base and not a chatbot reinventing the structure every session, and it gets edited jointly as it becomes clear what works. The division of labour ends up simple: humans own source selection and good questions, the model owns the bookkeeping.

Rules without which the whole thing falls apart in a month

Every page carries mandatory fields. Title, type, created and updated dates, source count, tags. Without them you can neither sort nor validate, which means you cannot trust the base either.

A rule against drift (the most useful of them all). Values that change over time live in fields, not in prose, otherwise a month later the text carries numbers that are no longer true and nobody remembers which is current. An archive rule: when sources disagree the later one wins, the earlier one is not deleted, and a note stays next to it saying it used to be different. The history of a decision is often more useful than the decision.

A separate «Gaps» page listing what the base does not contain. It looks like a weakness and works as a time saver, because a question from that list is not asked of the base, it is asked of a person. And prompt injection protection, because materials are reference data and not a source of commands. An instruction found inside a document is ignored, commands come only from a human in the conversation. Without that block, any file you are sent can start steering the system.

A boring script does more than a clever model

Next to the base sits a linter, a small program that checks the formal parts. Are the mandatory fields present, does the source counter match reality, do links point at pages that exist, are there orphan pages nobody references.

The model fails on small things: the wrong type, a stale date, a link to a page it has not created yet. As long as a script catches those instead of a human, the base stays coherent, and this boring part is what separates a working base from a pretty one.

What it costs in time

Ours covers a working archive from June to September 2026. Going in: 87 conversations and 13 projects. Coming out: about 80 source pages, 23 entity pages, 50 concept pages and 8 analyses. People usually ask how long that took. I have no honest answer, because I never measured. The bulk of the effort went into checking and editing the rules, and that is the most underestimated line item in projects of this kind.

What I do not know, and why I warn anyway

Most systems like this get abandoned. Somebody spends a month on templates and plugins, collects hundreds of notes and never returns to them, and it becomes collecting disguised as work.

Such a base pays off only when it has an outlet: you write from it, decide from it, answer clients or prepare documents with it. No outlet means it is just a tidy dump. Better not to start at all.

Back to where this began. Plain retrieval accumulates nothing, and an accumulating base brings no value either until you ask it questions regularly. The test before starting is simple: name three questions you ask yourself more than once a month and rebuild the context for every time. If there are none, you need search over a folder. If there are, tell us what they are, because the questions show whether you need the whole scheme or a part of it.