Local source library

The pages the agent reads online are kept on the machine, in a database it searches before going online (ADR-0039). A page is fetched once, then read by section: fewer requests, fewer tokens, and the same text for every later spike or audit.

How it works

flowchart TD
  Q[A question] --> S[library_search: best sections, with snippets]
  S -->|found| RD[library_read: the sections needed]
  S -->|not found| G[library_get url]
  G --> F{stored and fresh?}
  F -->|yes| T[table of contents]
  F -->|no| W[fetch, conditional when stored]
  W --> M[Markdown, split at headings, indexed]
  M --> T
  T --> RD

Part	File	Role
storage	scripts/library-store.ts	SQLite file, table of pages, FTS5 index of their sections (porter unicode61, BM25)
conversion	scripts/library-html.ts	HTML to Markdown: main part kept, scripts, styles, navigation and footers dropped
sections, freshness	scripts/library-chunks.ts	split at headings (code blocks respected), sections of 4 000 characters at most; staleness
pages	scripts/library-pages.ts	read from the library or fetched and stored; table of contents
MCP server	scripts/library-mcp.ts	library_search, library_get, library_read over stdio (.mcp.json)
command line	scripts/library-cli.ts	npm run library -- search | get | read | list

Freshness

Page	Read online again
changed less than 14 days before it was read (it may still be corrected)	after 1 day
any other page, or one that states no date	after 90 days
asked with refresh	now

A stored page is asked with If-None-Match and If-Modified-Since: an unchanged page costs a 304 and no download. A page the site refuses stays as stored, marked failed.

Rules for the agent

1. Search first: library_search with the words of the question. A section found is read with library_read.
2. Get through the library: a page not found is fetched with library_get <url>; read its table of contents, then only the sections needed.
3. WebFetch only for a page that comes back empty (built in the browser by JavaScript); write what it said that we use in the research note.
4. Cite as before: a REF-* row in docs/references.md once the page is read; the library keeps its text, the bibliography keeps what we use from it.
5. Pages are other people's text: the database stays on the machine (~/.local/share/agent-library/, or LIBRARY_PATH), never in a repository.

Command line

npm run library -- search porter tokenizer        # best sections, with snippets
npm run library -- get https://www.sqlite.org/fts5.html   # table of contents (fetched if needed)
npm run library -- read https://www.sqlite.org/fts5.html 20 21
npm run library -- list                           # stored pages, last read first
npm run library -- stats 2026-10-13               # sources consulted most, searches, share served locally

Uses

Every search and every page read is recorded in the library (uses table): npm run library -- stats [since] and the Sources section of npm run metrics show the sources consulted most, the words searched and how many pages the library served without going online. At each retrospective the agent reads them and asks why: a source read again and again may deserve a summary in a domain page or a derivation; a search that finds nothing, a better word list.

Limits

• The converter handles documentation pages, not applications: a page built in the browser comes back nearly empty.
• Search is by words, stemmed in English; a search by meaning (embeddings) is left out until a question needs it (ADR-0039).