Cedwyn library

Architecture

The core catalogue database, section data, federation and the no-duplication rule.

Three layers

LayerWhere it livesWhat it holds
Section sourceOne branch per sectionThe section's pages, styles, scripts and its own data files (JSON, CSV, Markdown).
Assembled siteBranch siteEvery section copied under /<id>/, links rewritten, shared files stored once in /_blob/.
Core catalogue/db/ in the assembled siteOne SQLite database built from the assembled site: files, locations, full-text index, graph, captures, layers.

The core catalogue database

Built by tools/build_db.py on every deploy. Size at the last build: 672 MB, split into 2,565 chunk files of 256 KB (/db/cedwyn.sqlite3.000 onwards). Counts: 11,829 artifacts, 18,857 locations, 4,042 stored blobs, 1,166 nodes, 271,768 edges, 8,481 full-text rows, 183 captures. Live figures: /db/stats.json.

TableOne row perKey columns
artifactsunique file (by sha256)id, sha, kind, mime, bytes, title, year_min, year_max, has_blob
locationsplace a file appearsaid → artifacts.id, theme (section id), path, url
blobsstored file bodysha, data (raw bytes: images, PDFs, data files up to 2.5 MB each)
ftspage or text fileFTS5 index: title, body, aid, theme, path
nodes, edgesgraph node / relationpage→embeds/links→artifact, artifact→mentions→year, artifact→cites→host, artifact→in_theme→section
capturesarchived web addresskey, url, title, latest, captured
layers, layer_items, layer_refs, layer_depsreading over the evidenceitems point at artifact ids; they never copy content
section_recordsrecord from a section's own databasetheme, kind, id, seq, data (JSON)
evidence_gradesevidence recordev, grade, origin, path, note
themes, metasection / build factsection id, title, source, file and page counts; build statistics

Shared without duplication

  • Content addressing. A file that appears in several sections is stored once, at /_blob/<sha>.<ext>, and once in blobs. Each appearance is a row in locations.
  • References, not copies. A layer such as 1000 Cuts stores artifact ids. A section that needs a core record links to it or reads it from the database.
  • Served from the database. After the build, data and evidence files whose exact bytes are in the database are removed from the deploy and served from the database chunks by the service worker (/cedwyn-sw.js) and the page fallback (/_cedwyn/runtime.js with /_cedwyn/dbfs.js). Files that browsers fetch before a service worker runs stay as files.

Federation: section data and the core

Each section is the source of truth for its own data. The convention for a section's own database: the branch ships public/_db/section.sqlite, a small SQLite file with one table

CREATE TABLE records(kind TEXT, id TEXT, seq INTEGER, data TEXT,  -- data = one JSON object
                     PRIMARY KEY(kind, id));
CREATE TABLE meta(k TEXT, v TEXT);

holding every word and record the section shows (interface strings are rows too, for example kind = 'words'). The section's database works on its own and is served at /<id>/_db/section.sqlite. On every build, tools/build_db.py merges each section database into the core table section_records(theme, kind, id, seq, data) and adds graph edges: in_theme, derived_from (a record made from a core artifact) and shows (an image the record displays). The section's pages read their rows back from the core: SELECT kind, id, data FROM section_records WHERE theme = ? ORDER BY seq.

So a section owns its data, the core holds one merged copy for reading, search, graph and export, and a record that already exists in the core (an image, a document) is referenced by path or sha256, never copied into the section. Sections with live writes keep a small store of their own (see Sections). Every section can be taken out as a slice: /db/export/sections/<id>.json.gz holds its row, its own database rows and every artifact it shows (see Data export).

Why one built database rather than a live database per section: it costs nothing to serve, it cannot drift out of sync, and it stays inside the free limits of the host. Live per-section databases add servers, quota and replication with no gain at the current scale.

Graph and GraphQL

The nodes and edges tables are a ready graph for retrieval and analysis. A GraphQL schema is resolved in the browser against the same database: see API.