The model

The grains the system is built out of, what writes each one, and what each one grows with.

Maintained. This is the vocabulary the code uses, and is expected to be true. When it and the code disagree, the code is right and this needs fixing. Last checked 30 August 2026.

Every grain here exists. The evolution plan is about what does not exist yet, and uses this vocabulary to say so.

Why a vocabulary document

Six words in this system mean something specific and are easy to blur: a protocol is not a connector, a source is not a dataset instance, a profile is not guidance, a concept row is not the English word, and a layer is not necessarily spatial. Every one of those distinctions decides where a fact gets written and who is allowed to write it.

The grains

"A person" in that column means a person deciding, not a person typing. The reading, the drafting and most of the research behind a connector, a holder route or a concept row are done with Claude; what makes the row curated is that somebody chose it and put their name to it. The line that matters is not who produced the characters. It is whether a judgement was made and reviewed, or whether the system wrote the row down while it was busy doing something else — a harvest matching a catalogue, a fetch recording a profile, a call leaving an observation. Curated rows are trusted by default; generated rows are provisional and say so, however good they look.

Layer is not a spatial word

Layer already carries kind: raster | vector | table in lib/contracts.ts, and HAPI lands as a table with a null bbox. A layer is anything the workspace holds. Nothing about the grain is spatial; geometry is the most common reason for using it, not the definition.

Concept, the row, and concept, the word

concepts is a table of a few dozen rows — pnpm state for the count. This document uses the word in its ordinary sense too. Where it means the table, it says concept row or names a value like population_count.

The vocabulary is load-bearing under concept-first retrieval, so it cannot grow freely: population_count, total_population and pop_est as three rows is a join that has stopped joining. Three things hold it together, and they are the same three that hold the source rows together. A row carries authored_by, reviewed_at and superseded_by, so a harvested row ranks in retrieval immediately and stays out of conflict and combination detection until a person promotes it. A row carries parent_id, so a households_with_piped_water joins at water_access's grain without waiting for that review — the conflict view pairs on coalesce(parent_id, id). And a concept row is never coined in the answer path: harvest matches against existing ids and coins nothing, a run may not create one at all, and a person adds rows in seed-concepts.ts and nowhere else. That file is meant to be a bottleneck. A subject area enters the registry when somebody decides it has.

A tag is a finding aid; the description is the claim

A source's concept rows say what it is filed under. Its description says what it measures. The two are written by different acts — tagging is a judgement made once, description is prose the author wrote about the thing — and they drift apart in one direction: a source acquires a capability, or the tagger's attention ran out, and the tags under-report.

That drift produces a specific failure, and it is worse than a stale tag sounds. nasa_geos_cf_v2 describes itself as producing "hourly surface concentrations of PM2.5, NO2 and other pollutants", carries a working connector, and was tagged pm25_concentration and nothing else. A run searching for NO2 ranked it first — retrieval reads the description — then read the tags, concluded nothing reachable measured NO2, and told a planner so, naming ESA as the holder. A missing tag does not produce a silence. It produces a confident gap, with a named holder and a route, which is the shape this system uses for truth.

Two things follow. The description reaches the model beside the tags, so the claim travels with the filing. And pnpm db:concepts:audit reads the two accounts against each other — each concept's own label and definition go in as a retrieval query, and a source that outranks a tagged row without being tagged is printed for a person. No keyword list per concept: that would be one more hand-maintained surface, which is the thing that failed.

The description is also the ranking surface

description is embedded, so it is doing two jobs at once: it states the claim a planner reads, and it is what separates this source from the others in the ranking. Those pull in different directions, and the second is the one that has been under-served.

Concept and place are exact. The concept rows are ids the model picks from a list it is shown; the country is a predicate over coverage_iso3 and coverage_geom. A description that spends its words restating either is spending them on axes already handled — and worse than merely wasted, because several sources measure one concept for one country and the words that would have told them apart went on saying which country it was.

So a description earns its place by discriminating: vintage, method, resolution, what the source is good at and where it breaks. Not "population of Ethiopia" but "modelled onto a 100 m grid from census totals and building footprints, reissued annually, so it is current and it is an estimate; totals will not match the census it was disaggregated from".

The place still belongs in the title, claimed_place and coverage_note, which is where the existing rule puts it. This is the same rule read from the other end.

A measurement carries how much of it there was

A statistic over an area is two numbers, and the system used to report one. The mean is the answer; the fraction of the area it was computed over is what says whether it is an answer about the area or about a corner of it.

This is not a raster detail. It is the same rule as evidence type: a modelled estimate and an official count are different claims wearing the same digits, and so are a mean over a whole subcity and a mean over 0.7% of one.

Measured, on one Landsat scene over Addis Ababa: the scene declared 21.8% cloud, the qa_pixel band removed 72% of the city window, and per subcity the surviving fraction ran from 0.7% to 48%. Without the mask the zonal means put one subcity at -10.5 °C, because a thermal band reports a cloud top as an apparent surface temperature near 250 K. With the mask but without the fractions, an answer reported all ten subcities "covered essentially completely" and called them "directly comparable". A wrong coverage figure is worse than none: it is the sentence the reader trusts before the number.

Which bits of a quality band mean "not ground" is a property of the product family, so it is a named scheme in code and a row on the endpoint — the same shape as an SDMX path form. A second collection with the same bit layout is a registry row, not a branch.

An artefact is what the code made of the bytes

A cached artefact is not the provider's file. It is a clipped window, scaled, masked, with a manifest entry and a profile beside it — a product of the connector, under the configuration the registry gave it.

The cache key has to say so, and for a while it did not: it carried the source, the version, the geometry, the period and the filters, all of which describe the request. Add a cloud mask to a connector and every cached raster keeps answering without it, with a summary, a provenance and a set of numbers that are internally consistent because the old code produced them together. A stale artefact is not a wrong answer wearing an error. It is a wrong answer wearing the right answer's provenance.

So the key carries two more things: sources.endpoint, which is the per-source half of the connector's configuration, and a hash of the connector code itself. The hash is computed rather than declared — a constant somebody bumps by hand is the same shape as the problem, because the person who forgets is the person who just changed what a connector writes.

The same key without the version is the request key, and a call reaches the cache through it first. A request answered before is served without asking the provider for a version: always for immutable_by_id, and for revision_checked within a day of a probe agreeing. A narrowing miss is stored under it too, because the list of what a versioned source holds is as fixed as its bytes. The code hash is in the request key, so a connector fix misses there as well. What it cannot see — a provider correcting a file in place, a registry field other than the endpoint — is cleared by hand with pnpm cache:purge --source.

How they nest

One dataset is reached through an endpoint, which speaks a protocol, for which there is one connector. The endpoint row is where a deployment's own oddities live — a path form, a quality-band layout, which credential it wants — so a fourth agency with the same deviation is a row and never a branch in the client.

The registry is memory, not a catalogue

A row is not a dataset we know about. It is something we have learned, which is why the columns that matter are profile, guidance and the observations behind it, and why a row that has never been called is a candidate rather than a source.

That is what find_sources being two-tier is for: the registry first, upstream on a miss. CKAN and STAC answer a query at question time, so a dataset earns a row the first time it is actually used rather than in a bulk import. The registry grows in the direction real questions pull it, and every row in it has been called at least once.

Bulk enumeration would fill the registry with rows nobody has tested, which is exactly what the observed-against-claimed distinction exists to expose. A pre-discovery pass to warm a new country is then an optimisation rather than a prerequisite.

Profile and guidance

Both describe a source. They have opposite lifecycles, and keeping them separable is the whole point.

The test between them: if refetching the source would regenerate it, it is profile. If it took a person or a failed run to learn it, it is guidance.

A profile that contradicts the previous one is information, not noise: it means the publisher changed a column, which is this registry's thesis arriving through a different door. Surface the disagreement rather than overwriting silently.

A profile says what the source is, not what one fetch found. The sandbox profiles the landed file, and half of that is the request: the clip's width and height, the features inside it, and the values found there. Stored whole, those read as facts about the source — WorldPop "was" 320×318 cells and Kontur's name_en "held" Addis Ababa, and a question about Jakarta would have been shown both. So storeProfile keeps only what does not change with the area asked about, and a clip of a new city is never a change. The values in a column are one tgis.load away once a layer has landed.

One shape per layer. A GTFS feed holds stops and routes; a boundary archive holds several admin levels; WorldPop comes at two resolutions. One profile per source meant the last fetch overwrote the others, and a run asking for stops was shown the routes' columns. The layer key is read from what landed — provenance.upstream's file, member, layer, variable, asset or resolution — not from the filter typed, and admin levels are normalised, so adm1, admin1 and eth_admin1.geojson are one layer.

What a fetch can ask for, and what a place contains

Two facts reach the model before it fetches, so a fetch is right the first time rather than after a refusal.

filters on each find_sources result. takes says what a key accepts, declared once per connector in tgis/filters.py — WorldPop's resolution is 1km or 100m, a CKAN dataset's resource is one of its files. options lists what the source itself published, read from the narrowing misses the cache already stores, so it fills in as sources are used. It is a view over stored rows, not a new store. The declarations live outside tgis/connectors/ because every file there is in the cache key, and pnpm tgis:card:check fails when a connector reads a key the file does not declare.

units_inside on resolve_place. The spine's units one level down, named up to twenty with the full count — Jakarta's five kota, Oromia's seventeen zones. A child covering 95% or more of its parent is the same ground under another level's name, as Addis Ababa's only ADM2 is, and is left out. Null means the spine holds nothing smaller, so units a question needs come from a boundary source.

vintage on resolve_place. The year the boundary represents, as geoBoundaries states it per country and level: 1995 to 2023, and once a range. A unit created after it is not in the spine — Ethiopia's ADM1 is 2016, with SNNPR and no Sidama — so a place that does not resolve may be younger than its boundary file. A plain year also fills the place's provenance.period.

The countries a fetch's area touches, for a source published one file per country. An endpoint with {iso3} in it is filled from the spine's ADM0 outlines within 0.01° of the clip box, so a box across a border reads both files: Namanga gave 4,397 Kenyan and 6,251 Tanzanian VIDA footprints. It is computed per fetch, from the geometry, for every kind of handle.

Guidance is a document, not a list of facts

One markdown document per source, revised by whoever learns something. Prose organises itself — related facts sit together, a caveat attaches to the sentence it qualifies — and a model revising a document deduplicates at write time, which an append-only list of items does not.

Structured columns per kind were the obvious alternative and they do not survive contact with real sources. HAPI needs a narrowing rule, WorldPop an operation-validity rule, Overture a connection-lifetime rule, ESA CCI an epoch rule; three field sets for table, raster and vector would have become four for NetCDF and five for paginated APIs. A document absorbs a new kind of fact without a migration, which is the property needed for dataset types not yet met. What keeps it affordable is retrieval: find_sources returns at most eight rows out of the whole registry, so guidance never has to be small in aggregate, only per row.

Review happens on a diff, not on a sentence. reviewed points at a version. Unreviewed edits render as provisional, the same way harvested source rows already do.

Two rules make it safe:

• Read at the start of a run, written at the end. A run never edits guidance that affects itself.
• Revise only what is contradicted or missing. The model has the current document in context, so it can tell.

And one that keeps the registry honest: a source's guidance describes that source and may not name another one. The moment a document says "for Ethiopian population questions, prefer HAPI", per-question routing has been written into the registry.

Two shapes of answer

The split is not spatial against tabular. It is one cell against a composition, and it is chosen per question, not per source.

The same source serves both. A connector that lands a layer may also offer a direct read; that is an added capability, not a fork in the architecture.

Population, households, vehicle ownership, employment, mode share, road length by class, fleet counts, crash counts: each is a number for an admin unit for a year, and each one used to pay sandbox boot, a Parquet write and model-written Python to read one cell. The sandbox earns its cost in the right-hand column and is overhead in the left, which is why cell: true on a fetch is a route rather than a shortcut.

Who writes what, and how fast it grows

Two rules follow from the last column of the grain table, and between them they decide where human judgement goes.

Everything curated should sit at a grain that grows slowly. Protocols, concept rows, holder routes. A holder route is decided once per country and read on every gap answer in that country, so an afternoon of research with Claude buys back a hundred answers.

Everything that grows fast is generated and marked provisional. Sources, profiles, endpoint enumerations. reviewed: false already renders as Provisional in the answer UI, so the data model already holds unreviewed rows honestly.

Create in the answer path, revise outside it. A run may insert a source row for a candidate it actually fetched, because that row is a record of something that happened. It may not revise an existing row, and it may not coin a concept row. profile is the one field a run overwrites, because refetching would regenerate it anyway; guidance is queued during a run and applied when the run closes.

Guidance is the only grain that is genuinely both, which is why it carries a version log and a review pointer rather than a single author.