TGIS v.0
New Chat

TGIS evolution plan

Where the system goes after v0, and what each move costs.

Active plan. The only forward-looking document still maintained. What the system does today is in the model and the architecture; what is queued is NEXT.md. The short version for people is Up next on the site. Almost nothing on this page is built, and where something is, it says so.

Rewritten 30 August 2026. The ten packages this document used to sequence are built. The measurements that argued for them, and the alternatives they beat, moved into the two documents that describe what those packages produced — a rejected alternative is worth keeping next to the thing it explains, not in a plan for work that is finished. The packages themselves are in git log.

What we are building

A planner, an analyst or a budget holder asks a question about a city. The system assembles an answer from data they would not otherwise have reached: geospatial, tabular and eventually unstructured. Every figure in it is traceable, understandable and honest about what it is. What cannot be reached is named, with the institution that holds it.

v0 tests that for any place, with sources worldwide; city-level sources are deepest in Addis Ababa, where discovery started. It was built around one corridor question, which widened to flood exposure on 13 October 2026. Everything below is the distance from there.

The scope this is built for

Settled 29 August 2026, widened to every country on 20 October: transport questions, worldwide, both shapes of answer. Focused questions — a number for a place and a period — and broader ones that compose several sources over a boundary. Both shapes are a requirement rather than a guess about what planners ask, which is why both paths exist rather than one.

Three consequences carry into everything below.

  • Every country in the spine; sources by use. The place spine holds every country geoBoundaries publishes, ADM0–ADM2, so a place name anywhere resolves to a unit before any data is looked for. Sources do not follow: the corpus grows with datasets as fast as with countries, and a country with no sources answers with gaps and named holders.
  • Transport is the sector. Protocols and endpoints are ranked by countries × transport concepts unlocked, not by breadth in general.
  • The registry grows by use. Pre-warming a new country is an optimisation, not a prerequisite.

How the system grows

Three ways a question that cannot be answered today becomes one that can. They are different acts with different costs, and conflating them is how a catalogue ends up being maintained by hand.

A new dataset — no code, and often no person

Built, and the cheapest of the three. find_sources returns candidates beside the rows: datasets an endpoint says it has and nothing has ever called. A candidate carries no profile and no coverage record, and that absence is the point — it is a claim, not evidence. Fetching one by its candidate_id lands it, and the row it becomes carries whatever the fetch found.

The person's job here is review, not entry. A row a run created is authored_by: 'agent' and reviewed_at: null, it ranks in retrieval immediately, and it renders as provisional until somebody reads it.

A new connector — a discovery session, then a row

The expensive one. Not because of the typing — most of the reading, the probing and the first draft happen with Claude — but because somebody has to decide it is worth having and stand behind what it claims. So it is scored before it is written.

Protocols are the unit of coverage. Count protocols connected, not datasets registered. One SDMX connector reaches the World Bank, Eurostat, UNICEF, the ILO, the OECD and most national statistical offices: hundreds of indicators, every country, over an HTTP GET. One CKAN connector reaches every portal running CKAN. So does Socrata. So does STAC for the raster catalogues. Roughly six protocols cover both shapes of answer globally — SDMX, CKAN, Socrata, STAC, Overture and a boundary spine — against nine HDX dataset rows for one city.

That framing has already been paid for once. Registering datasets one at a time out of a city scan is what produced the zero-percent transfer rate to a second city, and the corpus has still not moved much: nine connectors now rather than two, but ckan ×12 and planetary_computer ×8 are most of what is reachable.

And stop chasing the hard ones. The Addis commission with no public DNS and the ministries that resolve and time out stay as holder routes. That is the gap rail, already the part of the product nobody else has, and a route is written once per country rather than once per question. Engineering effort goes into protocols; institutional research goes into routes.

Once a connector is worth writing, the session that establishes what it must handle has an order, and it is the order because reversing it has cost real time.

The five steps of a data discovery session, in order, and what skipping each one has cost: read the documentation, call it, check what came back, write the finding to the registry, re-measure. A data discovery session, in the order that costs least Each step is here because skipping it has cost real time. The right column is what it cost. 1 Read the provider’s documentation A session that probed first spent an hour concluding two providers were broken. Both documents said otherwise. 2 Call it What a catalogue claims is not what a deployment does. 3 Check what came back A 200 is not a success. A well-formed empty is a false miss wearing a real one’s clothes. 4 Write the finding to the registry A deployment that addresses itself unusually gets a row. Never a branch in the client. 5 Re-measure Budgets read the profile. Nothing here is a flag somebody set. The test for where a finding goes: would a fourth agency with the same deviation be a row, or a diff?

Path forms, quality-band layouts and reference values are shapes, and they live in the client as named sets any deployment can claim. Which shape a given deployment takes is a row. If adding the fourth provider means editing the client, the finding was written in the wrong place.

A new data type — the one that is not built

The schema has carried modality: text from the start and six rows already use it: Addis Ababa's transport master plans, the BRT corridor designs, the zoning regulation, the air quality management plan. Every one of them is a gap with a holder, because nothing can read a document.

They are also the sources that answer the half of a corridor question the geospatial half cannot. Where the population is and where the stops are does not tell a planner that a corridor is already committed on that alignment, at what cost, with whose money. That is in a PDF.

A document is a layer. kind gains a fourth value beside raster, vector and table. Its profile is what a fetch can see without reading it — page count, whether there is a text layer or only scans, the tables detected, the section headings. The same rule as every other profile: if refetching regenerates it, it is profile.

A figure from a document is a citation, not a computation, and this is the part that needs care. Every other Value in this system is minted by code that did arithmetic. A number in a master plan is minted by locating it: the extraction is code, the choice of passage is the model's, and the provenance has to carry the page, the character span and the verbatim sentence so a planner can check the number against the words around it. A figure whose provenance is a quote is a different kind of claim from one whose provenance is a zonal sum, and the card has to say which.

evidence_type already has the vocabulary for it. stated_intention — what someone said they would do — exists precisely for this and is currently unreachable. A committed corridor and a measured passenger count are not the same kind of evidence, which is the same argument that separates an official count from a modelled estimate.

Three things to settle before building it:

  • Retrieval inside a document. A hundred-page plan does not fit in context. Chunk-and-embed is the obvious answer and it puts a second retrieval system in the loop, with its own evaluation.
  • A quote is not a fact. A document says a corridor is planned; whether it was built is a different question, and the answer must not let a stated intention harden into an outcome as it travels.
  • Scanned pages. OCR is a modelled reading of an image, so a figure from one is two inferences deep. Label it as such, or refuse it.

The rest of what is unbuilt

Smaller than the three above, and none of it is blocked on anything.

Import the concept vocabulary rather than inventing it. The rows are hand-written today and the file is deliberately the bottleneck — a run may not coin one, and harvest matches against existing ids. That holds for governance and does not scale to every sector. SDMX publishes concept schemes and code lists that are already curated, already defined, already hierarchical, and already mapped to the APIs being connected first. The existing rows stay as the spine; parent_id is in the schema and is what lets an imported households_with_piped_water join at water_access's grain without waiting for review.

Live upstream search on the protocols that have it. CKAN and STAC answer a query at question time; SDMX does not, and costs a whole catalogue to try. Eurostat and the OECD are harvest-only by measurement because of it. Each further protocol connected decides for itself which of the two it is, and the endpoint row records the answer.

Discovery as a sub-agent. A live catalogue search returns two hundred results with long descriptions, and the main agent needs three source ids and one line on why. The reasoning, and the rule that bounds it, are in the architecture.

The lines

Four invariants. Every change has to pass all of them.

Per-source facts, yes. Per-question routing, never. "Column population holds the count, narrow by admin_level" is a fact about the data. "For population questions in Ethiopia, use HAPI" is a decision, and it is the same mistake as a hardcoded top three in resolve_place. Every step removed must come from removing an unknown, never from removing a choice.

Code mints figures. The model never types one. Which is why there is no http_get tool, and why a direct read is a connector method rather than a shortcut. See the invariant.

Sub-agents read. The model decides. Delegate work whose output is a shortlist or a digest. Never delegate a figure, a choice between sources, or a judgement about disagreement.

Read at the start, write at the end. Guidance is read when a run begins and revised when it ends, so no run edits the knowledge steering itself. A run may create a source row for something it actually fetched. It may never revise one, and it may never coin a concept row. The single exception is profile, which is generated rather than judged: the fetch that produced the object writes it, mid-run, and that is the only update the answer path makes.

The closest call in this design is a connector running zonal_sum because guidance named it. The guard is that the operation is visible in find_sources output and overridable by the model, not something that happens invisibly.

Open questions

How much of the retrieval should the model write itself?

The connectors are accumulating small functions that exist because a publisher did something ordinary and unanticipated. A CSV encoded cp1252 rather than utf-8. A resource named metadata that a filter for data also matches. An archive whose member is 18.7 GB. A level called adm1 in one file and admin1 in another. Each of these was a real failure, each fix is four lines, and each one is a thing a person had to notice.

A model writing its own read in the sandbox would fix most of them without being told: it can see the traceback, it can try a different encoding, it can list the archive and pick again. That is self-repair, and the cost of the alternative is this file growing a function per publisher quirk forever.

It runs straight into the second line. "Code mints figures, the model never types one" is why there is no http_get, and a model that fetches and parses is one step from a number that reached an answer because a model computed it in a scratch cell. The line exists because that substitution is the failure the whole system is built to prevent.

The distinction worth testing is between reaching and reading. A model that may not choose what to fetch, but may write the code that turns bytes it was handed into a table, keeps provenance intact — the connector still decided what was retrieved and still records where it came from — while moving the encoding-and-parsing class of problem to something that repairs itself. Whether that distinction survives contact with a real run is the question; a model given a parse loop will eventually want one more byte than it was handed.

Not scheduled, and not a small change. Worth deciding before the connector directory doubles again.

Is harvest-then-review acceptable to the people testing it, given every harvested row shows as Provisional until a person reviews it? A research-design question, not an engineering one. Not on the critical path: pre-warming the registry is an optimisation, and growth by use does not need one.

Does live upstream search belong in the planner's latency budget? A CKAN search is fast; an SDMX structure query is not. If it costs seconds on a miss, it may belong in a curation pod rather than in the run.

Does discovery-as-subagent pay? It buys context and costs a model round trip. Measure it against the two-hundred-result bloat rather than assuming; the budget assertions in pnpm agent:dry exist for exactly this kind of question.

How much unstructured data, and when? Six rows already carry modality: text and every one is a gap. The row shape and the evidence vocabulary carry over, but the claim that it needs no new machinery did not survive being thought through: a document is a fourth layer kind, and a figure read out of one is a citation rather than a computation, which is a provenance shape this system does not have. Nothing here blocks it and nothing here schedules it.