Country data & seeding

How a registry gets its country's geography and code lists from Master Data, and the three kinds of data a registry install can create

New home: GitLab. The registry repositories are now developed at gitlab.com/openg2p/registry.

A registry's structure — its tables, fields, validation rules and reports — is built into its extension. Everything country-specific comes from the Master Data Service (MDS), which is seeded from a country pack.

This page covers what the registry does. For what a country pack is, where packs live, what P-codes are and how a country is configured end to end, see Country Data Architecture.

The registry has no country setting

A registry never names a country. It reads whatever the MDS beside it was seeded with, and adapts:

What the registry takes	From	Effect
The geo hierarchy	g2p_geo_levels, g2p_geo_level_values	Address dropdowns match the country's levels, however many there are
The code lists	g2p_attributes, g2p_attribute_values	Gender, education, water source… become the country's own
The sample people	g2p_sample_individuals, g2p_sample_households	Demo records that belong to the country

This is what lets one registry image serve any country. Changing country is a matter of pointing the registry at an MDS seeded with a different pack — there is no registry rebuild and no registry-side country value to keep in step.

Then why do the variant charts have expectedCountry?

It is a guard, not a selector. Setting analytics.bulkSample.expectedCountry: ETH does not make the registry Ethiopian; it makes the seed job fail loudly if the MDS it finds holds a different country than this deployment was set up for. Left empty (the default), no check is made. Nothing in the registry chooses the country — only MDS does.

Because MDS is read at install time and copied into the registry's own tables, MDS is not a runtime dependency: it being unavailable later does not stop a registration.

Three different kinds of data

These are routinely confused. They have different purposes, different volumes, and different switches.

	Sample data	Bulk data	Sanity data
Purpose	Demonstrations — records a person actually reads	Volume so reports and dashboards have something to show	Verifying the deploy works
How much	A few dozen people	Tens or hundreds of thousands	A handful of fixtures
Coherent with the country?	Yes — comes from the pack	No — invented names and numbers	No
Where defined	MDS (people) + the registry (its own fields)	Generated by the registry at install	Created by the sanity suite
Left behind afterwards	Yes	Yes	Yes — deliberately, for inspection
Switch	dbSeed.loadSampleData	analytics.bulkSample.enabled	sanity.enabled + sanity.runE2e

All three are for demonstration and testing. A production install should create none of them. See An empty install.

The platform ships machinery, not data

The reference registry is deliberately minimal. The platform's db-seed image carries the loaders and the ordering; it carries no sample people, no bulk generator and no country content.

In the platform image	What it is
entrypoint.sh	Runs the ordered steps, entirely from environment variables
load_attributes_from_mds.py	Copies the country's code lists from MDS
sync_geo_widgets.py	Rewrites geo dropdowns to match the country's levels
load_geo_data.py	Legacy slug-path geo loader — see the warning below
upload_templates.py	Jinja templates → MinIO

There is deliberately no load_sample_data.py and no bulk generator in the platform. The platform owns the hook and the order; each registry variant ships its own loader, because sample and bulk data write that variant's own tables. A bare platform install therefore has no sample data at all.

Bulk generators are not shared between registries for the same reason: a generator writes its registry's own extension tables and has to move in lockstep with that registry's reporting views. Each registry keeps its own copy.

What lives in the platform, and what does not

The platform is machinery only. Everything that produces or presents data lives in the individual registry:

	Registry Platform	NSR / Farmer Registry
Seeding machinery, hook order, switches	yes	inherited
Code-list SQL fixtures	reference registry's own	its own
Sample-data loader	none	its own
Sample content (the domain overlay)	none	its own
Bulk data generator	none	its own
Reporting views	the generator (generate_reporting_views.py)	its declaration (reporting.yaml) and any hand-written SQL
Superset dashboards	none	its own (bundle + import job)
analytics.* chart values	not defined	defined

So a bare platform install creates no sample data, no bulk data, no reporting views and no dashboards. Each registry defines all four, because each writes its own tables and its dashboards read its own views.

Code lists

Where they are defined

Every registry extension ships its code lists as SQL fixtures, applied on every install:

<extension>/meta_data/lookup-data/g2p_attributes.sql
<extension>/meta_data/lookup-data/g2p_attribute_values.sql

These are the registry's own defaults — programme names, livelihood categories, and so on. With no MDS involvement, these are what the registry validates against, and they are the same for every deployment of that registry.

How Master Data overrides them

With dbSeed.loadAttributes on, load_attributes_from_mds.py copies the country's lists from MDS after the fixtures have been applied. The copy is a per-list replacement, not a merge:

• For each code list MDS defines, the registry's existing values for that list are deleted and MDS's values inserted in their place.

• A list MDS does not define is left exactly as the extension shipped it.

Replacement is deliberate. Merging looks conservative and is worse: a country that deliberately omits a value would silently keep the extension's version of it, and the registry would accept a value the country does not recognise.

Values may also carry roles — semantic tags like head_of_household — so platform logic can ask for meaning rather than hardcoding a literal value that differs by country. The rationale is in Country Data Architecture.

How to use none of it

Leave dbSeed.loadAttributes off, which is the platform default. The registry then uses only its extension's own fixtures and never reads MDS attributes. Nothing breaks: an existing deployment that upgrades keeps the lists it has always had.

If loadAttributes is on but MDS holds no lists, the step logs that and moves on rather than failing the install.

A third switch decides whether the copy is enforced on writes:

Switch	Where	Default	Effect
geoSeed.load.codelists	openg2p-master-data chart	true	MDS loads the pack's lists
dbSeed.loadAttributes	registry chart	false	Registry copies them into its own tables
registry_core_validate_attribute_values	registry API config	false	Registry rejects writes with values not in its copy

The seeding switches

All of these live under registry.dbSeed.* in a variant chart (dbSeed.* in the platform chart itself).

Switch	Platform default	What it does
enabled	true	Run the db-seed Job at all. Meta-data SQL is applied unconditionally when on
loadAttributes	false	Copy the country's code lists from MDS
attributeDomains	[]	Extra domain lists, e.g. ["agriculture"] for a Farmer Registry
syncGeoWidgets	false	Match the geo dropdowns to the country's levels
loadSampleData	false	Load the demo people
loadImages	false	Sample profile photos — requires loadSampleData
loadTemplates	true	Jinja templates → MinIO
loadGeoData	false	Legacy — see below

loadGeoData must stay off. It loads a five-level country → village hierarchy from a CSV keyed by slug-paths. MDS now seeds its own geography from a country pack keyed by P-code, so enabling this writes a second hierarchy over the first, in a different id space and at a fixed depth that only ever described one country. It is kept only for a deployment that has no country pack at all.

These are registry.dbSeed.*, not global.*. The db-seed Job reads .Values.dbSeed.load* and there is no global fallback, so setting them under global silently does nothing.

syncGeoWidgets — why it matters

An extension's UI metadata names its geo dropdowns and fixes how many there are. A country whose pack disagrees — four levels where the metadata expects five — gets dropdowns that silently return nothing, with no error anywhere. Syncing rewrites those widgets from the hierarchy MDS actually holds.

Configuring from Rancher

The variant charts (NSR, Farmer Registry) carry no questions.yaml of their own. Rancher only reads questions.yaml from the root of the chart being installed, never from a subchart, so a wrapper chart would otherwise show an empty form. Instead the file is generated at package time from the platform chart's questions, by inherit-questions.sh in the packaging repository, which:

• prefixes the paths — dbSeed.loadSampleData becomes registry.dbSeed.loadSampleData, while global.* passes through unchanged;

• backfills defaults from the wrapper's own values.yaml, falling back to the platform's. Without this a field would render blank, and a blank boolean submits as false — silently disabling db-seed on install.

So in the Rancher form these appear under the DB Seed group:

Form field	Variable
Enable DB Seed	registry.dbSeed.enabled
Load Sample Data	registry.dbSeed.loadSampleData
Load Code Lists from Master Data	registry.dbSeed.loadAttributes
Domain Code Lists	registry.dbSeed.attributeDomains
Match Geo Dropdowns to Country	registry.dbSeed.syncGeoWidgets
Load Sample Images	registry.dbSeed.loadImages
Load Templates to MinIO	registry.dbSeed.loadTemplates
Load Legacy Geo Data (deprecated)	registry.dbSeed.loadGeoData

and under Sanity: registry.sanity.enabled, registry.sanity.runE2e, registry.sanity.failOnError.

Bulk data cannot currently be turned off from the Rancher form. The generated questions come only from the platform chart, and analytics.* is a variant-level key that the platform does not define. To disable bulk generation you must set analytics.bulkSample.enabled: false in values — for example in the YAML editor Rancher offers alongside the form.

An empty install, for production

A production registry should carry no demonstration data of any kind:

registry:
  dbSeed:
    enabled: true          # meta-data SQL is REQUIRED
    loadSampleData: false
    loadImages: false
    loadGeoData: false
    loadTemplates: true    # keep — see below
    loadAttributes: true   # keep if this deployment uses the country's code lists
    syncGeoWidgets: true   # keep — dropdowns must match the country
  sanity:
    enabled: true          # smoke only; creates nothing
    runE2e: false          # e2e SEEDS fixtures and leaves them behind

analytics:
  bulkSample:
    enabled: false

Do not turn off dbSeed.enabled or loadTemplates. The meta-data SQL defines the register itself — without it there is no registry. Without the Jinja templates in MinIO every record fails to render and a DCI search returns an empty result.

sanity.enabled: true is safe in production: with runE2e: false the suite is smoke-only — it checks that the Partner API is live and the DCI route is served, and creates nothing. It is runE2e: true that seeds fixtures, and those are never cleaned up.

The reporting layer

Seeding puts data in; reporting is how it comes back out, and the platform ships the machinery for that too.

generate_reporting_views.py lives in this platform's db-seed image, beside load_attributes_from_mds.py and load_sample_data.py, and follows the same principle: the platform supplies the mechanism, the registry supplies what is specific to it.

At install it reads the registry's own schema and Master Data's country pack, and creates a view per entity — geography inherited from the parent, workflow columns carried, personal data withheld and then verified absent. A registry that declares nothing still gets a complete reporting layer.

What each registry supplies is a short reporting.yaml: which entity hangs off which, what its columns mean, and which views it maintains by hand.

Why generated rather than written. Each registry used to hand-write its reporting SQL, so coverage was whatever somebody thought of, and the geographic unpacking — identical in intent everywhere — had already drifted between two registries. See Reporting views.

Everything about the reporting layer — the declaration, dashboards, the map drill-down, and what to do when a country pack or a schema changes — is under Reporting & Analytics.

What runs, in what order

The chart runs db-seed as a post-install,post-upgrade hook Job:

#	Step	Controlled by
1	meta-data SQL → registry DB — register definitions, UI metadata, code-list fixtures	always, when enabled
2	code lists from MDS → registry tables	loadAttributes
3	geo widgets rewritten from the MDS hierarchy	syncGeoWidgets
4	sample data → g2p_register_*	loadSampleData
5	images → MinIO	loadImages
6	templates → MinIO	loadTemplates
7	AWE seed → AWE DB	aweDbSeed

Code lists load before sample data on purpose, so seeded records reference values that exist.

Bulk generation and the sanity suite are separate Jobs at later hook weights, so a failure in one does not silently skip the others — but note that Helm stops at a failed hook, so a failing seed step does block everything after it.

See also Meta Data Seeding for the platform-level metadata framework.