Ctrlk
For the complete documentation index, see llms.txt. This page is also available as Markdown.

Data seeding

How the Farmer Registry is seeded at install — the seed content this repo owns and the seeding machinery it inherits from the platform.

New home: GitLab. farmer-registry is now developed at github.com/OpenG2P/farmer-registry.

Seeding follows the same split as everything else: the machinery is inherited from the platform's db-seed image, and the Farmer Registry supplies only its content.

The switches, the code-list mechanism, the Rancher form and the production ("empty") install are all platform-level and documented once in Country data & seeding. For what a country pack is, see Country Data Architecture. This page covers only what is specific to the Farmer Registry.

Three kinds of data the Farmer Registry can create

Sample data
Bulk data
Sanity data

How much

A few dozen people from the country pack, plus the farmer overlay below

100,000 farmers by default

A handful of fixtures

Purpose

Demonstrations — records a person reads

Volume for reports and dashboards

Verifying the deploy

Switch

registry.dbSeed.loadSampleData

analytics.bulkSample.enabled

registry.sanity.runE2e

Default here

true

true

false

Sample data

Where the people come from

The individuals and households come from Master Data, which holds the country pack's sample people (g2p_sample_individuals, g2p_sample_households), complete with their geo ancestry — so every record points at a real administrative unit in whatever country the deployment carries.

The Farmer Registry then augments each person with its own agricultural fields:

Farmer overlay
Rows

crops.json

757

lands.json

509

farmers.json

500

livestocks.json

471

farm_inputs.json

434

household_members.json

428

membership_details.json

122

scores.json

100

These live in docker/db-seed/seed-data and are linked to whoever was actually loaded, rather than to a fixed set of ids.

If Master Data holds no samples, the loader falls back to the shared demography CSV baked into the image and says so in its log. That CSV can only ever describe one country — it has five fixed level names — so the fallback produces people whose addresses do not match the deployment's country pack. Enable geoSeed.load.samples on the Master Data chart to get a pack-coherent set.

Code lists — including the agriculture domain

A Farmer Registry needs the country's agricultural vocabularies (crop types, livestock) on top of the core lists a social registry uses. Those live in a domain subtree of the country pack, and are opted into explicitly:

Leaving attributeDomains empty loads only the core lists. A country pack that carries no agriculture domain simply has none to load — the step logs it and continues.

Bulk data

The Farmer Registry generates 100,000 farmers at install so that reports and dashboards have something to show. Nobody reads an individual bulk row, so the names and phone numbers are invented and need not belong to the country.

Bulk generation still reads Master Data — it is not country-independent.

It is the people that are country-agnostic, not the geography. Every generated record must point at a real administrative unit or maps and drill-downs break, so the generator reads the hierarchy from the MDS database (g2p_geo_levels / g2p_geo_level_values) and fails if it is not there.

What this buys is that one generator serves every country — the same build generates for Ethiopia or Kamuntu unchanged. It does not mean MDS is optional.

analytics.bulkSample.expectedCountry is a guard, not a selector: set it and the job fails if MDS holds a different country than expected. Empty (the default) means no check.

Turning it off

This one is not in the Rancher form — the generated questions come from the platform chart, and analytics.* is the Farmer Registry's own key. Set it in the YAML editor alongside the form.

Sanity data

Completely separate from sample and bulk data, and created for a different reason: to prove the deploy actually works.

With sanity.runE2e: false (the default) the suite is smoke-only and creates nothing. With it on, a deploy-time job seeds the e2e's own fixtures — a sanity farmer (SANITY-FARMER-0001) injected by SQL rather than reused from the sample data, the suite's own Keycloak user with a non-temporary password, and that user as an approver on the change-request policy.

Sanity fixtures are never deleted. They are left in place for inspection after the run, so an environment that has ever had runE2e: true carries them permanently. Keep it off for production.

Reporting views

Dashboards and the map never read the register tables directly — they read reporting views, fr_rpt_*, one per entity, with geography and workflow columns and personal data withheld.

Most of them are generated, at install, by the platform's generate_reporting_views.py reading this registry's schema and the country's hierarchy from Master Data. This registry supplies two things:

  • reporting_views.sql — the views it maintains by hand: fr_rpt_farmer and fr_rpt_land, because they normalise a free-text land size into hectares, roll the parcels, crops, livestock and inputs up onto the farmer, and define what this registry means by "uses modern inputs" and where its age bands fall. Nothing can infer those from a schema.

  • reporting.yaml — a short declaration: which entity hangs off which, which views are hand-written, and what its own columns mean. (this registry's copy)

Everything else is generated: crop, livestock, farm inputs, membership, scores, households, household members, change requests and record history.

Who runs it: a job of this chart, <release>-fr-reporting-views, at hook weight 45 — after bulk data, so the first build has rows behind it. It runs the hand-written SQL first and the generator second, because a generated child reads its parent's columns.

Switched by analytics.reportingViews.enabled; generation alone by analytics.reportingViews.generate.

Why this changed. The reporting SQL used to be hand-written in full, here, so coverage was whatever somebody had thought of — livestock — the register holds 132,824 livestock records, and before these views the only thing reporting could say about them was whether a farmer kept any. Generation makes coverage structural instead. The reasoning, the declaration reference and the --discover workflow are in Reporting views.

Keeping them current

Some views are materialized, and Postgres never updates a materialized view when its base tables change. This chart therefore refreshes its own, on a schedule:

<release>-fr-reporting-views-refresh rebuilds every fr_rpt_* materialized view in dependency order resolved from the catalog, using REFRESH MATERIALIZED VIEW CONCURRENTLY so dashboards keep reading the previous snapshot while it runs. The cadence is on the Rancher form under Analytics.

Between refreshes, farmers registered since the last run do not appear in any report. Choose the interval accordingly.

Install sequence

Everything below runs as Helm post-install / post-upgrade hooks, in hook-weight order. Helm waits for each weight to succeed before creating the next, so this is a strict sequence — and a failure at any step blocks everything after it.

Read the diagram top to bottom: each box only starts once the one above it has succeeded. The two dotted arrows are the dependencies Helm cannot order, because they belong to other releases — see Cross-release dependencies.

Weight
Job
What it does
Runs when

—

(application pods)

The registry's own Deployments start and pass their probes

always

10

fr-db-seed

Meta-data SQL, code lists from Master Data, geo-widget sync, sample data, images, templates

dbSeed.enabled

11

fr-sanity-pm-seed

Registers the sanity partner in Partner Management

sanity.runE2e

12

fr-sanity-cm-seed

Consent Manager binding and policy for that partner

sanity.runE2e

13

fr-sanity-data-seed

Sanity fixtures — the test record, Keycloak user, approver rule

sanity.runE2e

20

fr-iam-register

Registers the registry's roles and permissions in IAM

always

25

fr-sanity

Runs the sanity suite

sanity.enabled

40

fr-fr-bulk-sample

Generates 100,000 farmers

analytics.bulkSample.enabled

45

fr-fr-reporting-views

Creates the reporting views the dashboards read

analytics.reportingViews.enabled

50

fr-fr-dashboards

Imports the dashboards into Superset

analytics.dashboards.enabled

The dashboards are loaded by this registry, not by the analytics platform. The dashboard bundle ships in this repository, and the weight-50 job imports it into the environment's Superset, points its database connection at this registry, publishes the dashboards and enables embedding on them.

The ordering is the point: dashboards are imported after bulk data and after the reporting views exist, so that a dashboard opened straight after install already has data behind it rather than rendering empty.

Cross-release dependencies

Two of these steps depend on releases Helm cannot order against, because they are separate installs:

  • Master Data must be seeded before weight 10 (code lists, geo, sample people) and before weight 40 (the bulk generator reads its geo hierarchy). The db-seed and bulk jobs each wait for it and fail with a clear message rather than producing records that point nowhere.

  • Superset must be reachable before weight 50. That job waits for it rather than failing immediately, so a Superset that is merely restarting does not lose the dashboard import.

Because Helm stops at a failed hook, a failure in bulk generation (weight 40) also prevents the reporting views and the dashboard import from ever running. If Superset has no dashboards after an install, check the earlier jobs first.

What the image contains

The Farmer Registry's db-seed image is a thin FROM of the platform's, clearing the reference registry's content and copying its own — the extension's meta_data/, awe_meta_data/ and templates/, the seed-data/*.json overlay above, generate_fr_bulk_sample.py and reporting_views.sql.

The bulk generator lives here, beside the reporting views it must move in step with, rather than in a shared repository.

Chart defaults

For a production install, see An empty install.

Beyond seeding. What is built on these views — the dashboards, the map drill-down, how to add your own, and what to do when a country pack or this registry's schema changes — is under Reporting & Analytics. Start with Setting up reporting if you are bringing this registry up for a country.

PreviousHelm chart
NextSanity testing

Last updated

Was this helpful?