> For the complete documentation index, see [llms.txt](https://docs.openg2p.org/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://docs.openg2p.org/products/registry/farmer-registry/deployment/data-seeding.md). # Data seeding How the Farmer Registry is seeded at install — the seed content this repo owns and the seeding machinery it inherits from the platform. {% hint style="info" %} **New home: GitLab.** **`farmer-registry`** is now developed at [github.com/OpenG2P/farmer-registry](https://github.com/OpenG2P/farmer-registry). {% endhint %} Seeding follows the same split as everything else: the **machinery** is inherited from the platform's `db-seed` image, and the Farmer Registry supplies only its **content**. {% hint style="info" %} The switches, the code-list mechanism, the Rancher form and the production ("empty") install are all platform-level and documented once in [**Country data & seeding**](/products/registry/registry/deployment-and-extension/country-data-and-seeding.md). For what a country pack is, see [**Country Data Architecture**](/platform/country-data-architecture.md). This page covers only what is specific to the Farmer Registry. {% endhint %} ## Three kinds of data the Farmer Registry can create | | Sample data | Bulk data | Sanity data | | ---------------- | ----------------------------------------------------------------------- | ---------------------------------------- | -------------------------------- | | **How much** | A few dozen people from the country pack, plus the farmer overlay below | **100,000 farmers** when on (adjustable) | A handful of fixtures | | **Purpose** | Demonstrations — records a person reads | Volume for reports and dashboards | Verifying the deploy | | **Switch** | `registry.dbSeed.loadSampleData` | `analytics.bulkSample.enabled` | `registry.sanity.runE2e` | | **Default here** | `false` | `false` | `false` | | **Rancher form** | DB Seed → *Load Sample Data* (+ *Load Sample Images*) | Analytics → *Generate bulk demo data* | Sanity → *Sanity - Run Full E2E* | ## Sample data ### Where the people come from The individuals and households come from **Master Data**, which holds the country pack's sample people (`g2p_sample_individuals`, `g2p_sample_households`), complete with their geo ancestry — so every record points at a real administrative unit in whatever country the deployment carries. The Farmer Registry then **augments** each person with its own agricultural fields: | Farmer overlay | Rows | | ------------------------- | ---- | | `crops.json` | 757 | | `lands.json` | 509 | | `farmers.json` | 500 | | `livestocks.json` | 471 | | `farm_inputs.json` | 434 | | `household_members.json` | 428 | | `membership_details.json` | 122 | | `scores.json` | 100 | These live in [`docker/db-seed/seed-data`](https://github.com/OpenG2P/farmer-registry/tree/develop/docker/db-seed/seed-data) and are linked to whoever was actually loaded, rather than to a fixed set of ids. {% hint style="warning" %} **If Master Data holds no samples, the loader falls back** to the shared demography CSV baked into the image and says so in its log. That CSV can only ever describe one country — it has five fixed level names — so the fallback produces people whose addresses do not match the deployment's country pack. Enable `geoSeed.load.samples` on the Master Data chart to get a pack-coherent set. {% endhint %} ## Code lists — including the agriculture domain Nothing to seed here: the Farmer Registry's coded dropdowns read their options **live from Master Data**, and the registry keeps no copy. Six farmer fields are bound to Master Data lists this way: | Field | Master Data list | Where it comes from | | --------------------- | --------------------- | ----------------------------------- | | `gender` | `GENDER` | the pack's core lists | | `marital_status` | `MARITAL_STATUS` | the pack's core lists | | `education_level` | `EDUCATION_LEVEL` | the pack's core lists | | `disability_type` | `DISABILITY_DOMAIN` | the pack's core lists | | `disability_severity` | `DISABILITY_SEVERITY` | the pack's core lists | | `source_of_income` | `SOURCE_OF_INCOME` | the pack's **`agriculture` domain** | The other dropdowns — crops, livestock, land use and the like — still carry fixed options in the section metadata. `SOURCE_OF_INCOME` lives in a domain subtree of the pack, which Master Data loads only when asked — on the **Master Data** chart, not the registry's: ```yaml masterData: # under commons-services geoSeed: domains: - agriculture ``` {% hint style="warning" %} **The domain must exist in the pack.** The ETH pack carries `agriculture`; naming a domain the pack lacks **fails the Master Data seed Job**. The fictitious XKM pack carries no code lists at all — on XKM leave `domains` empty, and expect all six coded dropdowns above to be empty. {% endhint %} Because the codes come from Master Data, everything the Farmer Registry writes into these fields uses them too: the sample loader maps its seed JSON onto them (e.g. income becomes `SOI_CROP_PRODUCTION`, gender `OTHER`), and the bulk generator and the inbound DCI template emit them directly. ## Bulk data When switched on, the Farmer Registry generates **100,000 farmers** (adjustable) at install so that load tests have volume and the dashboards have something to show. It is **off by default**, and it is independent of the reporting views and dashboards — those install either way, over whatever data the register holds. Nobody reads an individual bulk row, so the names and phone numbers are invented and need not belong to the country. {% hint style="warning" %} **Bulk generation still reads Master Data — it is not country-independent.** It is the *people* that are country-agnostic, not the *geography*. Every generated record must point at a real administrative unit or maps and drill-downs break, so the generator reads the hierarchy from the MDS API (through the platform's `mds_client.py`) and fails if it is not there. No Farmer Registry job reads or writes MDS's database. What this buys is that **one generator serves every country** — the same build generates for Ethiopia or Kamuntu unchanged. It does not mean MDS is optional. {% endhint %} `analytics.bulkSample.expectedCountry` is a **guard, not a selector**: set it and the job fails if MDS holds a different country than expected. Empty (the default) means no check. ### Turning it on In the Rancher form, under **Analytics**: tick **Generate bulk demo data**, and set **Bulk demo farmers** if 100,000 is not the size you want. In values: ```yaml analytics: bulkSample: enabled: true farmers: 100000 ``` ## Sanity data Completely separate from sample and bulk data, and created for a different reason: to prove the deploy actually works. With `sanity.runE2e: false` (the default) the suite is smoke-only and **creates nothing**. With it on, a deploy-time job seeds the e2e's *own* fixtures — a sanity farmer (`SANITY-FARMER-0001`) injected by SQL rather than reused from the sample data, the suite's own Keycloak user with a non-temporary password, and that user as an approver on the change-request policy. {% hint style="danger" %} **Sanity fixtures are never deleted.** They are left in place for inspection after the run, so an environment that has ever had `runE2e: true` carries them permanently. Keep it off for production. {% endhint %} ## Reporting views Dashboards and the map never read the register tables directly — they read **reporting views**, `fr_rpt_*`, one per entity, with geography and workflow columns and personal data withheld. **Most of them are generated**, at install, by the platform's `generate_reporting_views.py` reading this registry's schema and the country's hierarchy from Master Data. This registry supplies two things: * **`reporting_views.sql`** — the views it maintains by hand: `fr_rpt_farmer` and `fr_rpt_land`, because they normalise a free-text land size into hectares, roll the parcels, crops, livestock and inputs up onto the farmer, and define what this registry means by "uses modern inputs" and where its age bands fall. Nothing can infer those from a schema. * **`reporting.yaml`** — a short declaration: which entity hangs off which, which views are hand-written, and what its own columns mean. ([this registry's copy](https://github.com/OpenG2P/farmer-registry/blob/develop/docker/db-seed/reporting.yaml)) Everything else is generated: crop, livestock, farm inputs, membership, scores, households, household members, change requests and record history. **Who runs it:** a job of this chart, `-fr-reporting-views`, at hook weight 45 — after bulk data, so the first build has rows behind it. It runs the hand-written SQL first and the generator second, because a generated child reads its parent's columns. Switched by `analytics.reportingViews.enabled`; generation alone by `analytics.reportingViews.generate`. {% hint style="info" %} **Why this changed.** The reporting SQL used to be hand-written in full, here, so coverage was whatever somebody had thought of — livestock — the register holds 132,824 livestock records, and before these views the only thing reporting could say about them was whether a farmer kept any. Generation makes coverage structural instead. The reasoning, the declaration reference and the `--discover` workflow are in [Reporting views](/platform/platform-services/reporting-and-analytics/reporting-views.md). {% endhint %} ### Keeping them current Some views are materialized, and **Postgres never updates a materialized view when its base tables change**. This chart therefore refreshes its own, on a schedule: ```yaml analytics: reportingViews: refreshSchedule: "0 * * * *" # hourly; empty disables the CronJob ``` `-fr-reporting-views-refresh` rebuilds every `fr_rpt_*` materialized view in dependency order resolved from the catalog, using `REFRESH MATERIALIZED VIEW CONCURRENTLY` so dashboards keep reading the previous snapshot while it runs. The cadence is on the Rancher form under **Analytics**. Between refreshes, farmers registered since the last run do not appear in any report. Choose the interval accordingly. ## Install sequence Everything below runs as Helm **post-install / post-upgrade hooks**, in hook-weight order. Helm waits for each weight to succeed before creating the next, so this is a strict sequence — and **a failure at any step blocks everything after it**. ```mermaid flowchart TD A["Application pods
Deployments pass their probes"] --> B B["10 · db-seed
meta-data
sample data · images · templates"] --> C subgraph S ["Sanity fixtures & test — only when sanity.runE2e = true"] direction TB C["11 · sanity-pm-seed
partner in Partner Management"] --> D D["12 · sanity-cm-seed
Consent Manager binding + policy"] --> E E["13 · sanity-data-seed
test record · Keycloak user · approver rule
never deleted afterwards"] end E --> F["20 · iam-register
roles + permissions into IAM"] F --> G["25 · sanity
smoke, plus the e2e when runE2e = true"] subgraph AN ["Analytics — only when analytics.* enabled"] direction TB H["40 · bulk-sample
100,000 farmers
off by default"] --> I I["45 · reporting-views
views the dashboards read"] --> J J["50 · dashboards
import into Superset"] end G --> H MD(["Master Data
separate release"]) -.->|"must be seeded first"| B MD -.->|"geo hierarchy"| H SS(["Superset
separate release"]) -.->|"must be reachable"| J ``` Read the diagram top to bottom: each box only starts once the one above it has succeeded. The two dotted arrows are the dependencies Helm cannot order, because they belong to other releases — see [Cross-release dependencies](#cross-release-dependencies). | Weight | Job | What it does | Runs when | | ------ | ----------------------- | ------------------------------------------------------------------- | ---------------------------------- | | — | *(application pods)* | The registry's own Deployments start and pass their probes | always | | 10 | `fr-db-seed` | Meta-data SQL, **sample data**, images, templates | `dbSeed.enabled` | | 11 | `fr-sanity-pm-seed` | Registers the sanity partner in Partner Management | `sanity.runE2e` | | 12 | `fr-sanity-cm-seed` | Consent Manager binding and policy for that partner | `sanity.runE2e` | | 13 | `fr-sanity-data-seed` | **Sanity fixtures** — the test record, Keycloak user, approver rule | `sanity.runE2e` | | 20 | `fr-iam-register` | Registers the registry's roles and permissions in IAM | always | | 25 | `fr-sanity` | Runs the sanity suite | `sanity.enabled` | | 40 | `fr-fr-bulk-sample` | Generates **100,000 farmers** | `analytics.bulkSample.enabled` | | 45 | `fr-fr-reporting-views` | Creates the reporting views the dashboards read | `analytics.reportingViews.enabled` | | 50 | `fr-fr-dashboards` | **Imports the dashboards into Superset** | `analytics.dashboards.enabled` | {% hint style="info" %} **The dashboards are loaded by this registry, not by the analytics platform.** The dashboard bundle ships in this repository, and the weight-50 job imports it into the environment's Superset, points its database connection at this registry, publishes the dashboards and enables embedding on them. The ordering is the point: dashboards are imported **after** bulk data and **after** the reporting views exist, so that a dashboard opened straight after install already has data behind it rather than rendering empty. {% endhint %} ### Cross-release dependencies Two of these steps depend on releases Helm cannot order against, because they are separate installs: * **Master Data** must be seeded before weight 10 (code lists, geo, sample people) and before weight 40 (the bulk generator reads its geo hierarchy). The db-seed and bulk jobs each wait for it and fail with a clear message rather than producing records that point nowhere. * **Superset** must be reachable before weight 50. That job waits for it rather than failing immediately, so a Superset that is merely restarting does not lose the dashboard import. {% hint style="warning" %} Because Helm stops at a failed hook, a failure in bulk generation (weight 40) also prevents the reporting views and the dashboard import from ever running. If Superset has no dashboards after an install, check the **earlier** jobs first. {% endhint %} ## What the image contains The Farmer Registry's `db-seed` image is a thin `FROM` of the platform's, clearing the reference registry's content and copying its own — the extension's `meta_data/`, `awe_meta_data/` and `templates/`, the `seed-data/*.json` overlay above, `generate_fr_bulk_sample.py` and `reporting_views.sql`. The bulk generator lives here, beside the reporting views it must move in step with, rather than in a shared repository. ## Chart defaults ```yaml registry: dbSeed: loadSampleData: false # demo farmers — opt in from the form loadImages: false analytics: bulkSample: enabled: false # bulk demo data — opt in from the form farmers: 100000 # used only when enabled reportingViews: enabled: true # independent of bulk data ``` There is no geo-dropdown setting: register forms take their geo levels from Master Data when they load. `loadGeoData` and `syncGeoWidgets` were removed from the registry platform and take effect here once the chart's registry-platform version is bumped; until then leave `loadGeoData` off. The seed and analytics Jobs reach Master Data through its API (`MDS_API_URL` and a client-credentials token from the `-staff-portal` Keycloak client), so that client needs read access to MDS `/catalogue` and `/samples`. For a production install, see [An empty install](/products/registry/registry/deployment-and-extension/country-data-and-seeding.md#an-empty-install-for-production). *** {% hint style="info" %} **Beyond seeding.** What is built on these views — the dashboards, the map drill-down, how to add your own, and what to do when a country pack or this registry's schema changes — is under [**Reporting & Analytics**](/platform/platform-services/reporting-and-analytics.md). Start with [Setting up reporting](/platform/platform-services/reporting-and-analytics/setting-up-reporting.md) if you are bringing this registry up for a country. {% endhint %} --- # Agent Instructions This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com. ## Querying This Documentation If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question. Perform an HTTP GET request on the current page URL with the `ask` query parameter, and the optional `goal` query parameter: ``` GET https://docs.openg2p.org/products/registry/farmer-registry/deployment/data-seeding.md?ask=&goal= ``` `ask` is the immediate question: it should be specific, self-contained, and written in natural language. `goal` is optional and describes the broader end goal you are ultimately trying to accomplish on behalf of the user. GitBook uses it to tailor the answer towards what is most useful for that goal. The response will contain a direct answer to the question and relevant excerpts and sources from the documentation. Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.