Environment Setup
Setting up OpenG2P environments on an existing multi-node infrastructure.
This guide covers creating OpenG2P environments (namespace + services) on an existing multi-node infrastructure where Nginx, the Kubernetes cluster, and storage run on separate nodes.
Where you are in the flow. Stages 1–3 are done: VMs are provisioned, DNS+TLS are in place, and the platform (RKE2, Istio, Rancher with local auth, Wireguard, Nginx, NFS, host PostgreSQL) is installed and reachable. Production automation also scaffolds the environment (namespace, Istio Gateway, Helm repos, external-PG secret). This stage finishes by installing Commons from the Rancher UI only. After Commons, install the product modules your rollout delivers (Registry, PBMS, SPAR, G2P Bridge).
Before you start — procurement prerequisites
DNS records, TLS certificates, and server access for this environment must already be in place before Commons install. If you have not yet procured these, start with the Prerequisites & Procurement page — it contains a single fillable checklist (admin + production hostnames + certs + server access + firewall ports) you can hand to your IT / network / cert team. TLS issuance from sovereign or commercial CAs typically takes 2–4 weeks, so do this before running any installer.
Commons — Rancher UI only
Install openg2p-commons-base and openg2p-commons-services from the Rancher UI (Apps → Charts), where the chart's questions.yml provides a guided form for production parameters (external PostgreSQL, hostnames, storage classes, replicas, and so on). Scripts under automation/environment/ scaffold the namespace only — they do not install Commons. Production scaffolding (openg2p-prod.sh) already creates the namespace, Rancher Project, Istio Gateway, and commons-postgresql secret.
How this stage runs — production vs standalone scaffolding
Option A — Production automation scaffolding + Rancher UI Commons (recommended)
If you installed the platform with the production automation:
Scaffolding is Stage 4 of
openg2p-prod.sh(runs at the end of a full install wheninstall_environment: true). It creates the namespace, Rancher Project, Istio Gateway, Helm ClusterRepo (openg2p), and the external-PG secret. It uses an SSH tunnel to the Kubernetes API — Wireguard is not required for scaffolding.
./openg2p-prod.sh --config prod-config.yaml --stage environment
# or: ./openg2p-prod-env-install.sh --config prod-config.yamlCommons — connect Wireguard, open Rancher, and install openg2p-commons-base then openg2p-commons-services in the environment namespace (use the
commons-postgresqlsecret and host PostgreSQL on storage). Chart versions: Commons changelog.
Your only other manual actions:
Step 1 — DNS records (a procurement prerequisite; no script creates these).
Step 3 — citizen "go-public" exposure on the Reverse Proxy (add the public Nginx server block + open public
80/443), when you're ready to serve citizen traffic.
You do not need to write env-config.yaml for production scaffolding.
Option B — Standalone scaffolding (env-cluster.sh) + Rancher UI Commons
If you're setting up an environment separately — on infrastructure not built by openg2p-prod.sh — use automation/environment/env-cluster.sh for scaffolding only (namespace, Rancher Project, Istio Gateway), then install Commons from the Rancher UI. Follow the step-by-step below for DNS / TLS / Nginx; skip any Commons Helm CLI steps.
Which steps are manual?
Step
Production flow (openg2p-prod.sh + Rancher)
Standalone flow
1 — DNS records
Manual (prerequisite)
Manual
2 — TLS cert on the RP
Automated (done in Stage 3) — skip
Manual
3 — Citizen exposure on the RP
Manual (the go-public action)
Manual
Cluster scaffolding (namespace, project, gateway, repo, PG secret)
Automated (openg2p-prod.sh env stage)
env-cluster.sh (scaffolding only) or manual
Commons (commons-base + commons-services)
Rancher UI only
Rancher UI only
Architecture
In a multi-node setup, each environment gets its own domain, namespace, and full set of services. The Nginx node handles TLS termination and proxies traffic to the cluster's Istio ingress gateway.
The setup has two parts:
Nginx setup (Steps 1-3)
On the Nginx node (manual)
DNS, TLS certificate, Nginx server block, open public firewall
Cluster scaffolding
Workstation / production scripts
Namespace, Rancher project, Istio gateway
Commons
Rancher UI only
openg2p-commons-base then openg2p-commons-services
Prerequisites
Infrastructure
Nginx node, K8s cluster, Istio, and Rancher are all running
Procurement complete
DNS records, TLS cert, and Nginx access — see Prerequisites & Procurement
DNS records
<base_domain> and *.<base_domain> A records pointing to the Nginx node
TLS cert on Nginx node
Wildcard cert at /etc/openg2p/certs/<base_domain>/{fullchain.pem,privkey.pem} (customer-provided)
Nginx node
nginx running, istio_ingress upstream configured
Workstation
kubectl and helm installed, plus the base toolchain (bash 4+, ssh, openssl, git). See Operator's workstation for the canonical list, supported OSes, and per-OS install commands. Kubeconfig with admin access to the cluster is also required.
Step-by-Step Guide
Step 1: Verify DNS records (procured up front)
Production: manual — a procurement prerequisite (no script creates DNS records). Standalone: manual.
DNS records should have been procured as part of the Prerequisites & Procurement step. For this environment you need:
A
qa.openg2p.org
<nginx_node_ip>
A
*.qa.openg2p.org
<nginx_node_ip>
Verify DNS resolution before proceeding:
Step 2: Place the customer-provided TLS certificate
Production: automated — the cert was installed on the Reverse Proxy in Stage 3 (Infrastructure); skip this step. Standalone: manual.
The wildcard certificate for *.<base_domain> (covering the apex too) is procured from the customer's chosen CA — commercial (DigiCert, GlobalSign, Sectigo) or national / sovereign — as listed in the procurement checklist. Let's Encrypt is acceptable only for sandbox / PoC; see the note at the end of this step.
On the Nginx node, place the cert files at the standard path:
Verify the cert covers the expected hostnames:
Step 3: Expose the environment on the Reverse Proxy
Production & standalone: manual — this is the citizen "go-public" action on the Reverse Proxy (the env stage is laptop-side and does not touch the RP). Do it when you're ready to serve citizen traffic.
This is the step that opens the system to citizens. It has two parts: an Nginx server block for the environment's hostnames, and opening the public channel at the firewall. Until now the Reverse Proxy served only the admin tools (Rancher, Keycloak) on the private channel — this step adds the public, citizen-facing channel alongside them.
3a. Nginx server block (citizen channel)
On the Reverse-Proxy / Nginx node, create the server block that references the cert from Step 2. Note the listen address and the absence of an allowlist — both deliberate:
Bind to the RP's private IP. Public traffic to the Elastic IP (AWS) or your DNAT address (on-prem behind a firewall) arrives NAT'd to this private IP, so binding here serves it — and it coexists cleanly with the admin blocks already on <rp_private_ip>:443 (different server_name, same socket — no conflict).
If the RP holds its public IP directly on the NIC (no upstream NAT), bind the citizen block to that public IP. Admin blocks remain on the private IP, so there is no listen conflict.
The istio_ingress upstream must already exist in your Nginx config (the infrastructure automation creates it, pointing at the cluster node's Istio ingress NodePort):
Enable the site and reload Nginx:
3b. Open the public channel (firewall)
The infrastructure setup deliberately left public 80/443 closed (only SSH + Wireguard were open). Open them now so citizens can reach the environment. The per-host firewall (ufw) already allows 80/443 from the private subnet; this step opens them at the network boundary.
Add inbound rules to the Reverse-Proxy's Security Group (<project>-reverse-proxy):
At your perimeter firewall / router, allow inbound 80/tcp and 443/tcp from the internet to the Reverse Proxy. If the RP sits behind NAT, also DNAT those ports to the RP's private IP (the address the Nginx citizen block listens on).
No host-level change is needed — the automation already configured ufw to accept 80/443 from the private subnet, and the citizen block's source has no allowlist.
Step 4: Scaffold the cluster environment (if needed)
Production: skip — already done by openg2p-prod.sh. Standalone: run scaffolding only.
If the namespace / Rancher Project / Istio Gateway are not already present:
env-cluster.sh creates the namespace, Rancher Project, and Istio Gateway only. It does not install Commons.
Step 5: Install Commons from the Rancher UI
Production and standalone: required — Commons is installed from Rancher only.
Connect Wireguard (or otherwise reach Rancher) and open
https://rancher.<domain>.Select the environment namespace / project.
Apps → Charts → openg2p-commons-base — install with production values (external PostgreSQL via the
commons-postgresqlsecret when using production scaffolding, hostnames, storage class, and so on).Then install openg2p-commons-services in the same namespace.
Pick chart versions from the Commons changelog.
Configuration reference (scaffolding)
When using standalone env-cluster.sh, env-config.yaml needs:
environment
Namespace and Rancher project name
base_domain
Domain for the Istio Gateway hosts
Commons chart options are configured in the Rancher Apps UI (not in env-config.yaml).
Next: install your OpenG2P modules
At this point you have a working environment with commons-base + commons-services installed — the shared infrastructure (PostgreSQL, Kafka, MinIO, Redis, Keycloak, etc.) plus baseline cross-cutting services (eSignet, Superset, ODK). What you don't yet have is the OpenG2P product modules a specific deployment actually delivers (registry, payments, beneficiary onboarding, etc.). Each product has its own Helm chart and deployment guide — install whichever modules your rollout requires:
Farmer Registry — Social / Farmer / generic registry (Gen2). Helm Chart 4.x.
PBMS — Payment & Beneficiary Management System.
SPAR — Single Payee Account Repository.
G2P Bridge — government-to-payer bridge (treasury / bank disbursement integration).
Each product page documents its Helm-chart version, deployment commands, Keycloak client setup, and domain-name requirements. Install only the modules required for your use case — none of them are mandatory infrastructure dependencies of the others.
Creating Multiple Environments
To create additional environments (e.g., staging) on the same cluster:
Create DNS records for
staging.openg2p.organd*.staging.openg2p.orgpointing to the Nginx IPOn the Nginx node: obtain a new certificate (Step 2) and add a new server block (Step 3a) with the new domain. The firewall (Step 3b) is already open from the first environment — no need to repeat it.
Create a new config file with
environment: stagingandbase_domain: staging.openg2p.organd runenv-cluster.shfor scaffolding (or use production--stage environmentwith a differentenvironment.name)Install Commons for that environment from the Rancher UI
Each environment gets its own namespace, Rancher project, Istio gateway, and Commons install.
Uninstallation
To tear down an environment's Helm releases (including Commons installed from Rancher), use env-cluster-uninstall.sh or production's openg2p-prod-env-uninstall.sh. Reinstall Commons afterward from the Rancher UI.
Uninstalls all Helm releases in the namespace and deletes all data (Secrets, PVCs, PVs). Preserves the namespace, Istio Gateway, and Rancher Project so you can reinstall Commons quickly from the Rancher UI.
Deletes:
ALL Helm releases in the namespace —
commons-services,commons, and any other module charts (Registry, PBMS, SPAR, G2P Bridge, custom charts, etc.). Thecommonsrelease is uninstalled last since other modules depend on its infrastructure.All Jobs (hook leftovers)
All Secrets in the namespace
All PVCs + associated PVs
Chart-owned ConfigMaps only — those labelled
app.kubernetes.io/managed-by: Helm, those carrying ameta.helm.sh/release-nameannotation, or those named<release>-*
Preserved ConfigMaps: hand-created ConfigMaps (seed data, migration payloads, debug patches) are kept and listed in the output, so a shared namespace does not lose operator artifacts. Cluster- and mesh-owned ConfigMaps (kube-root-ca.crt, istio-*) are never touched.
Preserves:
Namespace, Istio Gateway, Rancher Project
Nginx config, certificates, DNS records
Everything in the default mode, plus the Istio Gateway, Rancher Project, and the namespace itself. Leaves only infra-level resources.
Also deletes:
Istio Gateway(s) in the namespace
Rancher Project association (and the project itself, if Rancher is on this cluster)
The namespace itself
Preserves:
Nginx config on the Nginx node
Let's Encrypt certificates
DNS records
Cluster / Rancher / Istio installations
See what would be deleted without actually deleting anything:
The script previews everything that will be deleted and asks for confirmation before proceeding.
Default mode requires typing
yes--fullmode requires typing the namespace name (prevents accidental wipes of the wrong environment)
Use --yes to skip confirmation for automation/CI.
Uninstall CLI options
--namespace <name>
Target Kubernetes namespace to tear down (required)
--full
Also delete Istio Gateway, Rancher Project, and namespace
--yes
Skip confirmation prompt (for automation)
--dry-run
Show what would be deleted without actually deleting
--help
Show help message
Why ConfigMaps are filtered rather than wiped
Helm-labelled ConfigMaps that have lost their meta.helm.sh/release-name annotation are not removed by helm uninstall — it cannot associate them with any release. Left behind, they cause invalid ownership metadata failures the next time a chart tries to create a ConfigMap of the same name, so the uninstall script removes them explicitly.
It deliberately stops short of kubectl delete configmap --all: on a shared namespace that would silently destroy hand-applied seed data and migration payloads that no chart will ever recreate. Anything not chart-owned is listed in the output so you can remove it by hand if you want.
Accessing host PostgreSQL from your laptop
Production PostgreSQL is the host install on the storage node, firewalled to the compute node only — so you reach it over SSH, not a direct connection (which fails even on Wireguard, because WG NATs your traffic to the reverse-proxy's IP). The full how-to — on-box psql, the SSH-tunnel routes, GUI clients, and where the credentials live — is the day-2 operational guide:
➡️ Access a Database from Outside the Cluster → Host PostgreSQL (production)
Quick reference: the superuser password is on the storage node at /etc/openg2p/secrets/postgres-superuser.env and in the installer's final summary (automation/production/setup-output/SETUP-SUMMARY.txt).
File Structure
Troubleshooting
Certificate issues (on Nginx node)
Nginx issues (on Nginx node)
Cluster issues (from workstation)
eSignet / mock-identity crashloop — relation "key_alias" does not exist
After installing commons-services, esignet and mock-identity-system may be in CrashLoopBackOff with this in their logs:
Cause. eSignet and mock-identity each embed the keymanager library, which needs the keymanager schema (key_alias, key_store, …) in their own database. Each ships its schema-init as a helm.sh/hook: post-install Job, which deadlocks helm --wait: the pods can't become Ready until the schema exists, but the post-install hook that creates the schema only runs after the release is Ready. So the hook never runs and the release ends as failed. (Standalone keymanager is unaffected — its init runs as a regular resource.) This is a chart-level issue in openg2p-commons-services.
Materialise the schema-init Jobs by hand (replace qa with your namespace):
The init is idempotent (it skips tables that already exist), so re-running is safe.
Last updated
Was this helpful?