Ctrlk
For the complete documentation index, see llms.txt. This page is also available as Markdown.

Operations

Day-to-day backup operations — install, run, verify, list, status, wal-health, daily-report, group toggling.

The orchestrator is automation/backups/openg2p-backup.sh. Every subcommand takes --config backup-config.yaml. Most also accept --component <name> to target a single group (all, pg, etcd, rancher, nfs, configs, objectstore).

Subcommands

./openg2p-backup.sh install       # one-time / idempotent bootstrap
./openg2p-backup.sh run           # execute backups now (also used by cron)
./openg2p-backup.sh verify        # cheap integrity checks, no restore
./openg2p-backup.sh drill         # weekly drill (verify + dry-run-restore)
./openg2p-backup.sh list          # what's in each repo
./openg2p-backup.sh restore       # see Restoration page
./openg2p-backup.sh status        # last-run + last-drill state
./openg2p-backup.sh wal-health    # independent PG WAL / archiver probe
./openg2p-backup.sh daily-report  # email .status.json summary (if SMTP enabled)
./openg2p-backup.sh help

--component accepts one value (or all). Do not pass multiple group names on one line.

install

# Full bootstrap (all enabled groups + cron + PrometheusRule when monitoring.enabled)
./openg2p-backup.sh install --config backup-config.yaml

# Re-run a single group's install steps (e.g. after changing rancher storage)
./openg2p-backup.sh install --config backup-config.yaml --component rancher

--component is supported for install as well as run / verify / list. Per-group install skips the other groups — useful when re-provisioning rancher storage without triggering etcd_install's rke2-server restart. Backup-host bootstrap and cron redeploy still run on every install invocation.

What it does, in order:

  1. SSH probes for backup, plus production nodes touched by enabled groups.

  2. Backup-host preflight — Ubuntu version, CPU (≥4, hard), RAM (≥8 GB, hard), root disk (≥64 GB, hard), repo data volume (≥1 TB, warn-only). Skip with --skip-preflight when re-running against a known-good host.

  3. Passphrase resolution — reads each *_passphrase_file from the keystore. Generates if missing (with a banner reminder to move into the keystore).

  4. Backup-host bootstrap — apt installs, repo dirs, lib files pushed to /opt/openg2p-backup/, SSH key generated for orchestrating compute/storage/RP, authorized on those nodes, wrapper scripts installed:

    • /usr/local/bin/openg2p-backup-run

    • /usr/local/bin/openg2p-backup-drill

    • /usr/local/bin/openg2p-backup-status

    • /usr/local/bin/openg2p-backup-wal-health

    • /usr/local/bin/openg2p-backup-daily-report

  5. Per-group install — gated by groups.<name> toggle and --component (when not all):

    • pg: pgBackRest on backup + storage, archive_command on PG, stanza-create, first full backup

    • etcd: RKE2 snapshot schedule (every 6h), initial on-demand snapshot, rsync-pull SSH trust + first pull to backup host

    • rancher: rancher-backup operator (chart 110.0.1+up11.0.2 default — matches Rancher 2.15.1 / RKE2 v1.35.8+rke2r1; override via versions.rancher_backup_chart), static NFS PV openg2p-rancher-backup-store, encryption Secret, ResourceSet + in-cluster Schedule CR. Older 107.x pins often fail the chart’s patch-sa post-upgrade Job (BackoffLimitExceeded); install deletes stuck Jobs and, if needed, patches the default SA and retries with --no-hooks.

    • nfs: storage-node NFS export + ufw allow for backup host; read-only NFS mount on backup host via _nfs_ensure_ro_mount (stops stale automounts, rewrites fstab without x-systemd.automount, remounts; falls back to /mnt/openg2p-nfs-ro-dr after DR IP changes); restic repo init

    • configs: restic repo for configs

    • objectstore (opt-in): rclone + restic for MinIO/S3 — skipped when groups.objectstore: false (default)

  6. SMTP install (optional) — copies alerting.smtp_env_file to /etc/openg2p-backup/smtp.env when present.

  7. PrometheusRule — when monitoring.enabled and monitoring.apply_prometheusrule are true, applies manifests/prometheusrule-backup.yaml.template into cattle-monitoring-system (warns and continues if CRD/namespace missing).

  8. Optional encryption-at-rest — only if --enable-secret-encryption flag is passed. Restarts kube-apiserver.

  9. Cron deploy — renders cron.template with active schedules, installs at /etc/cron.d/openg2p-backup.

install is idempotent — re-running it is safe. Pass --force where individual steps honour it. After pulling new orchestrator code, re-run install to refresh /opt/openg2p-backup/lib and the wrappers.

run

Used by cron on the backup host (via openg2p-backup-run <group>). PG honors PGBR_TYPE=full|diff — the cron file passes full on Sundays and diff other days.

If a group is disabled in config, run --component <that-group> exits with a warning rather than failing.

Failures don't stop other groups — run attempts every enabled group and surfaces per-group results in the status file. Failed groups also trigger failure email when alerting.email_enabled: true. Each run updates Prometheus textfile metrics under $backup_repo_root/metrics/ when monitoring.enabled: true.

rancher is special. The nightly rancher backup is driven by an in-cluster Schedule CR, not by the cron file on the backup host. run --component rancher from the laptop creates an ad-hoc Backup CR — useful before upgrades, but not the routine cadence. Each ad-hoc run also re-applies openg2p-resource-set (custom ResourceSets are often wiped when rancher-backup-crd is upgraded).

nfs / configs self-heal on run. run --component nfs re-asserts the storage /etc/exports + ufw rules for the backup host before mounting (avoids mount.nfs: Connection timed out after storage rebuild). run --component configs initialises the configs restic repo if it is missing.

verify

Cheap integrity checks. No data restored. When --component all, every enabled group is checked; a failure in one group does not stop the others (the orchestrator exits non-zero if any group failed).

Group
Verify command

pg

pgbackrest verify

etcd

Finds the latest etcd-snapshot-* / on-demand-* on the backup host (auto-pulls from compute if empty; if compute also has none — common before the first 6h schedule tick — takes rke2 etcd-snapshot save then pulls). Runs etcdctl/etcdutl snapshot status locally; if the distro etcd-client cannot read RKE2's snapshot format, falls back to RKE2-bundled tools on compute

rancher

Resolves the static NFS path (/srv/nfs/<cluster>/rancher-backup by default), SSHes to the storage node, confirms the latest *.tar.gz or *.tar.gz.enc is present and non-zero. Encrypted tarballs skip gzip -t (ciphertext, not plain gzip)

nfs

restic check --read-data-subset=5% on the NFS repo

configs

restic check --read-data-subset=5% on the configs repo

objectstore

restic snapshots --latest 1 + restic check --read-data-subset=1% (when enabled)

drill

Weekly. Runs every group's <group>_drill function (verify + canary restore + tear down) and writes results to /var/lib/openg2p-backup/.status.json on the backup host. See Drills.

list

Inventory per repo:

  • pg → pgbackrest info (backup sets, dates, sizes)

  • etcd → ls -lh of pulled snapshots

  • rancher → kubectl get backup.resources.cattle.io -A (CR inventory; tarballs live on NFS under /srv/nfs/<cluster>/rancher-backup/)

  • nfs / configs / objectstore → restic snapshots --compact (for NFS, prefer --tag nfs for data; pvc-manifest snapshots are sidecar-only)

For NFS, the sidecar .pvc-mapping.yaml is what tells you which PVC each UUID belongs to. Read it directly:

status

Tabular per-group last-run and last-drill state, read from /var/lib/openg2p-backup/.status.json:

Disabled groups show disabled and dashes. The same JSON drives daily email summaries and complements Prometheus metrics — see Alerting.

wal-health

Independent of pg backup runs. SSHes to the storage node, measures pg_wal size and pg_stat_archiver, writes metrics, and (when email is enabled) notifies if monitoring.wal.* thresholds are exceeded. Cron runs this every 5 minutes via openg2p-backup-wal-health.

daily-report

Emails operators a summary of .status.json when alerting.email_enabled: true.

After install, this runs automatically on the backup host via cron (schedules.daily_report, default 0 7 * * * → /usr/local/bin/openg2p-backup-daily-report). The laptop command above is for smoke-testing only.

Failure mails are separate: they are sent automatically when a backup/drill run fails (no need to invoke daily-report). See Alerting.

Group toggles

Edit backup-config.yaml:

Re-run install to apply. The cron file is regenerated and the disabled group's lines are commented out (kept for reference). objectstore defaults to off when the key is absent so existing installs are not surprised.

Forcing a re-run / resetting state

install uses state markers under automation/backups/.state/ (laptop side) and /var/lib/openg2p/deploy-state/ (remote side). --force ignores them where applicable:

To reset only the laptop state, delete the .state/ directory.

Re-running aws-provision to add the backup node

The Backup node is required for production. If you brought the platform up before provisioning it (the recommended order is to have backups in place before go-live), add it as follows:

Logs

Location
Content

automation/backups/logs/openg2p-backup-<timestamp>.log

Laptop orchestrator invocation

/var/log/openg2p-backup.log on backup host

Cron group runs, drill, daily-report

/var/log/openg2p-backup-wal.log on backup host

WAL-health cron

/var/lib/openg2p-backup/metrics/*.prom on backup host

Prometheus textfile gauges

/var/lib/openg2p-backup/.status.json on backup host

Last-run / last-drill JSON

PreviousConfiguration
NextDrills

Last updated

Was this helpful?