Alerting (Phase 2)

Backup health monitoring — Prometheus dead-man's switch, WAL probes, operator email, and Slack via Alertmanager.

The backup automation emits Prometheus metrics, runs an independent WAL probe outside the nightly pg job, can email operators who do not have SSH access, and applies a PrometheusRule into Rancher Monitoring (cattle-monitoring-system). Slack (and other Alertmanager receivers) are configured on the platform monitoring stack, not in smtp.env.

Status JSON at /var/lib/openg2p-backup/.status.json remains the source of truth for per-component last-run / last-drill results.

What is alerted

Event	Mechanism	Severity
Any backup component last run failed	OpenG2PBackupRunFailed + failure email	critical
No successful backup in >26h (dead-man)	OpenG2PBackupMissed	critical
Component hasn't run in >48h	OpenG2PBackupComponentStale	warning
Backup disk <25GiB / <10GiB	OpenG2PBackupDiskLow / OpenG2PBackupDiskCritical	warning / critical
Backup metrics missing from Prometheus	OpenG2PBackupMetricsAbsent	warning
WAL archive stalled / failing / growing	OpenG2PWAL* + optional WAL threshold email	critical / warning
WAL metrics missing	OpenG2PWALMetricsAbsent	warning
Daily operator summary	SMTP daily-report cron	info

Prometheus-named alerts fire only when metrics are scraped (or pushed). Failure / daily emails work from the backup host alone once SMTP is configured.

Prometheus rules (automated)

install applies manifests/prometheusrule-backup.yaml.template into cattle-monitoring-system when:

monitoring:
  enabled: true
  namespace: "cattle-monitoring-system"
  apply_prometheusrule: true

If the PrometheusRule CRD or namespace is missing, install logs a warning and continues. Re-run install after Rancher Monitoring is up.

No manual sed | kubectl apply is required.

kubectl -n cattle-monitoring-system get prometheusrule | grep openg2p-backup

Textfile metrics on the backup host

Each run (when monitoring.enabled: true) writes gauges under $backup_repo_root/metrics/ (default /var/lib/openg2p-backup/metrics/). Configure node_exporter on the backup host with:

--collector.textfile.directory=/var/lib/openg2p-backup/metrics

and ensure Rancher Monitoring scrapes that node (or use Pushgateway — below). Without a scrape path, rules exist but never fire.

Dead-man's switch

openg2p_backup_master_last_success_timestamp advances whenever any component reports last_run_result=ok. Alert OpenG2PBackupMissed fires if that gauge is older than 26 hours.

Optional Pushgateway

If the backup host is not scraped by node_exporter, set monitoring.pushgateway_url. Metrics are POSTed after each emit. If your Pushgateway does not honor labels, adapt rule selectors to exported_job / exported_instance.

Independent WAL health

openg2p-backup-wal-health runs every 5 minutes (outside pg backup runs). It SSHes to the storage node, measures pg_wal size and pg_stat_archiver, and writes:

• openg2p_pg_wal_size_bytes

• openg2p_pg_archiver_failed_count

• openg2p_pg_archiver_last_archive_age_seconds

Threshold emails use monitoring.wal.* in config; paging should use the PrometheusRules (OpenG2PWAL*).

./openg2p-backup.sh wal-health --config backup-config.yaml

Email reports (operators without SSH)

alerting:
  email_enabled: true
  smtp_env_file: "~/.openg2p/keystore/smtp.env"

1. Copy automation/backups/roles/backup-host/smtp.env.example → ~/.openg2p/keystore/smtp.env (mode 0600).

2. Fill SMTP_* and MAIL_TO. Always quote SMTP_FROM when it contains spaces or angle brackets — the file is sourced by bash (SMTP_FROM="OpenG2P Backup <[e-mail]>"). Unquoted <…> causes a syntax error.

3. Set email_enabled: true and re-run install (copies to /etc/openg2p-backup/smtp.env on the backup host).

Automatic vs manual

Event	When it fires
Daily .status.json summary	Cron on the backup host (schedules.daily_report, default 0 7 * * *) — no laptop action needed
Backup / drill failure	Automatic on non-zero exit from openg2p-backup-run / drill (and laptop run)
Ad-hoc test	./openg2p-backup.sh daily-report --config backup-config.yaml

MAIL_SENT in the log means the backup host handed the message to SMTP_HOST. It does not guarantee the recipient inbox received it — check the SMTP/relay logs if mail is missing.

Recommended: authenticated smarthost (port 587)

On AWS and many clouds, outbound TCP 25 is blocked. The in-cluster mail chart (Exim) can accept mail from the backup host and then fail when delivering to Gmail/Google Workspace MX (Network is unreachable / Connection timed out on :25). Prefer a real smarthost:

SMTP_HOST=smtp.gmail.com          # or org SES/SendGrid/relay
SMTP_PORT=587
SMTP_STARTTLS=true
SMTP_USER=[e-mail]
SMTP_PASS=your-app-or-smtp-password
SMTP_FROM="OpenG2P Backup <[e-mail]>"
MAIL_TO=[e-mail]

Email with in-cluster OpenG2P mail

The platform mail chart (openg2p/smtp) exposes SMTP as ClusterIP port 25 and typically allows relay only from the pod CIDR (RELAY_NETWORKS: :10.0.0.0/8). That works for Alertmanager inside the cluster (mail.<ns>.svc.cluster.local:25), but the backup host (VPC 172.29.x) cannot use ClusterIP DNS.

For backup emails via that pod (only if outbound :25 works in your environment):

1. Include the VPC in relay, e.g. RELAY_NETWORKS: ':10.0.0.0/8:172.29.0.0/16'.

2. Expose the Service (NodePort is simplest) and point smtp.env at the compute private IP + NodePort.

3. Use plain SMTP (SMTP_STARTTLS=false, empty SMTP_USER / SMTP_PASS).

4. If MAIL_SENT appears but no inbox mail, check kubectl -n <mail-ns> logs deploy/mail for MX/:25 timeouts — then switch to a 587 smarthost (above).

SMTP_HOST=172.29.1.177          # compute private IP
SMTP_PORT=<mail-NodePort>
SMTP_STARTTLS=false
SMTP_USER=
SMTP_PASS=
SMTP_FROM="OpenG2P Backup <backup@YOUR_MAILNAME>"
MAIL_TO=[e-mail]

Slack (via Alertmanager)

Backup scripts do not post to Slack themselves. Slack is configured on Rancher Monitoring Alertmanager, which receives the backup PrometheusRules once metrics are scraped.

In prod-config.yaml (production automation):

alert_slack_webhook_url: "https://hooks.slack.com/services/…"
alert_slack_channel: "#alerts"

Re-apply the monitoring/alerting values so Alertmanager picks up the receiver (see also alerting/ in the deployment repo for webhook / Alertmanager secret patching patterns). Then firing rules such as OpenG2PBackupMissed, OpenG2PBackupRunFailed, and OpenG2PWAL* can notify Slack.

Channel	Configured where	Covers
Email	backup-config.yaml → alerting.* + smtp.env	Daily summary + script failure mails
Slack	prod-config.yaml → alert_slack_* + Alertmanager	Prometheus-fired backup / WAL alerts

You can enable both; they complement each other.

How to test

Check	Command / action	Pass look like
Metrics files	run --component configs then cat /var/lib/openg2p-backup/metrics/*.prom on backup host	openg2p_backup_run_status, openg2p_backup_master_last_success_timestamp present
WAL probe	./openg2p-backup.sh wal-health --config …	WAL size / archiver gauges written
PrometheusRule	kubectl -n cattle-monitoring-system get prometheusrule | grep openg2p-backup	rule object exists
Prom scrape	Query openg2p_backup_master_last_success_timestamp in Prometheus	series returned (if empty, fix scrape/Pushgateway first)
Failed-run alert	Force one group to fail, wait ≥5m	OpenG2PBackupRunFailed in Alertmanager (and Slack if configured)
Daily email	./openg2p-backup.sh daily-report --config …	MAIL_SENT and message in inbox — also check Spam / Promotions / Updates; if only MAIL_SENT, check relay logs / use port 587

Email works without scrape. Cluster / Slack alerts need rule + scrape both.

Operator checklist after upgrade

1. Merge new keys from backup-config.example.yaml into your backup-config.yaml (especially versions.rancher_backup_chart — use 110.0.1+up11.0.2 or newer for Rancher 2.15 / RKE2 1.35).

2. Re-run install (refreshes cron wrappers + libs under /opt/openg2p-backup, re-applies PrometheusRule).

3. After a Rancher / rancher-backup upgrade specifically: ./openg2p-backup.sh install --config backup-config.yaml --component rancher --force (restores CRDs, operator, and openg2p-resource-set).

4. Point node_exporter at $backup_repo_root/metrics (or set Pushgateway).

5. Confirm PrometheusRule: kubectl -n cattle-monitoring-system get prometheusrule | grep openg2p-backup.

6. Optionally enable email (prefer SMTP :587) and/or Slack via Alertmanager.

7. Smoke-test: ./openg2p-backup.sh run --config backup-config.yaml --component rancher (and nfs / configs if those groups are enabled).