> For the complete documentation index, see [llms.txt](https://docs.openg2p.org/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://docs.openg2p.org/operations/deployment/infrastructure-setup/backups/restoration/single-pvc.md). # Single PVC Restore one PersistentVolumeClaim's data from the NFS restic repo and rebind it to the target app. Use this when one application's data is corrupt or accidentally wiped, but the cluster and other apps are fine. Common cases: Keycloak themes overwritten, MinIO bucket data deleted, a CMS upload directory wiped. ## How the mapping works Each PVC's data lives in a per-volume subdirectory under the NFS export (e.g. `/srv/nfs/openg2p/--/` on the `nfs-csi` StorageClass, or a native NFS PV path). On its own restic just sees opaque directory names. The sidecar manifest at `/var/lib/openg2p-backup/nfs/.pvc-mapping.yaml` joins each directory against `kubectl get pv` (including CSI `volumeAttributes.subdir`) so restore knows which app a path belongs to. This file is regenerated every NFS run — always inspect the latest version. ## Step 1 — Find the PVC ```bash ssh ubuntu@ sudo cat /var/lib/openg2p-backup/nfs/.pvc-mapping.yaml | jq '.[] | select(.pvc_namespace == "keycloak")' ``` You'll see entries like: ```yaml - nfs_path: pvc-abc123-def456 pv_name: pvc-abc123-def456 pvc_namespace: keycloak pvc_name: keycloak-data pvc_size: 20Gi storage_class: nfs-csi app_label: app=keycloak backed_up_at: 2026-04-27T03:30:11Z ``` ## Step 2 — Dry-run ```bash ./openg2p-backup.sh restore \ --config backup-config.yaml \ --component nfs \ --target keycloak/keycloak-data \ --dry-run ``` Confirms the restic source and the **Bound** PV destination on storage it would push to. ## Step 3 — Restore (restic → Bound path on storage) Scale the app down first, then: ```bash kubectl --kubeconfig ~/.kube/openg2p-prod scale deploy keycloak -n keycloak --replicas=0 ./openg2p-backup.sh restore \ --config backup-config.yaml \ --component nfs \ --target keycloak/keycloak-data kubectl --kubeconfig ~/.kube/openg2p-prod scale deploy keycloak -n keycloak --replicas=1 ``` This: 1. Resolves the live **Bound** PV `subDir` (or native NFS basename) for `namespace/pvc`. 2. `restic restore … --tag nfs` on the backup host (fails if empty). 3. Pushes the restored tree over SSH onto `${nfs.export_root}/` on storage (moves any existing contents aside to `.precrash`). Bitnami MinIO dirs get `chown 1001:1001`. ### After a full rebuild (DR) Post-rebuild NFS cron may have already taken a new (empty) `--tag nfs` snapshot. Pin a **pre-disaster** snapshot: ```bash ./openg2p-backup.sh list --config backup-config.yaml --component nfs # on backup host: restic snapshots --tag nfs --compact restic ls | grep commons-services-keymanager ./openg2p-backup.sh restore \ --config backup-config.yaml \ --component nfs \ --target prod/commons-services-keymanager \ --point-in-time ``` The push still targets the **Bound** (new) UUID, even when restic contains the old UUID path. ## Step 4 — Push the data to the live NFS export (manual fallback) Only needed if the orchestrator SSH push failed. The NFS export is mounted **read-only** on the backup host. ### Plan A — replace the whole PVC's contents ```bash # Pause the app so it's not writing during the swap. kubectl --kubeconfig ~/.kube/openg2p-prod scale deploy keycloak -n keycloak --replicas=0 # Bound subDir (post-DR this is the NEW uuid): BOUND=$(kubectl --kubeconfig ~/.kube/openg2p-prod get pv -o json | jq -r ' .items[] | select(.status.phase=="Bound" and .spec.claimRef.namespace=="keycloak" and .spec.claimRef.name=="keycloak-data") | .spec.csi.volumeAttributes.subDir') ssh ubuntu@ sudo mkdir -p /srv/nfs/openg2p/"$BOUND" ssh ubuntu@ sudo mv /srv/nfs/openg2p/"$BOUND" /srv/nfs/openg2p/"$BOUND".precrash ssh ubuntu@ sudo mkdir -p /srv/nfs/openg2p/"$BOUND" # Staging path from a prior restic restore on the backup host ssh ubuntu@ "sudo tar -C /tmp/openg2p-nfs-restore/keycloak-keycloak-data-/ -czf - ." | \ ssh ubuntu@ "sudo tar -C /srv/nfs/openg2p/$BOUND -xzf -" ssh ubuntu@ sudo chown -R 1001:1001 /srv/nfs/openg2p/"$BOUND" # Bitnami MinIO; else match the app kubectl --kubeconfig ~/.kube/openg2p-prod scale deploy keycloak -n keycloak --replicas=1 ``` ### Plan B — selective file restore ```bash ssh ubuntu@ sudo find /tmp/openg2p-nfs-restore/keycloak-keycloak-data- -type f ssh ubuntu@ "sudo cat /tmp/openg2p-nfs-restore/.../themes/openg2p/login.ftl" | \ ssh ubuntu@ "sudo tee /srv/nfs/openg2p/\$BOUND/themes/openg2p/login.ftl" ``` ## Step 5 — Verify ```bash kubectl --kubeconfig ~/.kube/openg2p-prod -n keycloak logs deploy/keycloak --tail=50 # Browse the Keycloak admin to confirm restored content is present. ``` ## Restoring a deleted PVC (PV/PVC objects gone) If the PVC itself was deleted (not just the data): 1. Check `.pvc-mapping.yaml` for the PV/PVC names + size + storage class. 2. Restore Kubernetes objects via a rancher-backup `Restore` CR — see [full-rebuild.md](/operations/deployment/infrastructure-setup/backups/restoration/full-rebuild.md). The orchestrator always restores cluster-wide (`--target cluster` does not filter by namespace). For a single namespace, apply a manual `Restore` CR with rancher-backup's restore filters. 3. The restored PVC will bind to a freshly-provisioned PV (because the old PV is also gone). The new PV's `nfs.path` won't match the original. Two options: * Move the restic-restored data into the new PV's NFS path. * Edit the restored PV manifest (before applying) to point at the original NFS path, so it binds to the existing data. The second is faster but requires manual YAML edits. Operate carefully. ## Common gotchas * **Permissions** — NFS exports often run with `root_squash`. Restored files may show up as `nobody:nogroup` (or `nobody:1001`). Match what the workload expects: Bitnami MinIO needs `chown -R 1001:1001` on the PVC dir or the pod fails with `Permission denied` on `.root_user`. * **Bound UUID ≠ restored UUID after full rebuild** — helmfile creates new `nfs-csi` PVs; restic restores old UUID dirs. Put restored data under the **Bound** PV’s `subDir` (see `kubectl get pv … volumeAttributes.subDir`). Leaving data only under the old UUID leaves pods empty/failing. * **CSI volume source is immutable** — you cannot `kubectl patch pv` to change `spec.csi.volumeAttributes.server` / `subDir`, and you cannot add `spec.nfs` onto a CSI PV. Recreate the PV or (preferred) copy data into the Bound path. * **`mv` into a non-empty directory** — `mv chunks dest/chunks` fails with `Directory not empty`. `mv` has no `-r`. Replace the whole Bound directory, or `rm -rf dest/*` then `rsync -a src/ dest/`. * **Destination must exist before `tar -C`** — create with `mkdir -p` on storage or extract fails with `Cannot open: No such file or directory`. * **Same UUID on live NFS and in staging** — after rancher restore of native NFS PVs, staging nests data under `/tmp/openg2p-nfs-restore/--//`. Copy/move that **inner** directory, not the outer timestamped folder. * **Trailing slashes in `tar -C`** — the path is the *destination* directory. Always verify with `du -sh` / `ls` after the copy. * **Forgetting to scale the app down** — you can corrupt new files mid-restore. Always pause the consuming Deployment/StatefulSet first. * **The restored data is older** — by definition. Anything written after the last NFS backup snapshot is gone. Check the restored snapshot timestamp in `.pvc-mapping.yaml` (`backed_up_at`). * **Keycloak / Superset still CrashLoop after PG cutover** — on a storage rebuild they often still point at the **old** Postgres private IP in ConfigMaps/Secrets. Update those to the new storage IP, then restart — see [full-rebuild.md Step 9](/operations/deployment/infrastructure-setup/backups/restoration/full-rebuild.md#step-9-bounce-workloads--verify). --- # Agent Instructions This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com. ## Querying This Documentation If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question. Perform an HTTP GET request on the current page URL with the `ask` query parameter, and the optional `goal` query parameter: ``` GET https://docs.openg2p.org/operations/deployment/infrastructure-setup/backups/restoration/single-pvc.md?ask=&goal= ``` `ask` is the immediate question: it should be specific, self-contained, and written in natural language. `goal` is optional and describes the broader end goal you are ultimately trying to accomplish on behalf of the user. GitBook uses it to tailor the answer towards what is most useful for that goal. The response will contain a direct answer to the question and relevant excerpts and sources from the documentation. Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.