> For the complete documentation index, see [llms.txt](https://docs.openg2p.org/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://docs.openg2p.org/operations/deployment/infrastructure-setup/backups/restoration/etcd-in-place.md).

# Etcd in-place

Restore etcd in-place on the existing compute node from an RKE2 snapshot. Use when the control plane is broken but the compute node is reusable.

This is a **cluster reset operation**. It rolls back the entire Kubernetes control plane to the moment of the chosen snapshot. Helm releases, runtime-issued certs, user-created CRs that landed after the snapshot are gone. Plan a maintenance window.

Use this when:

* The compute node's disk is intact and the OS boots, but
* Etcd is corrupted, OR
* The control plane is in a bad state you can't recover (failed upgrade, accidental `kubectl delete` of critical resources).

If the compute node is destroyed or unreachable, use [Full rebuild](/operations/deployment/infrastructure-setup/backups/restoration/full-rebuild.md) instead.

## Pre-flight

* Have the backup host reachable from your laptop.
* Have the `restic.pass` passphrase from your keystore (needed if you have to also restore the `cred/` and `tls/` dirs).
* Pick the target snapshot. List with `./openg2p-backup.sh list --config backup-config.yaml --component etcd`. The most recent valid snapshot is usually correct.
* Decide if you also need to restore RKE2 filesystem state. See "When also restore the FS state" below.

## Step 1 — Stage the snapshot on the compute node

```bash
./openg2p-backup.sh restore \
    --config backup-config.yaml \
    --component etcd \
    --target latest
```

This copies the chosen snapshot to `/tmp/openg2p-etcd-restore/` on the **current** compute node from config (`compute_private_ip` — after a rebuild that is the **new** compute). It does **not** apply the snapshot. The orchestrator prints the cluster-reset commands to run next.

Uses the backup-host orch SSH path (`Host openg2p-compute` / `openg2p-backup-orch`), so `install --force` after DR rewires the destination IP automatically.

For a specific snapshot file:

```bash
./openg2p-backup.sh restore --config backup-config.yaml --component etcd --target etcd-snapshot-compute-1-1714000000.zip
```

## Step 2 — When also restore the FS state

Skip if: you're rolling back to a snapshot taken *after* the most recent change to `/var/lib/rancher/rke2/server/tls/` and `cred/`. The directories rarely change; usually only on initial install and on encryption-at-rest enable.

If the FS state on the compute node is intact and matches the era of the snapshot, **skip this step** and go to Step 3. The cluster CA in `tls/` is what signed all the certs etcd refers to.

If the FS state is broken (compute node had a partial wipe, or you're not sure it matches), restore from the configs repo — the orchestrator pushes onto **compute**:

```bash
./openg2p-backup.sh restore --config backup-config.yaml --component configs --target rke2-tls
./openg2p-backup.sh restore --config backup-config.yaml --component configs --target rke2-cred
./openg2p-backup.sh restore --config backup-config.yaml --component configs --target rke2-token
```

Each writes under the live RKE2 paths on compute (prior trees → `*.precrash`). Do **not** restart `rke2-server` here — continue to Step 3 (cluster reset).

## Step 3 — Cluster reset and restore

On the **compute node** (under sudo):

```bash
# Stop RKE2.
sudo systemctl stop rke2-server

# Cluster reset with the snapshot path.
sudo rke2 server \
    --cluster-reset \
    --cluster-reset-restore-path=/tmp/openg2p-etcd-restore/<filename>

# When the reset completes (you'll see "etcd cluster has been reset" on stderr),
# Ctrl+C, then start the service normally.
sudo systemctl start rke2-server
```

`--cluster-reset` is a one-shot operation — it doesn't run the server, it just resets etcd and exits. After that, normal `systemctl start rke2-server` brings the cluster back.

## Step 4 — Verify

```bash
# Wait for the API to come back.
export KUBECONFIG=/etc/rancher/rke2/rke2.yaml
sudo kubectl get nodes
sudo kubectl get ns
sudo kubectl -n cattle-system get pods   # Rancher should be coming up

# From your laptop:
kubectl --kubeconfig ~/.kube/openg2p-prod get nodes
```

The cluster will look exactly as it did at the moment of the snapshot. Workloads pinned to NodePort/Ingress will reconnect; pods that depend on resources created post-snapshot won't find them.

## Step 5 — Reconcile

After the restore:

* **Postgres**: not touched (lives on storage, not compute). Still in whatever state it was when you started the restore.
* **NFS data**: not touched. Apps' on-disk state is from the present, but the cluster's view of them is from the snapshot. PVCs created post-snapshot won't exist; their data still on NFS will look like orphans. Reconcile by either (a) recreating the PVC and pointing it at the existing NFS UUID dir, or (b) accepting the data loss.
* **Resources created post-snapshot** (Helm releases of new apps, user-added secrets, custom CRs) are gone. Re-run helmfile if appropriate, or restore via [full-rebuild.md](/operations/deployment/infrastructure-setup/backups/restoration/full-rebuild.md)'s rancher-backup step targeted at specific namespaces.

## When this won't work

* Snapshot is corrupted — `etcd_verify` would have caught this. Pick an older snapshot.
* Snapshot was taken on a different RKE2 minor version than what's running now. Match versions before restoring.
* Compute node IP / hostname has changed since the snapshot. Etcd has node identity baked in. You'll need to fully rebuild instead.

## Upstream reference

* [RKE2 — Restoring a snapshot to existing nodes](https://docs.rke2.io/backup_restore#restoring-a-snapshot-to-existing-nodes)
* [RKE2 — Cluster reset](https://docs.rke2.io/backup_restore#cluster-reset)


---

# Agent Instructions
This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com.

## Querying This Documentation
If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question.

Perform an HTTP GET request on the current page URL with the `ask` query parameter, and the optional `goal` query parameter:

```
GET https://docs.openg2p.org/operations/deployment/infrastructure-setup/backups/restoration/etcd-in-place.md?ask=<question>&goal=<endgoal>
```

`ask` is the immediate question: it should be specific, self-contained, and written in natural language.
`goal` is optional and describes the broader end goal you are ultimately trying to accomplish on behalf of the user. GitBook uses it to tailor the answer towards what is most useful for that goal.

The response will contain a direct answer to the question and relevant excerpts and sources from the documentation.

Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.