ETCD Backup and Restore

Guide to backup Kubernetes Master ETCD

RKE2 already takes scheduled etcd snapshots automatically. By default RKE2 saves an etcd snapshot every 12 hours and retains the last 5, under /var/lib/rancher/rke2/server/db/snapshots/ on each control-plane node. The production install runs a single control-plane node (Production-Minimum), so that node holds the cluster's etcd snapshots — copy them off-node (e.g. to the NFS/Storage node or object storage) so a node loss doesn't take the backups with it. This guide covers taking an on-demand snapshot and restoring from one.

Backup of the etcd data of all clusters to ensures recovery in case of a complete failure. Here is how to create and restore backups of the etcd data in an RKE2 cluster. Note: /var/lib/rancher/rke2 is the default data directory for rke2. In RKE2, snapshots are stored on each etcd node. If you have multiple etcd or etcd + control-plane nodes, you will have multiple copies of local etcd snapshots.

You can take a snapshot manually while RKE2 is running with the etcd-snapshot subcommand. For example: rke2 etcd-snapshot save --name pre-upgrade-snapshot.

Restoring a snapshot to existing nodes

When RKE2 is restored from backup, the old data directory will be moved to /var/lib/rancher/rke2/server/db/etcd-old-%date%/. RKE2 will then attempt to restore the snapshot by creating a new data directory and start etcd with a new RKE2 cluster with one etcd member.

1. You must stop RKE2 service on all server nodes if it is enabled via systemd. Use the following command to do so: systemctl stop rke2-server

2. Initiate the restore from the snapshot on the first server node with the following commands: rke2 server \ --cluster-reset \ --cluster-reset-restore-path=<PATH-TO-SNAPSHOT>

3. Once the restore process is complete, start the rke2-server service on the first server node as follows: systemctl start rke2-server

4. Remove the rke2 db directory on the other server nodes as follows: rm -rf /var/lib/rancher/rke2/server/db

5. Start the rke2-server service on other server nodes with the following command: systemctl start rke2-server

Restoring a snapshot to new nodes

Note: For all versions of rke2 v.1.20.9 and prior, you will need to back up and restore certificates first due to a known issue in which bootstrap data might not save on restore (Steps 1 - 3 below assume this scenario). See note below for an additional version-specific restore caveat on restore.

1. Back up the following: /var/lib/rancher/rke2/server/cred, /var/lib/rancher/rke2/server/tls, /var/lib/rancher/rke2/server/token, /etc/rancher

2. Restore the certs in Step 1 above to the first new server node.

3. Install rke2 v1.20.8+rke2r1 on the first new server node as in the following example: curl -sfL https://get.rke2.io | INSTALL_RKE2_VERSION="v1.20.8+rke2r1" sh -

4. Stop RKE2 service on all server nodes if it is enabled and initiate the restore from snapshot on the first server node with the following commands: systemctl stop rke2-server rke2 server \ --cluster-reset \ --cluster-reset-restore-path=<PATH-TO-SNAPSHOT>

5. Once the restore process is complete, start the rke2-server service on the first server node as follows: systemctl start rke2-server

6. You can continue to add new server and worker nodes to cluster.