Kubernetes

Kubernetes installation guide

OpenG2P modules and components are recommended to be run on Kubernetes (K8s), because of ease-of-use, management, and security features that K8s provides.

K8s cluster may be installed on the following infrastructures:

• Cloud-native (like EKS on AWS, or AKS on Azure)

• Non-cloud native, or on-prem (resources provisioned on a cloud or local data centre).

Here we provide instructions to set up K8s cluster on-prem.

Broadly, the steps to install are as follows:

1. Provision virtual machines

2. Set firewall rules

3. Set up the K8s cluster using Rancher's tool RKE2.

4. Install NFS client on the cluster

5. Install Istio on cluster

6. Import cluster into Rancher

7. Provide access to users

Virtual machines provisioning

Provision for virtual machines (VMs) as per configuration mentioned in Hardware Requirements. Make sure you have root privileges to the machines and have secure access to them.

Install the following tools on all machines including the one you are using to connect to the VMs.

• wget , curl , kubectl , istioctl , helm , jq

If you have SSH access to the VMs, and root privileges, you are the Super Admin. Make sure very limited access is given to the machines.

Firewall setup

Refer guide here.

Cluster installation

The following section uses RKE2 to set up the K8s cluster.

• Decide on the number of control-planes (RKE2 server). For high availability the minimum of nodes running control-plane should 3. If your cluster is < 3 nodes, run only 1 control-plane (odd number). Refer RKE2 docs. The rest of the nodes are Kubernetes workers (RKE2 agent).
• The following setup has to be done on each node on the cluster.

  • SSH into the node. Execute all the below commands as root user.

  • Create the rke2 config directory

    mkdir -p /etc/rancher/rke2

  • Create a config.yaml file in the above directory, using one of the following config file templates:

    • For the first control-plane node, use rke2-server.conf.primary.template. The token can be any arbitrary string.
    • For subsequent control-plane nodes, use rke2-server.conf.subsequent.template. (Make sure the token defined in the first node's control plane is used here too.)
    • For worker nodes, use rke2-agent.conf.template. (Make sure the token defined in the first node's control plane is used here too.)
  • Edit the above config.yaml file with the appropriate names, IPs, and tokens

  • Run the following to set the RKE2 version after referring to RKE2 Releases.

    export INSTALL_RKE2_VERSION="v1.28.9+rke2r1"

  • Run this to download rke2.

    curl -sfL https://get.rke2.io | sh -

  • Run this to start rke2:

    • On the control-plane node, run:

      systemctl enable rke2-server
      systemctl start rke2-server

    • On the worker node, run:

      systemctl enable rke2-agent
      systemctl start rke2-agent

• To export KUBECONFIG, run (only on control-plane nodes):

• echo -e 'export PATH="$PATH:/var/lib/rancher/rke2/bin"\nexport KUBECONFIG="/etc/rancher/rke2/rke2.yaml"' >> ~/.bashrc
  source ~/.bashrc
  kubectl get nodes    

• Download the Kubeconfig file rke2.yaml and keep it securely shared with only Super Admins. Rename it so that it can be identified with the cluster. This file will be used if cluster control via Rancher is unavailable.

• Additional Reference: RKE2 High Availability Installation

NFS client provisioner

This section assumes an NFS server has already been set up. Install NFS client provisioner on the cluster as follows:

• Clone https://github.com/OpenG2P/openg2p-deployment.
• From kubernetes/nfs-client directory, run: (Make sure to replace the <NFS Node Internal IP> and <cluster name> parameters appropriately below)

  NFS_SERVER=<NFS Node Internal IP> \
  NFS_PATH=/srv/nfs/<cluster_name> \
      ./install-nfs-csi-driver.sh

In StorageClass, when the reclaimPolicy is set to Retainit implies that when a PVC is deleted the PV will not get deleted. And even if PV is deleted the relevant folder in NFS is not deleted.

When reclaimPolicy is set to Delete,if a PVC is deleted, both the PV and the relevant folder in the NFS get deleted.

Longhorn (optional)

This installation only applies if Longhorn is used as storage. This may be skipped if you are using NFS.

Longhorn Install as a Rancher App

Istio

Refer guide here.

Cluster import to Rancher

This step assumes that a Rancher server has already been set up and operational.

• Navigate to the Cluster Management section in Rancher

• Click on Import Existing Cluster. Follow the steps to import the new OpenG2P cluster

• After importing, download kubeconfig file for the new cluster from rancher (top right on the main page), to access the cluster through kubectl from the user's machine (client), without SSH

Cluster access to users

Users may be given access to the cluster using Rancher's RBAC.

Rancher provides "Project" feature. This feature is not a standard Kubernetes feature and hence it is recommended to not use it for OpenG2P deployments.

Backups

Your cluster may hold critical data that needs to be backed up. The following minimal backups are highly recommended:

Backup item	Recommendations
NFS data	Refer here.
Persistent Volumes (PVs)	The PVs on NFS are stored as folders that are hard to associate with original application/pod. Download the YAML of critical PVs - like Postgres, Minio etc and keep it safely. This will be required in case a cluster has to be recreated, in which case, the corresponding PVs may be mounted back from the storage provided the name of the PV is known. On Rancher the PV YAMLs are available under Storage -> PersistentVolumes of a cluster. Alternatively, you may download PVs using command line utility kubectl.