Kubernetes Cluster

OpenG2P modules and components are recommended to be run on Kubernetes (K8s), because of ease-of-use, management, and security features that K8s provides.

K8s cluster may be installed on the following infrastructures:

• Cloud-native (like EKS on AWS, or AKS on Azure)

• Non-cloud native, or on-prem (resources provisioned on a cloud or local data centre).

Here we provide instructions to set up K8s cluster on-prem.

Broadly, the steps to install are as follows:

1. Provision virtual machines

2. Set firewall rules

3. Set up the K8s cluster using Rancher's tool RKE2.

4. Install NFS client on the cluster

5. Install Istio on cluster

6. Import cluster into Rancher

7. Provide access to users

Virtual machines provisioning

Provision for virtual machines (VMs) as per configuration mentioned in Hardware Requirements. Make sure you have root privileges to the machines and have secure access to them.

Install the following tools on all machines including the one you are using to connect to the VMs.

• wget , curl , kubectl , istioctl , helm , jq

If you have SSH access to the VMs, and root privileges, you are the Super Admin. Make sure very limited access is given to the machines.

Firewall setup

Refer guide here.

Cluster installation

The following section uses RKE2 to set up the K8s cluster.

• Decide the number of K8s control-plane nodes (server nodes) and worker nodes (agent nodes).

  • Choose an odd number of control-plane nodes. For example, for a 3-node k8s cluster, choose 1 control-plane node and 2 worker nodes. For a 7-node k8s cluster, choose 3 control-plane nodes and 4 worker nodes.

• The following setup has to be done on each node on the cluster.

  • SSH into the node

  • Create the rke2 config directory

    mkdir -p /etc/rancher/rke2

  • Create a config.yaml file in the above directory, using one of the following config file templates:

    • For the first control-plane node, use rke2-server.conf.primary.template

    • For subsequent control-plane nodes, use rke2-server.conf.subsequent.template. (Make sure the token defined in the first node's control plane is used here too.)

    • For worker nodes, use rke2-agent.conf.template. (Make sure the token defined in the first node's control plane is used here too.)

  • Edit the above config.yaml file with the appropriate names, IPs, and tokens

  • Run this to download rke2.

    curl -sfL https://get.rke2.io | sh -

  • Run this to start rke2:

    • On the control-plane node, run:

      systemctl enable rke2-server
      systemctl start rke2-server

    • On the worker node, run:

      systemctl enable rke2-agent
      systemctl start rke2-agent

• To export KUBECONFIG, run (only on control-plane nodes):

• > echo -e 'export PATH="$PATH:/var/lib/rancher/rke2/bin"\nexport KUBECONFIG="/etc/rancher/rke2/rke2.yaml"' >> ~/.bashrc
  > source ~/.bashrc
  > kubectl get nodes    

• Download the Kubeconfig file rke2.yaml and keep it securely shared with only Super Admins. Rename it so that it can be identified with the cluster. This file is will be used if cluster control via Rancher is unavailable.

• Additional Reference: RKE2 High Availability Installation

NFS client provisioner

This section assumes an NFS server has already been set up. The NFS client provisioner runs on the cluster and connects seamlessly to the NFS server. Install NFS client provisioner on the cluster as follows:

• Log in to each node and run apt install nfs-common.

• On your machine run https://github.com/OpenG2P/openg2p-deployment/blob/main/kubernetes/nfs-client/install-nfs-client-provisioner.sh

• Provide the internal IP address of the NFS server (the IP must be accessible from Kubernetes nodes).

• Provide the path on which NFS was mounted on the NFS server (noted during NFS Server installation)

Longhorn

This installation only applies if Longhorn is used as storage. This may be skipped if you are using NFS.

Longhorn Install as a Rancher App

Istio

• The following setup can be done from the client machine. This installs Istio Operator, Istio Service Mesh, Istio Ingressgateway components.

• From kubernetes/istio directory, configure the istio-operator.yaml, and run;

  istioctl operator init
  kubectl apply -f istio-operator.yaml

  • If an external Loadbalancer is being used, then use the istio-operator-external-lb.yaml file.

    kubectl apply -f istio-operator-external-lb.yaml

  • Configure the operator.yaml with any further configuration

• Gather Wildcard TLS certificate and key and run; Note: To create TLS certificates refer here

  kubectl create secret tls tls-openg2p-ingress -n istio-system \
      --cert=<CERTIFICATE PATH> \
      --key=<KEY PATH>

• Create istio gateway for all hosts using this command:

  kubectl apply -f istio-gateway.yaml

  • If using external loadbalancer/external TLS termination, use the istio-gateway-no-tls.yaml file

    kubectl apply -f istio-gateway-no-tls.yaml

Cluster import to Rancher

This step assumes that a Rancher server has already been set up and operational.

• Navigate to the Cluster Management section in Rancher

• Click on Import Existing Cluster. Follow the steps to import the new OpenG2P cluster

• After importing, download kubeconfig file for the new cluster from rancher (top right on the main page), to access the cluster through kubectl from the user's machine (client), without SSH

Cluster access to users

Users may be given access to the cluster using Rancher's RBAC.

Rancher provides "Project" feature. This feature is not a standard Kubernetes feature and hence it is recommended to not use it for OpenG2P deployments.