# OpenG2P K8s Cluster Setup ## Introduction OpenG2P modules and components are recommended to be run on Kubernetes (K8s), because of ease-of-use, management, and security features that K8s provides. This document provides instructions to set up a K8s Cluster on which OpenG2P Modules and other components can be installed. ## Prerequisites * [Hardware Requirements](/1.2/deployment/infrastructure-setup/k8s-cluster-requirements.md) * The following tools are installed on all the nodes and the client machine: * `wget` , `curl` , `kubectl` , `istioctl` , `helm` , `jq` ## Firewall Requirements * Set up firewall rules on each node according to the following table. The exact method to set up the firewall rules will vary from cloud to cloud and on-prem. (For example on AWS, EC2 security groups can be used. For on-prem cluster, ufw can be used. Etc.)
ProtocolPortShould be accessible by onlyDescription
TCP22SSH
TCP80Postgres ports
TCP443Postgres ports
TCP5432Postgres port
TCP9345RKE2 agent nodesKubernetes API
TCP6443RKE2 agent nodesKubernetes API
UDP8472RKE2 server and agent nodesRequired only for Flannel VXLAN
TCP10250RKE2 server and agent nodeskubelet
TCP2379RKE2 server nodesetcd client port
TCP2380RKE2 server nodesetcd peer port
TCP30000:32767RKE2 server and agent nodesNodePort port range
* For example, this is how you can use `ufw` to set up the firewall on each cluster node. * SSH into each node, and change to superuser * Run the following command for each rule in the above table ``` ufw allow from to any port proto ``` * Example: ``` ufw allow from any to any port 22 proto tcp ufw allow from 10.3.4.0/24 to any port 9345 proto tcp ``` * Enable ufw: ``` ufw enable ufw default deny incoming ``` * Additional Reference: [RKE2 Networking Requirements](https://docs.rke2.io/install/requirements#networking) ## Installation on AWS cloud If you are using AWS only to get EC2 nodes, and you want to set up the K8s cluster manually, move to the [On-premises Setup](#installation-on-premises-on-prem). ## Installation on-premises (on-prem) ### k8s cluster setup The following section uses [RKE2](https://docs.rke2.io) to set up the K8s cluster. * Decide the number of K8s Control plane nodes(server nodes) and worker nodes(agent nodes) * Choose an odd number of control-plane nodes. For example, for a 3-node k8s cluster, choose 1 control-plane node and 2 worker nodes. For a 7-node k8s cluster, choose 3 control-plane nodes and 4 worker nodes. * The following setup has to be done on each node on the cluster: * SSH into the node * Create the rke2 config directory: ``` mkdir -p /etc/rancher/rke2 ``` * Create a `config.yaml` file in the above directory, using one of the following config file templates: * For the first control-plane node, use [rke2-server.conf.primary.template](https://github.com/OpenG2P/openg2p-deployment/blob/main/kubernetes/rke2/rke2-server.conf.primary.template) * For subsequent control-plane nodes, use [rke2-server.conf.subsequent.template](https://github.com/OpenG2P/openg2p-deployment/blob/main/kubernetes/rke2/rke2-server.conf.primary.template) * For worker nodes, use [rke2-agent.conf.template](https://github.com/OpenG2P/openg2p-deployment/blob/main/kubernetes/rke2/rke2-agent.conf.template) * Edit the above `config.yaml` file with the appropriate names, IPs, and tokens * Run this to download rke2. ``` curl -sfL https://get.rke2.io | sh - ``` * Run this to start rke2: * On the control-plane node, run: ``` systemctl enable rke2-server systemctl start rke2-server ``` * On the worker node, run: ``` systemctl enable rke2-agent systemctl start rke2-agent ``` * To export KUBECONFIG, run (only on control-plane nodes): * ``` echo -e 'export PATH="$PATH:/var/lib/rancher/rke2/bin"\nexport KUBECONFIG="/etc/rancher/rke2/rke2.yaml"' >> ~/.bashrc source ~/.bashrc ``` * ``` kubectl get nodes ``` * Additional Reference: [RKE2 High Availability Installation](https://docs.rke2.io/install/ha) ### Cluster import into Rancher This section assumes a Rancher server has already been set up and operational. [Rancher Server Setup](/1.2/deployment/infrastructure-setup/rancher.md) in case not already done. * Navigate to Cluster Management section in Rancher * Click on `Import Existing` cluster. And follow the steps to import the new OpenG2P cluster * After importing, download kubeconfig for the new cluster from rancher (top right on the main page), to access the cluster through kubectl from user's machine (client), without SSH ### NFS provisioner setup This section assumes an NFS Server has already been set up and operational for providing storage volumes to this K8s cluster, with requirements as given in [NFS Server Setup](/1.2/deployment/infrastructure-setup/nfs-server.md). This section assumes an NFS server has already been set up and operational, which meets the requirements, as given in [NFS Server Setup](/1.2/deployment/infrastructure-setup/nfs-server.md). This NFS server is used to provide persistent storage volumes to this K8s cluster. ### Longhorn setup This installation only applies if Longhorn is used as storage. This may be skipped if you are using NFS. [Longhorn Install as a Rancher App](https://longhorn.io/docs/1.3.2/deploy/install/install-with-rancher/) ### Istio setup * The following setup can be done from the client machine. This install Istio Operator, Istio Service Mesh, Istio Ingressgateway components. * From [kubernetes/istio](https://github.com/OpenG2P/openg2p-deployment/tree/main/kubernetes/istio) directory, configure the istio-operator.yaml, and run; ``` istioctl operator init kubectl apply -f istio-operator.yaml ``` * If an external Loadbalancer is being used, then use the `istio-operator-external-lb.yaml` file. ``` kubectl apply -f istio-operator-external-lb.yaml ``` * Configure the operator.yaml with any further configuration * Gather Wildcard TLS certificate and key and run; ``` kubectl create secret tls tls-openg2p-ingress -n istio-system \ --cert= \ --key= ``` * Create istio gateway for all hosts using this command: ``` kubectl apply -f istio-gateway.yaml ``` * If using external loadbalancer/external TLS termination, use the `istio-gateway-no-tls.yaml` file ``` kubectl apply -f istio-gateway-no-tls.yaml ``` ### Adding new nodes * From [kubernetes/rke2](https://github.com/OpenG2P/openg2p-deployment/tree/main/kubernetes/rke2) directory, take either the `rke2-server.conf.subsequent.template` or `rke2-agent.conf.template` based on whether the new node is control plane node or Worker node. Copy this file to `/etc/rancher/rke2/config.yaml` in the new node. * Configure the the config.yaml with relevant values * Run this to download rke2. ``` curl -sfL https://get.rke2.io | sh - ``` * Run this to start rke2 node: ``` systemctl enable rke2-server systemctl start rke2-server ``` --- # Agent Instructions: Querying This Documentation If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question. Perform an HTTP GET request on the current page URL with the `ask` query parameter: ``` GET https://docs.openg2p.org/1.2/deployment/infrastructure-setup/cluster-setup.md?ask= ``` The question should be specific, self-contained, and written in natural language. The response will contain a direct answer to the question and relevant excerpts and sources from the documentation. Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.