Introduction

The guide here provides instructions to deploy OpenG2P on Kubernetes (K8s) cluster.

Pre-requisites

• K8s cluster is set up as given here.

Installation of OpenG2P

• This section assumes the OpenG2P docker is already packaged. See Packaging Instructions.

• Clone the https://github.com/OpenG2P/openg2p-packaging and go to charts/openg2p directory

  • Run, (This installs the ref-impl dockers):

    Copy

    ./install.sh \
        --set global.hostname=openg2p.sandbox.net \
        --set global.selfServiceHostname=selfservice.openg2p.sandbox.net

  • If use different docker image or tag use:

    Copy

    ./install.sh \
        --set odoo.image.repository=<docker image name> \
        --set odoo.image.tag=<docker image tag> \
        --set global.hostname=openg2p.sandbox.net \
        --set global.selfServiceHostname=selfservice.openg2p.sandbox.net

Installation of ODK

• From the charts/odk-central directory, run the following to install ODK helm chart.

  Copy

  ./install.sh \
      --set global.hostname=openg2p.sandbox.net

• Note: The above helm chart uses the following docker images built from https://github.com/getodk/central/tree/v2023.1.0, since ODK Central doesn't provide pre-built docker images for these.

  Copy

  openg2p/odk-central-backend:v2023.1.0
  openg2p/odk-central-frontend:v2023.1.0
  openg2p/odk-central-enketo:v2023.1.0

• Post installation:

  • Exec into the service pod, and create user (and promote if required).

    Copy

    kubectl exec -it <service-pod> -- odk-cmd -u <email> user-create
    kubectl exec -it <service-pod> -- odk-cmd -u <email> user-promote

• Uninstallation:

  • To uninstall, just delete the helm installation of odk-central. Example:

    Copy

    helm -n odk delete odk-central

Hardware requirements

For sandbox setups

For staging setups

For production setups

TBD

Networking requirements

• All the machines in the same network.

• Public IP assigned to the Wireguard machine.

DNS requirements

The following domain names and mappings will be required. Examples:

Certificate requirements

One wildcard certificate is required at least, depending on the above domain names used. This can also be generated using letsencrypt.

Introduction

The following guide uses RKE2 to set up the Kubernetes (K8s) cluster.

Prerequisites

• The requirements for setting up the cluster are met as given here.

• The following tools are installed on all the nodes and the client machine.

  • ufw , wget , curl , kubectl , istioctl , helm , jq

Firewall setup

• Set up firewall rules on each node. The following uses ufw to setup firewall.

  • SSH into each node, and change to superuser.

  • Run the following command for each rule in the following table

  Copy

  ufw allow from <from-ip-range-allowed> to any port <port/range> proto <tcp/udp>

  • Example

  Copy

  ufw allow from any to any port 22 proto tcp
  ufw allow from 10.3.4.0/24 to any port 9345 proto tcp

  • Enable ufw.

  Copy

  ufw enable
  ufw default deny incoming

  • Additional Reference: RKE2 Networking Requirements

K8s setup

• The following setup has to be done for each cluster node.

• Choose odd number of server nodes. Example if there are 3 nodes, choose 1 server node and two agent nodes. If there are 7 nodes, choose 3 server nodes and 4 agent nodes.

• Clone the https://github.com/OpenG2P/openg2p-packaging and go to infra directory.

• For the first server node:

  • Configure rke2-server.conf.primary.template,

  • SSH into the node. Place the file to this path: /etc/rancher/rke2/config.yaml. Create the directory if not present already. mkdir -p /etc/rancher/rke2 .

  • Run this to download rke2.

    Copy

    curl -sfL https://get.rke2.io | sh -

  • Run this to start rke2 server:

    Copy

    systemctl enable rke2-server
    systemctl start rke2-server

• For subsequent server and agent nodes:

  • Configure rke2-server.conf.subsequent.template or rke2-agent.conf.template, with relevant ips for each node.

  • SSH into each node place the relevant file to this path: /etc/rancher/rke2/config.yaml, based on whether its a worker node, or control-plane node. (If worker use agent file. If control-plane use server file).

  • Run this to get download rke2.

    Copy

    curl -sfL https://get.rke2.io | sh -

  • To start rke2, use this

    Copy

    systemctl enable rke2-server
    systemctl start rke2-server

    or, based on server or agent.

    Copy

    systemctl enable rke2-agent
    systemctl start rke2-agent

• Execute these commands on a server node.

  • Copy

    echo -e 'export PATH="$PATH:/var/lib/rancher/rke2/bin"\nexport KUBECONFIG="/etc/rancher/rke2/rke2.yaml"' >> ~/.bashrc
    source ~/.bashrc

  • Copy

    kubectl get nodes

• Additional Reference: RKE2 High Availabilty Installation

Cluster import into Rancher.

• This section assumes a Rancher server has already been setup and operational. Rancher Server Setup in case not already done.

• Navigate to Cluster Management section in Rancher.

• Click on Import Existing cluster. And follow the steps to import the newly created cluster.

• After Rancher import, do not use the the kubeconfig from server anymore. Use it only via downloading kubeconfig from rancher.

Longhorn setup

• Use this to install longhorn. Longhorn Install as a Rancher App

Istio setup

• The following setup can be done from the client machine. This install Istio Operator, Istio Service Mesh, Istio Ingressgateway components.

• From infra directory, configure the istio-operator.yaml, and run;

  Copy

  istioctl operator init
  kubectl apply -f istio-operator.yaml

• Gather Wildcard TLS certificate and key and run;

  Copy

  kubectl create secret tls tls-openg2p-ingress -n istio-system \
      --cert=<CERTIFICATE PATH> \
      --key=<KEY PATH>

• Create istio gateway for all hosts using this command:

  Copy

  kubectl apply -f istio-gateway.yaml

Adding new nodes

• From infra directory, take either the rke2-server.conf.subsequent.template or rke2-agent.conf.template based on whether the new node is control plane node or Worker node. Copy this file to /etc/rancher/rke2/config.yaml in the new node.

• Configure the the config.yaml with relevant values.

• Run this to download rke2.

  Copy

  curl -sfL https://get.rke2.io | sh -

• Run this to start rke2 node:

  Copy

  systemctl enable rke2-server
  systemctl start rke2-server

Introduction

NFS-based storage is recommended for backing up DB and other data in developer/UAT/pilot environments.

Generate certificates

• Install letsencrypt and certbot.

sudo apt install letsencrypt certbot

• Generate Certificate.

sudo certbot certonly --agree-tos --manual --preferred-challenges=dns -d *openg2p.sandbox.net -d openg2p.sandbox.net

• The above command will ask for _acme-challenge, since the chosen challenge is of type DNS. Create the _acme-challenge TXT DNS record accordingly, and continue with the above prompt to certs generation.

• The generated certs should be present in /etc/letsencrypt directory.

Renew certificates

• Run the same generate certs command to renew certs.

sudo certbot certonly --agree-tos --manual --preferred-challenges=dns -d *openg2p.sandbox.net -d openg2p.sandbox.net

• The above command will generate new pair of certificates. The DNS challenge needs to be performed again, as prompted.

• Run the following to upload new certs back to Kubernetes Cluster. Adjust the certs path in the below command.

kubectl delete secret tls-openg2p-ingress -n istio-system
kubectl create secret tls tls-openg2p-ingress -n istio-system \
  --cert=/etc/letsencrypt/live/openg2p.sandbox.net-renewed/fullchain.pem \
  --key=/etc/letsencrypt/live/openg2p.sandbox.net-renewed/privkey.pem

Introduction

Rancher is used to managing multiple clusters. Being a critical component of cluster administration it is highly recommended that Rancher itself runs on a Kubernetes cluster with sufficient replication for high availability and avoiding a single point of failure.

Kubernetes cluster setup

• Set up a new RKE2 cluster. Refer to the guide.

  • Do not remove the stock ingress controller in the server config.

  • No need to install Istio.

Rancher installation

• To install Rancher use this (hostname to be edited in the below command):

  Copy

  helm repo add rancher-latest https://releases.rancher.com/server-charts/latest
  helm repo update
  helm install rancher rancher-latest/rancher \
    --namespace cattle-system \
    --create-namespace \
    --set hostname=rancher.openg2p.org \
    --set ingress.tls.source=tls-rancher-ingress

  • Configure/Create TLS secret accordingly.

  Copy

  kubectl create secret tls tls-rancher-ingress -n cattle-system \
      --cert=path/to/cert/file \
      --key=path/to/key/file

Longhorn Setup

Keycloak setup

• Copy

  helm repo add bitnami https://charts.bitnami.com/bitnami
  helm repo update
  helm install keycloak bitnami/keycloak \
    -n keycloak \
    --create-namespace \
    --version "7.1.18" \
    --set ingress.hostname=keycloak.openg2p.org \
    --set ingress.extraTls[0].hosts[0]=keycloak.openg2p.org \
    -f rancher-keycloak-values.yaml

Integrate Rancher and Keycloak