LogoLogo
1.1
1.1
  • 🏠Overview
  • 🏢Architecture
  • 📋Registration
    • Registration Interfaces
      • ODK Collect App
      • Self Service Portal
      • API Interface
      • Manual Entry
      • Bulk Export From CSV
    • ID Verification
    • Registry
  • 🅿️Program Management
    • Eligibility Assessment
    • Proxy Means Test
    • Enrolment
    • Deduplication
    • Program Cycles
    • Entitlement
    • Notifications
  • 💵Payments
    • Payment Modes
    • Payment Manager
    • Payment Batches
    • ID Account Mapper
    • Payment Rails
    • Reimbursements
      • Service Provider Portal
  • 🔐Privacy & Security
  • 📊Monitoring and Reporting
  • 🔧System Administration
    • Developer Mode
    • Role-Based Access Control
    • Internationalization (i18n)
    • Document Store
    • Apps
  • ⛓️Workflows
    • Cash grant scenario
    • Immediate Individual Assistance On-Demand
    • Service Provider Reimbursement
  • 🔗Integrations
    • M-PESA Integration
    • Mojaloop Integration
    • MOSIP Integration
      • ODK MTS Connector
      • Registry MTS Connector
      • Integration with e-Signet
  • 📦Releases
    • 1.1.0
      • Release Notes
  • ↔️REST API
  • ⚒️Testing
  • 👥Community
    • Contributing
    • Code of Conduct
  • 📗License
  • Guides
    • 📔How-To Guides
      • Install WireGuard Client on Desktop/Laptop
      • Install WireGuard Client on Android Device
      • Create User and Assign Role
      • Create a Custom Group
      • Create Program
      • Create Self-Service Portal Form
      • Map Self-Service Portal Form
      • Self Register Online
      • Integrate with MOSIP e-Signet
      • Create ODK Form
      • Provide Form Access to Field Agent
      • Download Form on ODK Collect
      • Register Offline
      • Create MTS Connector
        • Create ODK MTS Connector
        • Create OpenG2P Registry MTS Connector
      • Create Eligibility Manager Types
        • Create Default Eligibility Manager
        • Create ID Document Eligibility Manager
        • Create Phone Number Eligibility Manager
      • Create Eligibility Manager under Program
      • Configure Proxy Means Test
      • Enrol Registrants into Program
      • Configure ID Types
      • Create Deduplication Manager under Program
      • Create Deduplication Manager Types
        • Create ID Deduplication Manager
        • Create Phone Number Deduplication
      • Deduplicate Registrants
      • Create Notification Manager Types
        • Create SMS Notification Manager
        • Create Email Notification Manager
        • Create Fast2SMS Notification Manager
      • Create Notification Manager under Program
      • Send Notifications to Individual Registrants
      • Configure Program Manager under Program
      • Create and Approve Program Cycle
      • Multi-Stage Approval
      • Create Entitlement Voucher Template
      • Create Entitlement Manager Type
        • Create Default Entitlement Manager
        • Create Voucher Entitlement Manager
      • Configure Entitlement Manager under Program
      • Create Payment Manager Types
        • Create Payment Hub EE Payment Manager
        • Create Payment Interoperability Layer Payment Manager
        • Create Default Payment Manager
        • Create Cash Payment Manager
        • Create File Payment Manager
      • Configure Payment Manager in Program
      • Prepare and Send Payment
      • Install SmartScanner App
      • Submit Reimbursement Using the Service Provider Portal
      • Reimburse the service provider
    • 📙Documentation Guides
      • Creating Diagrams
      • OpenG2P Module Doc Template
      • Writing Guidelines For How-To Guides
    • 👩‍💻Developer Zone
      • Getting Started
      • Installing OpenG2P On Linux
      • Technology Stack
      • Modules
        • G2P Entitlement In-kind
        • G2P SelfServicePortal
        • OpenG2P Program Payment: G2P Connect Payment Manager
        • G2P Notifications: Wiserv SMS Service Provider
        • G2P Registry: Rest API
        • G2P Formio
        • G2P Registry: Base
        • G2P Registry: Individual
        • G2P: Proxy Means Test
    • 📘Deployment Guide
      • Deployment on Kubernetes
        • K8s Cluster Requirements
        • K8s Cluster Setup
        • Rancher Server Setup
        • NFS Server Setup
        • SSL Certificates using Letsencrypt
      • Post Install Configuration
      • Access to Deployed Setup
      • Packaging OpenG2P Docker
Powered by GitBook
LogoLogo

Copyright © OpenG2P. This work is licensed under Creative Common Attribution (CC-BY-4.0) International license unless otherwise noted.

On this page
  • Introduction
  • Prerequisites
  • Firewall setup
  • K8s setup
  • Cluster import into Rancher.
  • Longhorn setup
  • Istio setup
  • Adding new nodes
  1. Guides
  2. Deployment Guide
  3. Deployment on Kubernetes

K8s Cluster Setup

PreviousK8s Cluster RequirementsNextRancher Server Setup

Last updated 1 year ago

Introduction

The following guide uses to set up the Kubernetes (K8s) cluster.

Prerequisites

  • The requirements for setting up the cluster are met as given .

  • The following tools are installed on all the nodes and the client machine.

    • ufw , wget , curl , kubectl , istioctl , helm , jq

Firewall setup

  • Set up firewall rules on each node. The following uses ufw to setup firewall.

    • SSH into each node, and change to superuser.

    • Run the following command for each rule in the following table

    ufw allow from <from-ip-range-allowed> to any port <port/range> proto <tcp/udp>
    • Example

    ufw allow from any to any port 22 proto tcp
    ufw allow from 10.3.4.0/24 to any port 9345 proto tcp
    • Enable ufw.

    ufw enable
    ufw default deny incoming
    • Additional Reference:

Protocol
Port
Should be accessible by only
Description

TCP

22

SSH

TCP

80

Postgres ports

TCP

443

Postgres ports

TCP

5432

Postgres port

TCP

9345

RKE2 agent nodes

Kubernetes API

TCP

6443

RKE2 agent nodes

Kubernetes API

UDP

8472

RKE2 server and agent nodes

Required only for Flannel VXLAN

TCP

10250

RKE2 server and agent nodes

kubelet

TCP

2379

RKE2 server nodes

etcd client port

TCP

2380

RKE2 server nodes

etcd peer port

TCP

30000:32767

RKE2 server and agent nodes

NodePort port range

K8s setup

  • The following setup has to be done for each cluster node.

  • Choose odd number of server nodes. Example if there are 3 nodes, choose 1 server node and two agent nodes. If there are 7 nodes, choose 3 server nodes and 4 agent nodes.

  • For the first server node:

    • Configure rke2-server.conf.primary.template,

    • SSH into the node. Place the file to this path: /etc/rancher/rke2/config.yaml. Create the directory if not present already. mkdir -p /etc/rancher/rke2 .

    • Run this to download rke2.

      curl -sfL https://get.rke2.io | sh -
    • Run this to start rke2 server:

      systemctl enable rke2-server
      systemctl start rke2-server
  • For subsequent server and agent nodes:

    • Configure rke2-server.conf.subsequent.template or rke2-agent.conf.template, with relevant ips for each node.

    • SSH into each node place the relevant file to this path: /etc/rancher/rke2/config.yaml, based on whether its a worker node, or control-plane node. (If worker use agent file. If control-plane use server file).

    • Run this to get download rke2.

      curl -sfL https://get.rke2.io | sh -
    • To start rke2, use this

      systemctl enable rke2-server
      systemctl start rke2-server

      or, based on server or agent.

      systemctl enable rke2-agent
      systemctl start rke2-agent
  • Execute these commands on a server node.

    • echo -e 'export PATH="$PATH:/var/lib/rancher/rke2/bin"\nexport KUBECONFIG="/etc/rancher/rke2/rke2.yaml"' >> ~/.bashrc
      source ~/.bashrc
    • kubectl get nodes

Cluster import into Rancher.

  • Navigate to Cluster Management section in Rancher.

  • Click on Import Existing cluster. And follow the steps to import the newly created cluster.

  • After Rancher import, do not use the the kubeconfig from server anymore. Use it only via downloading kubeconfig from rancher.

Longhorn setup

Istio setup

  • The following setup can be done from the client machine. This install Istio Operator, Istio Service Mesh, Istio Ingressgateway components.

  • istioctl operator init
    kubectl apply -f istio-operator.yaml
    • If an external Loadbalancer is being used, then use the istio-operator-external-lb.yaml file.

      kubectl apply -f istio-operator-external-lb.yaml
    • Configure the operator.yaml with any further configuration.

  • Gather Wildcard TLS certificate and key and run;

    kubectl create secret tls tls-openg2p-ingress -n istio-system \
        --cert=<CERTIFICATE PATH> \
        --key=<KEY PATH>
  • Create istio gateway for all hosts using this command:

    kubectl apply -f istio-gateway.yaml
    • If using external loadbalancer/external TLS termination, use the istio-gateway-no-tls.yaml file.

      kubectl apply -f istio-gateway-no-tls.yaml

Adding new nodes

  • Configure the the config.yaml with relevant values.

  • Run this to download rke2.

    curl -sfL https://get.rke2.io | sh -
  • Run this to start rke2 node:

    systemctl enable rke2-server
    systemctl start rke2-server

Clone the and go to directory.

Additional Reference:

This section assumes a Rancher server has already been setup and operational. in case not already done.

Use this to install longhorn.

From directory, configure the istio-operator.yaml, and run;

From directory, take either the rke2-server.conf.subsequent.template or rke2-agent.conf.template based on whether the new node is control plane node or Worker node. Copy this file to /etc/rancher/rke2/config.yaml in the new node.

📘
RKE2
here
RKE2 Networking Requirements
https://github.com/OpenG2P/openg2p-deployment
kuberenetes/rke2
RKE2 High Availabilty Installation
Rancher Server Setup
Longhorn Install as a Rancher App
kuberenetes/istio
kuberenetes/rke2