Patching the CaaS Deployment Using the TCX CaaS Installer

Patching refers to applying an update to one or many components of the CaaS deployment. The update could be a simple configuration change like changing the location of a log file or a major upgrade of the Load Balancer, Storage interfaces or the Harbor container registry. The CaaS installer is implemented using Ansible so the patching process leverages the idempotency, failure resiliency and capabilities to skip/select specific tasks provided by Ansible and its modules.

Depending on the component(s) to be patched, it may involve an execution of all the playbooks or a limited execution of specific roles and tasks using Ansible tags.

Prerequisites

• The same pre-requisites as Day 0 CaaS installation applies. In addition, the following pre-requisites must be met.
  • vars.yml file
    • Your vars file is required for **all** patching tasks.
  • Inventory file
    • The inventory file is required for **all** patching tasks.
    • If you don't have the inventory file from your Day 0 installation, run the following command to re-generate it. The file will be generated at /root/k8s-installer/inventory/<your-cluster-name>/hosts.yml inside the deployment container.

      root [ ~ ]# cd $HOME/k8s-installer/
      root [ ~/k8s-installer ]# ansible-playbook scripts/ansible/prepare.yml -e @scripts/ansible/vars.yml --become --tags generate-inventory

      Note: You do not need to run this step separately if you are patching the system by doing an end-to-end re-installation. You can perform the following procedure..

Procedure

• In many cases, it is recommended to just re-run all the playbooks similar to an end-to-end installation specified in the Deployment Procedure section of the Kubernetes Cluster Deployment chapter in the VMware Telco Cloud Service Assurance Deployment Guide..
  • Here are few examples of such cases
    • If the patch has new container images for the core Kubernetes pods, the storage/networking interfaces (MetalLB, vsphere CSI, etc.) or the Harbor container registry.
    • If the patch has new linux binaries for the non-containerized components of CaaS layer (kubeadm, etcd, kubelet, etc.).
    • If the patch updates several components in CaaS layer.
• For simpler cases where the patch applies a new configuration to a single component, it is more optimal to limit the scope of execution to just that component.
  • Here are examples of such cases
    • Redeploying the Harbor registry with a new IP address.
    • Fixing a mistake in the vcenter configuration of the vsphere-csi plugin in vars.yml.
  • For such simple cases, it is inefficient and time consuming to re-run all the playbooks especially at larger scale. Here are the list of commands for patching individual components
    • Patch Harbor (Container Registry)

      root [ ~ ]# cd $HOME/k8s-installer/
      root [ ~/k8s-installer ]# ansible-playbook -i inventory/<your-cluster-name>/hosts.yml scripts/ansible/post_install.yml -u <your-SSH-username> --become -e @scripts/ansible/internal_vars.yml -e @scripts/ansible/vars.yml --tags harbor,postgres-ha

    • Regenerate Harbor certificates, redeploy Harbor and copy new certificates to all nodes.
      Note: Harbor certificate regeneration is needed if new IP is updated for harbor.

      root [ ~ ]# cd $HOME/k8s-installer/
      root [ ~/k8s-installer ]# ansible-playbook -i inventory/<your-cluster-name>/hosts.yml scripts/ansible/post_install.yml -u <your-SSH-username> --become -e @scripts/ansible/internal_vars.yml -e @scripts/ansible/vars.yml --tags harbor,postgres-ha,registry-cert-generation,copy-registry-cert

    • Metal LB (Load Balancer)

      root [ ~ ]# cd $HOME/k8s-installer/
      root [ ~/k8s-installer ]# ansible-playbook -i inventory/<your-cluster-name>/hosts.yml scripts/ansible/post_install.yml -u <your-SSH-username> --become -e @scripts/ansible/internal_vars.yml -e @scripts/ansible/vars.yml --tags load-balancer

    • vSphere CSI plugin (Storage)

      root [ ~ ]# cd $HOME/k8s-installer/
      root [ ~/k8s-installer ]# ansible-playbook -i inventory/<your-cluster-name>/hosts.yml scripts/ansible/post_install.yml -u <your-SSH-username> --become -e @scripts/ansible/internal_vars.yml -e @scripts/ansible/vars.yml --tags vsphere-csi