Shutting Down and Restarting Tanzu Kubernetes Grid Integrated Edition

This topic lists and describes the shutdown and startup sequence for VMware Tanzu Kubernetes Grid Integrated Edition including Tanzu Kubernetes Grid Integrated Edition-provisioned Kubernetes cluster nodes, TKGI components, and (vSphere only) vSphere hosts.

Many of these operations use your IaaS dashboard, such as vSphere Client, Azure Portal, AWS Management Console, or GCP Console.

Shutdown Sequence and Tasks

To perform a graceful shutdown of all Kubernetes, Tanzu Kubernetes Grid Integrated Edition, and infrastructure components, complete the following tasks in sequence.

Step 1: Deactivate BOSH Resurrection

If you have the Enable VM Resurrector Plugin checkbox selected in the BOSH Director tile > Director Config pane, you must turn BOSH resurrection off before restarting TKGI, to prevent BOSH from recreating VMs.

To do this, run the command bosh update-resurrection off.

Step 2: Delete All PodDisruptionBudgets

To ensure that all workloads are drained as the worker nodes shutdown, remove all PodDisruptionBudgets before deleting your apps.

1. To confirm the names of your existing PodDisruptionBudgets:

   kubectl get poddisruptionbudgets -A
   

2. Back up all PodDisruptionBudgets.
   
   To back up a single PodDisruptionBudget:

   kubectl get poddisruptionbudget PDB-NAME -o yaml > PDB-NAME.yaml
   

   Where PDB-NAME is the name of one of your PodDisruptionBudgets.

3. Delete each PodDisruptionBudget until you have removed all PodDisruptionBudgets.
   
   To delete a single PodDisruptionBudget:

   kubectl edit poddisruptionbudget PDB-NAME
   

   Where PDB-NAME is the name of one of your PodDisruptionBudgets.

Step 3: Shut Down Customer Apps

Shut down all customer apps running on Tanzu Kubernetes Grid Integrated Edition-provisioned Kubernetes clusters.

Note: This task is optional. Perform it after considering the types of apps you have deployed. For example, stateful, stateless, or legacy apps.

Step 4: Shut Down Kubernetes Clusters

Shut down all Tanzu Kubernetes Grid Integrated Edition-provisioned Kubernetes clusters following the procedure defined in the How to shutdown and startup a Multi Control Plane Node TKGI cluster knowledge base article.

For each Kubernetes cluster that you intend to shut down, do the following:

1. Using the BOSH CLI, retrieve the BOSH deployment name of your Tanzu Kubernetes Grid Integrated Edition clusters by running the following command:

   bosh deployments
   

   Kubernetes cluster deployment names begin with service-instance_ and include a unique BOSH-generated hash.

2. Using the BOSH CLI, stop the Kubernetes worker nodes:

   • For a Linux worker:

     bosh -d service-instance_CLUSTER-UUID stop worker
     

   • For a Windows worker:

     bosh -d service-instance_CLUSTER-UUID stop windows-worker
     

   Where CLUSTER-UUID is the BOSH deployment name of your Tanzu Kubernetes Grid Integrated Edition cluster.
   
   For example:

    $ bosh -d service-instance_aa1234567bc8de9f0a1c stop worker 

   Note: When you use the BOSH stop command, all processes on the Kubernetes node are stopped. BOSH marks them stopped so that when the VM is powered back on, the processes do not start automatically.

3. Using the BOSH CLI, stop the Kubernetes control plane nodes by running the following command:

   bosh -d service-instance_CLUSTER-UUID stop master
   

   Where CLUSTER-UUID is the BOSH deployment name of your Tanzu Kubernetes Grid Integrated Edition cluster. For example:

    $ bosh -d service-instance_aa1234567bc8de9f0a1c stop master

4. Using your IaaS dashboard, shut down all Kubernetes node VMs. To do this, perform the following steps:

   1. Verify the node type by checking the “job” name in the Custom Attributes pane.

   2. Perform a graceful shutdown by right-clicking the target VM and selecting Power > Shut Down Guest OS.

      

      View a larger version of this image.

Step 5: Stop the TKGI Control Plane

To shut down the TKGI control plane, stop and shut down the TKGI API and TKGI Database VMs as follows:

1. Stop TKGI Control Plane Processes
2. Shut Down the TKGI API and Database VMs

Stop TKGI Control Plane Processes

To stop Tanzu Kubernetes Grid Integrated Edition control plane processes and services, do the following:

1. Using the BOSH CLI, retrieve the BOSH deployment ID of your Tanzu Kubernetes Grid Integrated Edition deployment by running the following command:

   bosh deployments
   

   The Tanzu Kubernetes Grid Integrated Edition deployment ID is pivotal-container-service- followed by a unique BOSH-generated hash.

2. Stop the TKGI control plane VM by running the following command:

   bosh -d pivotal-container-service-DEPLOYMENT-ID stop
   

   Where DEPLOYMENT-ID is the BOSH-generated ID of your Tanzu Kubernetes Grid Integrated Edition deployment.
   
   For example:

    $ bosh -d pivotal-container-service-1bf7b02738056cdc37e6 stop 

Shut Down the TKGI API and Database VMs

To shut down the TKGI API and TKGI Database VMs, do the following:

1. Run the bosh vms command to list your Tanzu Kubernetes Grid Integrated Edition control plane VMs.

   bosh -d pivotal-container-service-DEPLOYMENT-ID vms
   

   Where DEPLOYMENT-ID is the BOSH-generated ID of your Tanzu Kubernetes Grid Integrated Edition deployment.
   
   For example:

    $ bosh -d pivotal-container-service-1bf7b02738056cdc37e6 vms

2. From the bosh vms output, record:

   • The TKGI API VM name, listed under Instances as pivotal-container-service/ followed by a unique BOSH-generated hash.
   • The TKGI Database VM name, listed under Instances as pks-db/ followed by a unique BOSH-generated hash.

3. For each TKGI Database VM listed, run bosh stop to shut it down:

   bosh -d TKGI-DATABASE-VM-ID stop
   

   Where TKGI-DATABASE-VM-ID is the name of the TKGI Database VM.

4. Using your IaaS dashboard, locate and gracefully shut down the TKGI API VMs:

   • The TKGI API VMs
   • The TKGI Database VMs

   

   View a larger version of this image.

Step 6: Shut Down VMware Harbor Registry (vSphere Only)

To shut down the Harbor Registry VM, do the following:

1. Using the BOSH CLI, retrieve the BOSH deployment ID of your Harbor Registry deployment by running the following command:

   bosh deployments
   

   Harbor Registry deployment names begin with harbor-container-registry and include a unique BOSH-generated hash.

2. Using the BOSH CLI, stop the Harbor Registry VM by running the following command:

   bosh -d harbor-container-registry-DEPLOYMENT-ID stop
   

   Where DEPLOYMENT-ID is the BOSH-generated ID of your Harbor Registry deployment. For example:

    $ bosh -d harbor-container-registry-b4023f6857207b237399 stop

3. Using vCenter, locate and gracefully shut down the Harbor Registry VM.

   

   View a larger version of this image.

Step 7: Shut Down BOSH Director

Using your IaaS dashboard, locate and gracefully shut down the BOSH Director VM.

View a larger version of this image.

Step 8: Shut Down Ops Manager

Using your IaaS dashboard, locate and gracefully shut down the Ops Manager VM.

View a larger version of this image.

Step 9: Shut Down NSX-T Components (vSphere NSX-T Only)

Using vCenter, gracefully shut down all NSX-T VMs in the following order:

1. NSX-T Manager
2. NSX-T Controllers
3. NSX-T Edge Nodes

View a larger version of this image.

Step 10: Shut Down vCenter Server (vSphere Only)

To shut down the vCenter Server VM, do the following:

1. Navigate to the vCenter Appliance Management Interface at https://YOUR-VCENTER-HOSTNAME-OR-IP-ADDRESS>:5480, where YOUR-VCENTER-HOSTNAME-OR-IP-ADDRESS is the hostname or IP address that you use to connect to vCenter through the vSphere Web Client.
2. Log in as root.
3. Select Actions > Shutdown from the menu and confirm the operation.

For more information about how to shut down the vCenter Server VM, see Reboot or Shut Down the vCenter Server Appliance in the vSphere documentation and the How to stop, start, or restart vCenter Server 6.x services KB article.

Note: After you shut down this vCenter VM, the vSphere Web Client will not be available.

View a larger version of this image

Step 11: Shut Down ESXi Hosts (vSphere NSX-T Only)

To shut down each ESXi host in the vSphere cluster, do the following:

1. Put the ESXi host into maintenance mode by doing the following:

   1. Using a browser, navigate to the HTTPS IP address of the ESXi host, for example: https://10.196.146.20/.
   2. Log in using vSphere administrative credentials.
   3. Put the ESXi host in maintenance mode by selecting Actions > Enter maintenance mode.
      View a larger version of this image.

2. Power off the ESXi host. To do this, you have two options:

   • Use the EXSi web interface and select Actions > Shut down.
   • Use the remote management console for the host, such as Dell IDRAC or HP iLO.
     View a larger version of this image.

Startup Sequence and Tasks

To restart all Kubernetes, Tanzu Kubernetes Grid Integrated Edition, and infrastructure components, complete the following tasks in the sequence presented.

Step 1: Start ESXi Hosts (vSphere NSX-T Only)

To start the ESXi hosts, do the following:

1. Using the remote management console, such as Dell IDRAC or HP iLO, power on each ESXi host.
2. Connect to the web interface of each ESXi host and exit maintenance mode.

Step 2: Start vCenter (vSphere Only)

Connect to the web interface of the ESXi server that hosts the vCenter VM. Select the vCenter VM, and click Power On.

Step 3: Start NSX-T Components (vSphere NSX-T Only)

To start the NSX-T components, perform the following steps:

1. Log into vCenter using the vSphere Client.
2. Power on the following VMs in the following order:
   1. NSX-T Manager
   2. NSX-T Controllers
   3. NSX-T Edge Nodes

Step 4: Start Ops Manager

1. Using your IaaS dashboard, power on the Ops Manager VM.
2. Using a browser, go to the Ops Manager URL.
3. Enter the Ops Manager passphrase.
4. Log in to the Ops Manager UI.

Step 5: Start the BOSH Director

Using your IaaS dashboard, power on the BOSH Director VM.

Note: BOSH is aware that all the VMs under its control were stopped. BOSH does not attempt to resurrect any VMs, which is the desired behavior.

It may take approximately 90 minutes for BOSH to start properly.

To speed up the BOSH startup process:

1. Obtain the BOSH Director VM Credentials from Ops Manager. For information about doing this, see Retrieving Credentials from Your Deployment in the Ops Manager documentation.
2. SSH to the BOSH Director VM.

3. On the BOSH Director VM, run the following commands:

   sudo -i
   monit summary
   

4. If you see messages such as Process uaa Connection failed and Process credhub not monitored, then run the following command:

   monit restart uaa
   

5. After a few minutes, run the following command again:

   monit summary
   

   You should see that the uaa and credhub processes are now running. At this point, the BOSH Director should be fully up and running.

Step 6: Start the TKGI Control Plane

To start the TKGI Control Plane, do the following:

1. Using your IaaS dashboard, power on the TKGI Database VMs.

2. Restart the TKGI Database VMs as follows, depending on whether they are scaled at 1 or 3 as determined in Stop the TKGI Control Plane:

   • TKGI Database Scaled at 1:

     1. To start the TKGI Database VM:

        bosh -d TKGI-DB-VM-ID start
        

        Where TKGI-DB-VM-ID is the name of the TKGI Database VM listed by the bosh vms command.

     2. To start the TKGI Database deployment:

        bosh -d DEPLOYMENT-ID start pks-db
        

        Where DEPLOYMENT-ID is the BOSH-generated ID of the Tanzu Kubernetes Grid Integrated Edition deployment.

   • TKGI Database Scaled at 3:

     Run the bootstrap errand:

     bosh -d tkgi-db-DEPLOYMENT-ID run-errand bootstrap
     

     Where DEPLOYMENT-ID is the BOSH-generated ID of the Tanzu Kubernetes Grid Integrated Edition deployment.

     Note: For more information about the bootstrap errand, see Run the Bootstrap Errand in the VMware Tanzu SQL with MySQL for VMs documentation.

3. Using your IaaS dashboard, power on the TKGI API VMs.

4. To start each TKGI API VM listed by the bosh vms command:

   bosh -d TKGI-API-VM-ID start
   

   Where TKGI-API-VM-ID is the name of a TKGI API VM. Your environment might have as many as three TKGI API VMs.

5. To start the TKGI API deployment:

   bosh -d DEPLOYMENT-ID start pivotal-container-service
   

   Where DEPLOYMENT-ID is the BOSH-generated ID of the Tanzu Kubernetes Grid Integrated Edition deployment.

Step 7: Start Harbor Registry (vSphere Only)

To start Harbor Registry, do the following:

1. Using vCenter, power on the Harbor VM.

2. Using the BOSH CLI, start the Harbor process on the VM by running the following command:

   bosh -d harbor-container-registry-DEPLOYMENT-ID start
   

   Where DEPLOYMENT-ID is the BOSH-generated ID of your Harbor Registry deployment. For example:

    $ bosh -d harbor-container-registry-b4023f6857207b237399 start

Step 8: Start the Kubernetes Clusters

For each Kubernetes cluster that you intend to start, follow the steps in the section below that corresponds to the number of control plane nodes in the cluster:

• Start a Cluster with Three Control Plane Nodes
• Start a Cluster with Five Control Plane Nodes

Start a Cluster with Three Control Plane Nodes

Use the BOSH CLI to run the commands below. For more information about the BOSH CLI, see the BOSH CLI Documentation.

1. Retrieve the BOSH deployment name of your Tanzu Kubernetes Grid Integrated Edition cluster:

   bosh deployments
   

   Tanzu Kubernetes Grid Integrated Edition cluster deployment names begin with service-instance_ followed by the BOSH deployment name.

2. Start etcd on the master/0 node. A cluster with three members must have at least two members running to satisfy quorum, so you must first start one etcd instance.

   bosh -d DEPLOYMENT-NAME ssh master/0 "sudo monit start etcd"
   

   Where DEPLOYMENT-NAME is the BOSH deployment name retrieved in the previous step.

3. View the status of etcd on the master/0 node:

   bosh -d DEPLOYMENT-NAME ssh master/0 "sudo monit summary"
   

   Where DEPLOYMENT-NAME is the BOSH deployment name retrieved in the previous step. Wait until etcd on master/0 is running before executing the next step.

4. Start the master/1 node:

   bosh -d DEPLOYMENT-NAME start master/1
   

   Where DEPLOYMENT-NAME is the BOSH deployment name retrieved in the previous step. Wait until master/1 has status Ready before executing the next step.

5. Start the master/2 node:

   bosh -d DEPLOYMENT-NAME start master/2
   

   Where DEPLOYMENT-NAME is the BOSH deployment name retrieved in the previous step. Wait until master/2 has status Ready before executing the next step.

6. Stop etcd on the master/0 node:

   bosh -d DEPLOYMENT-NAME ssh master/0 "sudo monit stop etcd"
   

   Wait until etcd on master/0 has stopped and has status not monitored before executing the next step.

7. Start the master/0 node:

   bosh -d DEPLOYMENT-NAME start master/0
   

   Wait until master/0 has status Ready before executing the next step.

8. Start the Kubernetes worker nodes:

   bosh -d DEPLOYMENT-NAME start worker
   

Start a Cluster with Five Control Plane Nodes

Use the BOSH CLI to run the commands below. For more information about the BOSH CLI, see the BOSH CLI Documentation.

1. Retrieve the BOSH deployment name of your Tanzu Kubernetes Grid Integrated Edition cluster:

   bosh deployments
   

   Tanzu Kubernetes Grid Integrated Edition cluster deployment names begin with service-instance_ followed by the BOSH deployment name.

2. Start etcd on the master/0 and master/1 nodes. A cluster with five members must have at least three members running to satisfy quorum, so you must first start two etcd instance.

   bosh -d DEPLOYMENT-NAME ssh master/0 "sudo monit start etcd"
   bosh -d DEPLOYMENT-NAME ssh master/1 "sudo monit start etcd"
   

   Where DEPLOYMENT-NAME is the BOSH deployment name retrieved in the previous step.

3. View the status of etcd on the master/0 and master/1 nodes:

   bosh -d DEPLOYMENT-NAME ssh master/0 "sudo monit summary"
   bosh -d DEPLOYMENT-NAME ssh master/1 "sudo monit summary"
   

   Where DEPLOYMENT-NAME is the BOSH deployment name retrieved in the previous step. Wait until etcd on master/0 and master/1 is running before executing the next step.

4. Start the master/2 node:

   bosh -d DEPLOYMENT-NAME start master/2
   

   Where DEPLOYMENT-NAME is the BOSH deployment name retrieved in the previous step. Wait until master/2 has status Ready before executing the next step.

5. Start the master/3 node:

   bosh -d DEPLOYMENT-NAME start master/3
   

   Wait until master/3 has status Ready before executing the next step.

6. Start the master/4 node:

   bosh -d DEPLOYMENT-NAME start master/4
   

   Wait until master/4 has status Ready before executing the next step.

7. Stop etcd on the master/1 node:

   bosh -d DEPLOYMENT-NAME ssh master/1 "sudo monit stop etcd"
   

   Wait until etcd on master/1 has stopped and has status not monitored before executing the next step.

8. Start the master/1 node:

   bosh -d DEPLOYMENT-NAME start master/1
   

   Wait until master/1 has status Ready before executing the next step.

9. Stop etcd on the master/0 node:

   bosh -d DEPLOYMENT-NAME ssh master/0 "sudo monit stop etcd"
   

   Wait until etcd on master/0 has stopped and has status not monitored before executing the next step.

10. Start the master/0 node:

    bosh -d DEPLOYMENT-NAME start master/0
    

    Wait until master/0 has status Ready before executing the next step.

11. Start the Kubernetes worker nodes:

    bosh -d DEPLOYMENT-NAME start worker
    

Step 9: Start Customer Apps

Start all apps running on the Tanzu Kubernetes Grid Integrated Edition-provisioned Kubernetes clusters.

Step 10: Restore All PodDisruptionBudgets

After you have restarted your apps, restore your PodDisruptionBudgets.

1. To restore your PodDisruptionBudgets, re-create each of the PodDisruptionBudgets you backed up before stopping your apps.
   To re-create a single PodDisruptionBudget:

   kubectl apply -f PDB-CONFIG-NAME
   

   Where PDB-CONFIG-NAME is the name of one of the backup PodDisruptionBudget YAML configuration files that you created before stopping your apps.

Step 11: Re-enable BOSH Resurrection

Turn BOSH resurrection back on by running the command bosh update-resurrection on.