Upgrading Tanzu Mission Control Self-Managed

To upgrade Tanzu Mission Control (TMC) Self-Managed, download the installer and follow the steps provided below.

Upgrade Tanzu Mission Control Self-Managed

Do the following to upgrade Tanzu Mission Control Self-Managed.

Download the installer to the bootstrap computer (Jumpbox VM).

You can download the installer from Broadcom Support.

Extract the contents of the tarball to a new directory.

mkdir tmc-sm-1.3.0
tar -xf tmc-self-managed-1.3.0.tar -C ./tmc-sm-1.3.0

Push the images to the project in the container registry that was set up for the initial installation.
```
tmc-sm-1.3.0/tmc-sm push-images harbor --project {{harbor-instance}}/{{harbor-project}} --username {{username}} --password {{password}}
```
After the push command completes successfully, you see the file pushed-package-repository.json in the tmc-sm-1.3.0 directory. The file includes the following contents:
```
{"repositoryImage":"{{harbor-instance}}/{{harbor-project}}/package-repository","version":"1.3.0"}
```
Make a note of the repositoryImage and version from the pushed-package-repository.json file.

If you use the Tanzu Standard package repository to deploy packages on workload clusters through TMC, you must also update the version of the Tanzu Standard package repository in your environment, as described in Copy Tanzu Standard package repository images. For example:

TANZU_STANDARD_VERSION=v2024.5.16
IMAGE_REGISTRY_CERT_PATH=ca.crt
imgpkg copy -b --registry-ca-cert-path=${IMAGE_REGISTRY_CERT_PATH}  projects.registry.vmware.com/tkg/packages/standard/repo:${TANZU_STANDARD_VERSION}" --to-repo "${TMC_PROJECT}/498533941640.dkr.ecr.us-west-2.amazonaws.com/packages/standard/repo"

You can optionally configure TMC Self-Managed to use a specific version of the Tanzu Standard package repository. For more information, see Create a values.yaml file.

Update the package repository version for tanzu-mission-control-packages in the tmc-local namespace.

tanzu package repository update tanzu-mission-control-packages --url "{{repositoryImage}}:{{version}}" --namespace tmc-local

After kapp-controller reconciles the Tanzu packages in the repository, verify the package version.
```
tanzu package repository list --namespace tmc-local
```

Update the installed package with the new version.

tanzu package installed update tanzu-mission-control -p tmc.tanzu.vmware.com --version "{{version}}" --values-file /home/kubo/values.yaml --namespace tmc-local

Notes
- Use the same values.yaml created during the initial installation.
- Use the version from the JSON file created in step 3.

You might see one or more Template failed errors like the one shown below. You can safely ignore errors like this during the upgrade process.

Template failed 
  | ytt: Error: Overlaying data values (in following order: additional data values): 
  | One or more data values were invalid
  | ====================================
  | 
  | Given data value is not declared in schema
  | values.yaml:
  |      |
  |   73 | tanzuStandard:
  |      |
  | 
  |      = found: tanzuStandard
  |      = expected: a key declared in map (from schema.yaml:2)
  | 
  | 
  | 
  | Templating dir: Error (see .status.usefulErrorMessage for details)

After kapp-controller reconciles the Tanzu packages in the repository, verify the package version.
```
tanzu package repository list --namespace tmc-local
```

Troubleshooting TMC Self-Managed upgrade

The following describes how to troubleshoot if you encounter failures during the upgrade.

Package reconciliation failure

If the package upgrade fails indicating Reconcile failed, use the following command to get more details.

kubectl -n tmc-local describe packageinstall \<packageinstall-name\>

Occasionally, some services take a little longer to come up. This causes the failure during the upgrade for packageinstall/tanzu-mission-control when the package is in Reconciling state for more than 15 minutes.

Verify the failure by checking the status.usefulErrorMessage field in the output of the following command for the package that encountered failure.

kubectl -n tmc-local describe packageinstall \<packageinstall-name\>

From the error message, note the service name and then verify the status of the pod with the following commands:

kubectl -n tmc-local get pods | grep service-name
kubectl -n tmc-local describe pod \<pod-name\>
kubectl -n tmc-local logs \<pod-name\>

Monitoring service failure

Upgrade can fail when the Prometheus service fails to start. You will see the following error:

Error: packageinstall/tanzu-mission-control (packaging.carvel.dev/v1alpha1) namespace: tmc-local: Reconciling: kapp: Error: waiting on reconcile packageinstall/tmc-local-monitoring (packaging.carvel.dev/v1alpha1) namespace: tmc-local:
Finished unsuccessfully (Reconcile failed: (message: Error (see .status.usefulErrorMessage for details))). Reconcile failed: Error (see .status.usefulErrorMessage for details)

If you see this error, try the following steps:

Verify the status of the Prometheus pod.

kubectl -n tmc-local get pod prometheus-server-tmc-local-monitoring-tmc-local-0

If the pod is in CrashLoopBackOff state, retrieve the pod logs with the following command:
```
kubectl -n tmc-local logs prometheus-server-tmc-local-monitoring-tmc-local-0
```
If the logs show failure with the error panic: preallocate: no space left on device, then the PVC associated with the prometheus requires resizing.

Update the size of the PVC. Set data-prometheus-server-tmc-local-monitoring-tmc-local-0 to 10 Gi and restart the pod.

kubectl -n tmc-local edit pvc data-prometheus-server-tmc-local-monitoring-tmc-local-0
kubectl -n tmc-local delete pod prometheus-server-tmc-local-monitoring-tmc-local-0