Adding Infrastructure Password Changes to the Tanzu Kubernetes Grid Integrated Edition Tile

This topic describes how to manage VMware Tanzu Kubernetes Grid Integrated Edition (TKGI) after changing a BOSH Director or TKGI service account password.

Manage Your Service Account Passwords

When you installed Tanzu Kubernetes Grid Integrated Edition you created two service accounts:

BOSH/Ops Manager Service Account: This service account is configured in the BOSH Director tile.
Master Node Service Account: This service account is configured in the Tanzu Kubernetes Grid Integrated Edition tile.

You must update a tile’s copy of a service account password after changing the password on your network.

Step 1: Update Your Service Account Passwords

To update BOSH Director with a new BOSH/Ops Manager Service Account password, perform the following steps:

Access the Installation Dashboard in Ops Manager.
Select the BOSH Director tile.
Select your IaaS’ Config tab.
Click Change, the link beneath the IaaS Password field, to modify the password.
Enter the new service account password.
Note:The vCenter user name must have the format user@domainname, for example: “[email protected]”. It cannot contain a backslash (\) character; TKGI does not support user names of the format Domain\User.
Click Save to save the new password to the BOSH Director tile.

To update Tanzu Kubernetes Grid Integrated Edition with a new Master Node Service Account password, perform the following steps:

Access the Installation Dashboard in Ops Manager.
Select the Tanzu Kubernetes Grid Integrated Edition tile.
Select the Kubernetes Cloud Provider tab.
Click Change, the link beneath your IaaS’ Master Credentials field, to modify the password.
Enter the new control plane node service account password.
Note:The vCenter user name must have the format user@domainname, for example: “[email protected]”. It cannot contain a backslash (\) character; TKGI does not support user names of the format Domain\User.
Click Save to save the new password to the Tanzu Kubernetes Grid Integrated Edition tile.

Step 2: Deploy Your New Service Account Passwords

After updating an Ops Manager tile’s service account password you must also deploy the new password.

To deploy a new password to BOSH Director and Tanzu Kubernetes Grid Integrated Edition, perform the following steps:

Access the Installation Dashboard in Ops Manager.
Click Review Pending Changes.
In the Errands section for Tanzu Kubernetes Grid Integrated Edition, select Update all clusters errand.
Click Apply Changes to update the Tanzu Kubernetes Grid Integrated Edition installation with the new passwords.

Note: The Update all clusters errand must be enabled to update the Kubernetes cloud provider password stored in Kubernetes clusters.

Manage Your NSX Manager Password (vSphere and vSphere with NSX-T only)

If you are on vSphere or vSphere with NSX-T only, you also configured the NSX Manager Account and password when you installed Tanzu Kubernetes Grid Integrated Edition. This service account is configured in the BOSH Director tile.

After changing the password on your network, you must also update the BOSH Director tile’s copy of the NSX Manager Account password.

To update the BOSH Director with the new NSX Manager password, perform the following steps:

Access the Installation Dashboard in Ops Manager.
Select the BOSH Director tile.
Select the vCenter Config tab.
Click Change, the link beneath the NSX Username field, to modify the password.
Enter the new password.
Click Save to save the changes to the BOSH Director tile.
On the Ops Manager Installation Dashboard, select Review Pending Changes.
Click Apply Changes.

Troubleshooting

‘Failed to authenticate user’ Error When Cluster Service Account Authenticates

Symptom

Your cluster control plane node does not authenticate with your vCenter even though the cluster’s /var/vcap/jobs/kube-controller-manager/config/cloud-provider.ini file includes the correct vCenter credentials.

You see errors similar to the following in your logs:

Service account errors in the TKGI logs:

error ... Failed to authenticate user ...

Authentication errors in the BOSH tasks logs:

WARN -- [req_id ...]: Error running method 'Login'. Failed with message 'Cannot complete login due to an incorrect user name or password.'.
Rescued Unknown: Cannot complete login due to an incorrect user name or password.. backtrace: /var/vcap/data/packages/vsphere_cpi/.../lib/cloud/vsphere/retryer.rb:13:in `try'

Additionally, the cluster control plane repeatedly attempts to authenticate with vCenter, degrading the state of the STS service, and your vCenter becomes inaccessible.

Solution

To resolve this issue:

Confirm that the correct credentials are included in your cluster’s /var/vcap/jobs/kube-controller-manager/config/cloud-provider.ini file. If the credentials are incorrect, update the configuration with the valid credentials.
If the configured credentials are correct, review your cluster’s actual vCenter authentication credentials:
```
bosh ssh -d DEPLOYMENT-NAME master/0 "sudo cat /var/vcap/jobs/csi-controller/config/csi-vsphere.conf"  
```
Where DEPLOYMENT-NAME is the BOSH deployment name for the cluster that does not authenticate.
If the configured credentials and actual credentials are different, you must upgrade the cluster.