Issues Registering a Supervisor Cluster

If you’re having trouble registering a Supervisor Cluster running in vSphere with Tanzu with Tanzu Mission Control, there are a number of steps you can take to identify the root cause and fix the problem.

Problem

You might suspect you’re having trouble registering your Supervisor Cluster if:

• When registering your Supervisor Cluster, the verification step fails.

• If you’re registering a Supervisor Cluster through the CLI, the operation times out. 

• If you’re registering a Supervisor Cluster through the API, the status of the cluster remains pending for longer than 7 minutes.

Cause

Cause 

There are a few common reasons you might have trouble registering your Supervisor Cluster with Tanzu Mission Control. You can follow the solution described to identify which of these causes is the issue and help you correct it. 

• You might need to rerun the registration step if the agent installation failed. Common reasons for failure:

  • You might have tried to attach instead of register the Supervisor Cluster.

  • You might have tried to register a Supervisor Cluster using the Tanzu Kubernetes Grid procedure and seen a message like “no such resource found.”

• You might have a resource scheduling issue or permission issue preventing the Tanzu Mission Control agent from installing successfully. 

• You might have a problem with connectivity or proxy configuration.

Solution

1. Check the format of the registration URL to verify you’re using the procedure to register a vSphere with Tanzu. You should see source=registration and type=tkgs. For example:

   https://myorgname.tmc.cloud.vmware.com/installer?id=1234&source=registration&type=tkgs.

   If you see attach or tkgm in the URL, you need to restart the registration process. Make sure you follow the procedure described in Register a Management Cluster with Tanzu Mission Control.

2. Check if the problem is your access rights or network connection to the by running a simple kubectl command like get namespaces.

   kubectl get namespaces

   If you cannot access the cluster, contact your kubernetes administrator.

3. Verify that the tmc namespace exists. The tmc namespace is in the form of svc-tmc-N.

   kubectl get namespaces | grep svc-tmc

   svc-tmc-c8                                  Active   30h

   If a tmc namespace doesn't exist, you might be registering an unsupported version of Tanzu Kubernetes Grid.

4. Check if the problem is the agent installation failure. Reinstall the agent if the installation failed.
   1. Verify if the agent install succeeded by checking if AgentInstall exists.

      kubectl get agentinstall -n svc-tmc-aN

      NAME                         AGE
      
      tmc-agent-installer-config   4s

      If the AgentInstall doesn’t exist, try again to register your supervisor cluster.

   2. Check the AgentInstall status field to understand if there were any issues with the installation and how to correct them.

      kubectl describe agentinstall -n svc-tmc-aN  tmc-agent-installer-config

      Name:         tmc-agent-installer-config
      
      Namespace:    svc-tmc-aN
      
      ...
      
      Status:  
      
      Message:      Installation inprogress  
      
      Retry Count:  5  
      
      Status:       INSTALLATION_IN_PROGRESS
      
      Events:         <none>

5.  Check if there was an issue applying the agent configuration.

   kubectl get configmap -n svc-tmc-aN stack-config  

   NAME           DATA   AGE
   
   stack-config   9      9d

   If the agent configuration doesn’t exist:

   • Delete the agent kubectl delete AgentInstall -n svc-tmc-aN tmc-agent-installer-config

   • Try again to register your supervisor cluster.

6. Investigate the agent pods to locate the source of the problem.
   1. Check the status of the agent pods.

      kubectl get pods -n svc-tmc-aN -l "tmc-extension-name in (extension-manager,extension-updater,agent-updater)"

      NAME                                   READY   STATUS              RESTARTS      AGE
      
      agent-updater-cc9c67b4d-sp6q9          0/1    CLBO             0             3m
      
      agentupdater-workload-27478576-8rxvk   0/1    CLBO           0             60s
      
      extension-manager-85f9cf5497-rk8ms     0/1     CLBO             0             3m
      
      extension-updater-6c4c95c5bf-9wl6p     0/1    CLBO             0             3m

   2. If the pods are in a CLBO (crash loop backoff), retry the registration procedure and verify you’re using the vSphere with Tanzu registration procedure instead of the Tanzu Kubernetes Grid registration procedure.
   3. If any of the pods are in a failed state, fetch the logs for the failed pod to learn more information about the problem. For example:

      kubectl logs -l "tmc-extension-name=extension-updater" -n svc-tmc-aN

      For help solving pod failures, see Common Log Errors and Solutions for Pod Failure in the Examples section of this page.

   4. If the pods have not been scheduled (they appear as “pending” or not at all), inspect the deployments of extension-manager, extension-updater and agent-updater to find any deployments that are not in a ready state.

      kubectl get deployments -n svc-tmc-aN -l "tmc-extension-name in (extension-manager,extension-updater,agent-updater)"

      NAME                READY   UP-TO-DATE   AVAILABLE   AGE
      
      agent-updater       1/11            1           3m
      
      extension-manager   1/11            1           3m
      
      extension-updater   1/11            1           3m

   5. If any of the deployments show as 0/1 , describe the deployment to understand how to correct the issue. For example:

      kubectl describe deployment extension-updater -n svc-tmc-aN

      These kubernetes errors are usually resource related. For example, you might get <output>OutOfDisk.

      Ask your kubernetes administrator to solve these types of errors by expanding resources or relaxing policies.

7. If the deployments are all showing 1/1 READY, look for Events messages in the replicasets to understand how to correct the issue. 
   1. Look for replicasets that are not in the ready status. For example:

      kubectl get replicaset -n svc-tmc-aN -l "tmc-extension-name in (extension-manager,extension-updater,agent-updater)"

   2. Inspect any replicasets that are not in ready status and check the events messages in the replicasets to understand how to correct the issue. For example:

      kubectl describe replicaset -n vmware-system-tmc extension-updater-6c4c95c5bf

   The Events messages should contain enough information for you to solve the issue.

Example

The following tables contain examples of common errors and how to solve them.

Table 1. How to Solve Common AgentInstall Status Messages

Status	Message	Solutions
INSTALLATION_IN_PROGRESS	Installation in progress	Wait until the installation completes.
INSTALL_FAILED	Registration link domain name verification failed	Check the registration link. Make sure you see something like, https://myorgname.tmc.cloud.vmware.com/installer?id=1234&source=registration&type=tkgs.
INSTALL_FAILED	Registration link is not an HTTPS link	Check the registration link. Make sure you see something like, https://myorgname.tmc.cloud.vmware.com/installer?id=1234&source=registration&type=tkgs.
INSTALL_FAILED	Get proxy options	Verify the proxy configured in the cluster.
INSTALL_FAILED	Download manifest from link	Make sure you downloaded and prepared the YAML from the Tanzu Mission Control Console as described in Complete the Registration of a Supervisor Cluster in vSphere with Tanzu. You might need to retry your registration procedure from the beginning.
INSTALL_FAILED	Parse manifest	Make sure you downloaded and prepared the YAML from the Tanzu Mission Control Console as described in Complete the Registration of a Supervisor Cluster in vSphere with Tanzu. You might need to retry your registration procedure from the beginning.
INSTALL_FAILED	Apply manifest	It's likely there aren't enough resources available for the agent to install successfully. Contact your vSphere administrator for help.

Table 2. Common Log Errors and Solutions for Pod Failure

Application	Log	Solutions
Extension-Updater	Failed to discover cluster metadata	Make sure your cluster is CNCF conformant. You can check the CNCF site for a list of Platform Certified Kubernetes - Distribution Providers.
Extension-Updater	Failed to set up manager:get Tanzu Mission Control connection	Make sure the cluster has internet connectivity.
Extension-Updater	Pre-flight checks failed Or something like tmc.extension-flag.register.tkg.version.block: 1.4.0	Make sure your cluster is using a version of kubernetes that is supported by Tanzu Mission Control.

What to do next

If you can’t solve the issue on your own, you can create a log bundle to share with support. Run the following commands against your management cluster:

1. Get the logs of the extension-updater kubectl logs -n svc-tmc-aN -l "tmc-extension-name=extension-updater"

2. Get the logs of the extension-manager <userinput> kubectl logs -n svc-tmc-aN -l "tmc-extension-name=extension-manager”

3. Save the logs and create a bundle to share with support.

See How to Get Support if you need more information about contacting them.