If you’re having trouble bringing an existing cluster under Tanzu Mission Control management, there are a number of steps you can take to identify and fix the problem.

Problem

You might suspect you’re having trouble attaching a cluster if: 

  • When attaching the cluster, you see the verify connection fail.

  • On the cluster detail page, your cluster is stuck in the pending status Pending icon.

  • In the CLI, you see the attach operation timeout and receive a message telling you the cluster didn’t achieve the desired state. 

  • If you’re attaching a cluster using the API, you see the status stuck in pending for longer than 7 minutes.

Cause

There are a few common reasons for cluster attach issues. Follow the solution described to identify which of these causes is the issue and help you correct it. 

  • You might need to rerun the cluster agent installation if it failed. 

  • You might have a resource scheduling issue preventing the cluster agent from being installed.

  • You might have a problem with the cluster connectivity or proxy configuration.

  • You might be attaching a windows-only cluster, and Tanzu Mission Control only supports clusters that have at least some Linux-based nodes.

  • You might be attaching an ARM cluster, which is currently unsupported.

  • You might be attaching a cluster on an unsupported version of kubernetes. 

  • You might be attaching a cluster that isn’t CNCF conformant, and Tanzu Mission Control only supports CNCF conformant clusters.  

Solution

  1. Check if the problem is your access rights or network connection to the cluster by running a simple kubectl command like get namespaces.
    kubectl get namespaces

    If you cannot access the cluster, contact your kubernetes administrator.

  2. Check if the problem is a cluster agent installation failure.
    1. Verify that the cluster agent installed successfully by checking if the vmware-system-tmc namespace exists.
      kubectl get namespace vmware-system-tmc
      NAME                STATUS   AGE
      
      vmware-system-tmc   Active   1m
    1. If the namespace doesn’t exist, try reinstalling the agent.
      Try reinstalling the agent if the namespace doesn't exist.

      For example:

      kubectl create -f "https://myorgname.tmc.cloud.vmware.com/installer?id=4...9&source=attach"
  3. Investigate the agent pods to locate the source of the problem.
    1. Check the status of the agent pods.
      kubectl get pods -n vmware-system-tmc -l "tmc-extension-name in (extension-manager,extension-updater,agent-updater)"
      NAME                                   READY   STATUS              RESTARTS      AGE
      
      agent-updater-cc9c67b4d-sp6q9          1/1   Running             0             3m
      
      agentupdater-workload-27478576-8rxvk   0/1   Completed           0             60s
      
      extension-manager-85f9cf5497-rk8ms     1/1   Failed             0             3m
      
      extension-updater-6c4c95c5bf-9wl6p     1/1   Pending             0             3m
    2. If any of the pods are in a failed state, fetch the logs for the failed pod to learn more information about the problem. For example:
      kubectl logs -l "tmc-extension-name=extension-updater" -n vmware-system-tmc

      For help solving pod failures, see Common Log Errors and Solutions for Pod Failure in the Examples section of this page.

    3. If the pods have not been scheduled (they appear as “pending” or not at all), inspect the deployments of extension-manager, extension-updater and agent-updater to find any deployments that are not in a ready state. For example:
      kubectl get deployments -n vmware-system-tmc -l "tmc-extension-name in (extension-manager,extension-updater,agent-updater)"
      NAME                READY   UP-TO-DATE   AVAILABLE   AGE
      
      agent-updater       1/11            1           3m
      
      extension-manager   1/11            1           3m
      
      extension-updater   0/11            1           3m
    4. If any of the deployments show as 0/1 , describe the deployment to understand how to correct the issue. 

      For example:

      kubectl describe deployment extension-updater -n vmware-system-tmc

      These kubernetes errors are usually resource related. For example, you might get <output>OutOfDisk.

      Ask your kubernetes administrator to solve these types of errors by expanding resources or relaxing policies.

  4. If the deployments are all showing 1/1 READY, look for Events messages in the replicasets to understand how to correct the issue. 
    1. Look for replicasets that are not in the ready status. For example:
      kubectl get replicaset -n vmware-system-tmc -l "tmc-extension-name in (extension-manager,extension-updater,agent-updater)"
    2. Inspect any replicasets that are not in ready status and check the events messages in the replicasets to understand how to correct the issue. For example:
      kubectl describe replicaset -n vmware-system-tmc extension-updater-6c4c95c5bf

    The Events messages should contain enough information for you to solve the issue.

Example

Table 1. Common Log Errors and Solutions for Pod Failure

Application

Log

Solutions

Extension-Updater

failed to discover cluster metadata

Make sure your cluster is CNCF conformant. You can check the CNCF site for a list of Platform Certified Kubernetes - Distribution Providers.

Extension-Updater

failed to set up manager:get Tanzu Mission Control connection
  1. Make sure the cluster has internet connectivity.

  2. If you’re behind a firewall, set up or validate the configuration of your proxy. See Connecting through a Proxy for more information.

Extension-Updater

pre-flight checks failed

Make sure your cluster is using a version of kubernetes that is supported by Tanzu Mission Control.

What to do next

If you can’t solve the issue on your own, you can create a log bundle to share with support. Run the following commands against the cluster you’re trying to attach:

  1. Print a report of components present in the cluster. tmc cluster validate -h

  2. Archive the logs of all components in this cluster. tmc cluster logs -h

See How to Get Support if you need more information about contacting them.