If you’re having trouble bringing an existing cluster under Tanzu Mission Control management, there are a number of steps you can take to identify and fix the problem.
Problem
You might suspect you’re having trouble attaching a cluster if:
On the cluster detail page, your cluster is stuck in the pending status .
In the CLI, you see the attach operation timeout and receive a message telling you the cluster didn’t achieve the desired state.
If you’re attaching a cluster using the API, you see the status stuck in pending for longer than 7 minutes.
Cause
There are a few common reasons for cluster attach issues. Follow the solution described to identify which of these causes is the issue and help you correct it.
You might need to rerun the cluster agent installation if it failed.
You might have a resource scheduling issue preventing the cluster agent from being installed.
-
You might have a problem with the cluster connectivity or proxy configuration.
You might be attaching a windows-only cluster, and Tanzu Mission Control only supports clusters that have at least some Linux-based nodes.
You might be attaching an ARM cluster, which is currently unsupported.
You might be attaching a cluster on an unsupported version of kubernetes.
You might be attaching a cluster that isn’t CNCF conformant, and Tanzu Mission Control only supports CNCF conformant clusters.
Solution
- Check if the problem is your access rights or network connection to the cluster by running a simple kubectl command like get namespaces.
If you cannot access the cluster, contact your kubernetes administrator.
- Check if the problem is a cluster agent installation failure.
- Verify that the cluster agent installed successfully by checking if the vmware-system-tmc namespace exists.
kubectl get namespace vmware-system-tmc
NAME STATUS AGE
vmware-system-tmc Active 1m
- If the namespace doesn’t exist, try reinstalling the agent.
For example:
kubectl create -f "https://myorgname.tmc.cloud.vmware.com/installer?id=4...9&source=attach"
- Investigate the agent pods to locate the source of the problem.
- Check the status of the agent pods.
kubectl get pods -n vmware-system-tmc -l "tmc-extension-name in (extension-manager,extension-updater,agent-updater)"
NAME READY STATUS RESTARTS AGE
agent-updater-cc9c67b4d-sp6q9 1/1 Running 0 3m
agentupdater-workload-27478576-8rxvk 0/1 Completed 0 60s
extension-manager-85f9cf5497-rk8ms 1/1 Failed 0 3m
extension-updater-6c4c95c5bf-9wl6p 1/1 Pending 0 3m
- If any of the pods are in a failed state, fetch the logs for the failed pod to learn more information about the problem. For example:
kubectl logs -l "tmc-extension-name=extension-updater" -n vmware-system-tmc
For help solving pod failures, see Common Log Errors and Solutions for Pod Failure in the Examples section of this page.
- If the pods have not been scheduled (they appear as “pending” or not at all), inspect the deployments of extension-manager, extension-updater and agent-updater to find any deployments that are not in a ready state. For example:
kubectl get deployments -n vmware-system-tmc -l "tmc-extension-name in (extension-manager,extension-updater,agent-updater)"
NAME READY UP-TO-DATE AVAILABLE AGE
agent-updater 1/11 1 3m
extension-manager 1/11 1 3m
extension-updater 0/11 1 3m
- If any of the deployments show as 0/1 , describe the deployment to understand how to correct the issue.
For example:
kubectl describe deployment extension-updater -n vmware-system-tmc
These kubernetes errors are usually resource related. For example, you might get <output>OutOfDisk.
Ask your kubernetes administrator to solve these types of errors by expanding resources or relaxing policies.
- If the deployments are all showing 1/1 READY, look for Events messages in the replicasets to understand how to correct the issue.
- Look for replicasets that are not in the ready status. For example:
kubectl get replicaset -n vmware-system-tmc -l "tmc-extension-name in (extension-manager,extension-updater,agent-updater)"
- Inspect any replicasets that are not in ready status and check the events messages in the replicasets to understand how to correct the issue. For example:
kubectl describe replicaset -n vmware-system-tmc extension-updater-6c4c95c5bf
The Events messages should contain enough information for you to solve the issue.
Example
Table 1.
Common Log Errors and Solutions for Pod Failure
Application |
Log |
Solutions |
Extension-Updater |
failed to discover cluster metadata |
Make sure your cluster is CNCF conformant. You can check the CNCF site for a list of Platform Certified Kubernetes - Distribution Providers. |
Extension-Updater |
failed to set up manager:get Tanzu Mission Control connection |
Make sure the cluster has internet connectivity.
-
If you’re behind a firewall, set up or validate the configuration of your proxy. See Connecting through a Proxy for more information.
|
Extension-Updater |
pre-flight checks failed |
Make sure your cluster is using a version of kubernetes that is supported by Tanzu Mission Control. |
What to do next
If you can’t solve the issue on your own, you can create a log bundle to share with support. Run the following commands against the cluster you’re trying to attach:
Print a report of components present in the cluster. tmc cluster validate -h
Archive the logs of all components in this cluster. tmc cluster logs -h
See How to Get Support if you need more information about contacting them.