This topic provides information and guidance for understanding AWS credential states and how to troubleshoot credential issues related to management of EKS clusters and Tanzu Mission Control accounts required for use of the EKS lifecycle management features.

Credential Phases and AWS Resources

Credential phases can be one of the following:

Figure 1. EKS Credential Phases
Credential Phases can be one of CREATED VALIDATING, VALID, INVALID, DELETING, ERROR

For more information about credential phases, see vmware.tanzu.manage.v1alpha1.account.credential.Status in Tanzu Mission Control API Reference.

Note that in the Tanzu Mission Control console, the DELETING phase appears as REMOVING.

Once a credential has been created, Tanzu Mission Control creates the following in the region where the Cloud Formation stack was initially created.

  1. Data sync lambda (responsible for syncing data about EKS clusters in the AWS account back to Tanzu Mission Control).
  2. Resource retriever lambda (responsible for syncing data about AWS account from enabled regions back to Tanzu Mission Control - e.g., region, VPCs, subnets, etc.).
  3. A CloudWatch event (for logging)
  4. An AWS Systems Manager or SSM Parameter (containing an authentication token for communicating with Tanzu Mission Control).

Once all of these have been deployed successfully and the lambdas have been able to run and sync back to Tanzu Mission Control, only then is a credential marked as VALID. This is the desired state of a healthy and ready to use credential.

Credentials in CREATED State

If a credential is in a CREATED state this means that the create request has been successful, however, it does not mean that the credential is useable. It is going through the pre-validating phase and will be marked either VALIDATING or INVALID shortly. If a credential is in a CREATED state for 20 minutes or more, you will need to reach out to customer support for further assistance.

Credentials in VALIDATING State

If a credential is stuck in a VALIDATING state, this means that the lambdas deployed to the account have not been able to communicate with Tanzu Mission Control.

Check that each of the resources listed in the section above was created successfully.

  • To check the lambdas, go to the AWS console and in the Services menu search for lambda in the region where the Cloud Formation stack was initially created:
    • Search by the tag account.tmc.cloud.vmware.com=credential-name to see the lambdas associated with your credential.
    • Check that both the data-sync and resource-retriever lambda for the credential have been created and have run successfully.
  • Once you confirm the lambdas are there, you can check the CloudWatch logs and metrics for each of them:
    • Click into each lambda to open up the full view of each one.
    • Scroll down and open the Monitor tab, then selection Metrics, Logs or Trace as appropriate.
    • You can also opt to select "View the Logs in CloudWatch" if you are more familiar with that view.
  • To check via the AWS console that the SSM parameter is present in the region where the Cloud Formation stack was created initially:
    • Search for Systems Manager in the Services menu.
    • Go to Application Management > Parameter Store.
    • Search by the tag account.tmc.cloud.vmware.com=credential-name to see the parameter associated with your credential.
    • You won't be able to view the actual authentication token, but you can verify that it is present and when it was last updated by looking at the History tab.

Credentials in INVALID State

The table below provides information about error messages you may receive if a credential is in an INVALID state, along with suggested fixes for the errors.

Error Message What does it mean? Who can fix it? Fix
Credential is not associated with Tanzu Mission Control This means that Tanzu Mission Control was not set for the credential. User (if using the API or CLI), or Customer Support if using the console

Ensure that the LCM capability is included in the specification of the credential request. For example:

...

"capability": "MANAGED_K8S_PROVIDER"

}

...
Failed to get credentials for account <account-name> with err <error> The service failed to get the credential from Tanzu Mission Control User

To workaround:

Try deleting and recreating a new credential (with a different name, in case the bad state of the original credential causes issues with the deletion).

received empty account credentials The meta section of the spec in the credential request is empty User (if using the API or CLI), or Customer Support if using the console

Make sure that there is a provider specified in the meta section of the credential request. For example:

...

"meta": {

"provider": 5

},

...

My credential was VALID but now it's INVALID - What happened?

Tanzu Mission Control continually reconciles the accounts (a.k.a credentials) associated with it. There are some use cases where a credential that was previously marked as VALID can be later marked as INVALID.

For example:

  • If the cluster options have not been updated in a long time, then we know the lambdas in the AWS account have lost connectivity with Tanzu Mission Control. This could happen in the case of an AWS outage even, in which case when connectivity resumes, the credential will be marked VALID again. However, if there is no such outage, you need to check that the lambdas are set up correctly and working and that the Tanzu Mission Control authentication token in SSM is still valid.

Credentials in DELETING/REMOVING State

If a credential is stuck in a DELETING state (or in the case of console users, in a REMOVING state) then there may have been a problem with the credential deletion that caused it to fail.

When a credential is deleted, the account-controller deletes the four items that it created in the AWS account:

  • Data sync lambda
  • Resource retriever lambda
  • CloudWatch event rule
  • SSM parameter

If you encounter issues trying to delete these, contact Customer Support.

Getting more Information about the State of a Credential

While the console only provides the high level state of the credential (for example, CREATING, ERROR, REMOVING, etc.), the Tanzu Mission Control API provides more information by including conditions.

You can use the List Credentials API to see what the conditions are for your credential. (For more information about, Tanzu Mission Control APIs, see Tanzu Mission Control API Reference.)

Example: 

# Using the GET Credential API endpoint
# {{hostname}}/v1alpha1/account/credentials/{{credential_name}}
         
"status": {
            "phase": "VALID",
            "conditions": {
                "Ready": {
                    "type": "Ready",
                    "status": "TRUE",
                    "severity": "INFO",
                    "lastTransitionTime": "2022-10-05T20:58:06.131349Z",
                    "reason": "CredentialCreated"
                },
                "Scheduled": {
                    "type": "Scheduled",
                    "status": "TRUE",
                    "severity": "INFO",
                    "lastTransitionTime": "2022-10-05T20:58:06.131349Z",
                    "reason": "CreatingCredential"
                }
            }
        }