Troubleshooting procedure for infrastructure automation.

Cluster Creation Failure

The following can cause cluster creation failure:
  • etcdserver leader change.
  • etcdserver timeout.
  • Internet connectivity problem (DNS configuration is incorrect in the bootstrapper VM), due to which the images for the containers cannot be downloaded from the internet.
    • For air-gapped configuration - not able to connect to the airgap server and the airgap server cannot reach out to the cluster nodes.
  • Unable to connect to Bootrapper Service.

  • Cluster creation timeout issues.

  • Delete Cluster and perform Resynchronisation from Infrastructure Automation
    Note:
    • If you have already created bootstrapper cluster, then delete the bootstrapper cluster before deleting the management cluster.
    • To delete VMware Telco Cloud Automation Cluster, use clusterType=MANAGEMENT

    • To delete Bootstrapper Cluster, use clusterType=WORKLOAD

    Steps to Delete a VMware Telco Cloud Automation cluster
    1. Use the GET API of the appliance manager to fetch the cluster details.
    2. Use the id from the API and pass it to the DELETE API.
    3. Use the status API to monitor the delete cluster.
    4. Use the force delete option to remove from the local DB to reuse the controlPlaneEndpointIP.

Service Installation Failure

The following can cause service installation failures:
  • Helm API failed: release: <service_name> not found
  • Helm API failed: release name <service_name> in namespace <namespace_name> is already in use
  • Release "service_name" failed: etcdserver: leader changed
  • Failed to deploy <service_name>-readiness

Workaround

Uninstall the failed service manually and perform Resynchronisation from Infrastructure Automation.

Steps to Uninstall the failed service

To check the installation failure of the service on the bootstrapper virtual machine (VM), execute the command helm ls -n <Namespace-name>

Example

helm ls -n tca-mgr

[root@tca-b-cdc1 /home/admin]# helm ls -n tca-mgr
NAME            NAMESPACE   REVISION    UPDATED                                 STATUS      CHART               APP VERSION
istio-ingress   tca-mgr     1           2021-11-20 17:18:25.309656522 +0000 UTC deployed    istio-ingress-2.0.0 1.10.3    
kafka           tca-mgr     1           2021-11-20 17:15:43.734275545 +0000 UTC deployed    kafka-2.0.0         2.12-2.5.0
mongodb         tca-mgr     1           2021-11-20 17:11:17.794039072 +0000 UTC deployed    mongodb-2.0.0       3.2.5     
redisoperator   tca-mgr     1           2021-11-20 17:15:55.431688075 +0000 UTC deployed    redisoperator-2.0.0 1.0.0     
redisservice    tca-mgr     1           2021-11-20 17:16:58.087135033 +0000 UTC deployed    redisservice-2.0.0  6.0-alpine
tca             tca-mgr     1           2021-11-20 17:18:46.328884407 +0000 UTC deployed    tca-2.0.0           2.0.0     
zookeeper       tca-mgr     1           2021-11-20 17:15:34.075735519 +0000 UTC deployed    zookeeper-2.0.0     3.4.9

To recover from the installation failure of the service, uninstall the failed service specifically on the Bootstrapper VM terminal. Use the command helm uninstall <Helm Service Name> -n <Namespace-name>

Example:

helm uninstall tca -n tca-mgr

To verify the successful uninstallation of the service, re-execute the command helm uninstall <Helm Service Name> -n <Namespace-name>. If the list does not shows Helm service, the uninstallation is successful.

[root@tca-b-cdc1 /home/admin]# helm ls -n tca-mgr
NAME            NAMESPACE   REVISION    UPDATED                                 STATUS      CHART               APP VERSION
istio-ingress   tca-mgr     1           2021-11-20 17:18:25.309656522 +0000 UTC deployed    istio-ingress-2.0.0 1.10.3    
kafka           tca-mgr     1           2021-11-20 17:15:43.734275545 +0000 UTC deployed    kafka-2.0.0         2.12-2.5.0
mongodb         tca-mgr     1           2021-11-20 17:11:17.794039072 +0000 UTC deployed    mongodb-2.0.0       3.2.5     
redisoperator   tca-mgr     1           2021-11-20 17:15:55.431688075 +0000 UTC deployed    redisoperator-2.0.0 1.0.0     
redisservice    tca-mgr     1           2021-11-20 17:16:58.087135033 +0000 UTC deployed    redisservice-2.0.0  6.0-alpine    
zookeeper       tca-mgr     1           2021-11-20 17:15:34.075735519 +0000 UTC deployed    zookeeper-2.0.0     3.4.9

Perform the resynchronisation through Infrastructure Automation using Resync.

Site-Pairing Failure

The following can cause the site-pairing issue:
  • etcdserver leader change
  • etcdserver timeout
  • Socket-timeout issue

Workaround

Perform the resynchronisation through Infrastructure Automation using Resync.

Cloud Builder Validation Failure

During the Management or Workload domain deployment, Cloudbuilder 4.3 performs a set of validations. Some validations could fail as expected, but have no impact of the domain deployment. For example, cloudbuilder validates the gateway configuration for vMotion and vSAN network. It is possible that the user may not have configured the gateway for vMotion and vSAN as they are configured in the same respective L2 domains. In such a situation, while Infrastructure Automation fails the domain deployment (due to cloudbuilder validation failing), the user can skip cloudbuilder validation using the following procedure, after which a user can perform a resync on the failed domain to continue further.

For VM based deployment, use the following procedure:
  1. Login to VMware Telco Cloud Automation manager using SSH.
  2. Switch to the root user.
  3. Open the file /common/lib/docker/volumes/tcf-manager-config/_data/cloud_spec.json.
  4. Set the field validateCloudBuilderSpec to false. 
       "settings": {
        "ssoDomain": "vsphere.local",
        "pscUserGroup": "Administrators",
        "saas": "10.202.228.222",
        "enableCsiZoning": true,
        "validateCloudBuilderSpec": true,
        "csiRegionTagNamingScheme": "region-{domainName}",
        "clusterCsiZoneTagNamingScheme": "zone-{domainName}",
        "hostCsiZoneTagNamingScheme": "zone-{hostname}",
        "dnsSuffix": "telco.example.com",
        "ntpServers": [
            "10.166.1.120"
        ],
  5. Resync the failed domain.
For HA based deployment, use the following procedure:
  1. Login to the bootstrapper VM using SSH.
  2. Switch to the root user.
  3. Open the file /common/lib/docker/volumes/tcf-manager-config/_data/cloud_spec.json.
  4. Set the field validateCloudBuilderSpec to false. 
       "settings": {
        "ssoDomain": "vsphere.local",
        "pscUserGroup": "Administrators",
        "saas": "10.202.228.222",
        "enableCsiZoning": true,
        "validateCloudBuilderSpec": true,
        "csiRegionTagNamingScheme": "region-{domainName}",
        "clusterCsiZoneTagNamingScheme": "zone-{domainName}",
        "hostCsiZoneTagNamingScheme": "zone-{hostname}",
        "dnsSuffix": "telco.example.com",
        "ntpServers": [
            "10.166.1.120"
        ],
  5. Resync the failed domain.
For Management or Workload domain deployment, use the following procedure:
  1. Login to the bootstrapper VM using SSH.
  2. Switch to the root user.
  3. Navigate to the tcf-manager docker using the kubectl exec -it tca-tcf-manager-0 -n tca-mgr bash command.
  4. Open the file /opt/vmware/tcf/config/cloud_spec.json.
  5. Set the field validateCloudBuilderSpec to false. 
       "settings": {
        "ssoDomain": "vsphere.local",
        "pscUserGroup": "Administrators",
        "saas": "10.202.228.222",
        "enableCsiZoning": true,
        "validateCloudBuilderSpec": true,
        "csiRegionTagNamingScheme": "region-{domainName}",
        "clusterCsiZoneTagNamingScheme": "zone-{domainName}",
        "hostCsiZoneTagNamingScheme": "zone-{hostname}",
        "dnsSuffix": "telco.example.com",
        "ntpServers": [
            "10.166.1.120"
        ],
  6. Resync the failed domain.