Troubleshoot Crossplane

This topic explains how you troubleshoot issues related to Crossplane on Tanzu Application Platform (commonly known as TAP).

For the limitations of Crossplane, see Crossplane limitations.

Resource already exists error when installing Crossplane

Symptom:

Installation of Crossplane, or a Tanzu Application Platform profile that includes Crossplane, fails with the error:

Resource already exists

Explanation:

Crossplane is already installed on the cluster. You cannot install the Crossplane package on a cluster that already has Crossplane installed on it by using another method, such as Helm install.

Solution:

Exclude the Crossplane package in the tap-values.yaml file. For more information, see Use your existing Crossplane installation.

The validatingwebhookconfiguration is not removed when you uninstall the Crossplane Package

Symptom:

The Crossplane Package deploys a validatingwebhookconfiguration named crossplane during installation. This resource is not deleted when you uninstall the Package.

Solution:

Delete the validatingwebhookconfiguration manually by running:

kubectl delete validatingwebhookconfiguration crossplane

Claims never transition to READY=True after reinstallation to the same cluster

Symptom:

After you uninstall the Crossplane package and reinstall it on the same cluster, service claims you create never transition to READY=True. If you inspect the underlying Crossplane managed resource, you see a TLS certificate verification error similar to the following:

Warning  ComposeResources   39s (x23 over 17m)  defined/compositeresourcedefinition.apiextensions.crossplane.io  cannot compose
resources: cannot run Composition pipeline step "patch-and-transform": cannot run Function "function-patch-and-transform": rpc error:
code = Unavailable desc = last connection error: connection error: desc = "transport: authentication handshake failed: tls: failed to
verify certificate: x509: certificate signed by unknown authority (possibly because of \"crypto/rsa: verification error\" while trying
to verify candidate authority certificate \"Crossplane\")"

Explanation:

This issue occurs due to the way Crossplane manages the life cycles of TLS certificates, in particular the root CA certificate in the crossplane-root-ca secret in the crossplane-system namespace. This certificate signs all other certificates used by Crossplane.

When you uninstall Crossplane, the root CA certificate is deleted but the certificates that it signed are not deleted. After Crossplane is reinstalled, a new root CA certificate is generated. As a result, the certificates stored in secrets that were not deleted are no longer valid.

This behavior is described in Function: certificate signed by unknown authority “Crossplane” #5456 in GitHub.

Solution:

As a workaround, delete all secrets in the crossplane-system namespace when you uninstall the Crossplane package. The certificates are then regenerated when reinstalling the package. A fix for this issue is planned for a future Tanzu Application Platform release.

Claims never transition to READY=True

Symptom:

On rare occasions, service claims you create do not transition to READY=True. If you inspect the underlying Crossplane managed resource, you find a TLS certificate verification error similar to the following:

Warning  ComposeResources   39s (x23 over 17m)  defined/compositeresourcedefinition.apiextensions.crossplane.io  cannot compose
resources: cannot run Composition pipeline step "patch-and-transform": cannot run Function "function-patch-and-transform": rpc error:
code = Unavailable desc = last connection error: connection error: desc = "transport: authentication handshake failed: tls: failed to
verify certificate: x509: certificate signed by unknown authority (possibly because of \"crypto/rsa: verification error\" while trying
to verify candidate authority certificate \"Crossplane\")"

Explanation:

This issue occurs due to the way Crossplane manages the life cycle of various TLS certificates, in particular, the root CA certificate found in the crossplane-root-ca secret in the crossplane-system namespace. This certificate signs all of the other certificates that Crossplane uses.

Occasionally, this certificate can get corrupted during the Crossplane installation. This behavior is described in Function: certificate signed by unknown authority “Crossplane” #5456 in GitHub.

Solution:

As a workaround:

Delete all secrets in the crossplane-system namespace.
Recreate all pods in the crossplane-system namespace by deleting the existing ones.

Crossplane component fails to reconcile during upgrade

Symptom:

The Crossplane package fails to reconcile when upgrading the package. The providers and function-patch-and-transform fail with error message:

cannot resolve package dependencies: missing node in tree

You can see the error message if you run these commands:

kubectl get provider.pkg -n crossplane-system -o yaml
kubectl get functions -n crossplane-system -o yaml

Explanation:

When upgrading the Crossplane package, this issue can occur if the source of the package has changed without changing its version.

Solution:

As a workaround:

Edit the lock and update the package source for provider-helm, provider-kubernetes, and function-patch-and-transform with the new package source as follows:

kubectl edit locks.pkg.crossplane.io lock

packages:
- dependencies: []
  name: provider-helm-919772e449f3
  source: tanzu.packages.broadcom.com/tanzu-application-platform/tap-packages:provider-helm
  type: Provider
  version: ...
- dependencies: []
  name: function-patch-and-transform-7799bf5e4e7f
  source: tanzu.packages.broadcom.com/tanzu-application-platform/tap-packages:function-patch-and-transform
  type: Function
  version: ...
- dependencies: []
  name: provider-kubernetes-28e95f897554
  source: tanzu.packages.broadcom.com/tanzu-application-platform/tap-packages:provider-kubernetes
  type: Provider
  version: ...

Verify that the providers and functions are healthy, and that Crossplane package has reconciled by running:

kubectl get provider.pkg -n crossplane-system
kubectl get functions -n crossplane-system
kubectl get packageinstall crossplane -n tap-install