Troubleshoot Crossplane
This topic explains how you troubleshoot issues related to Crossplane on Tanzu Application Platform (commonly known as TAP).
For the limitations of Crossplane, see Crossplane limitations.
Resource already exists error when installing Crossplane
Symptom:
Installation of Crossplane, or a Tanzu Application Platform profile that includes Crossplane, fails with the error:
Resource already exists
Explanation:
Crossplane is already installed on the cluster. You cannot install the Crossplane package on a cluster that already has Crossplane installed on it by using another method, such as Helm install.
Solution:
Exclude the Crossplane package in the tap-values.yaml file. For more information, see Use your existing Crossplane installation.
The validatingwebhookconfiguration is not removed when you uninstall the Crossplane Package
Symptom:
The Crossplane Package deploys a validatingwebhookconfiguration named crossplane during installation. This resource is not deleted when you uninstall the Package.
Solution:
Delete the validatingwebhookconfiguration manually by running:
kubectl delete validatingwebhookconfiguration crossplane
Claims never transition to READY=True after reinstallation to the same cluster
Symptom:
After you uninstall the Crossplane package and reinstall it on the same cluster, service claims you create never transition to READY=True. If you inspect the underlying Crossplane managed resource, you see a TLS certificate verification error similar to the following:
Warning ComposeResources 39s (x23 over 17m) defined/compositeresourcedefinition.apiextensions.crossplane.io cannot compose
resources: cannot run Composition pipeline step "patch-and-transform": cannot run Function "function-patch-and-transform": rpc error:
code = Unavailable desc = last connection error: connection error: desc = "transport: authentication handshake failed: tls: failed to
verify certificate: x509: certificate signed by unknown authority (possibly because of \"crypto/rsa: verification error\" while trying
to verify candidate authority certificate \"Crossplane\")"
Explanation:
This issue occurs due to the way Crossplane manages the life cycles of TLS certificates, in particular the root CA certificate in the crossplane-root-ca secret in the crossplane-system namespace. This certificate signs all other certificates used by Crossplane.
When you uninstall Crossplane, the root CA certificate is deleted but the certificates that it signed are not deleted. After Crossplane is reinstalled, a new root CA certificate is generated. As a result, the certificates stored in secrets that were not deleted are no longer valid.
This behavior is described in Function: certificate signed by unknown authority “Crossplane” #5456 in GitHub.
Solution:
As a workaround, delete all secrets in the crossplane-system namespace when you uninstall the Crossplane package. The certificates are then regenerated when reinstalling the package. A fix for this issue is planned for a future Tanzu Application Platform release.
Claims never transition to READY=True
Symptom:
On rare occasions, service claims you create do not transition to READY=True. If you inspect the underlying Crossplane managed resource, you find a TLS certificate verification error similar to the following:
Warning ComposeResources 39s (x23 over 17m) defined/compositeresourcedefinition.apiextensions.crossplane.io cannot compose
resources: cannot run Composition pipeline step "patch-and-transform": cannot run Function "function-patch-and-transform": rpc error:
code = Unavailable desc = last connection error: connection error: desc = "transport: authentication handshake failed: tls: failed to
verify certificate: x509: certificate signed by unknown authority (possibly because of \"crypto/rsa: verification error\" while trying
to verify candidate authority certificate \"Crossplane\")"
Explanation:
This issue occurs due to the way Crossplane manages the life cycle of various TLS certificates, in particular, the root CA certificate found in the crossplane-root-ca secret in the crossplane-system namespace. This certificate signs all of the other certificates that Crossplane uses.
Occasionally, this certificate can get corrupted during the Crossplane installation. This behavior is described in Function: certificate signed by unknown authority “Crossplane” #5456 in GitHub.
Solution:
As a workaround:
- Delete all secrets in the
crossplane-systemnamespace. - Recreate all pods in the
crossplane-systemnamespace by deleting the existing ones.
Crossplane component fails to reconcile during upgrade
Symptom:
The Crossplane package fails to reconcile when upgrading the package. The providers and function-patch-and-transform fail with error message:
cannot resolve package dependencies: missing node in tree
You can see the error message if you run these commands:
kubectl get provider.pkg -n crossplane-system -o yaml
kubectl get functions -n crossplane-system -o yaml
Explanation:
When upgrading the Crossplane package, this issue can occur if the source of the package has changed without changing its version.
Solution:
As a workaround:
-
Edit the
lockand update the package source forprovider-helm,provider-kubernetes, andfunction-patch-and-transformwith the new package source as follows:kubectl edit locks.pkg.crossplane.io lock packages: - dependencies: [] name: provider-helm-919772e449f3 source: tanzu.packages.broadcom.com/tanzu-application-platform/tap-packages:provider-helm type: Provider version: ... - dependencies: [] name: function-patch-and-transform-7799bf5e4e7f source: tanzu.packages.broadcom.com/tanzu-application-platform/tap-packages:function-patch-and-transform type: Function version: ... - dependencies: [] name: provider-kubernetes-28e95f897554 source: tanzu.packages.broadcom.com/tanzu-application-platform/tap-packages:provider-kubernetes type: Provider version: ... -
Verify that the providers and functions are healthy, and that Crossplane package has reconciled by running:
kubectl get provider.pkg -n crossplane-system kubectl get functions -n crossplane-system kubectl get packageinstall crossplane -n tap-install
