Troubleshoot Local Source Proxy

This topic helps you troubleshoot issues you might encounter with Local Source Proxy (LSP).

View Local Source Proxy server logs

Symptom

You encounter an error and need to view the Local Source Proxy server logs to investigate it.

Solution

Run

kubectl -n tap-local-source-system logs deployments/local-source-proxy

Use -f to follow the log output.

View Apps CLI health messages

Symptom

You need to read the Apps CLI health messages to assess the status of Local Source Proxy and its connectivity with the upstream repository.

Solution

Run

tanzu apps lsp health

Example:

$ tanzu apps lsp health
user_has_permission: true
reachable: true
upstream_authenticated: true
overall_health: true
message: All health checks passed

User does not have RBAC permission to list services

Symptom

You encounter any of these error messages:

$ tanzu apps workload apply
Error: Either Local Source Proxy is not installed on the Cluster or you don't have permissions to access it
Reason: The current user does not have permission to access the local source proxy.
Messages:
- services "http:local-source-proxy:5001" is forbidden: User "[email protected]" cannot get resource "services/proxy" in API group "" in the namespace "tap-local-source-system": requires one of ["container.services.proxy"] permission(s).

$ tanzu apps lsp health
user_has_permission: false
reachable: false
upstream_authenticated: false
overall_health: false
message: |-
  The current user does not have permission to access the local source proxy.
  Messages:
  - services "http:local-source-proxy:5001" is forbidden: User "[email protected]" cannot get resource "services/proxy" in API group "" in the namespace "tap-local-source-system": requires one of ["container.services.proxy"] permission(s).

Cause

Typically, this situation arises when a custom user or group is specified within the rbac_subjects_for_proxy_access section of tap-values.yaml.

Solution

Ensure that the user or group listed is valid. For more information about overriding default RBAC permissions to access the proxy service, see Override default RBAC permissions to access the proxy service.

Missing repository in Tanzu Application Platform values

Symptom

You encounter one of these error messages:

$ tanzu apps workload apply
Error: Local source proxy failed to upload source to the repository
Reason: Local source proxy is not healthy.
Messages:
- registry server configuration in the cluster is invalid

$ tanzu apps lsp health
user_has_permission: true
reachable: true
upstream_authenticated: false
overall_health: false
message: |
  Local source proxy is not healthy.
  Messages:
  - registry server configuration in the cluster is invalid

Cause

The cause might be that tap-values.yaml lacks a valid value for the repository.

Solution

Add a valid repository value to tap-values.yaml and wait for the app reconciliation to complete.

Missing or misconfigured registry secret

Symptom

You encounter one of these error messages:

$ tanzu apps workload apply
Error: Local source proxy failed to upload source to the repository
Reason: Local source proxy was unable to authenticate against the target registry.
Messages:
- GET https://gcr.io/v2/token?scope=repository:abc-playground/lsp-source:pull,push&service=gcr.io: UNAUTHORIZED: You don't have the needed permissions to perform this operation, and you may have invalid credentials. To authenticate your request, follow the steps in: https://cloud.google.com/container-registry/docs/advanced-authentication

$ tanzu apps lsp health
user_has_permission: true
reachable: true
upstream_authenticated: false
overall_health: false
message: |-
  Local source proxy was unable to authenticate against the target registry.
  Messages:
  - GET https://gcr.io/v2/token?scope=repository:abc-playground/lsp-source:pull,push&service=gcr.io: UNAUTHORIZED: You don't have the
 needed permissions to perform this operation, and you may have invalid credentials. To authenticate your request, follow the steps in: https:/
/cloud.google.com/container-registry/docs/advanced-authentication

Cause

Potential causes include:

• A missing registry secret:
  • push_secret information is not available in the local_source_proxy section of tap-values.yaml.
  • image_registry.secret information is not available in the shared section of tap-values.yaml.
• If push_secret is used, the secret was not exported to the Local Source Proxy namespace. The credentials used in the secret do not match the configured external registry.

Solution

1. Ensure that at least one of the following entries is found in tap-values.yaml:

   • push_secret information in the local_source_proxy section
   • image_registry.secret information in the shared section

2. If push_secret is used, make sure that it can be exported to the Local Source Proxy namespace.

3. Ensure that the credentials used in the secret match the configured external registry.

Invalid credentials

Symptom

You encounter one of these error messages:

$ tanzu apps workload apply
Error: Local source proxy failed to upload source to the repository
Reason: Local source proxy was unable to authenticate against the target registry.
Messages:
- GET https://gcr.io/v2/token?scope=repository:abc-playground/lsp-source:pull,push&service=gcr.io: UNAUTHORIZED: Not Authorized.

$ tanzu apps lsp health # when using Harbor
user_has_permission: true
reachable: true
upstream_authenticated: false
overall_health: false
message: |-
  Local source proxy was unable to authenticate against the target registry.
  Messages:
  - 401 Unauthorized

$ tanzu apps lsp health # when using GCR
user_has_permission: true
reachable: true
upstream_authenticated: false
overall_health: false
message: |-
  Local source proxy was unable to authenticate against the target registry.
  Messages:
  - GET https://gcr.io/v2/token?scope=repository:abc-playground/lsp-source:pull,push&service=gcr.io: UNAUTHORIZED: Not Authorized.

Cause

The cause is the use of invalid credentials.

Solution

Change the credentials used in the secret to match those in the configured external registry.

Local Source Proxy doesn’t automatically detect changes to podspec

Symptom

Local Source Proxy doesn’t automatically detect changes to podspec.

Cause

AWS Elastic Container Registry (ECR) is configured as the external registry in tap-values.yaml.

Solution

Delete the old pods so that the new pods can mount the expected podspec, enabling access to the registry through the Identity and Access Management (IAM) role Amazon Resource Name (ARN).

Error: unknown command “lsp” for “apps”

Symptom

When running tanzu apps lsp health the CLI returns the error message

Error: unknown command "lsp" for "apps"

Cause

Tanzu CLI and the apps plug-in are out of date.

Solution

1. Install Tanzu CLI v0.12.0 or later.

2. Upgrade the apps plug-in by running:

   tanzu plugin upgrade apps
   

Error: i/o timeout

Symptom

When you run tanzu apps lsp health or tanzu apps workload apply, after a few minutes the CLI returns the following error message:

connect: i/o timeout

Cause

TCP port 5002 is not open between your control plane nodes and your worker nodes.

Solution

1. Open the port.
2. Run the command again.