You can set up integrations with third-party observability tools to use logging, metrics, and tracing with Cloud Native Runtimes for Tanzu. These observability integrations allow you to monitor and collect detailed metrics from your clusters on Cloud Native Runtimes. You can collect logs and metrics for all workloads running on a cluster. This includes Cloud Native Runtimes components or any apps running on Cloud Native Runtimes. The integrations in this topic are recommended by VMware, however you can use any Kubernetes compatible logging, metrics, and tracing platforms to monitor your cluster workload.

Logging

You can collect and forward logs for all workloads on a cluster, including Cloud Native Runtimes components or any apps running on Cloud Native Runtimes. You can use any logging platform that is compatible with Kubernetes to collect and forward logs for Cloud Native Runtimes workloads. VMware recommends using Fluent Bit to collect logs and then forward logs to vRealize. The following sections describe configuring logging for Cloud Native Runtimes with Fluent Bit and vRealize as an example.

Configure Logging with Fluent Bit

You can use Fluent Bit to collect logs for all workloads on a cluster, including Cloud Native Runtimes components or any apps running on Cloud Native Runtimes. For more information about using Fluent Bit logs, see Fluent Bit Kubernetes Logging in the Fluent Bit documentation.

Fluent Bit lets you collect logs from Kubernetes containers, add Kubernetes metadata to these logs, and forward logs to third-party log storage services. For more information about collecting logs, see Logging in the Knative documentation.

If you are using Tanzu Mission Control (TMC), vSphere 7.0 with Tanzu, or Tanzu Kubernetes Cluster to manage your cloud native environment, you must set up a role binding that grants required permissions to Fluent Bit containers in order to configure logging with any integration. Then, you can follow the instructions in the Fluent Bit documentation to complete the logging configuration. For more information about configuring Fluent Bit logging, see Installation in the Fluent Bit documentation.

To configure logging with Fluent Bit for your Cloud Native Runtimes environment:

  1. VMware recommends that you add any integrations to the ConfigMap in both your Knative Serving and Knative Eventing namespaces. Follow the logging configuration steps in the Fluent Bit documentation to create the Namespace, ServiceAccount, Role, RoleBinding, and ConfigMap. To view these steps, see Installation in the Fluent Bit documentation.

  2. If you are using TMC, vSphere with Tanzu, or Tanzu Kubernetes Cluster to manage your cloud native environment, create a role binding in the Kubernetes namespace where your integration will be deployed to grant permission for privileged Fluent Bit containers. For information about creating a role binding on a Tanzu platform, see Add a Role Binding in the TMC documentation. For information about viewing your Kubernetes namespaces, see Viewing Namespaces in the Kubernetes documentation. Create the following role binding:

    apiVersion: rbac.authorization.k8s.io/v1
    kind: RoleBinding
    metadata:
      name: fluentbit-psp-rolebinding
      namespace: FLUENTBIT-NAMESPACE
    roleRef:
      apiGroup: rbac.authorization.k8s.io
      kind: ClusterRole
      name:  PRIVILEGED-CLUSTERROLE
    subjects:
    - kind: ServiceAccount
      name: FLUENTBIT-SERVICEACCOUNT
      namespace: FLUENTBIT-NAMESPACE
    

    Where:

    • FLUENTBIT-NAMESPACE is your Fluent Bit namespace.
    • PRIVILEGED-CLUSTERROLE is the name of your privileged cluster role.
    • FLUENTBIT-SERVICEACCOUNT is your Fluent Bit service account.
  3. To verify that you have configured logging successfully, run the following to access logs through your web browser:

    kubectl port-forward --namespace logging service/log-collector 8080:80
    

    For more information about accessing Fluent Bit logs, see Logging in the Knative documentation.

Forward Logs to vRealize

After you configure log collection, you can forward logs to log management services. vRealize Log Insight is one service you can use with Cloud Native Runtimes. vRealize Log Insight is a scalable log management solution that provides log management, dashboards, analytics, and third-party extensibility for infrastructure and apps. For more information about vRealize Log Insight, see the VMware vRealize Log Insight Documentation.

To forward logs from your Cloud Native Runtimes environment to vRealize, you can use a new or existing instance of Tanzu Kubernetes Cluster. For information about how to configure log forwarding to vRealize from Tanzu Kubernetes Cluster, see the Configure Log forwarding from VMware Tanzu Kubernetes Cluster to vRealize Log Insight Cloud blog.

Metrics

Cloud Native Runtimes integrates with Prometheus and Tanzu Observability by Wavefront to collect metrics on components or apps. For more information about integrating with Prometheus, see Overview in the Prometheus documentation and Kubernetes Integration in the Wavefront documentation.

You can configure Prometheus endpoints on Cloud Native Runtimes components in order to be able to collect metrics on your components or apps. For information on annotations required to collect metrics on apps, see Per-Pod Prometheus Annotations in the WeaveWorks documentation.

You can use annotation based discovery with Prometheus to define which Kubernetes objects in your Cloud Native Runtimes environment to add metadata and collect metrics in a more automated way. For more information about using annotation based discovery, see Annotation based discovery in GitHub.

You can then use the Wavefront Collector for Kubernetes collector to dynamically discover and scrape pods with the prometheus.io/scrape annotation prefix. For information about the Kubernetes collector, see Wavefront Collector for Kubernetes in GitHub.

Note: All Cloud Native Runtimes related metrics are emitted with the prefix tanzu.vmware.com/cloud-native-runtimes.*.

Tracing

Tracing is a method for understanding the performance of specific code paths in apps as they handle requests. You can configure tracing to collect performance metrics for your apps or Cloud Native Runtimes components. You can trace which aspects of Cloud Native Runtimes and workloads running on Cloud Native Runtimes are performing poorly.

Configuring Tracing

You can configure tracing for your apps on Cloud Native Runtimes. To do this, you configure tracing for both Knative Serving and Eventing by editing the ConfigMap for your Knative namespace.

To configure tracing, do the following:

  1. Configure the config-tracing ConfigMap in your Knative component namespace. VMware recommends that you add any integrations to the ConfigMap in both your Serving and Eventing namespaces. For information on how to enable request traces in each component, see the following Knative documentation:

Forwarding Trace Data to a Data Visualization Tool

You can use the OpenTelemetry integration with Tanzu Observability by Wavefront to forward trace data to Tanzu Observability by Wavefront. For information about forwarding trace data, see Sending Metrics Data to Wavefront in the Wavefront documentation.

To configure to send trace data to Cloud Native Runtimes tracing with Tanzu Observability by Wavefront and the OpenTelemetry integration, do the following:

  1. Use the following documentation to configure the OpenTelemetry Integration to send trace data to with Cloud Native Runtimes. For more information about sending trace data with OpenTelemetry, see OpenTelemetry Integration in the Wavefront documentation.

  2. Deploy the Wavefront Proxy. For more information about wavefront proxies, see Deploy a Wavefront Proxy in Kubernetes in the Wavefront documentation.

    • Use the following .yaml file to install the Wavefront proxy in your Kubernetes cluster: wavefront.yaml.
    • Provide the URL of your Wavefront instance and a Wavefront token.
    • Uncomment the lines indicated in the yaml file to enable consumption of Zipkin traces.

Sending Trace Data to an Observability Platform

You can send trace data to an observability and analytics platform to view and monitor your trace data in dashboards.

One way to do this is to integrate Tanzu Observability by Wavefront with your Cloud Native Runtimes environment. To view your trace data in Wavefront, you configure Cloud Native Runtimes to send traces to the Wavefront proxy and then configure the Wavefront proxy to consume Zipkin spans.

For more information about using Zipkin for tracing, see the Zipkin documentation.

You can send trace data from Cloud Native Runtimes to Wavefront by using Zipkin as the backend and defining the Zipkin endpoint as the Wavefront proxy URL listening over port 9411. You configure Cloud Native Runtimes to send traces directly to the Wavefront proxy by editing the zipkin-endpoint property in the ConfigMap to point to the Wavefront proxy URL. You can configure the Wavefront proxy to consume Zipkin spans by listening to port 9411.

To send trace data to Tanzu Observability by Wavefront:

  1. Edit the ConfigMap to enable the Zipkin tracing integration. VMware recommends that you add any integrations to the ConfigMap in both your Serving and Eventing namespaces. Edit the Knative config-tracing ConfigMap to set backend to zipkin and pass the Wavefront proxy URL in the zipkin-endpoint field:

    Kubectl edit configmap config-tracing —namespace knative-serving apiVersion: v1
    kind: ConfigMap
    metadata:
    name: config-tracing
    ...
    data:
    backend: "zipkin"
    zipkin-endpoint: "http://wavefront-proxy.default.svc.cluster.local:9411/api/v2/spans"  ...
    

Use Wavefront Dashboards

Cloud Native Runtimes provides two Wavefront dashboards in JSON format. You can use these dashboard to monitor your apps and investigate performance issues. For information about configuring dashboards, see Create and Customize Dashboards in the Wavefront documentation.

The following Wavefront dashboards are compatible with Cloud Native Runtimes: - Application Operator Service View. See app-operator-service-view.json in the Cloud Native Runtimes installation .tar file. - Application Operator Revision View. See app-operator-revision-view.json in the Cloud Native Runtimes installation .tar file.

To import a dashboard JSON file, use one of the following methods: - Wavefront REST API - Wavefront CLIs.

You must provide the URL of your Wavefront instance and a Wavefront token. For more information about Wavefront tokens, see Generating an API Token in the Wavefront documentation.

Import Wavefront Dashboards

You can import the Wavefront dashboards using either the Wavefront API or the Ruby Wavefront CLI. For more information about Wavefront dashboard, see Import Dashboards with the Wavefront API or Import with a Ruby Wavefront CLI below.

Import Dashboards with the Wavefront API

To import a Wavefront dashboard with the Wavefront API, run:

curl -H "Content-Type: application/json" -H 'Authorization: Bearer <wavefront-token>' \
    https://<wavefront-instance>.wavefront.com/api/v2/dashboard -d @observability/wavefront/app-operator-service-view.json

curl -H "Content-Type: application/json" -H 'Authorization: Bearer <wavefront-token>' \
    https://<wavefront-instance>.wavefront.com/api/v2/dashboard -d @dashboards/wavefront/app-operator-revision-view.json

After you run the import code, the Wavefront API creates two dashboards with the following names and URLs:

  • Title: Cloud Native Runtimes App Operator - Service View

    URL: https://<wavefront-instance>.wavefront.com/dashboards/App-Operator-Service-Level

  • Title: Cloud Native Runtimes App Operator - Revision View

    URL: https://<wavefront-instance>.wavefront.com/dashboards/App-Operator-Revision-Level

Import with the Ruby Wavefront CLI

To import a Wavefront dashboard with the Ruby Wavefront CLI, run:

export WAVEFRONT_TOKEN=<wavefront-token>
export WAVEFRONT_ENDPOINT=<wavefront-instance>.wavefront.com

wf config envvars
wf dashboard import observability/wavefront/app-operator-service-view.json
wf dashboard import dashboards/wavefront/app-operator-revision-view.json

After you run the import code, the Ruby Wavefront CLI creates two dashboards with a name and URL.

The Service View of the Cloud Native Runtimes App Operator dashboard will have the following title and URL:

  • Title: Cloud Native Runtimes App Operator - Service View

    URL: https://<wavefront-instance>.wavefront.com/dashboards/App-Operator-Service-Level

The Revision View of the Cloud Native Runtimes App Operator dashboard will have the following title and URL:

  • Title: Cloud Native Runtimes App Operator - Revision View

    URL: https://<wavefront-instance>.wavefront.com/dashboards/App-Operator-Revision-Level

check-circle-line exclamation-circle-line close-line
Scroll to top icon