Configuring Monitoring for VMware Spring Cloud® Data Flow for Kubernetes

Here you will learn how to configure monitoring for VMware Spring Cloud® Data Flow for Kubernetes (SCDF for Kubernetes). SCDF for Kubernetes can use either Wavefront or Prometheus.

Configure monitoring with Wavefront

You can configure SCDF for Kubernetes to send metrics from the Data Flow server application, along with metrics from deployed stream and task applications, to the Wavefront service. For information about using Wavefront with SCDF for Kubernetes, see the following sections.

Wavefront account

To use Wavefront with SCDF for Kubernetes, you must have a Wavefront account. You can sign up for an account at the VMware Tanzu Observability by Wavefront website.

After creating an account, you can generate an API token for your account. To generate the token, visit your Wavefront user profile page and click the API Access tab.

Wavefront API secret

To send metrics to the Wavefront service, you must create a Kubernetes secret containing the API token generated in the previous section. This secret must be created in the namespace where you will install SCDF for Kubernetes.

Create the secret using the following command, where <WAVEFRONT_API_TOKEN> is your API token:

$ kubectl create secret generic wavefront-api --from-literal=wavefront-api-token=<WAVEFRONT_API_TOKEN>

Next, mount the secret, to make it available for use by the Data Flow server configuration. Add the secret mount settings in the deployment-patch.yaml file for the Data Flow server application:

      containers:
      - name: scdf-server
        volumeMounts:
          - name: wavefront-api
            mountPath: /workspace/runtime/secrets/wavefront-api
            readOnly: true
  ...

      volumes:
        - name: wavefront-api
          secret:
            secretName: wavefront-api

Configure monitoring with Wavefront

See the following section for an example of the configuration to use when enabling Wavefront for SCDF for Kubernetes applications. After the Data Flow server starts up with this configuration, metrics from the applications will be sent to Wavefront.

Enable Wavefront for the Data Flow Server and stream and task applications

To enable Wavefront for the Data Flow server, edit the application.yaml file for the Data Flow application. Add the following configuration:

management:
  metrics:
    web:
      server:
        request:
          autotime:
            enabled: true
        metric-name: spring.cloud.dataflow.http.server.requests
    export:
      wavefront:
        enabled: true
        api-token: ${wavefront-api-token}
        uri: https://<YOUR_WAVEFRONT_URI>.wavefront.com
        source: <YOUR_SOURCE_NAME>

In this example configuration, replace the <YOUR_WAVEFRONT_URI> and <YOUR_SOURCE_NAME> placeholders with the relevant values. The value of the api-token field will be automatically resolved from the secret.

Configure monitoring with Prometheus

SCDF supports both short-lived application monitoring and long-lived application monitoring with Prometheus. Spring Cloud Task and Spring Cloud Stream applications deployed by Data Flow and Skipper can use the Prometheus RSocket Proxy.

Note: SCDF for Kubernetes also supports other monitoring technologies that use Micrometer. This section describes how to configure the Prometheus RSocket Proxy only.

The SCDF for Kubernetes dashboard provides a link to custom Streams and Tasks dashboards for use with Grafana. You can download these dashboards and install them into your Grafana server.

The following example configures monitoring for all deployed Spring Cloud Task and Spring Cloud Stream applications that use Prometheus over an RSocket gateway. Add this configuration to the application.yaml file for the Data Flow application:

spring:
  cloud:
    dataflow:
      applicationProperties:
        stream:
          management:
            metrics:
              export:
                prometheus:
                  enabled: true
                  rsocket:
                    enabled: true
                    host: prometheus-proxy
                    port: 7001
        task:
          management:
            metrics:
              export:
                prometheus:
                  enabled: true
                  rsocket:
                    enabled: true
                    host: prometheus-proxy
                    port: 7001
      grafana-info:
        url: 'https://grafana:3000'

In this example configuration, replace the Grafana url and Prometheus host and port values with the appropriate values for your specific environment. See the Spring Cloud Data Flow microsite for more information about stream monitoring and task monitoring.

These common application properties are passed to the deployed Spring Cloud Task and Spring Cloud Stream applications. If you are using an alternative monitoring tool that is also supported by Micrometer, you can set the appropriate Spring Boot configuration values for your monitoring infrastructure. Add this configuration to the applicationProperties.stream and applicationProperties.task sections of the application.yaml file for the Data Flow application.

Next: Configure multiple platforms support

To configure multiple platform support in SCDF for Kubernetes, see Configuring Multiple Platforms Support for SCDF for Kubernetes.