You can export API audit logs collected in Tanzu Service Mesh to Splunk Cloud Platform. To export the logs, you need to provide the information about the HTTP Event Collector (HEC) on Splunk Cloud Platform through the Tanzu Service Mesh API.

All calls that users make to the Tanzu Service Mesh APIs are logged for audit purposes. You can send the API audit logs to your Splunk Cloud Platform instance for analysis and visualization of the API events and for retrieving relevant data through search.

Note:

Currently, you can export API audit logs only to Splunk Cloud Platform using an HTTP Event Collector (HEC) input.

Tanzu Service Mesh sends the logs to a specified HTTP Event Collector (HEC) input on Splunk Cloud Platform. For Tanzu Service Mesh to connect to the HEC input, you must configure an external Splunk account through the Tanzu Service Mesh API and provide the following information about the HEC in the account configuration:

  • HEC host - The URL that hosts the HEC.

  • HEC port - The port configured for the HEC endpoint.

  • HEC endpoint - The HEC endpoint to use. Typically, you use the /services/collector/event endpoint for JavaScript Object Notation (JSON)-formatted events or the services/collector/raw endpoint for raw events.

  • HEC token - The token for Tanzu Service Mesh to use to authenticate connections to HEC. For more information about HEC tokens, see the Splunk Cloud Platform documentation.

You must also create an external audit storage configuration for the Splunk account through the TSM API.

The following procedure describes how to provide the required HEC information to Tanzu Service Mesh through the API.

Prerequisites

  • Deploy a Splunk Cloud Platform instance.

Note:

Export of API audit logs from Tanzu Service Mesh has been tested with Splunk Cloud version 9.0.2209.3.

  • Secure your Splunk Cloud Platform instance with certificates. For more information, see the Splunk documentation. You must use certificates signed by a trusted third-party certificate authority (CA).

Note:

  • Tanzu Service Mesh does not support Splunk configured with self-signed certificates.

  • Update the Splunk configuration to use your certificates. The procedure below includes steps on how to update the Splunk configuration.

  • Configure an HEC input on Splunk Cloud Platform. The setup includes enabling HEC on Splunk Cloud Platform to allow use of HEC inputs. During the setup, ensure that you select Enable SSL for your HEC input because Tanzu Service Mesh supports export of logs over HTTPS only. Configure the remaining optional settings as necessary. Be sure to create an HEC token. For more information about configuring HEC on Splunk Cloud Platform, see the Splunk documentation.

  • Verify the setup of HEC by sending data to HEC. For information about sending data to HEC, see the Splunk documentation. Also see Example of sending data to HEC with an HTTP request.

Note:

When testing sending data to HEC, use https and ensure that you do not allow insecure connections with the -k or --insecure argument when using a curl command.

  • Know the URL of your Splunk Cloud Platform instance and the port on which it is accessible, and have the HEC token string ready.

  • Verify that you are familiar with Splunk concepts and terminology. To become familiar with Splunk concepts and terminology, see the Splunk Quick Reference Guide.

Procedure

  1. To create an external account for Splunk through the API, perform these steps.
    1. On the bottom bar of the Tanzu Service Mesh Console UI, click API Explorer.


      For information about how to access the Tanzu Service Mesh Console UI, see Access the Tanzu Service Mesh Console.

    2. On the API Explorer page, click the Rest HTTP tab.
    3. Go to the External Accounts API, click the PUT /v1alpha1/external-accounts/{id} endpoint, and click Try It Out.
    4. Under Parameters, enter an ID for the external Splunk account, for example, splunkaccount.
    5. Under Request Body, provide the following parameters.

      Parameter

      Description

      name

      Enter the ID of the external Splunk account that you entered under Parameters, for example, splunkaccount.

      description

      Enter a description of the account.

      provider

      Set to SPLUNK_ENT.

      provider_url

      Provide the URL of your Splunk Cloud Platform instance and the port on which it is accessible in the format https://{host}:{port}/{hec-endpoint}.

      • {host} is the name of the Splunk Cloud Platform instance that runs HEC.

      • {port} is the HEC port number.

      • {hec-endpoint} is the HEC endpoint to use. The services/collector/event endpoint is typically used to collect JSON-formatted events.

      Tanzu Service Mesh will use this parameter as the HEC host and port to connect to. An example of provider_url:

      https://http-inputs.my-splunk-inst.splunkcloud.com:8088/services/collector/event

      Important:

      Make sure that the URL uses the HTTPS protocol.

      authentication_type

      Set to TOKEN.

      auth_password (inside authentication)

      This parameter is not needed. Delete it.

      auth_token (inside authentication)

      Set access_key inside auth_token to the HEC token string.

      Note:

      • secret_access_key is not needed, so delete it.

      • Make sure that access_key contains only the HEC token string, such as B5A79AAD-D822-46CC-80D1-819F80D7BFB0. Do not add Splunk or any spaces before the token string.

      certificate_id

      Set to an empty string ("").
    6. Click Execute.

      The API returns a 201 status code to indicate that the external Splunk account was created.

  2. To create an external audit storage configuration for the Splunk account through the API, perform these steps.
    1. Go to the External Audit Storage API, click the PUT /v1alpha1/external-audit/storage endpoint, and click Try It Out.
    2. Under Request Body, provide the following parameters.

      Parameter

      Description

      infrastructure_account_id

      Set this parameter to the ID of the external Splunk account that you created in step 2.

      storage_type

      Set to SPLUNK_ENT.

      storage_config

      This parameter is not needed. Delete it from the request body.

      certificate_id

      Set to an empty string.
    3. Click Execute.

      The API returns a 200 status code to indicate that the external audit storage configuration was created and saved.

Results

Tanzu Service Mesh will send API audit logs to the specified HEC input on Splunk Cloud Platform.

Note:

  • If you no longer want to export API audit logs to your Splunk instance, delete the external Splunk account and the external audit storage configuration for Splunk through the API.

To delete the external Splunk account:

  1. In the API Explorer, go to the External Accounts API.

  2. Click the DELETE /v1alpha1/external-accounts/{id} endpoint and click Try It Out.

  3. Provide the ID of the Splunk account and click Execute.

To delete the external audit storage configuration for Splunk:

  1. Go to the External Audit Storage API.

  2. Click the DELETE /v1alpha1/external-audit/storage endpoint and click Try It Out.

  3. Click Execute.

  • If you want to modify the external Splunk account (for example, you need to provide a different HEC token in the account), delete the existing Splunk account and its external audit storage configuration and create a new account with the parameters you want. For instructions on deleting a Splunk account and its external audit storage configuration, see the first note above. For instructions on creating an external Splunk account, see steps 1 and 2 in the procedure.