To start the data flow for VMware Telco Cloud Automation Pipeline reports, you must configure the VMware Telco Cloud Service Assurance with Kafka brokers in Airflow, or Edge Kafka, or any external Kafka.

For information about how to view the VMware Telco Cloud Automation reports, see View Pipeline Reports in VMware Telco Cloud Service Assurance User Guide.

Prerequisites

Ensure that you configure either Kafka brokers in Airflow, or Edge Kafka, or any external Kafka for ingesting data into VMware Telco Cloud Service Assurance.
  • If you are using Kafka brokers in Airflow for ingesting the data into VMware Telco Cloud Service Assurance, then perform the following steps:
    1. For Kafka brokers, navigate to Airflow > Variables and configure kafka_brokers variable.
      Note: Airflow used here is a third-party platform.
      Following is an example of kafka_brokers configuration in Airflow: 
      [{
    "bootstrap_servers": ["kafka-edge:32092 "],
    "topic": "tca-stages",
    "security_protocol": "SASL_SSL",
    "additional_properties" : {
        "sasl_mechanism": "SCRAM-SHA-512",
        "sasl_plain_username": "kafka-scram-sha-512-client-credentials",
        "sasl_plain_password" : "RDFYek9vSTJCNGRYCg=="
    },
    "password": "Y2EucGFzc3dvcmQK",
    "ssl_cafile": "LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSUZMVENDQXhXZ0F3SUJBZ0lVVlcrT2lEYVlUM3QyeWh6T3lidHJEelFrcXB3d0RRWUpLb1pJaHZjTkFRRU4KQlFBd0xURVRNQkVHQTFVRUNnd0thVzh1YzNSeWFXMTZhVEVXTUJRR0ExVUVBd3dOWTJ4MWMzUmxjaTFqWVNCMgpNREFlRncweU1qQTVNVE14TWpJMU5UVmFGdzB5TXpBNU1UTXhNakkxTlRWYU1DMHhFekFSQmdOVkJBb01DbWx2CkxuTjBjbWx0ZW1reEZqQVVCZ05WQkFNTURXTnNkWE4wWlhJdFkyRWdkakF3Z2dJaU1BMEdDU3FHU0liM0RRRUIKQVFVQUE0SUNEd0F3Z2dJS0FvSUNBUUMyR2l6cTl0aEs2aHhqaS9PeDRlNHprVVdDK3hWQzVNTW5ybGlzSWUrWApyMXFVWllnNTJxYWxJc0NaeGlER1FiM3BMb3R4bTJGNEhOQm10NlJuVWQ3QkJwaHFkcys3MjFSanpRZ1BZbmYyCkVtU0doOHMrai8wWlMzMmQ0ZUtKKzMwMVlSa0ljSkNQa3dTOWN5SkNVMU95OWoyTG9FZ21PNnlSVmplVm9YaFEKYjJ3ZmpqY2d1bTFkYmNYanNSY1R1cHRxclk4MlJVbWVoL0pPbkh1d016TUFFUElBYU9WZEpvVytrRW5oTFF0Zwp3cEUvMHltNncxelhWZm5iSjlXZUNkcnUwaEdJZ2ZkaWZ1MUdJTnM4UDVCcmU2TGZhcjRvZmlYMXJLNENHMHZWCkxKTTJoS29DYm1zRGhpNXc3azgxZ1V4b3hicDBKMTZWQWs5UWtCdnZwVG5zZnBhTDh5ZWMwMTBRS1ZtS0VUQWgKMHFVZFhxRy9XaUE5bVZlaU1lY1owSlp1RTViY2hsZnVrdTFoT0dBRlVvdVVDaDJuV1lYcHg5azdueW01VlN5ZQpEb3puSU8rSkpLMWREVlZwbUVJaUJSbzRrZTBySVhBZ0ZMc3F1U2tLNStlWEZBUkJLVEdISUlEWWYzeFZKdVJ5CkVpWkQ4NGdrK0RpSC9kTThGWkFrTXpjRGdCaUdKTjA0QysralZNYXZqd0pNRzlVV2FhaGgzV3UraHBlY2g3VnMKT0VtaXRHRGxTWUszWHdMNktYSnUrRjMwZkNyYlBGc2xIaTFmTjR5OVRGWWo0UmFOSmREUG0rQmRtMDEvR0taUQpjakhNbWU0VzhTa3J4TlhxLzhwWTJhdVE4cEVYWmlWdlNPRHhDRmMwbEQzVisxUXo0WVVrZXhQanBuNTVUQ1RtCjd3SURBUUFCbzBVd1F6QWRCZ05WSFE0RUZnUVVsR0traFpZTTNUN0FyQmM1SGRLT2w1SWJlSDh3RWdZRFZSMFQKQVFIL0JBZ3dCZ0VCL3dJQkFEQU9CZ05WSFE4QkFmOEVCQU1DQVFZd0RRWUpLb1pJaHZjTkFRRU5CUUFEZ2dJQgpBSG9MM2I5V1JWZUc5dElPQzROcCtZanVWclBiaEtoaTY0UU8vVEw0OVdJVjNRYkxsV0tlRkhacUtQNkJKV29qCi9EYkFtUVNaeWxmbkNzejZubll0RDNJcnFjN2w5WENGNjAwZEZaRCtGNkp1dnpCdlFxR3VJYlAza0czOHlDdU4KQWdyWHBiaXlRQXhRMzhXSnRFSlc4UHFzVzZtdnFSSEY2YUt4VXg4eDF4dWZKSHY4aWcxcG9PT1VqNWZ1Y044QwpQUWY1NGEzWm5YajA2K0QxM1Q2dnh6dzRlMVQyVllYWEtCSEFDL0hIZ095akZpbUhJeDg3eU9Dczduc2Q3c0NwCnU4RE9aT21URWRlUEkxcW95R2w5VmNBdjdVdERjdHI2dmNmdHFHZ2dMNUtEaTVFd09odDdjL1VuNEd0OFQwQTUKNDRWejJvdnZvcDFkU1JuNWplVHFocmxxUTRCVENrb0tSUFIxeEduMmFid1oxOWJFMEVzOXhjMkhWVEk4cDBDYwpJT085SW1mcVptVlBqQjc3cmZSZVpNWmRtTmpqOVVmeHBJMGZvcXZKN21kTTFYNzR1S2tCbTVkbG1sdUFRRE5rCk9BUFdhMnVvVXJEOGhkSUgrT0MrTzFueE02SW9Odm85aU5sdDFUaGl1RC9Mb2EvMUw0YWljQk9pK2dKbjk0UWEKclpQVlo5d2RxNjYvT0VoNlpyZmtadU1iMmVaRUZmZDJzN1dPYW54V0s1YXlNTTZKbmJLUUhjQ25iUWJZUG1lVQpNenF5c090SWxqeWQ1VnVQbVlzLyt1RUd5S1V6U2RIN1NQYTloeXFGWWJKUmJTY09yVFdhNTlTazlCZ1dwREtqClJ0ZkdHdzRieUZsd2FNbW5kYk11L25hU0M2V005eVVsSWJFdmJzWk9hTDUxCi0tLS0tRU5EIENFUlRJRklDQVRFLS0tLS0K"
}]
      Get the passwords and certificate by executing the following commands from the VMware Telco Cloud Service Assurance deployment VM: 
      ssl_cafile:
export CLUSTER_NAME=edge
EDGENS=kafka-edge
kubectl get secret -n $EDGENS $CLUSTER_NAME-cluster-ca-cert -o jsonpath='{.data.ca\.crt}'

password:
kubectl get secret -n $EDGENS $CLUSTER_NAME-cluster-ca-cert -o jsonpath='{.data.ca\.password}' > ca.password

sasl_plain_password:
export USER_NAME=kafka-scram-sha-512-client-credentials
export SCRAM_PASSWORD_FILE_PATH=user-scram.password
kubectl get secret -n $EDGENS $USER_NAME -o jsonpath='{.data.password}' > $SCRAM_PASSWORD_FILE_PATH
export SCRAM_PASSWORD=`cat $SCRAM_PASSWORD_FILE_PATH`
    2. Run the following statefulset commands:
      kubectl get statefulset -n airflow 
NAME                 READY   AGE
airflow-postgresql   1/1     78d
airflow-redis        1/1     78d
airflow-worker       3/3     78d
capv@airflow-cnc-control-plane-vjbx5 [ ~ ]$
  • If you are using Edge Kafka for ingesting data into VMware Telco Cloud Service Assurance, then perform the following steps from the deployment host.
    1. Install Kafka.
    2. Export KUBECONFIG=<KUBECONFIG-file-location>
      EDGENS=kafka-edge
kubectl get secret -n $EDGENS $CLUSTER_NAME-cluster-ca-cert -o jsonpath='{.data.ca\.crt}' | base64 --decode > ca.crt
kubectl get secret -n $EDGENS $CLUSTER_NAME-cluster-ca-cert -o jsonpath='{.data.ca\.password}' | base64 --decode > ca.password
export CERT_FILE_PATH=ca.crt
export CERT_PASSWORD_FILE_PATH=ca.password
export KEYSTORE_LOCATION=cacerts
export PASSWORD=`cat $CERT_PASSWORD_FILE_PATH`
export CA_CERT_ALIAS=strimzi-kafka-cert
keytool -noprompt -importcert -alias $CA_CERT_ALIAS -file $CERT_FILE_PATH -keystore $KEYSTORE_LOCATION -keypass $PASSWORD -storepass $PASSWORD
export USER_NAME=kafka-scram-sha-512-client-credentials
export SCRAM_PASSWORD_FILE_PATH=user-scram.password
kubectl get secret -n $EDGENS $USER_NAME -o jsonpath='{.data.password}' | base64 --decode > $SCRAM_PASSWORD_FILE_PATH
export SCRAM_PASSWORD=`cat $SCRAM_PASSWORD_FILE_PATH`
<<KAFKALOCATION>>/bin/kafka-console-producer.sh --broker-list kafka-edge:32092 --producer-property security.protocol=SASL_SSL --producer-property sasl.mechanism=SCRAM-SHA-512 --producer-property ssl.truststore.password=$PASSWORD --producer-property ssl.truststore.location=$PWD/cacerts --producer-property sasl.jaas.config="org.apache.kafka.common.security.scram.ScramLoginModule required username=\"$USER_NAME\" password=\"$SCRAM_PASSWORD\";" --topic metrics < ${5Gdatadump}
  • If you are using external Kafka to ingest data into VMware Telco Cloud Service Assurance, then perform the following steps:
    1. Install Kafka in any RHEL host.
    2. Start Zookeeper.
      ${KafkaInstallLocation}/bin/zookeeper-server-start.sh -daemon ${KafkaInstallLocation}/config/zookeeper.properties
    3. Start Kafka Server.
      ${KafkaInstallLocation}/bin/kafka-server-start.sh -daemon /${KafkaInstallLocation}/config/server.properties
    4. Start Kafka Producer.
      ${KafkaInstallLocation}/bin/kafka-console-producer.sh --bootstrap-server ${kafkahost}:${kafkaport} --topic ${KafkaTopicname} < ${5Gdatadump}

Procedure

  1. Configure Kafka Mapper for Pipeline reports. For information about configuring Kafka Mapper, see the Add Kafka Mapping section.
    For VMware Telco Cloud Automation Pipeline reports, you can use the following Kafka Mapper file content: 
    {
  "instance": "tca-stages",
  "tags.error_msg": "$.error_msg",
  "tags.event": "$.additional_data.[?(@.event)].event",
  "properties.entityType": "$.fsm_state",
  "processedTimestamp": "$.timestamp.transform_date(\"yyyy-MM-dd HH:mm:ss.SSSSSS\")",
  "tags.logs": "$.additional_data.[?(@.logs)].logs",
  "tags.task": "$.additional_data.[?(@.task)].task",
  "type": "KafkaCollector",
  "tags.site_id": "$.site_id",
  "properties.deviceName": "$.fsm_state",
  "metrics.preVal": "1",
  "tags.clustername": "$.additional_data.[?(@.clustername)].clustername",
  "metricType": "tca-stages",
  "properties.entityName": "$.fsm_state",
  "tags.fsm_status": "$.fsm_status",
  "tags.fsm_state": "$.fsm_state",
  "properties.dataSource": "$.site_id",
  "timestamp": "$.timestamp.transform_date(\"yyyy-MM-dd HH:mm:ss.SSSSSS\")",
  "properties.deviceType": "$.fsm_state",
  "tags.dagrunid": "$.additional_data.[?(@.dagrunid)].dagrunid"
}

  2. Configure Kafka Collector. For more information about configuring Kafka Collector, see Configuring the Kafka Collector topic.