Kafka-Strimizi and Kafka-Edge Service not Reconciled

Sometimes Kafka Strimzi service does not reconcile.

Following are the scenarios to be considered if the Kafka-Strimizi or Kafka-Edge does not reconcile:

• If the topic is created, but CR is not reconciled due to etcd issue, then after the etcd issue is resolved, you might have to restart the strimzi-cluster-operator, entity-operator, or edge-entity-operator by executing the following commands.

  kubectl get pods -n tps-system | egrep "strimzi-cluster-operator" | cut -f1 -d' ' | xargs kubectl delete pod -n tps-system
  kubectl get pods -n tps-system | egrep "internal-entity-operator" | cut -f1 -d' ' | xargs kubectl delete pod -n tps-system
  kubectl get pods -n kafka-edge | egrep "edge-entity-operator" | cut -f1 -d' ' | xargs kubectl delete pod -n kafka-edge

• If the topic is not created, you must restart the topic operator.
  For internal Kafka, run the following command.

  kubectl get pods -n tps-system | egrep "internal-entity-operator" | cut -f1 -d' ' | xargs kubectl delete pod -n tps-system

  For Edge Kafka, run the following command.

  kubectl get pods -n kafka-edge | egrep "edge-entity-operator" | cut -f1 -d' ' | xargs kubectl delete pod -n kafka-edge

• If the topic is created, but CR not reconciled, execute the following command:

  kubectl get kafka-topic -A
  NAME                                                      CLUSTER    PARTITIONS   REPLICATION FACTOR   READY
  vsa-alerts                                                internal   4            1                    
  vsa-alerts---57fba39f4d6436566c678036822b624a918e720f     internal   1            1                    True
  vsa-baseline                                              internal   1            1                    
  vsa-baseline---73960ad4f7a1fcf1fb6f1fff79846a8d1f4f496f   internal   1            1                    True

  In the output, vsa-alerts and vsa-baseline are not reconciled. Here the topic is created with default partition of 1 as seen in the output which might lead to a performance impact. However, there is no functional impact.

  The partition can be increased by editing the CR as follows.

  • For internal Kafka, execute the following command:

    kubectl edit kafka—topic -n tps-system vsa-alerts---57fba39f4d6436566c678036822b624a918e720f

    Change partition to 4.

  • For Edge Kafka, execute the following command:

    kubectl edit kafkatopic -n kafka-edge metrics-ext-data---2c3fd3a5b7407dddf679abdbf87726bfad6a4dad

    Change partition to 4.

Kafka Edge not getting reconciled as few Kafka topics are conflicting with auto-created topics

Problem

kubectl get kt -A
NAMESPACE    NAME                                                                                               CLUSTER    PARTITIONS   REPLICATION FACTOR   READY
kafka-edge   inv-topo-raw                                                                                       edge       4            3                   
kafka-edge   inv-topo-raw---12b38b0b5a182d693f9783a3dd5e8b8cf14ba5bb                                            edge       4            3                    True
kafka-edge   vsa-events-raw                                                                                     edge       5            3                   
kafka-edge   vsa-events-raw---68e94faa465ed3f2dbe939b97ad68eed2cfb72a7                                          edge       5            3                    True
kafka-edge   vsa-metrics-raw                                                                                    edge       6            3                   
kafka-edge   vsa-metrics-raw---9115db2760b5b9c3da2d2e28e3577aff10a9f2cc                                         edge       6            3                    True

Solution

• Stop all the Collectors and Enrichment.
• Delete the auto-created topics.

  kubectl delete kt -n kafka-edge inv-topo-raw---12b38b0b5a182d693f9783a3dd5e8b8cf14ba5bb vsa-events-raw---68e94faa465ed3f2dbe939b97ad68eed2cfb72a7 vsa-metrics-raw---9115db2760b5b9c3da2d2e28e3577aff10a9f2cc
  
  Output
  kafkatopic.kafka.strimzi.io "inv-topo-raw---12b38b0b5a182d693f9783a3dd5e8b8cf14ba5bb" deleted
  kafkatopic.kafka.strimzi.io "vsa-events-raw---68e94faa465ed3f2dbe939b97ad68eed2cfb72a7" deleted
  kafkatopic.kafka.strimzi.io "vsa-metrics-raw---9115db2760b5b9c3da2d2e28e3577aff10a9f2cc" deleted

Wait for Kafka Edge to reconcile.

kubectl get app kafka-edge
kafka-edge