Headless Service with StatefulSet

StatefulSets are used to manage stateful applications, such as databases or other applications, that keep track of their state. By using StatefulSets, a set of pods can be deployed and scaled within a global namespace, ensuring that they are ordered and unique. Headless service is a regular Kubernetes service where the spec.clusterIP is explicitly set to "None" and spec.type is set to "ClusterIP". Instead, SRV records are created for all the named ports of service's endpoints.

Prerequisite

Onboard the clusters where your services are deployed to Tanzu Service Mesh. For a global namespace, it can be a single cluster with multiple namespaces.
Create a global namespace to which you can add the headless service. For information about creating a global namespace and adding services to it, see Connect Services Across Clusters with a Global Namespace.
Create a Headless service with the label and set the clusterIP field to None.

Context

Deploying and replicating stateful applications poses the following challenges:

Pod replicas cannot be created or deleted at the same time. StatefulSet will not create the next pod until the previous pod is up and running.
Pods are not interchangable. Pod replicas have a persistent identifier across any rescheduling.
Continous data synchronization between pods is necessary to maintain same states.
If all pods die or the clusters running these pods crash, data will be lost.Each pod should have its own persistent storage with replicable data and pod state, stored in the pod's own storage, so that when a pod dies, the Persistent Pod Identifiers will reattach the volume to the replaced pod. Reattaching persistent volumes requires remote storage.
StatefulSet pods get fixed ordered names ($statefulsetname-$ordinal). For example, if you create a StatefulSet with a name of mongo with three replicas, the replicated pods get names mongo-0, mongo-1, and mongo-2.
Most importantly, the stateful workloads often has to reach a specific pod directly (for example, during database write operations) or have pod-pod communication, without load balancing.

Headless Services are the best solution for the above issues. Tanzu Service Mesh implements Headless service for stateful applications that allows clients to directly access pods without using Kubernetes' load balancing. If you deploy an application within a global namespace as StatefulSet with Headless services, Tanzu Service Mesh supports sending traffic to those services from other pods within the cluster or from remote clusters using the Headless service name. Each pod from StatefulSet gets its own DNS name of the form: ${podname}.{governing service name}. When a pod restarts, IP address changes but the name and endpoint remains the same.

When the StatefulSet is in the same global namespace as the source traffic, then the client will use the Kubernetes service associated with the statefulset to resolve the endpoint addresses. Hostnames resolve to all statefulset pod endpoint addresses.
When a client is in a remote cluster but within the same global namespace, it resolves to a number of virtual IPs corresponding to the StatefulSet replica count. Routing is handled by existing global namespace service entries and virtual services.

Creating a Headless Service

A headless service defined in one of the namespaces of a global namespace having the following definition can be extended by creating a selectorless headless service in the other namespaces of the global namespace.
```
apiVersion: v1 
kind: Service 
metadata: 
    name: foo 
spec: 
   clusterIP: None 
   selector: 
        key1: val1 
        key2: val2 
   ports: 
     - port: 8080 
       protocol: TCP
   type: ClusterIP
```

Create a headless service in the other global namespace namespaces. The selectors which would have been defined in the spec.selector section will be defined in a custom annotation:

apiVersion: v1 
kind: Service 
metadata: 
    name: foo 
    annotations: tsm.tanzu.vmware.com/endpoints.statefulset:    '{"key1": "val1", "key2": "val2"}' 
spec: 
    clusterIP: None 
    ports: 
        - port: 8080 
          protocol: TCP
    type: ClusterIP

Tanzu Service Mesh will auto manage the endpoints for this headless service in the namespaces where the selectorless headless service is defined.

Use case: Kafka Installation with Headless Services for access within a global namespace

Create two namespaces, 'nsOne' and 'nsTwo' on the cluster, 'clusterOne'.

k --context clusterOne create ns nsOne 
k --context clusterOne create ns nsTwo

Create a namespace 'nsOne' on the cluster, 'clusterTwo'.
```
k --context clusterTwo create ns nsOne 
```
Create a global namespace called 'kafka-gns' having the following members: 'clusterOne' / 'nsOne', 'clusterOne' / 'nsTwo', and 'clusterTwo' / 'nsOne'.

Install Kafka on 'clusterOne' / 'nsOne'.

#add helm repo
helm repo add bitnami https://charts.bitnami.com/bitnami 
#install kafka on clusterOne/nsOne
helm install kafka --kube-context=clusterOne -n nsOne --set replicaCount=3 bitnami/kafka

Create the following headless service with no selectors in the other namespaces of the global namespaces- 'clusterOne' / 'nsTwo' and 'clusterTwo' / 'nsOne'.

apiVersion: v1 
kind: Service 
metadata: 
  annotations: tsm.tanzu.vmware.com/endpoints.statefulset: '{"app.kubernetes.io/component":"kafka","app.kubernetes.io/instance":"kafka","app.kubernetes.io/name":"kafka"}' 
    name: kafka-headless 
spec: 
  clusterIP: None 
  ports: 
   - name: tcp-client 
    port: 9092 
    protocol: TCP 
    targetPort: kafka-client 
  - name: tcp-internal 
    port: 9093 
    protocol: TCP 
    targetPort: kafka-internal

Test the setup by running a producer and consumer from any of the global namespace members. The following command would start and open a prompt within a temporary kafka-client container.
```
kubectl --context=${cluster} --namespace ${ns} run kafka-client --image docker.io/bitnami/kafka:3.1.0-debian-10-r89 --rm -it --command -- sh
```

At the prompt within the temporary kafka-client container, create a producer and a consumer using the . service name.

# create topic and populate it with messages 
kafka-producer-perf-test.sh --topic topic-foo --num-records 10000 --throughput -1 --record-size 1000 --producer-props bootstrap.servers=kafka:9092 
# consume messages from the previously created topic 
kafka-consumer-perf-test.sh --bootstrap-server kafka:9092 --topic topic-foo --messages 10000 --timeout 10000

Advantages of Headless Services

Direct access to each pod.
Easy Pod discovery in the StatefulSet.
Pods can be addressed more generally by using their DNS names.
Utilizes each pod's sticky identity in a stateful service (i.e. you can address a specific pod by name).
Write operations are synchronized.