STIG and NSA/CISA Hardening

By default, Tanzu Kubernetes Grid (TKG) workload clusters deployed by standalone management clusters are hardened to the levels shown in STIG Results and Exceptions and CIS Results and Exceptions. This topic explains how to further harden the clusters.

The methods depend on whether the cluster is class-based or plan-based, as described as described in Workload Cluster Types.

TKG releases are continuously validated against the Defense Information Systems Agency (DISA) Kubernetes Security Technical Implementation Guide (STIG) and NSA/CISA Kubernetes Hardening Guide.

Additional Kubernetes Hardening in Class-Based Workload Clusters

To increase the STIG- and CIS-compliance of Kubernetes in class-based workload clusters, configure them as described in the sections below.

For more granular exception handling, you can follow the workarounds listed in the exceptions tables in STIG Results and Exceptions and CIS Results and Exceptions.

STIG Hardening

To harden Kubernetes in class-based workload clusters to STIG standards, do either of the following before creating the cluster:

• Include the variable settings below in the cluster configuration file

• Set the variables in your local environment, for example with export

  ETCD_EXTRA_ARGS: "auto-tls=false;peer-auto-tls=false;cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_256_GCM_SHA384"
  KUBE_CONTROLLER_MANAGER_EXTRA_ARGS: "tls-min-version=VersionTLS12;profiling=false;tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_256_GCM_SHA384"
  WORKER_KUBELET_EXTRA_ARGS: "streaming-connection-idle-timeout=5m;tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_256_GCM_SHA384;protect-kernel-defaults=true"
  APISERVER_EXTRA_ARGS: "tls-min-version=VersionTLS12;tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_256_GCM_SHA384"
  KUBE_SCHEDULER_EXTRA_ARGS: "tls-min-version=VersionTLS12;tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_256_GCM_SHA384"
  CONTROLPLANE_KUBELET_EXTRA_ARGS: "streaming-connection-idle-timeout=5m;tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_256_GCM_SHA384;protect-kernel-defaults=true"
  ENABLE_AUDIT_LOGGING: true
  
  

NSA/CISA Hardening

To harden Kubernetes in class-based workload clusters to CIS standards, do the following before creating the cluster:

1. Review the Event Rate limit configuration below. If you want to change any settings, save the code to event-rate-config.yaml and change the settings as desired:

   apiVersion: eventratelimit.admission.k8s.io/v1alpha1
   kind: Configuration
   limits:
   - type: Namespace
     qps: 50
     burst: 100
     cacheSize: 2000
   - type: User
     qps: 10
     burst: 50
   

2. If you created event-rate-config.yaml with custom settings, base64-encode the file by running the following and recording the output string:

   • Linux: base64 -w 0 event-rate-config.yaml
   • Mac: base64 -b 0 event-rate-config.yaml

3. Do either of the following before creating the cluster:

   • Include the variable settings below in the cluster configuration file

   • Set the variables in your local environment, for example with export

     ETCD_EXTRA_ARGS: "auto-tls=false;peer-auto-tls=false;cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_128_GCM_SHA256"
     KUBE_CONTROLLER_MANAGER_EXTRA_ARGS: "profiling=false;terminated-pod-gc-threshold=500;tls-min-version=VersionTLS12;profiling=false;tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_128_GCM_SHA256"
     WORKER_KUBELET_EXTRA_ARGS: "read-only-port=0;authorization-mode=Webhook;client-ca-file=/etc/kubernetes/pki/ca.crt;event-qps=0;make-iptables-util-chains=true;streaming-connection-idle-timeout=5m;tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_128_GCM_SHA256;protect-kernel-defaults=true"
     APISERVER_EXTRA_ARGS: "enable-admission-plugins=AlwaysPullImages,NodeRestriction;profiling=false;service-account-lookup=true;tls-min-version=VersionTLS12;tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_128_GCM_SHA256"
     KUBE_SCHEDULER_EXTRA_ARGS: "profiling=false;tls-min-version=VersionTLS12;tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_128_GCM_SHA256"
     CONTROLPLANE_KUBELET_EXTRA_ARGS: "read-only-port=0;authorization-mode=Webhook;client-ca-file=/etc/kubernetes/pki/ca.crt;event-qps=0;make-iptables-util-chains=true;streaming-connection-idle-timeout=5m;tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_128_GCM_SHA256;protect-kernel-defaults=true"
     APISERVER_EVENT_RATE_LIMIT_CONF_BASE64: "<EVENT-RATE-CONFIG>"
     ENABLE_AUDIT_LOGGING: true
     

   Where <EVENT-RATE-CONFIG> is the base64-encoded value from the last step, or the following if you did not change the Event Rate limit configuration:

   • YXBpVmVyc2lvbjogZXZlbnRyYXRlbGltaXQuYWRtaXNzaW9uLms4cy5pby92MWFscGhhMQpraW5kOiBDb25maWd1cmF0aW9uCmxpbWl0czoKLSB0eXBlOiBOYW1lc3BhY2UKICBxcHM6IDUwCiAgYnVyc3Q6IDEwMAogIGNhY2hlU2l6ZTogMjAwMAotIHR5cGU6IFVzZXIKICBxcHM6IDEwCiAgYnVyc3Q6IDUwCg==

Additional OS Hardening in Class-Based Workload Clusters

To harden Ubuntu OS v20.04 in class-based workload clusters to STIG or CIS standards, create custom hardened VM images for the clusters by running Image Builder with the ansible_user_vars settings for STIG or CIS hardening, as described in Build a Linux Image

Hardening Plan-Based Workload Clusters

Legacy, non class-based TKG workload clusters deployed by standalone management clusters can be hardened by using ytt overlays. For information on how to customize plan-based TKG clusters using ytt, see Legacy Cluster Configuration with ytt.

You can create legacy clusters by setting allow-legacy-cluster to true in your CLI configuration as describe in Features in Tanzu CLI Architecture and Configuration.

STIG Hardening

To further harden plan-based TKG clusters, VMware provides a STIG hardening ytt overlay.

The following snippet is an ytt overlay to set tls-min-version (STIG: V-242378) on the api-server.

#@ load("@ytt:overlay", "overlay")
#@ load("@ytt:data", "data")
#@overlay/match missing_ok=True,by=overlay.subset({"kind":"KubeadmControlPlane"})
---
spec:
  kubeadmConfigSpec:
    clusterConfiguration:
      apiServer:
        extraArgs:
        #@overlay/match missing_ok=True
        tls-min-version: VersionTLS12

NSA/CISA Hardening

To further harden plan-based TKG clusters for NSA/CISA, VMware provides the following Antrea object specifications:

• NSA/CISA hardening: Antrea ClusterNetworkPolicies:

  The following Antrea ClusterNetworkPolicies specification for Network policies control sets a default policy for all Pods to deny all ingress and egress traffic and ensure that any unselected Pods are isolated.

  apiVersion: security.antrea.tanzu.vmware.com/v1alpha1
  kind: ClusterNetworkPolicy
  metadata:
    name: default-deny
  spec:
    priority: 150
    tier: baseline
    appliedTo:
      - namespaceSelector: {}
    ingress:
      - action: Drop              # For all Pods in every namespace, drop and log all ingress traffic from anywhere
        name: drop-all-ingress
        enableLogging: true
    egress:
      - action: Drop              # For all Pods in every namesapces, drop and log all egress traffic towards anywhere
        name: drop-all-egress
        enableLogging: true
  

• NSA/CISA hardening: Antrea network policy:

  The following Antrea network policy allows tanzu-capabilities-manager egress to kube-apiserver ports 443 and 6443.

  apiVersion: security.antrea.tanzu.vmware.com/v1alpha1
  kind: NetworkPolicy
  metadata:
    name: tanzu-cm-apiserver
    namespace: tkg-system
  spec:
    priority: 5
    tier: securityops
    appliedTo:
      - podSelector:
          matchLabels:
          app: tanzu-capabilities-manager
    egress:
      - action: Allow
        to:
        - podSelector:
            matchLabels:
            component: kube-apiserver
          namespaceSelector:
            matchLabels:
              kubernetes.io/metadata.name: kube-system
        ports:
        - port: 443
          protocol: TCP
        - port: 6443
          protocol: TCP
        name: AllowToKubeAPI
  

• NSA/CISA hardening: OPA template and constraints:

  The following example of uses OPA gatekeeper to restrict allowed images repositories.

  • OPA template:

    apiVersion: templates.gatekeeper.sh/v1beta1
    kind: ConstraintTemplate
    metadata:
      name: k8sallowedrepos
      annotations:
        description: Requires container images to begin with a repo string from a specified
          list.
    spec:
      crd:
        spec:
          names:
            kind: K8sAllowedRepos
          validation:
            # Schema for the `parameters` field
            openAPIV3Schema:
              type: object
              properties:
                repos:
                  type: array
                  items:
                    type: string
      targets:
        - target: admission.k8s.gatekeeper.sh
          rego:|
            package k8sallowedrepos
    
            violation[{"msg": msg}] {
              container := input.review.object.spec.containers[_]
              satisfied := [good| repo = input.parameters.repos[_] ; good = startswith(container.image, repo)]
              not any(satisfied)
              msg := sprintf("container <%v> has an invalid image repo <%v>, allowed repos are %v", [container.name, container.image, input.parameters.repos])
            }
    
            violation[{"msg": msg}] {
              container := input.review.object.spec.initContainers[_]
              satisfied := [good| repo = input.parameters.repos[_] ; good = startswith(container.image, repo)]
              not any(satisfied)
              msg := sprintf("container <%v> has an invalid image repo <%v>, allowed repos are %v", [container.name, container.image, input.parameters.repos])
            }
    

  • OPA constraints:

    apiVersion: constraints.gatekeeper.sh/v1beta1
    kind: K8sAllowedRepos
    metadata:
      name: repo-is-openpolicyagent
    spec:
      match:
        kinds:
          - apiGroups: [""]
            kinds: ["Pod"]
      parameters:
        repos:
          - "<ALLOWED_IMAGE_REPO>"
    

• NSA/CISA hardening: OPA mutations:

  The following example uses OPA mutation to set allowPrivilegeEscalation to false if it is missing in the pod spec.

  apiVersion: mutations.gatekeeper.sh/v1alpha1
  kind: Assign
  metadata:
    name: allow-privilege-escalation
  spec:
      match:
        scope: Namespaced
        kinds:
          - apiGroups: ["*"]
            kinds: ["Pod"]
        excludedNamespaces:
        - kube-system
      applyTo:
      - groups: [""]
        kinds: ["Pod"]
        versions: ["v1"]
      location: "spec.containers[name:*].securityContext.allowPrivilegeEscalation"
      parameters:
        pathTests:
        - subPath: "spec.containers[name:*].securityContext.allowPrivilegeEscalation"
          condition: MustNotExist
        assign:
          value: false
  

Antrea CNI is used in this guide for network hardening as it provides fine-grained control of network policies using tiers and the ability to apply a cluster-wide security policy using ClusterNetworkPolicy.

Open Policy Agent (OPA) is used instead of pod security policies as they were deprecated in Kubernetes 1.21.

ytt overlays, network policies and OPA policies are created in a way to make it easy for cluster admins to opt out of hardening controls for certain workloads. We suggest not completely opting out of the hardening practices and instead isolating workloads in namespaces where these hardening controls are not applied. Opting out of hardening controls also depends on the risk appetite of the TKG deployment.

NSA/CISA Kubernetes Hardening Guidance

Title	Compliant By Default?	Can be resolved?	Explanation/Exception
Allow privilege escalation	No	Yes	Resolved with OPA Gatekeeper Policy as well as mutations
Non-root containers	No	Yes	Resolved with OPA Gatekeeper Policy as well as mutations. Exception Some pods such as contour/envoy need root in order to function. Tanzu System Ingress needs to interact with the network
Automatic mapping of service account	No	Yes	Resolved with OPA Gatekeeper Mutation. Exception Gatekeeper needs access to the API server so its service accounts are automounted
Applications credentials in configuration files	No	No	Exception All of the detected credentials in config files were false positives as they were public keys
Linux hardening	No	Yes	Resolved with OPA gatekeeper constraint as well as mutation to drop all capabilities Exception Some pods such as contour/envoy need advanced privileges in order to function. Tanzu System Ingress needs to interact with the network
Seccomp Enabled	No	Yes	Resolved with OPA gatekeeper mutation to set a seccomp profile for all pods
Host PID/IPC privileges	No	Yes	A gatekeeper constraint has been added to prohibit all pods from running with PID/IPC
Dangerous capabilities	No	Yes	A gatekeeper constraint has been added to prohibit dangerous capabilities and a mutation has been added to set a default. Exception Some pods such as contour/envoy need advanced privileges in order to function. Tanzu System Ingress needs to interact with the network
Exec into container	No	No	Kubernetes ships with accounts that have exec access to pods this is likely needed by admins and a customer facing solution would be advised. Such as removing exec in RBAC for normal end users
Allowed hostPath	No	Yes	A gatekeeper constraint has been added to prevent the host path from being mounted
hostNetwork access	No	Yes	A gatekeeper constraint has been added to prevent the host network from being used. Exception The Kapp controller needs access to the host for tanzu to function and is the only pod outside the control plane allowed host network access
Exposed dashboard	Yes
Cluster-admin binding	No	No	A cluster admin binding is needed for k8s to start and should be the only one in the cluster
Resource policies	No	Yes	Fixed by setting a default for all pods via a gatekeeper mutation
Control plane hardening	Yes
Insecure capabilities	No	Yes	A gatekeeper constraint has been added to prohibit dangerous capabilities and a mutation has been added to set a default. Exception Some pods such as contour/envoy need advanced privileges in order to function. Tanzu System Ingress needs to interact with the network
Immutable container filesystem	No	Yes	A gatekeeper constraint has been added to prevent readOnlyRootFilesystems from being disabled. Exception Pods created by contour/envoy, fluentd, the kapp controller, telemetry agents, and all other data services that need to run on k8s Caution This mutation can cause issues within the cluster and may not be the wisest to implement.
Privileged container	No	Yes	By default all pods have privileged set to false but a constraint has been added to enforce that a user does not enable it.
Ingress and Egress blocked	No	Yes	A default deny cluster network policy can be implemented in Antrea
Container hostPort	No	Yes	A gatekeeper constraint has been added to ensure users do not use hostPorts. Exception The Kapp controller needs access to the host for tanzu to function and is the only pod outside the control plane allowed host network access
Network policies	No	Yes	A suite of network policies can be installed to ensure all namespaces have a network policy
Fluent Bit Forwarding to SIEM	No	Yes	Fluent bit needs to be installed and point at a valid output location
Fluent Bit Retry Enabled	No	Yes	Fluent bit needs to be installed by the user with retries enabled in the config
IAAS Metadata endpoint blocked	No	Yes	A Cluster Network Policy can be implemented to restrict all pods from hitting the endpoint