When deploying NCP, it is important to take steps to secure both the Kubernetes and the NSX-T environments.
Restrict NCP to Run Only on Designated Nodes
NCP has access to the NSX-T management plane and should be restricted to run only on designated infrastructure nodes. You can identify these nodes with an appropriate label. A nodeSelector for this label should then be applied to the NCP ReplicationController specification/ For example,
nodeSelector: nsx-infra: True
You can also use other mechanisms, such as affinity, to assign pods to nodes. For more information, see https://kubernetes.io/docs/concepts/configuration/assign-pod-node.
Ensure that the Docker Engine is Up To Date
Docker periodically releases security updates. An automated procedure should be implemented to apply these updates.
Disallow NET_ADMIN and NET_RAW Capabilities of Untrusted Containers
Linux capabilities NET_ADMIN and NET_RAW can be exploited by attackers to compromise the pod network. You should disable these two capabilities of untrusted containers. By default, NET_ADMIN capability is not granted to a non-privileged container. Be wary if a pod specification explicitly enables it or sets the container to be in a privileged mode. In addition, for untrusted containers, disable NET_RAW by specifying NET_RAW in the list of dropped capabilities in the SecurityContext configuration of the container's specification. For example,
securityContext: capabilities: drop: - NET_RAW - ...
Role-Based Access Control
Kubernetes uses Role-Based Access Control (RBAC) APIs to drive authorization decisions, allowing administrators to dynamically configure policies. For more information, see https://kubernetes.io/docs/admin/authorization/rbac.
Typically, the cluster administrator is the only user with privileged access and roles. For user and service accounts, the principle of least privilege must be followed when granting access.
The following guidelines are recommended:
Restrict access to Kubernetes API tokens to pods which need them.
Restrict access to NCP ConfigMap and NSX API client certificate's TLS secrets to the NCP pod.
Block access to Kubernetes networking API from pods that do not require such access.
Add a Kubernetes RBAC policy to specify which pods can have access to the Kubernetes API.
Recommended RBAC Policy for the NCP Pod
Create the NCP pod under a ServiceAccount and give this account a minimum set of privileges. Additionally, do not allow other pods or ReplicationControllers to access the ConfigMap and TLS Secrets that are mounted as volumes for the NCP ReplicationController and NSX node agent.
The following example shows how to specify roles and role bindings for NCP:
Cluster wide role to read, watch and get resources ---- kind: ClusterRole # Set the apiVersion to v1 if running with OpenShift apiVersion: rbac.authorization.k8s.io/v1beta1 metadata: name: ncp-cluster-role rules: - apiGroups: - "" - extensions resources: - deployments - endpoints - pods - namespaces - networkpolicies - nodes - replicationcontrollers - services verbs: - get - watch - list ---- Cluster wide role to read, watch, get and modify ingresses ---- kind: ClusterRole # Set the apiVersion to v1 if running with OpenShift apiVersion: rbac.authorization.k8s.io/v1beta1 metadata: name: ncp-ingress-role rules: - apiGroups: - "" - extensions resources: - ingresses verbs: - get - watch - list - update - patch - apiGroups: - extensions resources: - ingresses/status verbs: - replace - update - patch ---- Bind roles to ServiceAccount belonging to NCP ---- # Set the apiVersion to v1 if running with OpenShift apiVersion: rbac.authorization.k8s.io/v1beta1 kind: ClusterRoleBinding metadata: name: ncp-cluster-role-binding roleRef: # Comment out the apiGroup if running with OpenShift apiGroup: rbac.authorization.k8s.io kind: ClusterRole name: ncp-cluster-role subjects: - kind: ServiceAccount name: ncp-svc-account namespace: ncp-deployed-ns-name ---- ---- # Set the apiVersion to v1 if running with OpenShift apiVersion: rbac.authorization.k8s.io/v1beta1 kind: ClusterRoleBinding metadata: name: ncp-ingress-role-binding roleRef: apiGroup: rbac.authorization.k8s.io kind: ClusterRole name: ncp-ingress-role subjects: - kind: ServiceAccount name: ncp-svc-account namespace: ncp-deployed-ns-name
The TLS Secret that is created using the Kubernetes API for the NSX-T client certificate and the private key pair are accessible to any pod that has access to the Kubernetes API server. Similarly, when a pod is created with no service account, it is automatically assigned the default service account in the same namespace which auto mounts the token to access Kubernetes API. Therefore, access to these tokens must be restricted to pods which need them.