This topic describes how to enable and use Pod Security Policies in VMware Tanzu Kubernetes Grid Integrated Edition (TKGI).
Note: When the PodSecurityPolicy option is enabled for a Kubernetes cluster plan, a cluster administrator must define the policy, role, and role binding that gives cluster users (developers) permission to deploy pods to the cluster. See Workflow for Enabling PSPs.
About Pod Security Policies
In Kubernetes, a Pod Security Policy (PSP) is a cluster-level resource that controls security sensitive aspects of the pod specification. PodSecurityPolicy objects define a set of conditions that a pod must run with in order to be accepted into the system, as well as defaults for related fields. For more information, see Pod Security Policies in the Kubernetes documentation.
There are various ways to implement the use of PodSecurityPolicy objects in Kubernetes. One common approach is the use of role based access control (RBAC) objects: roles and role bindings. For a PodSecurityPolicy to take effect, the Kubernetes cluster User or ServiceAccount launching a workload must have the use permission on the desired PodSecurityPolicy object.
A role contains rules that represent a set of permissions. A role can be defined within a namespace with a Role, or cluster-wide with a ClusterRole. For more information, see Role and ClusterRole in the Kubenetes documentation.
A role binding grants the permissions defined in a role to a user or set of users. A role binding holds a list of subjects (users, groups, or service accounts), and a reference to the role being granted. Permissions can be granted within a namespace with a RoleBinding, or cluster-wide with a ClusterRoleBinding. For more information, see RoleBinding and ClusterRoleBinding in the Kubenetes documentation.
Note: The most commonly used Kubernetes workloads (deployment, replication controller, for example) are spinning up pods using a service account. It is this entity that requires the use permission on the PSP. Just because the kubectl-user has the use permission on a PSP does not mean your workload will spin up (unless you are doing simple pod-level workloads using kubectl run). To grant a service account use permission for a PSP, the user attempting to create the role binding must also have use on that PSP.
Default Pod Security Policies in TKGI
Tanzu Kubernetes Grid Integrated Edition ships with two default PSPs: PKS Privileged and PKS Restricted.
| PSP | Description |
|---|---|
| PKS Privileged | Allows privileged access to pod containers, which allows the container to do almost everything a host can do. See Privileged in the Kubernetes PSP documentation for more information. |
| PKS Restricted | Restricts privileged access to pod containers. |
By default, when PSPs are enabled for a plan, the cluster administrator is bound to the PKS Privileged PSP. This policy grant gives the cluster administrator permission to deploy pods. Other users will not be able to deploy pods unless the cluster administrator creates and binds a PSP for such users. The PKS Restricted PSP is example for such purposes.
Note: The default PSPs will be updated and overwritten on upgrade. You should not edit the default PSPs.
Enabling Pod Security Policies in TKGI
Enabling PSPs is done during configuration of Tanzu Kubernetes Grid Integrated Edition in the Plan section of the tile configuration. Refer to the TKGI tile configuration documentation for your IaaS for details.
Note: To use PodSecurityPolicy features, you must use Ops Manager v2.10.17 or later.
Note: Windows in Kubernetes does not support the PodSecurityPolicy Admission Plugin feature.
A Kubernetes cluster created from a plan with the PodSecurityPolicy option enabled will require cluster users to have a binding that grants use on an appropriate PSP to deploy pods. Enabling the PodSecurityPolicy option is a security feature. The design goal is to make the Kubernetes cluster more secure. Once enabled, PodSecurityPolicy acts on creation and modification of pods in that cluster or namespace, and determines the actions that should be permitted based on the requested security context and the applied PSP. See PSP Policy Order for more information on PSP application order when multiple PSPs are in use.
WARNING: Enabling the PodSecurityPolicy option on install or upgrade of Tanzu Kubernetes Grid Integrated Edition will prevent developers from using the cluster as they would expect unless the proper PSP, role, and role binding are configured by the cluster administrator before cluster deployment.
New Installations of TKGI: PSPs Are Not Enabled for Any Plan
For new Tanzu Kubernetes Grid Integrated Edition installations, the PodSecurityPolicy option is not enabled by default in a plan. If the PodSecurityPolicy option is enabled, the cluster administrator will be able to deploy pods, but developers without a binding to a PSP will not. For new deployments, the cluster administrator will need to create one or more PSPs, roles, and role bindings for developers to deploy pods. Once you enable PSPs, you will need to define the RBAC objects and PSP for cluster users. See Workflow for Enabling PSPs and Configuring the pks-restricted PSP for Developers to Use with TKGI.
Upgrade of TKGI: PSPs Must Be Enabled per Plan
On upgrade of Tanzu Kubernetes Grid Integrated Edition, existing plans will not have the PodSecurityPolicy option enabled. If the PSP option is enabled for a plan that is in use and did not previously leverage PSPs, and the cluster is upgraded, the cluster administrator will need to create the appropriate PSPs, roles, and role bindings BEFORE upgrading the clusters associated to that plan. Cluster upgrades when enabling PSPs on an existing plan can have unpredictable results on workloads if the appropriate PSPs are not enabled. If you are considering enabling PSPs on an existing plan to apply to all associated clusters, the following set of tasks is recommended:
Workflow for Enabling PSPs
Before you select the PodSecurityPolicy check box and enable PSPs for a new or existing plan, complete the following tasks:
- Review all existing pods to collect their security requirements for each cluster associated with that plan. There are some open-source tools to assist in this task, such as kube-psp-advisor.
- Create the appropriate PSPs, roles, and role bindings for each cluster associated with that plan to allow the pods to run after upgrade. See Configuring PSP for Developers to Use as a starting point.
- Enable PSPs by selecting the PodSecurityPolicy checkbox in the appropriate plan.
- Review the pending changes and verify that Upgrade all clusters errand is enabled.
- Apply the changes to update the clusters that use the plan with PSPs enabled.
- Verify that the workloads are in desired state after upgrade.
- If Upgrade all clusters errand was not enabled, run it manually and redeploy.
Alternatively, instead of enabling Upgrade all clusters errand, you can upgrade individual Kubernetes clusters through the TKGI Command Line Interface (TKGI CLI). For instructions on upgrading individual Kubernetes clusters, see Upgrading Clusters.
Configuring the pks-restricted PSP for Developers to Use with TKGI
This section describes how to define and configure a pod security policy for developers to use to access a TKGI provisioned cluster that has enabled PSPs.
At a high-level, the steps for configuring a PSP with appropriate RBAC controls are as follows:
Step 1: Define the PSP
The first step is to define the PSP. We provide the PSP named pks-restricted for general development work in Kubernetes. To onboard cluster users (developers), the best practice is to start with the pks-restricted PSP. To create your own PSP, refer to the Kubernetes documentation.
To view the pks-restricted PSP, run the following command:
kubectl get psp pks-restricted -o yaml
apiVersion: policy/v1beta1
kind: PodSecurityPolicy
metadata:
annotations:
apparmor.security.beta.kubernetes.io/allowedProfileNames: runtime/default
apparmor.security.beta.kubernetes.io/defaultProfileName: runtime/default
kubectl.kubernetes.io/last-applied-configuration: |
{"apiVersion":"policy/v1beta1","kind":"PodSecurityPolicy","metadata":{"annotations":{"apparmor.security.beta.kubernetes.io/allowedProfileNames":"runtime/default","apparmor.security.beta.kubernetes.io/defaultProfileName":"runtime/default","seccomp.security.alpha.kubernetes.io/allowedProfileNames":"docker/default","seccomp.security.alpha.kubernetes.io/defaultProfileName":"docker/default"},"name":"pks-restricted"},"spec":{"allowPrivilegeEscalation":false,"fsGroup":{"ranges":[{"max":65535,"min":1}],"rule":"MustRunAs"},"hostIPC":false,"hostNetwork":false,"hostPID":false,"privileged":false,"readOnlyRootFilesystem":false,"requiredDropCapabilities":["ALL"],"runAsUser":{"rule":"MustRunAsNonRoot"},"seLinux":{"rule":"RunAsAny"},"supplementalGroups":{"ranges":[{"max":65535,"min":1}],"rule":"MustRunAs"},"volumes":["configMap","emptyDir","projected","secret","downwardAPI","persistentVolumeClaim"]}}
seccomp.security.alpha.kubernetes.io/allowedProfileNames: docker/default
seccomp.security.alpha.kubernetes.io/defaultProfileName: docker/default
creationTimestamp: 2019-03-28T02:33:07Z
name: pks-restricted
resourceVersion: "261"
selfLink: /apis/policy/v1beta1/podsecuritypolicies/pks-restricted
uid: d25bb0eb-5101-11e9-abc1-0a834658b12e
spec:
allowPrivilegeEscalation: false
fsGroup:
ranges:
- max: 65535
min: 1
rule: MustRunAs
requiredDropCapabilities:
- ALL
runAsUser:
rule: MustRunAsNonRoot
seLinux:
rule: RunAsAny
supplementalGroups:
ranges:
- max: 65535
min: 1
rule: MustRunAs
volumes:
- configMap
- emptyDir
- projected
- secret
- downwardAPI
- persistentVolumeClaim
Step 2: Create the Role
The following ClusterRole grants permission to use the pks-restricted PSP resource. You can use this role for onboarding cluster users and granting them cluster access.
Below is the system-provided ClusterRole.
kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1
metadata:
name: psp:restricted
rules:
- apiGroups:
- extensions
resources:
- podsecuritypolicies
resourceNames:
- pks-restricted # the psp we are giving access to
verbs:
- use
Note: Because this is a ClusterRole example, rather than a Role example, the namespace is omitted.
For more information about Role and ClusterRole, see Role and ClusterRole in the Kubernetes documentation.
Step 3: Create the Role Binding
For onboarding cluster users, you have two options based on the mode of authentication you have chosen for TKGI, either internal or external:
- Internal authentication uses a service account mechanism for role binding, such as
SERVICE-UUID-cluster-admin, for example. - External authentication uses the OpenID Connect (OIDC) identity provider to interface with an external system such as LDAP or AD. In this case the role binding references the
userorgroupname.
RoleBinding and ClusterRoleBinding configurations adhere to the following format:
apiVersion: rbac.authorization.k8s.io/v1
kind: ROLE-BINDING-TYPE
metadata:
name: ROLE-BINDING-NAME
namespace: NAMESPACE-NAME
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ROLE-REF-TYPE
name: BIND-TO
subjects:
- kind: ServiceAccount
name: ACCOUNT-ID
Where:
ROLE-BINDING-TYPEis the binding type, eitherRoleBindingorClusterRoleBinding.ROLE-BINDING-NAMEis your name for the role binding you are creating.NAMESPACE-NAMEis the name of the namespace the binding grants access within.namespaceis optional for aClusterRolebinding.ROLE-REF-TYPEis theroleReftype, eitherRoleorClusterRole. If the binding type isRole, thenamespaceproperty is required.BIND-TOis the name of the ClusterRole to bind to.ACCOUNT-TYPEis the type of account managed by the binding, eitherServiceAccount,User, orGroup.ACCOUNT-IDis the account managed by the binding. TheACCOUNT-IDis case sensitive.
For example, the following ClusterRoleBinding examples bind specific groups or users to the psp:restricted ClusterRole, which provides permissions to use the pks-restricted PSP:
-
Example of binding a service account user:
apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRoleBinding metadata: name: psp:restricted roleRef: apiGroup: rbac.authorization.k8s.io kind: ClusterRole # Must be Role or ClusterRole name: psp:restricted # The name of the ClusterRole to bind to subjects: - kind: ServiceAccount name: cluster-user -
Example of binding an externally authenticated user named
jane:apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRoleBinding metadata: name: psp:restricted roleRef: apiGroup: rbac.authorization.k8s.io kind: ClusterRole # This must be Role or ClusterRole name: psp:restricted # The name of the ClusterRole to bind to subjects: - kind: User name: jane # Name is case sensitive -
Example of binding externally authenticated group members:
apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRoleBinding metadata: name: psp:restricted roleRef: apiGroup: rbac.authorization.k8s.io kind: ClusterRole # This must be Role or ClusterRole name: psp:restricted # The name of the ClusterRole to bind to subjects: - kind: Group name: developers # Name is case sensitive
Note: Because these are ClusterRoleBinding examples, rather than RoleBinding examples, the namespace property is omitted.
For more information about RoleBinding and ClusterRoleBinding, see RoleBinding and ClusterRoleBinding in the Kubernetes documentation.
Administering PSPs, Roles, and RoleBindings
This section lists common kubectl commands for administering and managing PSPs and related objects in Kubernetes.
To view the default PSPs:
kubectl get psp
NAME PRIV CAPS SELINUX RUNASUSER FSGROUP SUPGROUP READONLYROOTFS VOLUMES
pks-privileged true * RunAsAny RunAsAny RunAsAny RunAsAny false awsElasticBlockStore,azureDisk,azureFile,cephFS,configMap,csi,downwardAPI,emptyDir,fc,flexVolume,flocker,gcePersistentDisk,glusterfs,iscsi,nfs,persistentVolumeClaim,projected,portworxVolume,quobyte,rbd,scaleIO,secret,storageos,vsphereVolume
pks-restricted false RunAsAny MustRunAsNonRoot MustRunAs MustRunAs false configMap,emptyDir,projected,secret,downwardAPI,persistentVolumeClaim
To view the details of a particular PSP:
kubectl describe psp <psp-name>
For example:
kubectl describe psp pks-privileged
Name: pks-privileged
Settings:
Allow Privileged: true
Allow Privilege Escalation: true
Default Add Capabilities: <none>
Required Drop Capabilities: <none>
Allowed Capabilities: *
Allowed Volume Types: awsElasticBlockStore,azureDisk,azureFile,cephFS,configMap,csi,downwardAPI,emptyDir,fc,flexVolume,flocker,gcePersistentDisk,glusterfs,iscsi,nfs,persistentVolumeClaim,projected,portworxVolume,quobyte,rbd,scaleIO,secret,storageos,vsphereVolume
Allow Host Network: true
Allow Host Ports: 0-65535
Allow Host PID: true
Allow Host IPC: true
Read Only Root Filesystem: false
SELinux Context Strategy: RunAsAny
User: <none>
Role: <none>
Type: <none>
Level: <none>
Run As User Strategy: RunAsAny
Ranges: <none>
FSGroup Strategy: RunAsAny
Ranges: <none>
Supplemental Groups Strategy: RunAsAny
Ranges: <none>
kubectl describe psp pks-restricted
Name: pks-restricted
Settings:
Allow Privileged: false
Allow Privilege Escalation: false
Default Add Capabilities: <none>
Required Drop Capabilities: ALL
Allowed Capabilities: <none>
Allowed Volume Types: configMap,emptyDir,projected,secret,downwardAPI,persistentVolumeClaim
Allow Host Network: false
Allow Host Ports: <none>
Allow Host PID: false
Allow Host IPC: false
Read Only Root Filesystem: false
SELinux Context Strategy: RunAsAny
User: <none>
Role: <none>
Type: <none>
Level: <none>
Run As User Strategy: MustRunAsNonRoot
Ranges: <none>
FSGroup Strategy: MustRunAs
Ranges: 1-65535
Supplemental Groups Strategy: MustRunAs
Ranges: 1-65535
To view the role bindings:
kubectl get rolebinding
To view the cluster role bindings:
kubectl get clusterrolebindings
In the results, the UUID-cluster-admin is a service account.
NAME AGE
9a605c3d-b2f3-4961-b2a4-804885d85eba-cluster-admin 23s
9e495241-65b0-43bb-ac3a-120cb4ee6533-cluster-admin 2d1h
c990e9cc-29fc-4696-8673-fc69a93904f4-cluster-admin 2d1h
cluster-admin 8d
system:basic-user 8d
telemetry-agent 8d
To view service accounts:
kubectl get sa
NAME SECRETS AGE
2c92e23f-6480-4265-9456-4c81bffaa9c6 1 2d1h
38cb4886-8e19-4da3-a522-3ebaa3d6e553 1 44h
default 1 8d
To view the YAML for a cluster role binding:
kubectl get clusterrolebinding 9a605c3d-b2f3-4961-b2a4-804885d85eba-cluster-admin -o yaml
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
creationTimestamp: 2019-04-05T18:54:04Z
labels:
generated: "true"
name: 9a605c3d-b2f3-4961-b2a4-804885d85eba-cluster-admin
resourceVersion: "983229"
selfLink: /apis/rbac.authorization.k8s.io/v1/clusterrolebindings/9a605c3d-b2f3-4961-b2a4-804885d85eba-cluster-admin
uid: 2f24d7f0-57d4-11e9-abc1-0a834658b12e
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: cluster-admin
subjects:
- kind: ServiceAccount
name: 9a605c3d-b2f3-4961-b2a4-804885d85eba
namespace: default
To view a cluster role:
kubectl get clusterrole cluster-admin
NAME AGE
cluster-admin 8d
To view the YAML for a clusterrole:
kubectl get clusterrole cluster-admin -o yaml
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
annotations:
rbac.authorization.kubernetes.io/autoupdate: "true"
creationTimestamp: 2019-03-28T02:32:53Z
labels:
kubernetes.io/bootstrapping: rbac-defaults
name: cluster-admin
resourceVersion: "41"
selfLink: /apis/rbac.authorization.k8s.io/v1/clusterroles/cluster-admin
uid: c9c20648-5101-11e9-abc1-0a834658b12e
rules:
- apiGroups:
- '*'
resources:
- '*'
verbs:
- '*'
- nonResourceURLs:
- '*'
verbs:
- '*'
Troubleshooting PSP Configuration
This section provides some troubleshooting tips for working with PodSecurityPolicy.
Verify Policy Order
The source of many common problems related to PodSecurityPolicy–including mutating, workloads passing validation, and workloads failing at runtime–can usually be traced to proper policy ordering. If you are experiencing such issues, see Policy Order in the Kubernetes documentation.
Verify Use Permission Grants
The most commonly used Kubernetes workloads (deployment and replication controller, for example) are spinning up pods using a service account. It is this entity that requires the use permission on the PSP. To grant a service account use permission for a PSP, the user attempting to create the role binding must also have use on that PSP.
PSP Files and Demo
Tanzu Kubernetes Grid Integrated Edition provides additional resources for implementing PSPs, including the two default PSPs provided with the product, example YAML files, and a PSP demo movie. Download the files at the following location: https://github.com/pivotal-cf/docs-pks/blob/master/demos/psp-demo.tar.gz.
