This topic describes how to enable and use Pod Security Policies in VMware Tanzu Kubernetes Grid Integrated Edition (TKGI).
Note: Kubernetes Pod Security Policy (PSP) support has been deprecated and PSP support will be entirely removed in Kubernetes v1.25. Kubernetes v1.23 and v1.24 provide beta support for Pod Security Admission. For more information, see Pod Security Admission and Enforce Pod Security Standards with Namespace Labels in the Kubernetes documentation.
In Kubernetes, a Pod Security Policy (PSP) is a cluster-level resource that controls security sensitive aspects of the pod specification. PodSecurityPolicy objects define a set of conditions that a pod must run with in order to be accepted into the system, as well as defaults for related fields. For more information, see Pod Security Policies in the Kubernetes documentation.
Note: When the PodSecurityPolicy option is enabled for a Kubernetes cluster plan, a cluster administrator must define the policy, role, and role binding that gives cluster users (developers) permission to deploy pods to the cluster. See Workflow for Enabling PSPs.
There are various ways to implement the use of PodSecurityPolicy objects in Kubernetes. One common approach is the use of role based access control (RBAC) objects: roles and role bindings. For a PodSecurityPolicy to take effect, the Kubernetes cluster User or ServiceAccount launching a workload must have the use
permission on the desired PodSecurityPolicy object.
A role contains rules that represent a set of permissions. A role can be defined within a namespace with a Role, or cluster-wide with a ClusterRole. For more information, see Role and ClusterRole in the Kubenetes documentation.
A role binding grants the permissions defined in a role to a user or set of users. A role binding holds a list of subjects (users, groups, or service accounts), and a reference to the role being granted. Permissions can be granted within a namespace with a RoleBinding, or cluster-wide with a ClusterRoleBinding. For more information, see RoleBinding and ClusterRoleBinding in the Kubenetes documentation.
Note: The most commonly used Kubernetes workloads (deployment, replication controller, for example) are spinning up pods using a service account. It is this entity that requires the use permission on the PSP. Just because the kubectl-user has the use permission on a PSP does not mean your workload will spin up (unless you are doing simple pod-level workloads using kubectl run). To grant a service account use permission for a PSP, the user attempting to create the role binding must also have use on that PSP.
Tanzu Kubernetes Grid Integrated Edition ships with two default PSPs: PKS Privileged and PKS Restricted.
PSP | Description |
---|---|
PKS Privileged | Allows privileged access to pod containers, which allows the container to do almost everything a host can do. See Privileged in the Kubernetes PSP documentation for more information. |
PKS Restricted | Restricts privileged access to pod containers. |
By default, when PSPs are enabled for a plan, the cluster administrator is bound to the PKS Privileged PSP. This policy grant gives the cluster administrator permission to deploy pods. Other users will not be able to deploy pods unless the cluster administrator creates and binds a PSP for such users. The PKS Restricted PSP is example for such purposes.
Note: The default PSPs will be updated and overwritten on upgrade. You should not edit the default PSPs.
Enabling PSPs is done during configuration of Tanzu Kubernetes Grid Integrated Edition in the Plan section of the tile configuration. Refer to the TKGI tile configuration documentation for your IaaS for details.
Note: To use PodSecurityPolicy features, you must use Ops Manager v2.10.17 or later.
Note: Windows in Kubernetes does not support the PodSecurityPolicy Admission Plugin feature.
A Kubernetes cluster created from a plan with the PodSecurityPolicy option enabled will require cluster users to have a binding that grants use
on an appropriate PSP to deploy pods. Enabling the PodSecurityPolicy option is a security feature. The design goal is to make the Kubernetes cluster more secure. Once enabled, PodSecurityPolicy acts on creation and modification of pods in that cluster or namespace, and determines the actions that should be permitted based on the requested security context and the applied PSP. See PSP Policy Order for more information on PSP application order when multiple PSPs are in use.
WARNING: Enabling the PodSecurityPolicy option on install or upgrade of Tanzu Kubernetes Grid Integrated Edition will prevent developers from using the cluster as they would expect unless the proper PSP, role, and role binding are configured by the cluster administrator before cluster deployment.
For new Tanzu Kubernetes Grid Integrated Edition installations, the PodSecurityPolicy option is not enabled by default in a plan. If the PodSecurityPolicy option is enabled, the cluster administrator will be able to deploy pods, but developers without a binding to a PSP will not. For new deployments, the cluster administrator will need to create one or more PSPs, roles, and role bindings for developers to deploy pods. Once you enable PSPs, you will need to define the RBAC objects and PSP for cluster users. See Workflow for Enabling PSPs and Configuring the pks-restricted PSP for Developers to Use with TKGI.
On upgrade of Tanzu Kubernetes Grid Integrated Edition, existing plans will not have the PodSecurityPolicy option enabled. If the PSP option is enabled for a plan that is in use and did not previously leverage PSPs, and the cluster is upgraded, the cluster administrator will need to create the appropriate PSPs, roles, and role bindings BEFORE upgrading the clusters associated to that plan. Cluster upgrades when enabling PSPs on an existing plan can have unpredictable results on workloads if the appropriate PSPs are not enabled. If you are considering enabling PSPs on an existing plan to apply to all associated clusters, the following set of tasks is recommended:
Before you select the PodSecurityPolicy check box and enable PSPs for a new or existing plan, complete the following tasks:
Alternatively, instead of enabling Upgrade all clusters errand, you can upgrade individual Kubernetes clusters through the TKGI Command Line Interface (TKGI CLI). For instructions on upgrading individual Kubernetes clusters, see Upgrading Clusters.
This section describes how to define and configure a pod security policy for developers to use to access a TKGI provisioned cluster that has enabled PSPs.
At a high-level, the steps for configuring a PSP with appropriate RBAC controls are as follows:
The first step is to define the PSP. We provide the PSP named pks-restricted
for general development work in Kubernetes. To onboard cluster users (developers), the best practice is to start with the pks-restricted
PSP. To create your own PSP, refer to the Kubernetes documentation.
To view the pks-restricted
PSP, run the following command:
kubectl get psp pks-restricted -o yaml
apiVersion: policy/v1
kind: PodSecurityPolicy
metadata:
annotations:
apparmor.security.beta.kubernetes.io/allowedProfileNames: runtime/default
apparmor.security.beta.kubernetes.io/defaultProfileName: runtime/default
kubectl.kubernetes.io/last-applied-configuration: |
{"apiVersion":"policy/v1","kind":"PodSecurityPolicy","metadata":{"annotations":{"apparmor.security.beta.kubernetes.io/allowedProfileNames":"runtime/default","apparmor.security.beta.kubernetes.io/defaultProfileName":"runtime/default","seccomp.security.alpha.kubernetes.io/allowedProfileNames":"docker/default","seccomp.security.alpha.kubernetes.io/defaultProfileName":"docker/default"},"name":"pks-restricted"},"spec":{"allowPrivilegeEscalation":false,"fsGroup":{"ranges":[{"max":65535,"min":1}],"rule":"MustRunAs"},"hostIPC":false,"hostNetwork":false,"hostPID":false,"privileged":false,"readOnlyRootFilesystem":false,"requiredDropCapabilities":["ALL"],"runAsUser":{"rule":"MustRunAsNonRoot"},"seLinux":{"rule":"RunAsAny"},"supplementalGroups":{"ranges":[{"max":65535,"min":1}],"rule":"MustRunAs"},"volumes":["configMap","emptyDir","projected","secret","downwardAPI","persistentVolumeClaim"]}}
seccomp.security.alpha.kubernetes.io/allowedProfileNames: docker/default
seccomp.security.alpha.kubernetes.io/defaultProfileName: docker/default
creationTimestamp: 2019-03-28T02:33:07Z
name: pks-restricted
resourceVersion: "261"
selfLink: /apis/policy/v1/podsecuritypolicies/pks-restricted
uid: d25bb0eb-5101-11e9-abc1-0a834658b12e
spec:
allowPrivilegeEscalation: false
fsGroup:
ranges:
- max: 65535
min: 1
rule: MustRunAs
requiredDropCapabilities:
- ALL
runAsUser:
rule: MustRunAsNonRoot
seLinux:
rule: RunAsAny
supplementalGroups:
ranges:
- max: 65535
min: 1
rule: MustRunAs
volumes:
- configMap
- emptyDir
- projected
- secret
- downwardAPI
- persistentVolumeClaim
The following ClusterRole grants permission to use the pks-restricted
PSP resource. You can use this role for onboarding cluster users and granting them cluster access.
Below is the system-provided ClusterRole.
kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1
metadata:
name: psp:restricted
rules:
- apiGroups:
- extensions
resources:
- podsecuritypolicies
resourceNames:
- pks-restricted # the psp we are giving access to
verbs:
- use
Note: Because this is a ClusterRole example, rather than a Role example, the namespace is omitted.
For more information about Role
and ClusterRole
, see Role and ClusterRole in the Kubernetes documentation.
For onboarding cluster users, you have two options based on the mode of authentication you have chosen for TKGI, either internal or external:
SERVICE-UUID-cluster-admin
, for example.user
or group
name.RoleBinding and ClusterRoleBinding configurations adhere to the following format:
apiVersion: rbac.authorization.k8s.io/v1
kind: ROLE-BINDING-TYPE
metadata:
name: ROLE-BINDING-NAME
namespace: NAMESPACE-NAME
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ROLE-REF-TYPE
name: BIND-TO
subjects:
- kind: ServiceAccount
name: ACCOUNT-ID
Where:
ROLE-BINDING-TYPE
is the binding type, either RoleBinding
or ClusterRoleBinding
.ROLE-BINDING-NAME
is your name for the role binding you are creating.NAMESPACE-NAME
is the name of the namespace the binding grants access within. namespace
is optional for a ClusterRole
binding.ROLE-REF-TYPE
is the roleRef
type, either Role
or ClusterRole
. If the binding type is Role
, the namespace
property is required.BIND-TO
is the name of the ClusterRole to bind to.ACCOUNT-TYPE
is the type of account managed by the binding, either ServiceAccount
, User
, or Group
.ACCOUNT-ID
is the account managed by the binding. The ACCOUNT-ID
is case sensitive.For example, the following ClusterRoleBinding examples bind specific groups or users to the psp:restricted
ClusterRole, which provides permissions to use the pks-restricted
PSP:
Example of binding a service account user:
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
name: psp:restricted
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole # Must be Role or ClusterRole
name: psp:restricted # The name of the ClusterRole to bind to
subjects:
- kind: ServiceAccount
name: cluster-user
Example of binding an externally authenticated user named jane
:
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
name: psp:restricted
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole # This must be Role or ClusterRole
name: psp:restricted # The name of the ClusterRole to bind to
subjects:
- kind: User
name: jane # Name is case sensitive
Example of binding externally authenticated group members:
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
name: psp:restricted
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole # This must be Role or ClusterRole
name: psp:restricted # The name of the ClusterRole to bind to
subjects:
- kind: Group
name: developers # Name is case sensitive
Note: Because these are ClusterRoleBinding examples, rather than RoleBinding examples, the namespace property is omitted.
For more information about RoleBinding
and ClusterRoleBinding
, see RoleBinding and ClusterRoleBinding in the Kubernetes documentation.
This section lists common kubectl commands for administering and managing PSPs and related objects in Kubernetes.
To view the default PSPs:
kubectl get psp
NAME PRIV CAPS SELINUX RUNASUSER FSGROUP SUPGROUP READONLYROOTFS VOLUMES
pks-privileged true * RunAsAny RunAsAny RunAsAny RunAsAny false awsElasticBlockStore,azureDisk,azureFile,cephFS,configMap,csi,downwardAPI,emptyDir,fc,flexVolume,flocker,gcePersistentDisk,glusterfs,iscsi,nfs,persistentVolumeClaim,projected,portworxVolume,quobyte,rbd,scaleIO,secret,storageos,vsphereVolume
pks-restricted false RunAsAny MustRunAsNonRoot MustRunAs MustRunAs false configMap,emptyDir,projected,secret,downwardAPI,persistentVolumeClaim
To view the details of a particular PSP:
kubectl describe psp <psp-name>
For example:
kubectl describe psp pks-privileged
Name: pks-privileged
Settings:
Allow Privileged: true
Allow Privilege Escalation: true
Default Add Capabilities: <none>
Required Drop Capabilities: <none>
Allowed Capabilities: *
Allowed Volume Types: awsElasticBlockStore,azureDisk,azureFile,cephFS,configMap,csi,downwardAPI,emptyDir,fc,flexVolume,flocker,gcePersistentDisk,glusterfs,iscsi,nfs,persistentVolumeClaim,projected,portworxVolume,quobyte,rbd,scaleIO,secret,storageos,vsphereVolume
Allow Host Network: true
Allow Host Ports: 0-65535
Allow Host PID: true
Allow Host IPC: true
Read Only Root Filesystem: false
SELinux Context Strategy: RunAsAny
User: <none>
Role: <none>
Type: <none>
Level: <none>
Run As User Strategy: RunAsAny
Ranges: <none>
FSGroup Strategy: RunAsAny
Ranges: <none>
Supplemental Groups Strategy: RunAsAny
Ranges: <none>
kubectl describe psp pks-restricted
Name: pks-restricted
Settings:
Allow Privileged: false
Allow Privilege Escalation: false
Default Add Capabilities: <none>
Required Drop Capabilities: ALL
Allowed Capabilities: <none>
Allowed Volume Types: configMap,emptyDir,projected,secret,downwardAPI,persistentVolumeClaim
Allow Host Network: false
Allow Host Ports: <none>
Allow Host PID: false
Allow Host IPC: false
Read Only Root Filesystem: false
SELinux Context Strategy: RunAsAny
User: <none>
Role: <none>
Type: <none>
Level: <none>
Run As User Strategy: MustRunAsNonRoot
Ranges: <none>
FSGroup Strategy: MustRunAs
Ranges: 1-65535
Supplemental Groups Strategy: MustRunAs
Ranges: 1-65535
To view the role bindings:
kubectl get rolebinding
To view the cluster role bindings:
kubectl get clusterrolebindings
In the results, the UUID-cluster-admin
is a service account.
NAME AGE
9a605c3d-b2f3-4961-b2a4-804885d85eba-cluster-admin 23s
9e495241-65b0-43bb-ac3a-120cb4ee6533-cluster-admin 2d1h
c990e9cc-29fc-4696-8673-fc69a93904f4-cluster-admin 2d1h
cluster-admin 8d
system:basic-user 8d
telemetry-agent 8d
To view service accounts:
kubectl get sa
NAME SECRETS AGE
2c92e23f-6480-4265-9456-4c81bffaa9c6 1 2d1h
38cb4886-8e19-4da3-a522-3ebaa3d6e553 1 44h
default 1 8d
To view the YAML for a cluster role binding:
kubectl get clusterrolebinding 9a605c3d-b2f3-4961-b2a4-804885d85eba-cluster-admin -o yaml
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
creationTimestamp: 2019-04-05T18:54:04Z
labels:
generated: "true"
name: 9a605c3d-b2f3-4961-b2a4-804885d85eba-cluster-admin
resourceVersion: "983229"
selfLink: /apis/rbac.authorization.k8s.io/v1/clusterrolebindings/9a605c3d-b2f3-4961-b2a4-804885d85eba-cluster-admin
uid: 2f24d7f0-57d4-11e9-abc1-0a834658b12e
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: cluster-admin
subjects:
- kind: ServiceAccount
name: 9a605c3d-b2f3-4961-b2a4-804885d85eba
namespace: default
To view a cluster role:
kubectl get clusterrole cluster-admin
NAME AGE
cluster-admin 8d
To view the YAML for a clusterrole:
kubectl get clusterrole cluster-admin -o yaml
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
annotations:
rbac.authorization.kubernetes.io/autoupdate: "true"
creationTimestamp: 2019-03-28T02:32:53Z
labels:
kubernetes.io/bootstrapping: rbac-defaults
name: cluster-admin
resourceVersion: "41"
selfLink: /apis/rbac.authorization.k8s.io/v1/clusterroles/cluster-admin
uid: c9c20648-5101-11e9-abc1-0a834658b12e
rules:
- apiGroups:
- '*'
resources:
- '*'
verbs:
- '*'
- nonResourceURLs:
- '*'
verbs:
- '*'
This section provides some troubleshooting tips for working with PodSecurityPolicy.
The source of many common problems related to PodSecurityPolicy–including mutating, workloads passing validation, and workloads failing at runtime–can usually be traced to proper policy ordering. If you are experiencing such issues, see Policy Order in the Kubernetes documentation.
The most commonly used Kubernetes workloads (deployment
and replication controller
, for example) are spinning up pods using a service account. It is this entity that requires the use
permission on the PSP. To grant a service account use
permission for a PSP, the user attempting to create the role binding must also have use
on that PSP.
Tanzu Kubernetes Grid Integrated Edition provides additional resources for implementing PSPs, including the two default PSPs provided with the product, example YAML files, and a PSP demo movie. Download the files at the following location: https://github.com/pivotal-cf/docs-pks/blob/master/demos/psp-demo.tar.gz.