This topic explains how you can use a Kubernetes profile in Tanzu Kubernetes Grid Integrated Edition (TKGI) to override the default Identity Provider (IDP).

Overview

The TKGI UAA pane configures a default IDP for all the clusters that TKGI creates. You can use a Kubernetes profile to override this default IDP.

The Kubernetes profile applies a custom OIDC-compatible IDP to a cluster by deploying an OIDC connector as a service pod on the cluster.

The following diagram provides an overview of how this configuration works:

OIDC diagram

  • OIDC service provider cluster with external hostname dex-host.example.com
    • Hosts a dex pod that accesses the LDAP server
    • Publishes its OIDC service to dex.example.com:32000
  • OIDC Kubernetes profile has the URL of the OIDC service
  • Host cluster with external hostname cluster.example.com
    • Uses the OIDC Kubernetes profile
    • Calls the OIDC service at dex.example.com:32000 to authenticate the user whenever a user requests an app hosted on the cluster

The Kubernetes profile in this topic deploys dex as an OIDC provider, but you can use any OIDC service.

For more information and other uses of Kubernetes profiles, see Using Kubernetes Profiles.

Prerequisites

To use UAA as your OIDC provider, the TKGI API Certificate to secure the TKGI API field on the TKGI tile must be a proper certificate chain and have a SAN field. For more information, see configuring TKGI API in the Installing TKGI topic for your IaaS.

Configure a Custom OIDC Provider

To configure a custom OIDC provider for TKGI clusters, complete the following:

  1. Set Up Dex Workload
  2. Set Up Communication Path
  3. Deploy and Expose Dex
  4. Create Kubernetes Profile
  5. Create Cluster
  6. Test Cluster Access

Set Up Dex Workload

To configure dex as an OIDC provider for an LDAP directory:

  1. Create a cluster in TKGI for installing dex as a pod:

    tkgi create-cluster dex -p small -e dex-host.example.com

  2. Run tkgi cluster for the cluster and record its Kubernetes Master IP address.

    For example:

     $ tkgi cluster dex
    
TKGI Version:             1.9.0-build.1
     Name:                    dex
     K8s Version:             1.24.3
     Plan Name:               small
    

    UUID:                    dbe1d880-478f-4d0d-bb2e-0da3d9641f0d
     Last Action:             CREATE
     Last Action State:       succeeded
     Last Action Description: Instance provisioning completed
     Kubernetes Master Host:  dex-host.example.com
     Kubernetes Master Port:  8443
     Worker Nodes:            1
     Kubernetes Master IP(s): 10.0.11.11
     Network Profile Name:
     Kubernetes Profile Name:
     Tags:

  3. Add the Kubernetes Master IP address to your local /etc/hosts file.

  4. Populate your ~/.kube/config with context for dex:

    tkgi get-credentials dex

  5. Switch to the admin context of the dex cluster:

    kubectl config use-context dex

  6. To deploy a dex workload on a Kubernetes cluster, follow the steps in Deploying dex on Kubernetes in the dex GitHub repo.

    • Use the dex.yaml example YAML file in GitHub to create a dex deployment that connects to an LDAP server.

Set Up Communication Path

To set up \etcd\hosts and TLS so that clusters can access dex securely:

  1. Add the /etc/hosts entry for the public IP and the hostname dex.example.com on your local workstation. This lets you retrieve a token to access your OIDC-profile cluster later.

    10.0.11.11 dex.example.com

  2. To generate TLS assets for the dex deployment, complete the steps in Generate TLS assets in the dex documentation.

  3. To add the generated TLS assets to the cluster as a secret, complete the steps in Create cluster secrets in the dex documentation.

Deploy and Expose Dex

To run dex as a local service within a pod and exposes its endpoint via an IP address:

  1. On a Kubernetes cluster, deploy dex using the example YAML file linked above.

  2. Wait for the deployment to succeed.

  3. Expose the dex deployment as a service named dex-service:

    kubectl expose deployment dex --type=LoadBalancer --name=dex-service

    For example:

     $ kubectl expose deployment dex –type=LoadBalancer –name=dex-service
     > service/dex-service exposed

  4. This should create a dex service with a public IP address that clusters can use as an OIDC issuer URL. Retrieve the IP address by running:

    kubectl get services dex-service

  5. Add the IP of the dex service to your /etc/hosts:

    35.222.29.10 dex.example.com
    • Ensure that you map the dex service to dex.example.com, which the dex binary expects as issuer_url and for TLS handshakes.
    • For this example, we set up the issuer URL as https://dex.example.com:32000.

Create Kubernetes Profile

To create a Kubernetes profile that lets a cluster’s kube-api-server connect to the dex service:

  1. Create a Kubernetes profile /tmp/profile.json containing your custom OIDC settings under the kube-apiserver component.

    For example:

     $ cat /tmp/profile.json
     {
     “name”: “oidc-config”,
     “description”: “Kubernetes profile with OIDC configuration”,
     “customizations”: [
     {
     “component”: “kube-apiserver”,
     “arguments”: {
     “oidc-client-id”: “example-app”,
     “oidc-issuer-url”: “https://dex.example.com:32000”,
     “oidc-username-claim”: “email”
     },
     “file-arguments”: {
     “oidc-ca-file”: “/tmp/oidc-ca.pem”
     }
    
}
    
]
    
}
    Of all the supported kube-apiserver flags, the following are specific to OIDC:
    * In the arguments block:
    * oidc-issuer-url: Set this to "https://dex.example.com:32000".
    * oidc-client-id
    * oidc-username-claim: Set this to "email" for testing with the example app below.
    * oidc-groups-claim
    * In the file-arguments block:
    * oidc-ca-file: Set this to a path in the local file system that contains a CA certificate file. The certificate must be a proper certificate chain and have a SAN field.

    For more information on kube-apiserver flags, see kube-apiserver in the Kubernetes documentation.

  2. Create the profile:

    tkgi create-kubernetes-profile /tmp/profile.json

In the example above, the file-path /tmp/oidc-ca.pem points to a CA certificate on the local file system, and the tkgi create-kubernetes-profile command sends this certificate to the API server when it creates the profile.

Create Cluster

To create a cluster using the Kubernetes profile created above:

  1. Run the following:

    tkgi create-cluster cluster-with-custom-oidc -e cluster.example.com -p small --kubernetes-profile oid-config

    Note: Use only lowercase characters when naming your cluster if you manage your clusters with Tanzu Mission Control (TMC). Clusters with names that include an uppercase character cannot be attached to TMC.

  2. Confirm the cluster has custom OIDC settings from the profile.

Test Cluster Access

To test that the cluster uses the OIDC provider to control access, install an app, generate an ID token, and test the cluster.

You can use an example app from the dex repo or test with a full-fledged application, such as Gangway, instead of the example app.

To test that the cluster uses the OIDC provider:

  1. To install an example app, complete the steps to install the example-app in Logging into the cluster in the dex documentation.

  2. Run the dex example app:

    ./bin/example-app --issuer https://dex.example.com:32000
--issuer-root-ca /tmp/ca.pem
    • The example app only provides the email scope.

  3. To fetch the token, complete the steps to generate an ID token in Logging into the cluster in the dex documentation.

  4. Log in using the Log in with Email option and enter the email and password of an account in your OIDC IDP.

  5. A page appears listing the ID Token, Access Token, Refresh Token, ID Token Issuer (iss claim), and other information.

    Page lists: ID Token with cert, Access Token with token, Claims with access structure for user alana, and Refresh Token token string

  6. Wait for the token to be generated.

  7. Edit your .kube/config file to add a new context for the test user:

    apiVersion: v1
clusters:
- cluster:
    certificate-authority-data: CA-CERT
    server: CLUSTER-URL
  name: TEST-CLUSTER
contexts:
- [EXISTING-CONTEXTS]
- context:
    cluster: TEST-CLUSTER
    user: TEST-USER
  name: TEST-CONTEXT
current-context: TEST-CONTEXT
kind: Config
preferences: {}
users:
- [EXISTING-USERS]
- name: TEST-USER
  user:
    token: ID-TOKEN

    Where:

    • CA is your CA certificate.
    • CLUSTER-URL is the address of the test service, such as https://cluster.example.com:8443.
    • TEST-CLUSTER is the name of the test cluster, such as cluster-with-custom-oidc.
    • TEST-USER is the test account username, such as alana.
    • TEST-CONTEXT is a name you create for the new context, such as cluster-with-custom-oidc-ldap-alana.
    • ID-TOKEN is the ID Token retrieved by the example-app app above.

    Include the cluster.server and user.token values retrieved using the example app.

  8. Create a ClusterRole YAML file that grants permissions to access services and pods in the default namespace:

    apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  namespace: default
  name: pod-reader-clusterRolebinding
rules:
- apiGroups: [""] # "" indicates the core API group
  resources: ["pods", "services"]
  verbs: ["get", "watch", "list"]

  9. Run kubectl apply or kubectl create to pass the ClusterRole spec file to the kube controller:

    kubectl apply -f ClusterRole.yml

  10. Create a ClusterRoleBinding YAML file that applies the ClusterRole role to the test user.

    For example:

    apiVersion: rbac.authorization.k8s.io/v1
# This role binding allows "[email protected]" to read pods in the "default" namespace.
kind: ClusterRoleBinding
metadata:
  name: read-pods-clusterRolebinding
  namespace: default
subjects:
- kind: User
  name: [email protected] # Name is case sensitive
  apiGroup: rbac.authorization.k8s.io
roleRef:
  kind: ClusterRole #this must be Role or ClusterRole
  name: pod-reader-clusterRolebinding # this must match the name of the Role or ClusterRole you wish to bind to
  apiGroup: rbac.authorization.k8s.io

  11. Run kubectl apply or kubectl create to pass the ClusterRoleBinding spec file to the kube controller:

    kubectl apply -f ClusterRoleBinding.yml

  12. Confirm the test user can run the following:

    kubectl get pods

    The cluster is successfully authenticating the user by connecting to the dex OIDC provider via OIDC if the test user can run kubectl get pods.

check-circle-line exclamation-circle-line close-line
Scroll to top icon