Auditing Tanzu Kubernetes Grid Integrated Edition Logs

This topic summarizes key auditable events in TKGI, and the content of the log entries that the events generate. Operators can use this information to audit event logs to see what users took what actions at what times. This is helpful for security, compliance, and troubleshooting.

Log content can either be downloaded or configured to be transported via syslog.

TKGI API events

The following log entry examples are produced by TKGI API events and correspond to key actions taken by a user logged into the TKGI CLI.

Cluster Creation

create-cluster
Description	A user has issued a create cluster command.
Identifying String	`Action ‘create-cluster’`
Example Log Entries	2019-05-16 14:59:34.897 INFO 7594 — [nio-9021-exec-7] io.pivotal.pks.cluster.ClusterService : Action ‘create-cluster’ by user ‘admin’, cluster name: ‘logs’, plan name: ‘small’. Details: class ClusterParameters { kubernetesMasterHost: logs.lathrop.cf-app.com kubernetesMasterPort: 8443 workerHaproxyIpAddresses: null kubernetesWorkerInstances: 3 authorizationMode: null nsxtNetworkProfile: null } 2019-05-16 14:59:34.911 INFO 7594 — [nio-9021-exec-7] io.pivotal.pks.telemetry.Agent : Telemetry - addCluster: cluster request: class ClusterRequest { name: logs planName: small networkProfileName: null parameters: class ClusterParameters { kubernetesMasterHost: logs.lathrop.cf-app.com kubernetesMasterPort: 8443 workerHaproxyIpAddresses: null kubernetesWorkerInstances: 3 authorizationMode: null nsxtNetworkProfile: null } }, cluster entity: ClusterEntity{name=‘logs’, uuid=‘f4e2b775-8be3-41b8-abe8-67f2265b957e’, owner=‘admin’, brokerOperationId=‘{“BoshTaskID”:479,“BoshContextID”:“256c3b65-2eae-48f7-81f0-caed7472fa5f”,“OperationType”:“create”,“PostDeployErrand”:{},“PreDeleteErrand”:{},“Errands”:[{“Name”:“apply-addons”,“Instances”:null},{“Name”:“vrops-errand”,“Instances”:null},{“Name”:“telemetry-agent”,“Instances”:null}]}’, lastActionDescription=‘Creating cluster’, planId=‘8A0E21A8-8072-4D80-B365-D1F502085560’, lastAction=‘CREATE’, lastActionState=‘in progress’, masterIps=‘[In Progress]’, parameters=io.pivotal.pks.cluster.data.ClusterParametersEntity@6efbedb6’, networkProfileUuid=null’, computeProfileUuid=null’, taskStartedAt=2019-05-16T14:59:34.804}, plan: class Plan { id: 8A0E21A8-8072-4D80-B365-D1F502085560 name: small description: Example: This plan will configure a lightweight kubernetes cluster. Not recommended for production workloads. workerInstances: 3 masterInstances: 1 allowPrivilegedContainers: false }

Cluster Deletion

delete-cluster
Description	A user has issued a delete cluster command.
Identifying String	`delete deployment for instance`
Example Log Entries	`2019-06-04T14:16:52-06:00 10.0.10.10 broker/rs2 [on-demand-service-broker] [2f71a161-5755-4a0d-9c21-5b8405209594] 2019/06/04 20:16:52.493286 BOSH task ID 132 status: processing delete deployment for instance 67f77801-3d15-4d65-b501-38a643055e69: Description: delete deployment service-instance_67f77801-3d15-4d65-b501-38a643055e69 Result:`

Successful Login

UserAuthenticationSuccess
Description	A user has successfully logged into Tanzu Kubernetes Grid Integrated Edition.
Identifying String	`UserAuthenticationSuccess`
Example Log Entries	`[2019-05-16 17:12:48.833] uaa - 7777 [https-jsse-nio-8443-exec-2] …. INFO — Audit: UserAuthenticationSuccess (‘admin’): principal=0074aab6-6ff7-4b4c-b821-49526a96ebcb, origin=[remoteAddress=207.126.127.114, clientId=pks_cli], identityZoneId=[uaa] [2019-05-16 17:12:48.873] uaa - 7777 [https-jsse-nio-8443-exec-2] …. INFO — Audit: TokenIssuedEvent (‘[“pks.clusters.admin”]’): principal=0074aab6-6ff7-4b4c-b821-49526a96ebcb, origin=[client=pks_cli, user=admin], identityZoneId=[uaa]`

Unsuccessful Login

UserAuthenticationFailure
Description	A user has failed a login attempt into Tanzu Kubernetes Grid Integrated Edition.
Identifying String	`UserAuthenticationFailure`
Example Log Entries	[2019-05-16 17:15:31.363] uaa - 7777 [https-jsse-nio-8443-exec-8] …. INFO — Audit: UserAuthenticationFailure (‘admin’): principal=0074aab6-6ff7-4b4c-b821-49526a96ebcb, origin=[remoteAddress=207.126.127.114, clientId=pks_cli], identityZoneId=[uaa] [2019-05-16 17:15:31.371] uaa - 7777 [https-jsse-nio-8443-exec-8] …. INFO — Audit: PrincipalAuthenticationFailure (‘null’): principal=admin, origin=[207.126.127.114], identityZoneId=[uaa] [2019-05-16 17:15:33.387] uaa - 7777 [https-jsse-nio-8443-exec-6] …. INFO — Audit: ClientAuthenticationSuccess (‘Client authentication success’): principal=pks_client, origin=[remoteAddress=127.0.0.1, cl

Successful Cluster Credential Retrieval

ClientAuthenticationSuccess
Description	A user has successfully gained access to a cluster in Tanzu Kubernetes Grid Integrated Edition.
Identifying String	`ClientAuthenticationSuccess`
Example Log Entries	[2019-05-16 17:15:31.363] uaa - 7777 [https-jsse-nio-8443-exec-8] …. INFO — Audit: UserAuthenticationFailure (‘admin’): principal=0074aab6-6ff7-4b4c-b821-49526a96ebcb, origin=[remoteAddress=207.126.127.114, clientId=pks_cli], identityZoneId=[uaa] [2019-05-16 17:15:31.371] uaa - 7777 [https-jsse-nio-8443-exec-8] …. INFO — Audit: PrincipalAuthenticationFailure (‘null’): principal=admin, origin=[207.126.127.114], identityZoneId=[uaa] [2019-05-16 17:15:33.387] uaa - 7777 [https-jsse-nio-8443-exec-6] …. INFO — Audit: ClientAuthenticationSuccess (‘Client authentication success’): principal=pks_client, origin=[remoteAddress=127.0.0.1, cl

User Creation

UserCreatedEvent
Description	An administrator has successfully created a new user for Tanzu Kubernetes Grid Integrated Edition.
Identifying String	`UserCreatedEvent`
Example Log Entries	`Jun 04 16:00:07 10.0.10.10 uaa/rs2: [2019-06-04 22:00:07.293] uaa - 18840 [https-jsse-nio-8443-exec-6] …. INFO — Audit: UserCreatedEvent (‘[“user_id=dc803130-15dc-4279-8b42-868fc80b8ca1”,“username=USERNAME2”]’): principal=dc803130-15dc-4279-8b42-868fc80b8ca1, origin=[client=admin, details=(remoteAddress=35.192.67.34, tokenType=bearertokenValue= , sub=admin, iss=https://api.tkgi.hawthorne.cf-app.com:8443/oauth/token)], identityZoneId=[uaa]`

User Deletion

UserDeletedEvent
Description	An administrator has successfully deleted a user for Tanzu Kubernetes Grid Integrated Edition.
Identifying String	`UserDeletedEvent`
Example Log Entries	`Jun 04 16:00:07 10.0.10.10 uaa/rs2: [2019-06-04 22:00:07.293] uaa - 18840 [https-jsse-nio-8443-exec-6] …. INFO — Audit: UserCreatedEvent (‘[“user_id=dc803130-15dc-4279-8b42-868fc80b8ca1”,“username=USERNAME2”]’): principal=dc803130-15dc-4279-8b42-868fc80b8ca1, origin=[client=admin, details=(remoteAddress=35.192.67.34, tokenType=bearertokenValue= , sub=admin, iss=https://api.tkgi.hawthorne.cf-app.com:8443/oauth/token)], identityZoneId=[uaa]`

Telemetry Collection

Telemetry Ping
Description	The optional telemetry system has successfully reached an external host for collecting product data for Tanzu Kubernetes Grid Integrated Edition. To learn more about the Tanzu Kubernetes Grid Integrated Edition telemetry program, see Telemetry.
Identifying String	`telemetry-server`
Example Log Entries	`2019-06-04T15:41:05-06:00 10.0.10.10 telemetry-server/rs2 2019-06-04 21:41:05 +0000 [debug]: #0 generating helo 2019-06-04T15:41:05-06:00 10.0.10.10 telemetry-server/rs2 2019-06-04 21:41:05 +0000 [debug]: #0 checking ping 2019-06-04T15:41:05-06:00 10.0.10.10 telemetry-server/rs2 2019-06-04 21:41:05 +0000 [debug]: #0 generating pong 2019-06-04T15:41:05-06:00 10.0.10.10 telemetry-server/rs2 2019-06-04 21:41:05 +0000 [debug]: #0 connection established address=“10.0.11.21” port=33366`

Kubernetes Audit Log Events

The Kubernetes control plane emits a standard log format every time a user takes action to query or change the state of the Kubernetes API. An example audit event log entry is below.

{
	"kind": "Event",
	"apiVersion": "audit.k8s.io/v1",
	"level": "Request",
	"auditID": "dc2bb4e9-4b85-42da-82a3-5ee47091207d",
	"stage": "ResponseStarted",
	"requestURI": "/apis/policy/v1/poddisruptionbudgets?resourceVersion=370506\u0026timeout=7m54s\u0026timeoutSeconds=474\u0026watch=true",
	"verb": "watch",
	"user": {
		"username": "system:kube-scheduler",
		"uid": "system:kube-scheduler",
		"groups": ["system:authenticated"]
	},
	"sourceIPs": ["10.0.11.10"],
	"userAgent": "kube-scheduler/v1.15.4 (linux/amd64) kubernetes/67d2fcf/scheduler",
	"objectRef": {
		"resource": "poddisruptionbudgets",
		"apiGroup": "policy",
		"apiVersion": "v1"
	},
	"responseStatus": {
		"metadata": {},
		"code": 200
	},
	"requestReceivedTimestamp": "2019-12-11T21:47:28.097065Z",
	"stageTimestamp": "2019-12-11T21:47:28.097491Z",
	"annotations": {
		"authorization.k8s.io/decision": "allow",
		"authorization.k8s.io/reason": "RBAC: allowed by ClusterRoleBinding \"system:kube-scheduler\" of ClusterRole \"system:kube-scheduler\" to User \"system:kube-scheduler\""
	}
}

For more information about Kubernetes Audit Event Log format see the Kubernetes documentation.