When adding Kubernetes components to an Automation Assembler cloud template, you can choose to add clusters or enable users to create namespaces in various configurations. Typically, this choice depends on your access control requirements, how you have configured your Kubernetes components, and your deployment requirements.

To add a Kubernetes component to a cloud template in Automation Assembler, select Design > Cloud Templates , click New, and then locate and expand the Kubernetes option on the left menu. Then, make the desired selection, either Cluster or KBS Namespace by dragging it to the canvas.

Adding a Kubernetes cluster that is associated with a project to a cloud template is the most straightforward method of making Kubernetes resources available to valid users. You can use tags on clusters to control where they are deployed just as you do with other Automation Assembler resources. You can use tags to select a zone and a VMware Tanzu Kubernetes Grid Integrated Edition (TKGI) plan during the allocation phase of cluster deployment.

Once you add a cluster in this way, it is automatically available to all valid users.

Cloud template examples

The first cloud template example shows a template for a simple Kubernetes deployment that is controlled by tagging. A Kubernetes zone was created with two deployment plans, configured on the New Kubernetes Zone page. In this case, a tag called placement:tag was added as a capability on the zone, and it was used to match the analogous constraint on the cloud template. If there were more than one zone configured with the tag, the one with the lowest priority number would be selected.

formatVersion: 1
inputs: {}
resources:
  Cluster_provisioned_from_tag:
    type: Cloud.K8S.Cluster
    properties:
      hostname: 109.129.209.125
      constraints:
	-tag: 'placement tag'
      port: 7003
      workers: 1
      connectBy: hostname 

The second cloud template examples shows how to set up a template with a variable called $(input.hostname) so that users can input the desired cluster hostname when requesting a deployment. Tags can also be used to select a zone and a TKGI plan during the resource allocation phase of cluster deployment.

formatVersion: 1
inputs:
  hostname:
    type: string
    title: Cluster hostname
resources:
  Cloud_K8S_Cluster_1:
    type: Cloud.K8S.Cluster
    properties:
      hostname: ${input.hostname}
      port: 8443
      connectBy: hostname
      workers: 1

If you want to use namespaces to mange cluster usage, you can set up a variable in the cloud template called name: ${input.name} to substitute for the namespace name which a user enters when requesting a deployment. For this sort of deployment, you would create a template something like the following example:

1 formatVersion: 1
2 inputs:
3 name:
4    type: string
5    title: "Namespace name"
6 resources:
7    Cloud_KBS_Namespace_1:
8        type: Cloud.K8S.Namespace
9        properties:
10            name: ${input.name}

Users can manage deployed clusters via kubeconfig files that are accessible from the Infrastructure > Resources > Kubernetes Clusters page. Locate the card on the page for the desired cluster and click Kubeconfig.

Supervisor Namespaces in VMware Cloud Templates

The following is the schema for a basic Supervisor namespace in an Automation Assembler cloud template.

{
  "title": "Supervisor namespace schema",
  "description": "Request schema for provisioning of Supervisor namespace resource",
  "type": "object",
  "properties": {
    "name": {
      "title": "Name",
      "description": "Alphabetic (a-z and 0-9) string with maximum length of 63 characters. The character ‘-’ is allowed anywhere except the first or last position of the identifier.",
      "type": "string",
      "pattern": "^.*\\$\\{.*\\}.*$|^((?!-)[a-z0-9-]{1,63}(?<!-))$",
      "ignoreOnUpdate": true
    },
    "description": {
      "title": "Description",
      "description": "An optional description of this Supervisor namespace.",
      "type": "string",
      "ignoreOnUpdate": true
    },
    "content": {
      "title": "Content",
      "description": "Kubernetes Yaml Content",
      "type": "string",
      "maxLength": 65000
    },
    "constraints": {
      "title": "Constraints",
      "description": "To target the correct resources, blueprint constraints are matched against infrastructure capability tags. Constraints must include the key name. Options include value, negative [!], and hard or soft requirement.",
      "type": "array",
      "recreateOnUpdate": true,
      "items": {
        "type": "object",
        "properties": {
          "tag": {
            "title": "Tag",
            "description": "Constraint definition in syntax `[!]tag_key[:tag_value][:hard|:soft]` \nExamples:\n```\n!location:eu:hard\n location:us:soft\n!pci\n```",
            "type": "string",
            "recreateOnUpdate": true
          }
        }
      }
    },
    "limits": {
      "title": "Limits",
      "description": "Defines namespace resource limits such as pods, services, etc.",
      "type": "object",
      "properties": {
        "stateful_set_count": {
          "title": "stateful_set_count",
          "description": "This represents the new value for 'statefulSetCount' option which is the maximum number of StatefulSets in the namespace.",
          "type": "integer",
          "recreateOnUpdate": false
        },
        "deployment_count": {
          "title": "deployment_count",
          "description": "This represents the new value for 'deploymentCount' option which is the maximum number of deployments in the namespace.",
          "type": "integer",
          "recreateOnUpdate": false
        },
        "cpu_limit_default": {
          "title": "cpu_limit_default",
          "description": "This represents the new value for the default CPU limit (in Mhz) for containers in the pod. If specified, this limit should be at least 10 MHz.",
          "type": "integer",
          "recreateOnUpdate": false
        },
        "config_map_count": {
          "title": "config_map_count",
          "description": "This represents the new value for 'configMapCount' option which is the maximum number of ConfigMaps in the namespace.",
          "type": "integer",
          "recreateOnUpdate": false
        },
        "pod_count": {
          "title": "pod_count",
          "description": "This represents the new value for 'podCount' option which is the maximum number of pods in the namespace.",
          "type": "integer",
          "recreateOnUpdate": false
        },
        "job_count": {
          "title": "job_count",
          "description": "This represents the new value for 'jobCount' option which is the maximum number of jobs in the namespace.",
          "type": "integer",
          "recreateOnUpdate": false
        },
        "secret_count": {
          "title": "secret_count",
          "description": "This represents the new value for 'secretCount' option which is the maximum number of secrets in the namespace.",
          "type": "integer",
          "recreateOnUpdate": false
        },
        "cpu_limit": {
          "title": "cpu_limit",
          "description": "This represents the new value for 'limits.cpu' option which is equivalent to the maximum CPU limit (in MHz) across all pods in the namespace.",
          "type": "integer",
          "recreateOnUpdate": false
        },
        "cpu_request_default": {
          "title": "cpu_request_default",
          "description": "This represents the new value for the default CPU request (in Mhz) for containers in the pod. If specified, this field should be at least 10 MHz.",
          "type": "integer",
          "recreateOnUpdate": false
        },
        "memory_limit_default": {
          "title": "memory_limit_default",
          "description": "This represents the new value for the default memory limit (in mebibytes) for containers in the pod.",
         "type": "integer",
          "recreateOnUpdate": false
        },
        "memory_limit": {
          "title": "memory_limit",
          "description": "This represents the new value for 'limits.memory' option which is equivalent to the maximum memory limit (in mebibytes) across all pods in the namespace.",
          "type": "integer",
          "recreateOnUpdate": false
        },
        "memory_request_default": {
          "title": "memory_request_default",
          "description": "This represents the new value for the default memory request (in mebibytes) for containers in the pod.",
          "type": "integer",
          "recreateOnUpdate": false
        },
        "service_count": {
          "title": "service_count",
          "description": "This represents the new value for 'serviceCount' option which is the maximum number of services in the namespace.",
          "type": "integer",
          "recreateOnUpdate": false
        },
        "replica_set_count": {
          "title": "replica_set_count",
          "description": "This represents the new value for 'replicaSetCount' option which is the maximum number of ReplicaSets in the namespace.",
          "type": "integer",
          "recreateOnUpdate": false
        },
        "replication_controller_count": {
          "title": "replication_controller_count",
          "description": "This represents the new value for 'replicationControllerCount' option which is the maximum number of ReplicationControllers in the namespace.",
          "type": "integer",
          "recreateOnUpdate": false
        },
        "storage_request_limit": {
          "title": "storage_request_limit",
          "description": "This represents the new value for 'requests.storage' which is the limit on storage requests (in mebibytes) across all persistent volume claims from pods in the namespace.",
          "type": "integer",
          "recreateOnUpdate": false
        },
        "persistent_volume_claim_count": {
          "title": "persistent_volume_claim_count",
          "description": "This represents the new value for 'persistentVolumeClaimCount' option which is the maximum number of PersistentVolumeClaims in the namespace.",
          "type": "integer",
          "recreateOnUpdate": false
        },
        "daemon_set_count": {
          "title": "daemon_set_count",
          "description": "This represents the new value for 'daemonSetCount' option which is the maximum number of DaemonSets in the namespace.",
          "type": "integer",
          "recreateOnUpdate": false
        }
      },
      "additionalProperties": false
    },
    "vm_classes": {
      "title": "VM classes",
      "description": "Defines set of Virtual Machine classes to be assigned to the namespace",
      "type": "array",
      "recreateOnUpdate": false,
      "items": {
        "type": "object",
        "properties": {
          "name": {
            "title": "Name",
            "description": "Name of the Virtual Machine class.",
            "type": "string",
            "recreateOnUpdate": false
          }
        }
      }
    },
    "storage": {
      "title": "Storage policies",
      "description": "Defines set of storage profiles to be used to assign storage policies to the namespace.",
      "type": "array",
      "recreateOnUpdate": false,
      "items": {
        "type": "object",
        "properties": {
          "profile": {
            "type": "object",
            "title": "Storage profile",
            "description": "Defines storage policies to be assigned to the namespace",
            "recreateOnUpdate": false,
            "properties": {
              "constraints": {
                "title": "Constraints",
                "description": "To target the correct storage profiles, blueprint constraints are matched against storage profile capability tags.",
                "type": "array",
                "recreateOnUpdate": false,
                "items": {
                  "type": "object",
                  "properties": {
                    "tag": {
                      "title": "Tag",
                      "description": "Constraint definition in syntax `[!]tag_key[:tag_value][:hard|:soft]` \nExamples:\n```\nlocation:eu:hard\n location:us:soft\n```",
                      "type": "string",
                      "recreateOnUpdate": false
                    }
                  }
                },
                "minItems":1
              },
              "limitMb": {
                "title": "Limit",
                "description": "The maximum amount of storage (in mebibytes) which can be utilized by the namespace for this storage policy. Optional. If unset, no limits are placed.",
                "type": "integer"
              }
            },
            "required": [
              "constraints"
            ]
          }
        }
      }
    }
  },
  "required": [
    "name"
  ]
}

VMware cloud templates support the use of limits with supervisor namespaces. Limits enable you to control resource usage for CPUs and memory as well as the maximum number of pods allowed in the namespace by deployed machines.

formatVersion: 1
inputs: {}
resources:
  Cloud_SV_Namespace_1:
    type: Cloud.SV.Namespace
    properties:
      name: '${env.deploymentName}'
      limits:
        - cpu_limit: 1000
          cpu_request_default: 800
          memory_limit: 2000
          memory_limit_default: 1500
          pod_count: 200

The following example shows how you could specify a storage policy using tags.

formatVersion: 1
inputs: {}
resources:
  Cloud_SV_Namespace_1:
    type: Cloud.SV.Namespace
    properties:
      name: 'ns-with-storage-policy'
      description: 'sample'
      storage: 
        - profile: 
            limitMb: 1000
            constraints: 
              - tag: 'storage:fast'
        - profile: 
            constraints: 
              - tag: 'storage:cheap'

Using arbitrary YAMLs with self-service namespace or cluster VCTs

As part of a cluster or namespace creation, you may want to run additional customizations. For example, you may want to add users (role/role binding) or create a pod security policy, or install agents. By using the YAML content property, you can define customized packages to provision on that cluster/namespace/supervisor namespace.

Each YAML content package associated with the content property must be separated with a triple dash (---). Also the content information must be a multi-line string. Refer to the following YAML example to see how content packages can be configured.

formatVersion: 1
inputs: {}
resources:
  Cloud_Tanzu_Cluster_1:
    type: Cloud.Tanzu.Cluster
    properties:
      name: ddonchev-tkc
      plan: small
      content: |-
        apiVersion: rbac.authorization.k8s.io/v1
        kind: ClusterRoleBinding
        metadata:
          name: psp:authenticated-from-yaml
        subjects:
        - apiGroup: rbac.authorization.k8s.io
          kind: Group
          name: system:authenticated
        roleRef:
          apiGroup: rbac.authorization.k8s.io
          kind: ClusterRole
          name: psp:vmware-system-privileged
        ---
        apiVersion: apiextensions.k8s.io/v1
        kind: CustomResourceDefinition
        metadata:
          # name must match the spec fields below, and be in the form: <plural>.<group>
          name: crontabs.stable.example.com
        spec:
          # group name to use for REST API: /apis/<group>/<version>
          group: stable.example.com
          # list of versions supported by this CustomResourceDefinition
          versions:
            - name: v1
              # Each version can be enabled/disabled by Served flag.
              served: true
              # One and only one version must be marked as the storage version.
              storage: true
              schema:
                openAPIV3Schema:
                  type: object
                  properties:
                    spec:
                      type: object
                      properties:
                        cronSpec:
                          type: string
                        image:
                          type: string
                        replicas:
                          type: integer
          # either Namespaced or Cluster
          scope: Namespaced
          names:
            # plural name to be used in the URL: /apis/<group>/<version>/<plural>
            plural: crontabs
            # singular name to be used as an alias on the CLI and for display
            singular: crontab
            # kind is normally the CamelCased singular type. Your resource manifests use this.
            kind: CronTab
            # shortNames allow shorter string to match your resource on the CLI
            shortNames:
            - ct

The YAML defined in the content property also appears on the Properties tab for the deployment.

Automation Assembler can only create content resources in the scope of the resource being deployed. For example: if you provision a kubernetes namespace,Automation Assembler cannot create a deployment inside a different namespace. Users have the same rights as if they were using the kubeconfig with kubectl.

After the virtual machine is provisioned, an installation of the kubernetes objects inside the content property will begin. If one of the resources referenced in the YAML content property fails to provision, Automation Assembler will roll back and delete all the previous kubernetes objects from the resource and the deployment will have a Failed status. The resource will still be provisioned and visible. Also, you can still use day2 actions, including trying to apply the content again.

You can enhance the content property with inputs from the cloud template as shown in the following example.

formatVersion: 1
inputs: {}
resources:
  Cloud_SV_Namespace_1:
    type: Cloud.SV.Namespace
    properties:
      name: sv-namespace-with-vm-classes
      vm_classes:
        - name: best-effort-2xlarge
        - name: best-effort-4xlarge
        - name: best-effort-8xlarge

In addition, you can provision custom resources such as the TanzuKubernetesCluster. This would fail as a day 1 operation, because the supervisor namespace will not contain the required virtual machine classes and storage classes. When the virtual machine classes and storage classes are bound to the supervisor namespace you can create the TanzuKubernetesCluster (or another resource) using the day 2 action.

Note: You can provision a resource without content, and you will still be able to add kubernetes objects as YAML with the day 2 action.

The content that appears in the YAML property defines what is provisioned on the resource. When you edit this content, the following table shows the possible results:

Action Result
If you add a kubernetes object and submit. The specified object is created on the resource.
If you remove a kubernetes object and submit. The specified object is deleted from the resource.
If you modify a kubernetes object and submit. The specified object is patched on the resource.

It is important to clarify what actions are considered as a modification to the current object. For example: if you modify the namespace field of an object, then a new object is created instead of the old one being patched.

The uniqueness of a resource is defined by the following fields: apiVersion, kind, metadata.name, metadata.namespace