Use Gitops delivery with Flux CD (beta)

This topic explains how you can deliver Carvel Packages, created by the Carvel Package Supply Chains, from a GitOps repository to one or more run clusters using Flux CD and Supply Chain Choreographer.

Prerequisites

To use Gitops Delivery with Flux CD, you must complete the following prerequisites:

  • You must create a Workload that uses either the source-to-url-package or basic-image-to-url-package Carvel Package Supply Chain. See the Carvel documentation. You must have at least one Carvel Package generated by this Workload stored in your GitOps repository.
  • You must have at least one run cluster. Run clusters are your deployment environments. They can either be Tanzu Application Platform clusters, or regular Kubernetes clusters, but they must have kapp-controller and Contour installed. See the Carvel documentation and the Contour documentation.
  • To use a build cluster to control the deployment on all the run clusters, you must create a build cluster that has network access to your run clusters. You must also install Flux CD Kustomize Controller. See Install Kustomize Controller Prerequisite. If you intend to deploy directly on the run cluster without a build cluster, a build cluster is only necessary for building the package.

Install Kustomize Controller Prerequisite

As mentioned earlier, to use a build cluster to control the deployment on run clusters, you must install Flux CD kustomize-controller. You can do the installation using the flux cli. Use release v0.41.2. (Caution: newer releases of Kustomize Controller are not compatible with TAP 1.7.x)

To install Flux CD:

# https://fluxcd.io/flux/get-started/
brew install fluxcd/tap/[email protected]  (for example, when on macOS)
flux install -n flux-system --components kustomize-controller --version v0.41.2

Set up run cluster namespaces

Each run cluster must have a namespace and ServiceAccount with the correct permissions to deploy the Carvel Packages, PackageInstalls and Kubernetes Secrets. Create a namespace and ServiceAccount with the following permissions:

---
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  namespace: <run-cluster-ns>
  name: app-package-and-pkgi-install-role
rules:
  - apiGroups: ["data.packaging.carvel.dev"]
    resources: ["packages"]
    verbs: ["get", "list", "create", "update", "delete", "patch"]
  - apiGroups: ["packaging.carvel.dev"]
    resources: ["packageinstalls"]
    verbs: ["get", "list", "create", "update", "delete", "patch"]
  - apiGroups: [""]
    resources: ["secrets"]
    verbs: ["get", "list", "create", "update", "delete", "patch"]

If your run cluster is a Tanzu Application Platform cluster, see Set up developer namespaces to use your installed packages.

If your run cluster is not a Tanzu Application Platform cluster, the ServiceAccount must also have the following permissions:

---
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  namespace: <run-cluster-ns>
  name: app-cr-role
rules:
- apiGroups: ["apps"]
  resources: ["deployments"]
  verbs: ["get", "list", "create", "update", "delete"]
- apiGroups: [""]
  resources: ["configmaps", "services"]
  verbs: ["get", "list", "create", "update", "delete"]
- apiGroups: ["networking.k8s.io"]
  resources: ["ingresses"]
  verbs: ["get", "list", "create", "update", "delete"]

Create Carvel PackageInstalls and secrets

For each Carvel Package and for each run cluster, you must create a Carvel PackageInstall and a Secret. The Carvel PackageInstall and the Secret are stored in your GitOps repository and deployed to run clusters by Flux CD.

The following example shows GitOps repository structure after completing this section:

app.default.tap/
  packages/
    20230321004057.0.0.yaml  # Package
  staging/
    packageinstall.yaml      # PackageInstall
    params.yaml              # Secret
  prod/
    packageinstall.yaml      # PackageInstall
    params.yaml              # Secret
  1. For each run cluster, create a Secret that has the values for each Package parameter. You can see the configurable properties of the Package by inspecting the Package CR’s valuesSchema, or in the Carvel Package Supply Chains documentation. Store the Secret in your GitOps repository at <package_name>/<run_cluster>/params.yaml.

    ---
    apiVersion: v1
    kind: Secret
    metadata:
      name: app-values
    stringData:
      values.yaml: |
        ---
        workload_name: app
        replicas: 2
        hostname: app.mycompany.com
    
    Note

    You must set a value for the workload_name parameter. You can skip setting other fields to use the default parameter values.

  2. For each run cluster, create a PackageInstall. Reference the Secret you created earlier. Store the PackageInstall in your GitOps repository at <package_name>/<run_cluster>/packageinstall.yaml.

    ---
    apiVersion: packaging.carvel.dev/v1alpha1
    kind: PackageInstall
    metadata:
      name: app
    spec:
      serviceAccountName: <run-cluster-ns-sa> # ServiceAccount on run cluster with permissions to deploy Package, see "Set up run Cluster Namespaces"
      packageRef:
        refName: app.default.tap # name of the Package
        versionSelection:
          constraints: 20230321004057.0.0 # version of the Package
      values:
      - secretRef:
          name: app-values # Secret created in previous step
    

    To continuously deploy the latest version of your Package, set versionSelection.constraints: >=0.0.0. To revert to a previous version, update the versionSelection.constraints: field and annotate the PackageInstall:

    packaging.carvel.dev/downgradable: ""
    

    See the Carvel documentation.

  3. Push the newly created PackageInstalls and Secrets to your GitOps repository.

Create Flux CD GitRepository and Flux CD Kustomizations on the build cluster

Configure Flux CD on the Build cluster to deploy your Packages, PackageInstalls, and Secrets to each of your run clusters.

  1. Give your Build cluster access to your run clusters. On the Build cluster, for each run cluster, create a Secret containing the run cluster’s kubeconfig. Create the Kubernetes Secret in the same namespace as the Kustomization resource:

    kubectl create secret generic <run-cluster>-kubeconfig \
       -n <build-cluster-ns> \
       --from-file=value.yaml=<path-to-run-cluster-kubeconfig>
    
    Note

    The kubeconfig must be self-contained and not rely on binaries, or environment and credential files from the kustomize-controller pod. Kubeconfigs that specify a command to provide client credentials such as gke-gcloud-auth-plugin won’t work without a custom per-provider installation of kustomize-controller.

  2. Configure your Build cluster to clone the GitOps repository. On the Build cluster, create the following Flux CD GitRepository:

    ---
    apiVersion: source.toolkit.fluxcd.io/v1beta2
    kind: GitRepository
    metadata:
      name: <package-name>-gitops-repo
      namespace: <build-cluster-ns>
    spec:
      url: # GitOps repo URL
      ignore: |
        !.git
      interval: 30s
      ref:
        branch: # GitOps repo branch
      timeout: 60s
    
       # only required if GitOps repo is private (recommended). The secret below should be present in the same namespace as the GitRepository.
       secretRef:
         name: <package-name>-gitops-auth
    
     # only required if GitOps repo is private (recommended)
     ---
     apiVersion: v1
     kind: Secret
     metadata:
       name: <package-name>-gitops-auth
       namespace: <build-cluster-ns>
     type: Opaque
     data:
       username: # base64 encoded GitHub (or other git remote) username
       password: # base64 encoded GitHub (or other git remote) personal access token
    
  3. Configure your Build cluster to deploy your Package to the run clusters. For each run cluster, on the Build cluster, create the following Flux CD Kustomization:

    ---
    apiVersion: kustomize.toolkit.fluxcd.io/v1beta2
    kind: Kustomization
    metadata:
      name: <package-name>-<run-cluster>-packages
      namespace: <build-cluster-ns>
    spec:
      sourceRef:
        kind: GitRepository
        name: <package-name>-gitops-repo
        namespace: <build-cluster-ns>
      path: "./<package-name>/packages"
      interval: 5m
      timeout: 5m
      prune: true
      wait: true
    
      # where to deploy
      kubeConfig:
        secretRef:
          name: <run-cluster>-kubeconfig
      targetNamespace: <run-cluster-ns>
      serviceAccountName: <run-cluster-ns-sa>
    
    Note

    The Kustomization resource does not accept a metadata.name field longer than 63 characters.

  4. Configure your Build cluster to deploy your PackageInstalls and Secrets to the run clusters. For each run cluster, on the Build cluster, create the following Flux CD Kustomization:

    ---
    apiVersion: kustomize.toolkit.fluxcd.io/v1beta2
    kind: Kustomization
    metadata:
      # for the second run cluster, for example hello-app-prod2-packages
      name: <package-name>-<run-cluster>-packageinstalls
      namespace: <build-cluster-ns>
    spec:
      sourceRef:
        kind: GitRepository
        name: <package-name>-gitops-repo
        namespace: <build-cluster-ns>
      path: "./<package-name>/<run-cluster>"
      interval: 5m
      timeout: 5m
      prune: true
      wait: true
    
      # where to deploy
      kubeConfig:
        secretRef:
          name: <run-cluster>-kubeconfig
      targetNamespace: <run-cluster-ns>
      serviceAccountName: <run-cluster-ns-sa>
    
    Note

    The Kustomization resource does not accept a metadata.name field longer than 63 characters.

Verify installation

To verify your installation:

  1. On your Build cluster, confirm that your Flux CD GitRepository and Kustomizations are reconciling:

    kubectl get gitrepositories,kustomizations -A
    
  2. Target a run cluster. Confirm that all Packages from the GitOps repository are deployed:

    kubectl get packages -A
    
  3. Target a run cluster. Confirm that all PackageInstalls are reconciled:

    kubectl get packageinstalls -A
    

Now you can access your application on each run cluster.

check-circle-line exclamation-circle-line close-line
Scroll to top icon