Here you will learn how to configure and deploy an extension in Spring Cloud Gateway for Kubernetes. For more information, see Extensions Development.

Extensions can be added with two simple steps to any Gateway Instance, including those already running. In short, you need to create a Kubernetes ConfigMap and activate it in the desired Gateway Instance.

Note This guide is to configure an extension for a Kubernetes deployment. See Launch the Gateway if you are running a Standalone installation.

Prerequisites

The requirements for these steps are:

  • SCG for K8s packaged extension (in JAR)
  • Docker command line tool if packaging extension as a OCI image
  • Kubernetes cluster
  • SCG Operator configured with the installation parameter gateway.enableExtensions=true

Deploying extensions

Extensions can be stored in Kubernetes ConfigMaps, OCI Images, or Volumes, each with their own considerations:

  • Using a ConfigMap has an advantage that SCG for K8s can detect when they change and update automatically. However, ConfigMaps have a 1MB size limit restriction.
  • Using an OCI image can be easier to automate, as it only requires building an image from a Dockerfile and pushing it to a registry.
  • Using Volumes avoid size limitations but have several restrictions and depend heavily on K8s and storage implementations provided in your environment.

Using ConfigMaps

Provided you have the JAR package of less than 1MB, create a ConfigMap as follows:

  $ kubectl create configmap extension-name --from-file=extension.jar -n gateway_namespace

The config map name will identify the extension later.

You can confirm that the ConfigMap was successfully created with the contents of the jar:

  $  kubectl get configmap extension-name -o yaml

You will see something that looks like this:

apiVersion: v1
binaryData:
  mycustomfilter-0.0.1-SNAPSHOT.jar: UEsDBAoAAAgIABV491IAAAAAAgAAAAAAAAAJAAAATUVUQS1JTkYvAwBQSwMECgAACAgAFXj3UrJ/Au4bAAAAGQAAABQAAABNRVRBLUlORi9NQU5JRkVTVC5NRvNNzMtMSy0u0Q1LLSrOzM+zUjDUM+Dl4uUCAFBLAwQKAAAICAAVePdSAAAAAAIAAAAAAAAABAAAAGNvbS8DAFBLAwQKAAAICAAVePdSAAAAAAIAAAAAAAAACwAAAGNvbS92bXdhcmUvAwBQSwMECgAACAgAFXj3UgAAAAACAAAAAAAAAA8AAABjb20vdm13YXJlL3NjZy8DAFBLAwQKAAAICAAVePdSAAAAAAIAAAAAAAAAGgAAAGNvbS92bXdhcmUvc2NnL2V4dGVuc2lvbnMvAwBQSwMECgAACAgAFXj3UgAAAAACAAAAAAAAACkAAABjb20vdm13YXJlL3NjZy9leHRlbnNpb25zL215Y3VzdG9tZmlsdGVyLwMAUEsDBAoAAAgIABV491IVhwu7iwUAAPgOAABUAAAAY29tL3Ztd2FyZS9zY2cvZXh0ZW5zaW9ucy9teWN1c3RvbWZpbHRlci9BZGRNeUN1c3RvbUhlYWRlckdhdGV3YXlGaWx0ZXJGYWN0b3J5LmNsYXNzrVdrWxNHFH4nCAvJAjGtUK0i2mgBxfXWagm1hRjBmngBC9Kb3exOkoXNbtwLmN7v9/pbavs8rfChP6BPf1OfntldIVzkweCH3ZmdOed9zzlzZs7sP/+t/A3gAh7E8RxeT1F/PIEsLqfAMJnAVbwlXtcSyKOQQkyMncRNCbfaMdWO6QRu4+04ZjDbgTjutGMugXfwbhzv4X0JHzC05W9MTOSmGFJ52ykrrlk6P6/k7XKZOxmGzsLc3cnc2OXc1N1ruTkhNK8uqoqpWmVl2nMMqyyEsrbleqrlzaimzwly1LAM7xJDy8DgDMOerK3TaHfesPh1v1rkzm21aPKAUVPNGdUxxHc0uMerGC7DzbxmV5XF6pLqcMXVygq/73HLNYhJqdY13/XsaskwPe4oY7peqGeDkUmu6tyZUD2+pNavBNNXVM2znTqZ2arWamadIT/Q4MSN4jzXvMxg6HxNeFRy1Cpfsp0FRTNtX1fKIZwS0a1DJ9g2zbZKRnl9bCJYhi5TrRZ1NR1wp88yPBjYkmqJFxWXO4tEMB00s7yYu69VCIxnmjQuW1ENi1xzeBADRbMpljW/aBpuhWQLtmWThe08omE4+/SmUVg1QcPw+q6spPzwazqN67lVc9bH7oyEDxmkcd8waY0Z5KuWRdqm6rqcEmZi67hWPK/22PogDsYij9yYpKkpfs/nrpeOUDMiXyUnHGS4/CwgN/pxmmFwG1sFRJjGbmhNJfxgOL4zJVrRUc2MdmDHtFG2VM93KJzazheoFG4aZazoeg71t9pRo5vT/RKRx6dt39E4iRLl0I725ikBxLB/yrc8o8pnDNego2DMsmxP9cSOJ6AtbXcJg9tevcaVrF2t2Ra3vIyM/TjAkBy3bU9YXytwr2Lrbncb1CSOUVOUoUGXwGWUUE6iQmNGEoqMeSxIMGWcRlWGBVtCTcjeIwEniXMyzoOMuf5sjyaGxJ3hQn04lKCNUK0Ph4s+vCgOVAmeDB+LEpZk3EddxvPYRyG56dgap+TX+6OEPdmv6jrXG8D6QxwJH8n4GJ9I+FTGZ/icofBMU0HCFwzju98rcXyJr+JUp76mYzXkZxgZ2MXhfPppDzQirvqUdtQZfQLxduoNu37fVmVmRnj3DZGomsZrdMQcJpJAzPcMUyn5liYyXhEV1a8GOBebtYFhbmBb7F2511oUXYbzTUSJ1DO7KBYMt5oootvXQbEu39IZ2UzEnr4grHHkDVFoYnaJIb3lvWS9KCmnd1IF6O5V8z2xOpsvbBsht+BIbrwK0r3MsEr2hqyOAEWZ6t2osXq4dZS59xjkWKN2ULofh3T9tXOk+TBL+E7G9/hJRhrHZJwQx/pZnGM4usZsWIv2AlfyQVmm8qCWVk/i6tqXhF/E9di2F/waw7+NpkcAYWWZVC3d5G46lMw8KeCbFW9T4WpuMqTcXndw82xWNc1pw+MZCb/SYuzIH4a+7eVwBOLXBPTvcUBUJuodQA966f/khWA0gTbqU1Gm94s0olDLqG0d+gvsYSBykN5twWAXDtFbDgXQh8PUMvQTSag8Qm1MSK+I5rcN2r2Bdk8oEWmL3lG8BJaMi5SIcEySaaG2b2gZLQwr2AMsozVGrzYqjydOLkOKNeD3kDUgJxi5kSCEJFnUQ7iCrz9EwnG8HPD1YQCDJNmDdgzRWIyYTwRuC+YMIQpbO4dWaJ74OmKYXQtDPDD+IIWgvyEUnXQlOQMQEOVyBHSVaAOnh1KJlPwInX+ia/Z3dKeSy9gbwxpkV2DfIYLqI5sON8D20oXmFWrb8erq8hwJZoCOVPwRUn+g+2GwBGtB3kuhuBCAX8RrQTtCToWzo9SeCjKghX5D72K4rYPszAbtD/gZP1IIL0Uhe4OeNzFGj+jlcAUTUe86bvwPUEsDBAoAAAgIABV491KTBtcyAwAAAAEAAAAWAAAAYXBwbGljYXRpb24ucHJvcGVydGllc+MCAFBLAQIUAwoAAAgIABV491IAAAAAAgAAAAAAAAAJAAAAAAAAAAAAEADtQQAAAABNRVRBLUlORi9QSwECFAMKAAAICAAVePdSsn8C7hsAAAAZAAAAFAAAAAAAAAAAAAAApIEpAAAATUVUQS1JTkYvTUFOSUZFU1QuTUZQSwECFAMKAAAICAAVePdSAAAAAAIAAAAAAAAABAAAAAAAAAAAABAA7UF2AAAAY29tL1BLAQIUAwoAAAgIABV491IAAAAAAgAAAAAAAAALAAAAAAAAAAAAEADtQZoAAABjb20vdm13YXJlL1BLAQIUAwoAAAgIABV491IAAAAAAgAAAAAAAAAPAAAAAAAAAAAAEADtQcUAAABjb20vdm13YXJlL3NjZy9QSwECFAMKAAAICAAVePdSAAAAAAIAAAAAAAAAGgAAAAAAAAAAABAA7UH0AAAAY29tL3Ztd2FyZS9zY2cvZXh0ZW5zaW9ucy9QSwECFAMKAAAICAAVePdSAAAAAAIAAAAAAAAAKQAAAAAAAAAAABAA7UEuAQAAY29tL3Ztd2FyZS9zY2cvZXh0ZW5zaW9ucy9teWN1c3RvbWZpbHRlci9QSwECFAMKAAAICAAVePdSFYcLu4sFAAD4DgAAVAAAAAAAAAAAAAAApIF3AQAAY29tL3Ztd2FyZS9zY2cvZXh0ZW5zaW9ucy9teWN1c3RvbWZpbHRlci9BZGRNeUN1c3RvbUhlYWRlckdhdGV3YXlGaWx0ZXJGYWN0b3J5LmNsYXNzUEsBAhQDCgAACAgAFXj3UpMG1zIDAAAAAQAAABYAAAAAAAAAAAAAAKSBdAcAAGFwcGxpY2F0aW9uLnByb3BlcnRpZXNQSwUGAAAAAAkACQCGAgAAqwcAAAAA
kind: ConfigMap
metadata:
  creationTimestamp: "2021-07-23T21:02:47Z"
  name: my-custom-header
  namespace: testing
  resourceVersion: "10535421"
  selfLink: /api/v1/namespaces/testing/configmaps/my-custom-header
  uid: 405e72ce-b025-4552-bee3-5d795c9013ce

Note It's possible to create a config map with multiple jars, for example if you need a third-party library. Ensure that these do not cause classpath conflicts with project template. In case of doubt check the "Runtime description for Extensions developers" for full list of included libraries.

Once it is running, if you need to update it, you can easily do it and automatically restart the gateway with:

$ kubectl create configmap extension-name --from-file=extension.jar -o yaml --dry-run=client | kubectl apply -f -

Using OCI image

Create a Dockerfile in the root of your project that contains the custom extension

FROM gradle:7-jdk17-alpine AS build
COPY --chown=gradle:gradle . /home/gradle/src
WORKDIR /home/gradle/src
RUN gradle build --no-daemon

FROM alpine

RUN mkdir -p /app/extensions

COPY --from=build /home/gradle/src/build/libs/*.jar /app/extensions/gateway-extension.jar

Now build the image using the docker command. You should specify the image registry you will push the extension to, this should be the same registry where the gateway and operator images reside. You can obtain it by running the command kubectl get deployment -o jsonpath="{..image}" -n spring-cloud-gateway.

docker build . -t myregistry.example.com/scg-test-extensions:dev

Then push the image to the registry.

docker push myregistry.example.com/scg-test-extensions:dev

Now you can use the custom extension by creating a gateway like this:

apiVersion: "tanzu.vmware.com/v1"
kind: SpringCloudGateway
metadata:
name: test-gateway
spec:
extensions:
  custom:
    - myregistry.example.com/scg-test-extensions:dev

This will create a gateway that uses a Kubernetes Init Container to load the custom extensions.

Using persistent volumes

For extensions larger than 1MB, you can load extensions from Kubernetes Volumes.

The example below shows how to setup a Persistent Volume using exclusively Kubernetes tools. Please refer to your storage implementations in your Cloud/K8s provider, ideal solutions would provide direct access to the storage backend in order to update the contents of the volumes without additional Kubernetes components.

Persistent Volumes comes with some restrictions:

  • An extra pod is required to access the storage backend (upload-extensions-pod in the example below).
  • The gateway expects a persistent volume claim.
  • The gateway expects the extensions to be located in /{mount_name}/extensions.
  • The contents of the directory should all be readable from the pod. For example, some storage providers may add a /{mount_name}/lost+found/ directory that is not readable. This potential issue is mitigated by expecting the extensions in the /{mount_name}/extensions/ subdirectory.
  • If the PersistentVolume has an access mode of ReadWriteOnce, the gateway pods must be scheduled on the same node to concurrently access the volume. For High-Availability scenarios (gateways across multiple cluster nodes), a StorageClass with support for ReadOnlyMany is required. Consult specific storage implementations for your Cloud/K8s provider.
  • If the PersistentVolume has an access mode of ReadWriteMany or ReadOnlyMany, the gateway pods can be scheduled under different nodes. However, a cloud provider may decide to have the PersistentVolume only be accessible within one Availability Zone (AZ), so the gateway pods need to be scheduled to the same AZ. Consult specific storage implementations for your Cloud/K8s provider.

To test this feature, create a PersistentVolumeClaim, along with a pod for uploading files. For details on how to create a matching compatible Volume, we suggest starting checking Kubernetes docs on Persistent Volumes.

---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: extensions-pvc
spec:
  storageClassName: standard
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 1Gi
---
apiVersion: v1
kind: Pod
metadata:
  name: upload-extensions-pod
spec:
  containers:
    - name: task-pv-container
      image: nginx
      volumeMounts:
        - mountPath: /mount
          name: extensions
  initContainers:
    - name: init-extensions-dir
      image: nginx
      command: ['sh', '-c', 'mkdir -p /mount/extensions']
      volumeMounts:
        - mountPath: /mount
          name: extensions
  volumes:
    - name: extensions
      persistentVolumeClaim:
        claimName: extensions-pvc

Next, copy your extension to /mount/extensions

  $ kubectl cp add-my-custom-header.jar upload-extensions-pod:/mount/extensions/add-my-custom-header.jar -c task-pv-container

Finally, specify the PersistentVolumeClaim in the custom extensions array of the gateway and the custom route filter in the route config, as shown in the next section.

Gateway configuration

With the extension deployed in the cluster, update or create a new Gateway with the extensions option.

apiVersion: "tanzu.vmware.com/v1"
kind: SpringCloudGateway
metadata:
  name: my-gateway
spec:
  extensions:
    custom:
      - extension-name

This will automatically restart the Gateway with the new extension(s) available.

Now that the extension is available, it can be used in the respective Route Configuration.

apiVersion: "tanzu.vmware.com/v1"
kind: SpringCloudGatewayRouteConfig
metadata:
  name: my-gateway-routes
spec:
  routes:
  - uri: https://httpbingo.org
    predicates:
      - Path=/add-header/**
    filters:
      - AddMyCustomHeader

If there's no mapping already, add it to complete the configuration.

apiVersion: "tanzu.vmware.com/v1"
kind: SpringCloudGatewayMapping
metadata:
  name: test-gateway-mapping
spec:
  gatewayRef:
    name: my-gateway
  routeConfigRef:
    name: my-gateway-routes

Validation and troubleshooting

With the deployment completed, activate traffic to the gateway kubectl port-forward service/my-gateway 8080:80 and open http://localhost:8080/add-header/get in your web browser.

You will be greeted with a response similar to the one below, for simplicity some data has been removed. There you should see the custom X-My-Header header.

{
  "args": {},
  "headers": {
    ...
    "Host": ["httpbingo.org"],
    "X-Forwarded-Host": ["localhost:8080"],
    "X-Forwarded-Prefix": ["/add-header"],
    "X-My-Header": ["my-header-value"]
  },
  "url": "https://localhost:8080/get"
}

If you cannot see the extension working:

  • Obtain the output of a gateway instance (kubectl logs statefulset.apps/my-gateway) to validate if your log traces appear.
  • See the Gateway events with kubectl describe scg my-gateway for diagnostics messages. If the extension could not be loaded you will see a message like ConfigMap '{extension_name}' not found. Skipping configuration.
  • Check the ConfigMap or PersistentVolumeClaim is available in the same namespace as the gateway.
  • Ensure the ConfigMap or PersistentVolumeClaim name matches the extension configuration in the Gateway.

High-availability deployments

Previous kubectl cp approach can cause issues with providers that don't support ReadWriteMany. For example Google Kubernetes Engine only supports ReadWriteOnce and ReadOnlyMany. That means that the upload pod cannot simultaneously run with multiple gateway instances in different nodes.

In those scenarios, to provide 100% availability you can use the automatic update features of SCG for K8s by switching to a different PersistentVolumeClaim. Provided you have:

  • 1 Volume and VolumeClaim with the old extension version
  • 1 running Spring Cloud Gateway for Kubernetes with 2 or more instances using the old extension
  • Jar file(s) of the new extensions version

You should:

  1. Create a new Volume and VolumeClaim
  2. Upload the new extension version as described in previous step
  3. Update the Gateway extensions.custom value with a new VolumeClaim name 
    apiVersion: "tanzu.vmware.com/v1"
kind: SpringCloudGateway
metadata:
  name: my-gateway
spec:
  extensions:
    custom:
      - extension-name-new

This will initiate a controlled update of the Gateway instances one by one ensuring no downtime.

check-circle-line exclamation-circle-line close-line
Scroll to top icon