Install External DNS in Workload Clusters Deployed by a Standalone Management Cluster

This topic explains how to deploy ExternalDNS into a workload cluster in Tanzu Kubernetes Grid.

The ExternalDNS service publishes DNS records for applications to DNS servers, using a declarative, Kubernetes-native interface. It is packaged as a CLI-managed package in Tanzu Kubernetes Grid.

In environments where Harbor is deployed in a shared services cluster with load balancing (AWS, Azure, and vSphere with NSX Advanced Load Balancer), ExternalDNS may be used to publish a DNS hostname for the Harbor service. This provides access to Harbor from other clusters. For more information, see Harbor Registry and ExternalDNS.

Note
As of v2.5, TKG does not support clusters on AWS or Azure. See the End of Support for TKG Management and Workload Clusters on AWS and Azure in the Tanzu Kubernetes Grid v2.5 Release Notes.

Prerequisites

A bootstrap machine with the following installed:
- Tanzu CLI, Tanzu CLI plugins, and kubectl.
- yq v4.5 or later.
You have deployed a management cluster on vSphere, Amazon Web Services (AWS), or Azure, in either an Internet-connected or Internet-restricted environment. If you are using Tanzu Kubernetes Grid in an Internet-restricted environment, you performed the procedure in Prepare to Deploy Management Clusters to an Internet-Restricted Environment before you deployed the management cluster.
You have logged in to the Tanzu CLI with the tanzu context use command.
You have determined the Fully Qualified Domain Names (FQDNs) for the services you want to expose with ExternalDNS to your DNS server.

Prepare the Cluster for ExternalDNS Deployment

The ExternalDNS service must be deployed into the same cluster as the services for which it will export DNS records.

To install Harbor, follow the prerequisites and procedure in Install Harbor for Service Registry.
ExternalDNS supports creating records for both Kubernetes Services and Contour HTTPProxy resources. If you want to create records for Contour HTTPProxy resources in a workload cluster, you must install the Contour package in the cluster. For instructions, see Install Contour for Ingress Control. The Contour package is also required by Harbor.

Prepare the Configuration File for the ExternalDNS Package

The ExternalDNS package has been validated with AWS (Route 53), Azure DNS, and RFC2136 (BIND). The configuration provided exports records for both Contour HTTPProxy resources and Kubernetes Services of type LoadBalancer.

The External DNS community maintains integrations with many DNS providers at varying levels of stability. Except where noted, VMware does not guarantee support for integrating the ExternalDNS package with specific providers.

AWS (Route 53)

To prepare the configuration file to deploy the ExternalDNS package on AWS,

Create a hosted zone within Route 53 with the domain that your services will be using.
Record the Hosted zone ID. You will use this ID later, when configuring ExternalDNS.

Create an IAM policy for ExternalDNS that allows ExternalDNS to update Route 53. In the AWS Console, go to the IAM dashboard and under Access Management, navigate to the Policies screen. Click Create Policy and switch to the JSON tab. Paste in the following policy. If needed, you may fine-tune the policy to allow updates to the hosted zone that you just created. Complete the wizard.

{
 "Version": "2012-10-17",
 "Statement": [
   {
     "Effect": "Allow",
     "Action": [
       "route53:ChangeResourceRecordSets"
     ],
     "Resource": [
       "arn:aws:route53:::hostedzone/*"
     ]
   },
   {
     "Effect": "Allow",
     "Action": [
       "route53:ListHostedZones",
       "route53:ListResourceRecordSets"
     ],
     "Resource": [
       "*"
     ]
   }
 ]
}

Create an IAM user for ExternalDNS with the policy that you created above. In the AWS Console, go to the Users screen and click Add users. Provide a name for the IAM user and ensure Programmatic access is enabled. On the Permissions screen of the wizard, click Attach existing policies directly and select the policy that you created in the previous step.
Continue to the final page of the wizard. Record the Access key ID and Secret access key. To make these Route 53 credentials available to ExternalDNS, you create a Kubernetes secret in the namespace where ExternalDNS will be running.
1. Set the context of kubectl to the cluster where you are deploying ExternalDNS. For example:
```
kubectl config use-context tkg-services-admin@tkg-services
```
2. Create the Kubernetes secret:
```
kubectl -n tanzu-system-service-discovery create secret generic route53-credentials \
--from-literal=aws_access_key_id=YOUR-ACCESS-KEY-ID \
--from-literal=aws_secret_access_key=YOUR-SECRET-ACCESS-KEY
```
  Where YOUR-ACCESS-KEY-ID and YOUR-SECRET-ACCESS-KEY are the credentials that you recorded above.
Create a configuration file for the ExternalDNS package by retrieving the default configuration of the package:
```
tanzu package available get external-dns.tanzu.vmware.com/PACKAGE-VERSION --default-values-file-output FILE-PATH
```
Where PACKAGE-VERSION is the version of the ExternalDNS package that you want to install and FILE-PATH is the location to which you want to save the configuration file, for example, external-dns-data-values.yaml.

Configure the following settings in the external-dns-data-values.yaml file. This file configures the ExternalDNS package.

---

# Namespace in which to deploy ExternalDNS pods.
namespace: tanzu-system-service-discovery

# Deployment-related configuration.
deployment:
args:
  - --source=service
  - --source=ingress
  - --source=contour-httpproxy # Provide this to enable Contour HTTPProxy support. Must have Contour installed or ExternalDNS will fail.
  - --domain-filter=DOMAIN # Makes ExternalDNS see only the hosted zones matching provided domain, omit to process all available hosted zones.
  - --policy=upsert-only # Prevents ExternalDNS from deleting any records, omit to enable full synchronization.
  - --registry=txt
  - --txt-owner-id=HOSTED-ZONE-ID
  - --txt-prefix=txt # Disambiguates TXT records from CNAME records.
  - --provider=aws
  - --aws-zone-type=public # Looks only at public hosted zones. Valid values are public, private, or no value for both.
  - --aws-prefer-cname
env:
  - name: AWS_ACCESS_KEY_ID
     valueFrom:
     secretKeyRef:
        name: route53-credentials
        key: aws_access_key_id
  - name: AWS_SECRET_ACCESS_KEY
     valueFrom:
     secretKeyRef:
        name: route53-credentials
        key: aws_secret_access_key
securityContext: {}
volumeMounts: []
volumes: []

Replace the placeholders in the external-dns-data-values.yaml file with your values. For more configuration options, see the ExternalDNS documentation.

Before setting any additional configuration options in the external-dns-data-values.yaml file, review the values schema of the ExternalDNS package. To retrieve the values schema, run:

tanzu package available get external-dns.tanzu.vmware.com/AVAILABLE-VERSION --values-schema

Where AVAILABLE-VERSION is the version of the ExternalDNS package. The --values-schema flag retrieves the valuesSchema section from the Package API resource for the ExternalDNS package. You can set the output format, --output, for the values schema to yaml, json, or table. For more information, see Packages in Install and Manage Packages.

RFC2136 (BIND) Server

The RFC2136 provider allows you to use any RFC2136-compatible DNS server as a provider for ExternalDNS, such as BIND.

Request or create a TSIG key for your server. This key must be authorized to update and transfer the zone you want to update. The key looks similar to the following:

key "externaldns-key" {
algorithm hmac-sha256;
secret "/2avn5M4ndEztbDqy66lfQ+PjRZta9UXLtToW6NV5nM=";
};

If you are managing your own DNS server, then you can create a TSIG key using the tsig-keygen -a hmac-sha256 externaldns command. Copy the output to your DNS servers configuration. For example, for BIND, you add the key to the named.conf file and configure the zone with the allow-transfer and update-policy fields. For example:

key "externaldns-key" {
algorithm hmac-sha256;
secret "/2avn5M4ndEztbDqy66lfQ+PjRZta9UXLtToW6NV5nM=";
};
zone "k8s.example.org" {
type master;
file "/etc/bind/zones/k8s.zone";
      allow-transfer {
      key "externaldns-key";
      };
      update-policy {
      grant externaldns-key zonesub ANY;
      };
};

The above assumes that you also have a zone file that might look similar to the following:

$TTL 60 ; 1 minute
@         IN SOA  k8s.example.org.  root.k8s.example.org. (
                  16  ; serial
                  60  ; refresh (1 minute)
                  60  ; retry (1 minute)
                  60  ; expire (1 minute)
                  60  ; minimum (1 minute)
                  )
            NS   ns.k8s.example.org.
ns           A    1.2.3.4

Create a configuration file for the ExternalDNS package by retrieving the default configuration of the package:
```
tanzu package available get external-dns.tanzu.vmware.com/PACKAGE-VERSION --default-values-file-output FILE-PATH
```
Where PACKAGE-VERSION is the version of the ExternalDNS package that you want to install and FILE-PATH is the location to which you want to save the configuration file, for example, external-dns-data-values.yaml.

Configure the following settings in the external-dns-data-values.yaml file. This file configures the ExternalDNS package.

---

# Namespace in which to deploy ExternalDNS pods.
namespace: tanzu-system-service-discovery

# Deployment-related configuration.
deployment:
args:
   - --source=service
   - --source=ingress
   - --source=contour-httpproxy # Provide this to enable Contour HTTPProxy support. Must have Contour installed or ExternalDNS will fail.
   - --domain-filter=DOMAIN # For example, k8s.example.org. Makes ExternalDNS see only the hosted zones matching provided domain, omit to process all available hosted zones.
   - --policy=upsert-only # Prevents ExternalDNS from deleting any records, omit to enable full synchronization.
   - --registry=txt
   - --txt-owner-id=k8s
   - --txt-prefix=external-dns- # Disambiguates TXT records from CNAME records.
   - --provider=rfc2136
   - --rfc2136-host=IP-ADDRESS-OF-RFC2136-DNS-SERVER
   - --rfc2136-port=53
   - --rfc2136-zone=DNS-ZONE # For example, k8s.example.org.
   - --rfc2136-tsig-secret=TSIG-SECRET-FROM-STEP-1
   - --rfc2136-tsig-secret-alg=hmac-sha256
   - --rfc2136-tsig-keyname=TSIG-KEY-NAME # For example, externaldns-key.
   - --rfc2136-tsig-axfr
env: []
securityContext: {}
volumeMounts: []
volumes: []

Replace the placeholders in the external-dns-data-values.yaml file with your values. For more configuration options, see the ExternalDNS documentation.

Before setting any additional configuration options in the external-dns-data-values.yaml file, review the values schema of the ExternalDNS package. To retrieve the values schema, run:

tanzu package available get external-dns.tanzu.vmware.com/AVAILABLE-VERSION --values-schema

Azure

To prepare the configuration file to deploy the ExternalDNS package on Azure,

Log in to the az CLI:
```
az login
```
Set your subscription:
```
az account set -s SUBSCRIPTION-ID-GUID
```

Create a service principal:

az ad sp create-for-rbac -n SERVICE-PRINCIPAL-NAME

The JSON output of the command looks similar to the following:

{
  "appId": "a72a7cfd-7cb0-4b02-b130-03ee87e6ca89",
  "displayName": "foo",
  "name": "http://foo",
  "password": "515c55da-f909-4e17-9f52-236ffe1d3033",
  "tenant": "b35138ca-3ced-4b4a-14d6-cd83d9ea62f0"
}

Assign permissions to the service principal:
1. Retrieve the ID of the resource group:
```
az group show --name RESOURCE-GROUP --query id
```
2. Assign the reader role to the service principal for the resource group scope. You will need the appId from the output of the az ad sp create-for-rbac command above.
```
az role assignment create --role "Reader" --assignee APP-ID-GUID --scope RESOURCE-GROUP-RESOURCE-ID
```
3. Retrieve the ID of the DNS zone:
```
az network dns zone show --name DNS-ZONE-NAME -g RESOURCE-GROUP-NAME --query id
```
4. Assign the contributor role to the service principal for the DNS zone scope:
```
az role assignment create --role "Contributor" --assignee APP-ID-GUID --scope DNS-ZONE-RESOURCE-ID
```
To connect the ExternalDNS service to the Azure DNS service, you create a configuration file named azure.json on your local machine with contents that look like the following:
```
{
 "tenantId": "01234abc-de56-ff78-abc1-234567890def",
 "subscriptionId": "01234abc-de56-ff78-abc1-234567890def",
 "resourceGroup": "MyDnsResourceGroup",
 "aadClientId": "01234abc-de56-ff78-abc1-234567890def",
 "aadClientSecret": "uKiuXeiwui4jo9quae9o"
}
```
Replace the values in the example above with you own values as follows:
- To retrieve tenantId, you can run the az account show --query "tenantId" command.
- To retrieve subscriptionId, you can run the az account show --query "id" command.
- resourceGroup is the name of the resource group that your DNS zone is within.
- aadClientId is the appId from the output of the service principal.
- aadClientSecret is the password from the output of the service principal.
To make your Azure credentials available to ExternalDNS, create a Kubernetes secret in the namespace where ExternalDNS will be running:
1. Set the context of kubectl to the cluster where you are deploying ExternalDNS. For example:
```
kubectl config use-context tkg-services-admin@tkg-services
```
2. Create the secret, using the azure.json configuration file from the previous step:
```
kubectl -n tanzu-system-service-discovery create secret generic azure-config-file --from-file=azure.json
```
Create a configuration file for the ExternalDNS package by retrieving the default configuration of the package:
```
tanzu package available get external-dns.tanzu.vmware.com/PACKAGE-VERSION --default-values-file-output FILE-PATH
```
Where PACKAGE-VERSION is the version of the ExternalDNS package that you want to install and FILE-PATH is the location to which you want to save the configuration file, for example, external-dns-data-values.yaml.

Configure the following settings in the external-dns-data-values.yaml file. This file configures the ExternalDNS package.

---

# Namespace in which to deploy ExternalDNS.
namespace: tanzu-system-service-discovery

# Deployment-related configuration.
deployment:
 args:
   - --source=service
   - --source=ingress
   - --source=contour-httpproxy # Provide this to enable Contour HTTPProxy support. Must have Contour installed or ExternalDNS will fail.
   - --domain-filter=DOMAIN # For example, k8s.example.org. Makes ExternalDNS see only the hosted zones matching provided domain, omit to process all available hosted zones.
   - --policy=upsert-only # Prevents ExternalDNS from deleting any records, omit to enable full synchronization.
   - --registry=txt
   - --txt-prefix=externaldns- # Disambiguates TXT records from CNAME records.
   - --provider=azure
   - --azure-resource-group=RESOURCE-GROUP # Azure resource group.
 env: []
 securityContext: {}
 volumeMounts:
   - name: azure-config-file
     mountPath: /etc/kubernetes
     readOnly: true
 volumes:
   - name: azure-config-file
     secret:
       secretName: azure-config-file

Replace the placeholders in the external-dns-data-values.yaml file with your values. For more configuration options, see the ExternalDNS documentation.

Before setting any additional configuration options in the external-dns-data-values.yaml file, review the values schema of the ExternalDNS package. To retrieve the values schema, run:

tanzu package available get external-dns.tanzu.vmware.com/AVAILABLE-VERSION --values-schema

Install the ExternalDNS Package

Set the context of kubectl to the cluster where you are deploying ExternalDNS. For example:
```
kubectl config use-context tkg-services-admin@tkg-services
```
If the cluster does not have a package repository with the ExternalDNS package installed, such as the tanzu-standard repository, install one:
```
tanzu package repository add PACKAGE-REPO-NAME --url PACKAGE-REPO-ENDPOINT --namespace tkg-system
```
Where:
- PACKAGE-REPO-NAME is the name of the package repository, such as tanzu-standard or the name of a private image registry configured with ADDITIONAL_IMAGE_REGISTRY variables.
- PACKAGE-REPO-ENDPOINT is the URL of the package repository.
  - For the TKG v2.5.1 release, the tanzu-standard URL is projects.registry.vmware.com/tkg/packages/standard/repo:v2024.4.12. See List Package Repositories to obtain this value from the Tanzu CLI, or in Tanzu Mission Control see the Addons > Repositories list in the Cluster pane.
Retrieve the name of the ExternalDNS package:
```
tanzu package available list -A
```

Retrieve the version of the ExternalDNS package:

tanzu package available list external-dns.tanzu.vmware.com -A

If your external-dns-data-values.yaml file contains comments, remove them before installing the package:
```
yq -i eval '... comments=""' external-dns-data-values.yaml
```
Install the package:
```
tanzu package install external-dns \
--package external-dns.tanzu.vmware.com \
--version AVAILABLE-PACKAGE-VERSION \
--values-file external-dns-data-values.yaml \
--namespace TARGET-NAMESPACE
```
Where:
- TARGET-NAMESPACE is the namespace in which you want to install the ExternalDNS package. For example, the my-packages or tanzu-cli-managed-packages namespace.
  - If the --namespace flag is not specified, the Tanzu CLI installs the package in the default namespace. The ExternalDNS pods and any other resources associated with the ExternalDNS component are created in the tanzu-system-service-discovery namespace; do not install the ExternalDNS package into this namespace.
  - The specified namespace must already exist, for example from running kubectl create namespace my-packages.
- AVAILABLE-PACKAGE-VERSION is the version that you retrieved above.
For example:
```
tanzu package install external-dns \
--package external-dns.tanzu.vmware.com \
--version 0.10.0+vmware.1-tkg.1 \
--values-file external-dns-data-values.yaml \
--namespace my-packages
```
Confirm that the external-dns package has been installed:
```
tanzu package installed list -A
```
To see more details about the package, you can also run:
```
tanzu package installed get external-dns --namespace PACKAGE-NAMESPACE
```
Where PACKAGE-NAMESPACE is the namespace in which the external-dns package is installed.
Confirm that the external-dns app has been successfully reconciled in your PACKAGE-NAMESPACE:
```
kubectl get apps -A
```
If the status is not Reconcile Succeeded, view the full status details of the external-dns app. Viewing the full status can help you troubleshoot the problem.
```
kubectl get app external-dns --namespace PACKAGE-NAMESPACE -o yaml
```
Where PACKAGE-NAMESPACE is the namespace in which you installed the package. If troubleshooting does not help you solve the problem, you must uninstall the package before installing it again:
```
tanzu package installed delete external-dns --namespace PACKAGE-NAMESPACE
```
Confirm that ExternalDNS pods are running in the tanzu-system-service-discovery namespace:
```
kubectl get pods -A
```

Validating ExternalDNS

If configured with Contour, ExternalDNS will automatically watch the specified namespace for HTTPProxy resources and create DNS records for services with hostnames that match the configured domain filter.

ExternalDNS will also automatically watch for Kubernetes Services with the annotation external-dns.alpha.kubernetes.io/hostname and create DNS records for services whose annotations match the configured domain filter.

For example, for a service with the annotation external-dns.alpha.kubernetes.io/hostname: foo.k8s.example.org, ExternalDNS will create a DNS record for foo.k8s.example.org. You can validate that the record exists by examining the zone that you created.

Update a Running ExternalDNS Deployment

If you need to make changes to the configuration of the ExternalDNS package after deployment, follow these steps to update your deployed ExternalDNS package.

Update the ExternalDNS configuration in external-dns-data-values.yaml.

Update the configuration of the installed package:

tanzu package installed update external-dns \
--version INSTALLED-PACKAGE-VERSION \
--values-file external-dns-data-values.yaml \
--namespace INSTALLED-PACKAGE-NAMESPACE

Where:

INSTALLED-PACKAGE-VERSION is the version of the installed ExternalDNS package.
INSTALLED-PACKAGE-NAMESPACE is the namespace in which the ExternalDNS package is installed.

For example:

tanzu package installed update external-dns \
--version 0.10.0+vmware.1-tkg.1 \
--values-file external-dns-data-values.yaml \
--namespace my-packages

The ExternalDNS package will be reconciled using the new value or values that you added. It can take up to five minutes for kapp-controller to apply the changes.

For more information about the tanzu package installed update command, see Update a Package in Install and Manage Packages. You can use this command to update the version or the configuration of an installed package.