VMware Data Services Manager 2.1 Release Notes

VMware Data Services Manager 2.1.0 \| 23 JUL 2024 \| Build 24132739 Check for additions and updates to these release notes.

VMware Data Services Manager 2.1.0 | 23 JUL 2024 | Build 24132739

Check for additions and updates to these release notes.

What's in the Release Notes

The release notes cover the following topics:

What's New
Supported Platforms
Component Versions
Resolved Issues
Known Issues

What's New

VMware Data Services Manager 2.1 release introduces the following new features and enhancements.

Note:

VMware Data Services Manager supports upgrades to version 2.1 only from version 2.0.3. If you use VMware Data Services Manager version earlier than 2.0.3, upgrade it to 2.0.3 before upgrading to 2.1. For more information, see What's New in Release 2.0.3.

Increased the Kubernetes APIs coverage to install and manage DSM through declarative API.

Improved installation process:
- Entire contents of DSM are now included in DSM OVA. A number of manual steps required to install DSM has reduced, which, as a result, significantly improves and simplifies the overall process of installing and configuring DSM.
- Ability to monitor databases from inside the vSphere Client DSM plugin.
- Lack of S3 storage no longer blocks installation. S3 is still required for backups and logs.
- Manual downloading and installation of database templates are no longer required. This step is now automated.

Improved day 2 troubleshooting:
- Significantly improved alerting. DSM 2.1 offers more and better defined alerts.
Certificate management:
- User configurable list of globally trusted root certificates.
- Automatic rotation of certificates.
- Ability to configure DNS names to be used in the SAN field of the Provider VM and database endpoints.
- Ability to add custom externally signed certificates for the Provider VM and databases through DSM API.
- Certificate rotation of external end points, such as S3, without disruption to DSM.
Database improvements:
- New minor database versions. Minor version database upgrades can be done manually and automatically.
- LDAP authentication for databases.
- Improved user creation workflow for databases.
- Database metrics are displayed inside the vSphere Client plugin.
- MySQL now supports three node deployment limited to a single vSphere cluster.
Log shipping improvements:
- Better formatting for Aria Operations for logs by supporting the Ingestion protocol (cfapi).
- TLS-encrypted TCP connection.
- UDP connection is available.
- Custom Log Server.
Password rotation for pgadmin.
Cloud Consumption Interface (CCI) within VMware Aria Automation is now initially available with VMware Data Services Manager User Interface that will enable users to rapidly create and consume Postgres DB workloads in DSM. For additional information, see Provisioning databases with Aria Automation, Cloud Consumption Interface and Data Services Manager.

Supported Platforms

The following table identifies the supported component versions for VMware Data Services Manager version 2.1.

Note:

Make sure to enable vSphere HA and vSphere DRS on the vSphere clusters that you use for the infrastructure policies. For information, see How vSphere HA Works in vSphere Availability and Create a vSphere DRS Cluster in vSphere Resource Management.

Component	Supported Versions
vCenter	7.0.3i and later
ESXi	7.0 and later
VMFS	5 and 6
PostgreSQL	16.3, 15.7, 14.12, 13.15, 12.19
MySQL	8.0.34, 8.0.32, 8.0.31, 8.0.29

Component Versions

The DSM 2.1 release includes the following software component versions:

Component	Version
antrea	v1.11.2_vmware.1
cert-manager-cainjector	v1.12.2_vmware.1
cert-manager-controller	v1.12.2_vmware.1
cert-manager-webhook	v1.12.2_vmware.1
cloud-provider-vsphere	v1.26.2_vmware.1
coredns	v1.10.1_vmware.17
csi-attacher	v4.5.0_vmware.1
csi-livenessprobe	v2.12.0_vmware.1
csi-node-driver-registrar	v2.10.0_vmware.1
csi-provisioner	v4.0.0_vmware.1
csi-snapshotter	v7.0.1_vmware.1
dsm-alloydb-instance	2.1.0-20240618T1153Z-87da465c
dsm-alloydb-operator	2.1.0-20240618T1153Z-87da465c
etcd	v3.5.11_vmware.4
fluent-bit	v2.1.6_vmware.1
kapp	v0.48.2_vmware.1
kube-apiserver	v1.27.11_vmware.1
kube-controller-manager	v1.27.11_vmware.1
kube-proxy	v1.27.11_vmware.1
kube-scheduler	v1.27.11_vmware.1
kube-vip	v0.5.7_vmware.2
kubernetes-csi_external-resizer	v1.10.0_vmware.1
Kubernetes node OS	ubuntu 20.04
mysql-operator	2.1.0-20240712T1313Z-14b08f06
pause	3.9
postgres-operator	2.1.0-20240712T1302Z-77c1a7d8
telegraf	1.29.0
telegraf-chart	1.1.2
volume-metadata-syncer	v3.3.1_vmware.1
vsphere-block-csi-driver	v3.3.1_vmware.1

Resolved Issues

The 2.1 release of VMware Data Services Manager resolves the following known issues that appeared in the previous releases.

To find detailed descriptions and workarounds of these issues, search the VMware Data Services Manager 2.0.x Release Notes.

If you configure external/database-backup storage and FQDN ends with .local, node DNS Config can't resolve it.
Edit operations fail on Alloy DB after disabling DSR during an upgrade.
If you fail to run additional steps required when upgrading the DSM appliance from earlier releases to 2.0.3, system disruptions might occur.
Domain name that ends with .local is not resolved by DSM Provider (Resolved in 2.0.2).
If admin username is "postgres", the Postgres DB creation remains in "InProgress" state.
Data services releases are not visible in the Version & Upgrade tab after successful release processing.
Operations/tasks remain stuck InProgress due to new desired state set.
After PostgreSQL cluster backup location change, the status of lastSuccessfulBackup and Wal archival are not immediately updated and might show stale information.
Databases cluster provisioning does not work on datacenter different from the datacenter used in the first configured infrastructure policy.
When using kubectl get -w against DSM Kubernetes API, the process might fail to start or reflect the resource events properly.
When vCenter resources are included in a folder, problems with infrastructure policies or resource pools might occur.

Known Issues

You cannot provision a database cluster using an Infrastructure Policy with a VM folder nested under another folder in the root VM folder of a vCenter datacenter

When you try to provision a database cluster using an Infrastructure Policy with a VM folder nested under another root VM folder of a vCenter data center, the database cluster never achieves status Ready and the VMs for this cluster are never created in the vSphere inventory tree.

Workaround: In the Infrastructure Policy placements, select a VM folder that is directly under the datacenter in the vSphere inventory tree.
You cannot provision database clusters using an Infrastructure Policy with a VM folder that contains special characters

When you try to provision a database cluster using an Infrastructure Policy with a VM folder that contains special characters like * ? [ ] \ in its name, the database cluster never achieves status Ready and the VMs for this cluster are never created in the vSphere inventory tree.

Workaround: Do not use special characters like * ? [ ] \ in the name of the folder selected in the Infrastructure Policy.
You cannot change the database owner to an LDAP User when the login ID and email are different

When you try to change the database owner to an LDAP user whose login ID (use principle name) and email are different, after the operation, the database appears to have no owner.

Workaround: Change the database owner using API.

1. Download the kubeconfig file, as a DSM Admin.

2. Change the owner using the following command:

kubectl --kubeconfig <path-to-downloaded-kubeconfig> annotate --overwrite postgrescluster <name-of-cluster> dsm.vmware.com/owner=<correct-user>
Creation of an Infrastructure Policy fails with an error

When you try to create an Infrastructure Policy from the VMware Data Services Manager Plug-in, the process fails with the following error:

Error: cannot invoke "vmware.tdm.sp.plugin.vc.PathObject.isGroupHead()" because "obj" is null

Workaround: See KB 376189.
The LDAP user doesn't have access to any database-related operations in the VMware Data Services Manager UI

When you log in to the the VMware Data Services Manager UI using an LDAP with the DSM User role, you do not see any pages in the UI that are related to database operations, such as Database Operations, Summary and so on.

Workaround: None.
MySQL database cluster remains In Progress indefinitely after you deactivate the Directory Service

If you deactivate the Directory Service, this causes the MySQL database cluster and its Modify Database Cluster task to become stuck.

Workaround: None.
Postgres HA database gets permanent alerts after you apply a custom certificate

After you apply a custom certificate without the X509v3 extension KeyUsage DigitalSignature to a Postgres HA instance, the database starts to get permanent alerts.

Workaround: Restore or clone to another database, using a certificate with the X509v3 extension KeyUsage DigitalSignature.
You cannot create databases that use a storage policy with encryption

When you try to provision a database that uses a storage policy with encryption in VMware Data Services Manager, the process fails. VM encryption is available with vSphere 8.0 Update 3 and VMware Cloud Foundation 5.2 and to be able to use it with VMware Data Services Manager, you must add additional permissions to the DSM Administrator role in vCenter.

Workaround: In vCenter, edit the DSM Administrator role by adding the privilege Cryptographic operations > Encrypt.

See Using vCenter Server Roles to Assign Privileges in the VMware vSphere Product Documentation.
You cannot log in to VMware Data Services Manager using LDAP credentials after upgrading from version 2.0.3 to 2.1

Upgrading VMware Data Services Manger from version 2.0.3 to 2.1 might cause a reset of the LDAP settings and as a result users are unable to log in to VMware Data Services Manager using LDAP credentials. In some cases VMware Data Services Manger cannot find the leaf (or root) certificate of the LDAP server which might lead to errors in migration. You might see the following or a similar error message:

<YYY-MM-DD> <HH:mm:ss.SSS> ERROR [main] .s.PostUpgradeProcessorService - Exception fetching cert by url, skipping DirectoryService migration com.vmware.tdm.sp.common.exception.TdmException: Certificate for the server ldaps://example_ldap_host:636 should be self-signed or issuer CA certificate should be added to the Trusted Root Certificates.
Workaround: Add the root certificates again and reconfigure the directory service.
1. Log in to the VMware Data Services Manager UI using a locally created user.
  
  If you need to create a local user, use the VMware Data Services Manager plug-in in vCenter.
2. To add the certificates, navigate to Settings > Trusted Root Certificates > Add.
3. To reconfigure the directory service, navigate to Directory Service Settings > Configure Directory Service.
4. Validate log in to VMware Data Services Manager UI using LDAP credentials.
The VMware Data Services Manager user interface does not load after upgrading to version 2.1

After you successfully upgrade VMware Data Services Manager to version 2.1 and then enter your credentials, the user interface is stuck in a loading screen. This issue is caused by a network interruption between AWS and VMware Data Services Manager during the bundle download process from the provider repository.
Workaround:
1. Establish an SSH connection to the VMware Data Services Manager appliance.
2. Extract the bearer token by running the following command:
  
  curl -k -v -location 'https:/<provider_ip>/provider/session' --header 'Content-Type: application/json' --data-raw '{ "email" : "YourEmailAddress", "password" : "YourPassword" }'
3. Copy the files from S3 to the local appliance by executing the following command:
  
  curl -k -v --location --request POST 'https://<PROVIDER_IP_ADDRESS>/updatemanager/copy-bundles' \--header 'Authorization: Bearer <BEARER TOKEN>' \--header 'Accept: application/json, text/plain, */*'
  
  The copying process might take between 1 and 2 hours.
Installation or upgrade of VMware Data Services Manager with two or more NTP servers fails with an error

When you try to install or upgrade VMware Data Services Manager with two or more NTP servers, the dsm-system-config file creation fails with the following error:

dsmsystemconfigs.system.dataservices.vmware.com "dsm-system-config" not found

The issue is observed when the NTP servers are not correctly separated with a comma in the config file after parsing by the installer.

Workaround: See KB 373070.
After an upgrade to DSM version 2.1, the vSphere Client shows the DSM version as 2.0.3
You can see version 2.0.3 on the following pages of the vSphere Client:
- In vCenter DSM plugin, click Configure > Version & Upgrade.
  
  You can observe that Current Control Plane Version in the Version Information pane displays version 2.0.3.
- In vCenter, click Administration > Client Plugins > Data Service Manager Plugin.
  
  Observe that the Instance version column shows 2.0.3.
Workaround:
1. Make sure that the DSM version shows correctly in the DSM console.
  1. In the DSM console, select Version & Upgrade from the left navigation pane, and click the DSM System tab.
  2. Check that Current Control Plane Version in the Version Information pane displays 2.1.0.XXXX.
2. Unregister and re-register the DSM plugin manually. See Unregistration and Re-registration of VMware Data Services Manager Plugin.
VMware Data Services Manager fails to start after installation to a vCenter instance that has outdated trusted root certificates

If your vCenter instance has trusted root certificates without X509v3 extension X509v3 Key Usage = Certificate Sign and you install VMware Data Services Manager, VMware Data Services Manager fails to start with the following error:

configmaps 'vcenter-ca' not found

Workaround 1: Delete the problematic trusted root certificate from the vCenter instance and retry the operation.

Workaround 2: Manually create a Kubernetes ConfigMap with name vcenter-ca in the dsm-system namespace and upload the vCenter trusted root certificates using the VMware Data Services Manager UI.
High CPU utilization on vSphere clusters running DSM databases might impact the ability to create, scale, or upgrade the databases

When DSM databases are running on vSphere clusters that are using large amounts of CPU, some database management operations my be impacted. This is especially true when high CPU alarm conditions have been raised on the vSphere clusters.

Workaround: Address the high CPU alarm condition raised on the vSphere cluster by adding more resources or migrating VMs to another vSphere cluster.
Attempts to disable log forwarding through the DSM UI console are not successful

When you disable log forwarding through the UI, logs still flow to external logs collector.

You can observe errors in the logs with the following text provider type can not be empty.
Workaround:

To disable log forwarding edit the dsmsystemconfig directly using the Gateway Kubernetes APIs.

Example 1: Disable log forwarding in an environment where it is currently enabled.

When you query the dsmsystemconfig object in the gatway, the log forwarding config should look similar to the following:
```
externalLogDestination:
      enabled: true      
      remoteLogDestinationProvider: "sys log"      
      remoteLogUrl: "udp://myurl"      
      trustBundle: {}
```
Change the value for enabled to false as in the following example.
```
externalLogDestination:
      enabled: false      
      remoteLogDestinationProvider: "sys log"      
      remoteLogUrl: "udp://myurl"      
      trustBundle: {}
```
Make sure to keep existing values for remoteLogDestinationProvider and remoteLogUrl.

Example 2: Disable log forwarding in an environment after failed attempts to disable it through the UI.

When you query the dsmsystemconfig object in the gatway, the log forwarding config should look similar to the following:
```
externalLogDestination:
      enabled: false      
      remoteLogDestinationProvider: ""      
      remoteLogUrl: ""      
      trustBundle: {}
```
Make the changes as in the following example:
```
externalLogDestination:
      enabled: false      
      remoteLogDestinationProvider: "sys log"      
      remoteLogUrl: ""      
      trustBundle: {}
```
Set the value for remoteLogDestinationProvider to sys log. This action successfully completes the disable log forwarding operation.
After a DSM provider VM has been restored, you might see a permissions denied error message
After a DSM provider VM has been restored, you can see the following error on the Data Services tab of the DSM console:
```
Message: Making database images available for database creation (failed to process manifest file in bundle: error opening zip file: open /data/bundles/dsm-data-plane/2.1.0/2.1.0-image-bundle-image-bundle-43bbb99332.zip:
permission denied)
```
Workaround:
1. SSH into provider.
2. Change permission of the file.
  
  For fresh installation of 2.1.0:
  
  chmod 775 /data/bundles/dsm-data-plane/2.1.0/2.1.0-image-bundle-image-bundle-43bbb99332.zip
  
  If provider environment has been updated from 2.0.3 to 2.1.0, change file permission as follows:
  
  chmod 775 /data/bundles/dsm-data-plane/2.0.0/2.0.0-image-bundle-image-bundle-2.0.1-4-ga3953bb.zip
  
  chmod 775 /data/bundles/dsm-data-plane/2.1.0/2.1.0-image-bundle-image-bundle-43bbb99332.zip
3. Run the following commands:
  
  systemctl restart dsm-tsql-provisioner.service
  
  Wait for two or three minutes and run:
  
  systemctl restart simplified-gateway.service
4. Observe the DSR Enable(InProgress) status on the DSM console.
Postgres backups and Write-Ahead Log (WAL) archiving might fail if the configured backup location's endpoint is an IP address

Postgres backups and Write-Ahead Log (WAL) archiving will fail if the configured backup location's endpoint is an IP address, and the certificate for the backup location doesn't list the IP address as a DNS entry within the certificate.
Workaround:
1. Update certificate being used by the storage endpoint to include the IP address as a DNS entry of the Subject Alternative Name (SAN).
  
  See the following example:
  
  X509v3 Subject Alternative Name: IP Address:10.218.130.143, DNS:10.218.130.143
2. Navigate to the storage settings of your DSM appliance, click the edit button on impacted backup location, and accept the new certificate.
After a fresh installation of DSM 2.1, data services are not showing up in the Data Services tab of Version & Upgrade

If you navigate to the Data Services tab of the Version & Upgrade page in the DSM console, you cannot see available data services listed on the tab.
Workaround:
1. In the DSM console, select Version & Upgrade from the left navigation pane, and click the DSM System tab.
2. Click CHECK FOR UPGRADE in the Available Upgrades & History pane.
3. Wait two or three minutes for the data services to appear in the Data Services tab.
During a MySQL database cluster creation, a replica might occasionally fails to join the cluster
In addition, you might observe the following issues.

The bootstrapping process keeps failing and restarting repeatedly. The cluster fails to start.

You can also see the following error message:
```
❯ kubectl get mysqlcluster st1s-myreuse-001 -o yaml    
    ....
    message: 'Database is accessible, but 0 member/s is/are not online or not active
      members of the cluster: . No failure tolerance: Cluster is NOT tolerant to any
      failures.'
    observedGeneration: 1
    reason: Degraded
    status: "True"
    type: DatabaseEngineReady
```
Workaround: If the problem continues for over 30 minutes, delete the cluster and create it again.
VMware Data Services Manager 2.1 does not support five-node topology (3 replicas) for PostgreSQL database clusters

Only clusters with three nodes (1 replica) or a single node (0 replicas) are supported by VMware Data Services Manager 2.1.

Workaround: If you have any existing five-node clusters in your environment created with VMware Data Services Manager 2.0.3, they will continue to be functional after upgrading to 2.1, but you cannot edit them. You can clone or restore your five-node clusters into one-node (0 replicas) or three-node (1 replica) clusters if you want them to be modifiable in 2.1.
Performing scale operations on a database cluster, such as changing a VM class from small to medium or upgrading a database cluster, might cause failures if underlying ESXi hosts have active CPU or memory alarms

These alarms might indicate that the host does not have enough resources for workload cluster nodes.

Workaround: Make sure to address these alarms before performing scale operations.