VMware Cloud Director Object Storage Extension 2.2.3 Release Notes

VMware Cloud Director Object Storage Extension 2.2.3 \| 30 NOV 2023 \| Build 22871129 Check for additions and updates to these release notes.

VMware Cloud Director Object Storage Extension 2.2.3 | 30 NOV 2023 | Build 22871129

Check for additions and updates to these release notes.

What's New

VMware Cloud Director Object Storage Extension version 2.2.3 includes the following new features:

VMware Cloud Director Object Storage Extension installation process optimization

VIP server and Kubernetes Backup and Restore deployer are no longer part of the VMware Cloud Director Object Storage Extension installation process.
API token authentication

Cloud providers can now use an API token, instead of system administrator credentials, to authenticate the VMware Cloud Director instance where the plug-in is installed.
Custom Storage User Mapping

Existing users in a supported S3 platform can now be mapped to a tenant organization in VMware Cloud Director Object Storage Extension. One tenant user can be mapped to multiple S3 storage users.
OSIS adapter name deprecation

The OSIS adapter name, returned from the API GET /api/info is no longer used and mapped to the local adapter name in VMware Cloud Director Object Storage Extension.
Enhancements
- VMware Cloud Director Object Storage Extension 2.2.3 now supports OpenSSL 3 for importing or generating certificates with the command-line script ose cert.
- VMware Cloud Director Object Storage Extension 2.2.3 expands the operating system versions and it supports and integrates DELL ECS 3.8.

Upgrade

You can upgrade directly to VMware Cloud Director Object Storage Extension 2.2.3 from versions 2.X. See Upgrading VMware Cloud Director Object Storage Extension.

From VMware Cloud Director Object Storage Extension 2.2.3, operating systems, such as CentOS 7, are no longer supported. Verify that your operating system is supported before installing or upgrading to VMware Cloud Director Object Storage Extension 2.2.3.

If you plan to install VMware Cloud Director Object Storage Extension on a new operating system, you can migrate your existing configuration by using the ose config export and ose config import commands.

Caveats and Limitations

When you access a tenant organization as a cloud provider, you can see only local resources

When you access a tenant organization, you can see only the local resources of this organizaiton. When you open the VMware Cloud Director Object Storage Extension Dashboard or Buckets page, you can see and select only the local organizations.
S3 API requests authenticated with application credentials do not support the following use cases:
- Accessing a shared bucket if another user grants you permissions for the bucket.
- Deleting multiple objects simultaneously with a single API request.
- Copying objects from buckets that you own.
If you are using ECS storage, you cannot remove object tags.

When you try to remove an object tag, the operation fails with an error.
VMware Cloud Director and the underlying storage systems have different limitations on user names. To use VMware Cloud Director Object Storage Extension, user names must comply with both the requirements of VMware Cloud Director and the underlying storage system. A best practice is to use short user names (under 50 bytes) and to use alphanumeric characters.
- If you are using Cloudian storage, the maximum length of user IDs is 255 bytes.
- If you are using Dell ECS 3.4 or earlier, the maximum length of user IDs is 91 bytes.
- If you are using Dell ECS 3.6, the maximum length of user IDs is 64 bytes.
Bucket synchronization supports up to 10 million objects per a single synchronization job

When the cloud provider enables bucket synchronization for a tenant in the provider portal, the synchronization can support up to 10 million objects for the tenant. VMware Cloud Director Object Storage Extension 2.1 does not support the synchronization for more than 10 million objects per a single bucket synchronization job.
If you are using ECS storage, S3 API, or the Find a Bucket feature, you cannot visit a bucket that belongs to a different tenant organization in the ECS platform.

Product Documentation

All documentation is available on the VMware Cloud Director Object Storage Extension Documentation page.

Resolved Issues

New - You cannot see all the VMware Cloud Director tenant organizations

When you open the VMware Cloud Director Object Storage Extension Provider portal, you see a limited list of tenant organizations. The issue is observed if the VMware Cloud Director REST API maximum page size is set to size smaller than 128.
New - Cluster backup protection fails with an error

When you try to set up a Kubernetes cluster backup protection, the process fails with the following error:

500 internal {"status":500,"code":"InternalError","message":"Rollback bucket tenant-<organization-name> due to failing to enable cluster protection urn:vcloud:entity:vmware:capvcdCluster:<cluster-ID>, please contact the system administrator.","resource":"/api/v1/k8s/clusters/urn:vcloud:entity:vmware:capvcdCluster:<cluster-ID>/protection","requestId":"<requestId>"}
New - Backup of persistent volumes is unsuccessful for template-fs volumes.

When you try to back up template-fs volumes, the process enters a partially failed status.
New - You cannot log in to VMware Cloud Director Object Storage Extension as a service administrator

When you try to log in to VMware Cloud Director Object Storage Extension as a service administrator, the login is unsuccessful. The problem is observed on a random basis and it occurs if you have multiple groups which are mapped to multiple VMware Cloud Director roles.
New - Bucket synchronization does not synchronize all objects when object paths contain special symbols

VMware Cloud Director Object Storage Extension synchronizes the first 1,000 objects to the database. Special symbols include characters that might be encoded in the URL.
New - You cannot push images to the VMware Harbor Registry

When you try to push an image to the VMware Harbor Registry and the VMware Harbor Registry server uses VMware Cloud Director Object Storage Extension S3 as a backend storage, the process fails. The problem is observed when VMware Harbor Registry is configured with VMware Cloud Director Object Storage Extension S3 endpoint and credentials.
New - You cannot create a bucket policy with a UI-generated service instance ID

When you try to create a bucket policy and use the UI to generate the service instance ID, the process fails.
New - Synchronization tasks of buckets with more than 60,000 objects fail with a timeout error

When you try to sync a bucket that contains more than 60,000 objects, the synchronization task expires after 24 hours.

Known Issues

New - When you use the VMware Cloud Director Object Storage Extension S3 endpoint as a backup location, an error occurs

When you use the VMware Cloud Director Object Storage Extension S3 endpoint as a backup location and Velero is the backup software, Velero must generate a presigned URL to expose the download URL, but the predesigned URL is denied. For example, if you execute the command velero backup logs ${backupName}, you will receive the following error:

An error occurred: request failed: <Error><details><cause>The authorization header you provided is invalid.</cause></details><Status>401</Status><Code>S3_TOKEN_AUTH_ERROR</Code><Message>Failed to log into s3 endpoint with token: 60073ebea9766a2c043e.</Message><RequestId>4699397a-0301-4713-8fa5-f13cf8732523</RequestId></Error>

The issue is observed, because the command generates a presigned URL as its download URL, but the download URL is denied.
Workaround:

Append the suffix /api/v1/s3 to the VMware Cloud Director Object Storage Extension S3 endpoint.

For example:

Change spec.config.s3Url from https://ose-endpont to https://ose-endpoint/api/v1/s3:
```
kubectl edit backupstoragelocations.velero.io bsl-aws -n velero

apiVersion: velero.io/v1
kind: BackupStorageLocation
metadata:
  creationTimestamp: "2024-06-20T02:45:02Z"
  ...
  uid: 8f11f302-f17a-469c-a404-8f03eddb80eb
spec:
  accessMode: ReadWrite
  config:
    insecureSkipTLSVerify: "true"
    region: us-east-1
    s3ForcePathStyle: "true"
    s3Url: https://ose-endpoint
  ...
  provider: aws
status:
  phase: Available
```
Deleting an object from an existing bucket after upgrading to VMware Cloud Director Object Storage Extension version 2.2.3, fails with an error

If you upgrade to VMware Cloud Director Object Storage Extension version 2.2.3, then try to delete an object from an existing bucket, the process fails with the following error:

Failed to exchange user info between Cloud Director and storage platform.

The issue is observed if the tenant user who attempts the operation has a user name that contains special characters.
Workaround:
1. Navigate to the Postgres Database that VMware Cloud Director Object Storage Extension uses.
2. In the table bucket_info, in the storage_user_id for buckets column, add the encoded tenant user name.
  
  You can find the encoded user name in the table platform_user_mapping, when you select the platform_user_id corresponding to user_name.
Backing up an entire cluster fails

When you try to back up a Kubernetes cluster, where a pod contains persistent volumes in the primary node, the process enters a partially failed status.

Workaround: Activate Scheduling Pods in the Kubernetes Control plane primary nodes by running the following commands:

kubectl taint nodes --all node-role.kubernetes.io/master-

kubectl taint nodes --all node-role.kubernetes.io/control-plane-
The Kubernetes cluster protection status remains as Restoring

After performing a restore task in the target Kubernetes cluster, the cluster protection status remains as Restoring. The problem might occur when VMware Cloud Director Object Storage Extension continues to monitor the status of the restoring task and the state of the task remains in an InProgress state.
Workaround: Manually delete the restoring task.
1. Get the name of the "InProgress" restore task by running the following command:
  
  velero -n velero-09ad8e66-1841-4933-ad50-162170ed0ae7 restore describe
2. Delete the restore task by name by running the following command:
  
  velero -n velero-09ad8e66-1841-4933-ad50-162170ed0ae7 restore delete {restoreName}
  
  It takes a few moments for the cluster protection status to return to its normal state.
The S3 service of VMware Cloud Object Storage Extension is unavailable

When you start or view VMware Cloud Object Storage Extension, the VMware Cloud Object Storage Extension service is active, but the S3 service is unavailable, with the following error message in the log file:

S3_TOKEN_AUTH_ERROR

The issue is observed if the time gap between the S3 client and the VMware Cloud Object Storage Extension VM is over 20 seconds.

Workaround 1: Change the gap time between the S3 client and the VMware Cloud Object Storage Extension VM to less than 20 seconds, for example, NTP for the VMware Cloud Object Storage Extension VM.

Workaround 2: Run the command oss.s3.request-expire-time=3600 and restart the VMware Cloud Object Storage Extension service.
Region metrics on the provider portal's tenant onboarding page does not distinguish region specific metrics data

With multi-region deployment, when multiple regions are activated for a tenant organization, active region cards show the global consumption metrics, not region-specific data. The problem is observed, because region specific metrics is not supported yet.

Workaround: None.