VMware Cloud Director Object Storage Extension 2.2.2 Release Notes

VMware Cloud Director Object Storage Extension 2.2.2 | 18 JUL 2023 | Build 22098306 Check for additions and updates to these release notes.

What's New

VMware Cloud Director Object Storage Extension version 2.2.2 includes the following new features:

• Automated tenant onboarding

  Offering automation capabilities for new VMware Cloud Director tenant organizations. Providing cloud providers with the flexibility to activate or deactivate tenants as needed.

• Bucket SSE-C support

  Enhancing the encryption capabilities by extending the S3 bucket encryption. Allowing for configurable bucket encryption to align with each individual tenant’s needs.

• Publish core open APIs

  VMware Cloud Director Object Storage Extension publishes a set of core APIs that allow the cloud providers and tenants to automate various object storage workflows. The set includes S3-compatible APIs for bucket and object operations, provider and tenant administration, as well as tenant operations.

• Advanced backup and restore options for Kubernetes clusters

  Providing more options for the backing up and restoring of Kubernetes clusters in tenant organizations.

• TLS support for Postgres connection

  Enable TLS connection to Postgres server for the backend server of Object Storage VMware Cloud Director Object Storage Extension.

Upgrade

You can upgrade directly to VMware Cloud Director Object Storage Extension 2.2.2 from versions 2.X. See Upgrading VMware Cloud Director Object Storage Extension.

Caveats and Limitations

• New - When you access a tenant organization as a cloud provider, you can see only local resources

  When you access a tenant organization, you can see only the local resources of this organizaiton. When you open the VMware Cloud Director Object Storage Extension Dashboard or Buckets page, you can see and select only the local organizations.

• S3 API requests authenticated with application credentials do not support the following use cases:

  • Accessing a shared bucket if another user grants you permissions for the bucket.

  • Deleting multiple objects simultaneously with a single API request.

  • Copying objects from buckets that you own.

• If you are using ECS storage, you cannot remove object tags.

  When you try to remove an object tag, the operation fails with an error.

• VMware Cloud Director and the underlying storage systems have different limitations on user names. To use VMware Cloud Director Object Storage Extension, user names must comply with both the requirements of VMware Cloud Director and the underlying storage system. A best practice is to use short user names (under 50 bytes) and to use alphanumeric characters.

  • If you are using Cloudian storage, the maximum length of user IDs is 255 bytes.

  • If you are using Dell ECS 3.4 or earlier, the maximum length of user IDs is 91 bytes.

  • If you are using Dell ECS 3.6, the maximum length of user IDs is 64 bytes.

• Bucket synchronization supports up to 10 million objects per a single synchronization job

  When the cloud provider enables bucket synchronization for a tenant in the provider portal, the synchronization can support up to 10 million objects for the tenant. VMware Cloud Director Object Storage Extension 2.1 does not support the synchronization for more than 10 million objects per a single bucket synchronization job.

• If you are using ECS storage, S3 API, or the Find a Bucket feature, you cannot visit a bucket that belongs to a different tenant organization in the ECS platform.

Product Documentation

All documentation is available on the VMware Cloud Director Object Storage Extension Documentation page.

Resolved Issues

• New - VMware Cloud Director Object Storage Extension fails to start

  If you manually change the endpoint of VMware Cloud Director Object Storage Extension by using the command ose endpoint set, VMware Cloud Director Object Storage Extension might fail to start due to the following error:

  Unsatisfied dependency expressed through field 'regionMgmtService'

Known Issues

• New - You cannot see all the VMware Cloud Director tenant organizations

  When you open the VMware Cloud Director Object Storage Extension Provider portal, you see a limited list of tenant organizations. The issue is observed if the VMware Cloud Director REST API maximum page size is set to size smaller than 128.

  Workaround: Set the VMware Cloud Director REST API maximum page size to size larger than 128 by setting restapi.queryservice.maxPageSize with VMware Cloud Director cell management tool. See Configure Your VMware Cloud Director Installation. For example:

  /opt/vmware/vcloud-director/bin/cell-management-tool manage-config -n restapi.queryservice.maxPageSize -v 128

• New - Cluster backup protection fails with an error

  When you try to set up a Kubernetes cluster backup protection, the process fails with the following error:

  500 internal {"status":500,"code":"InternalError","message":"Rollback bucket tenant-<organization-name> due to failing to enable cluster protection urn:vcloud:entity:vmware:capvcdCluster:<cluster-ID>, please contact the system administrator.","resource":"/api/v1/k8s/clusters/urn:vcloud:entity:vmware:capvcdCluster:<cluster-ID>/protection","requestId":"<requestId>"} 

  Workaround: See KB 94728.

• New - Backing up an entire cluster fails

  When you try to back up a Kubernetes cluster, where a pod contains persistent volumes in the primary node, the process enters a partially failed status.

  Workaround: Activate Scheduling Pods in the Kubernetes Control plane primary nodes by running the following commands:

  kubectl taint nodes --all node-role.kubernetes.io/master-

  kubectl taint nodes --all node-role.kubernetes.io/control-plane-

• New - Backup of persistent volumes is unsuccessful for template-fs volumes.

  When you try to back up template-fs volumes, the process enters a partially failed status.

  Workaround 1: Restore the partially failed backup.

  1. Get the Velero namespace by running the following command:

  kubectl get namespace | grep velero

  2. Restore the backup by running the following command:

  velero --namespace ${<velero-namespace>} restore create --from-backup ${<your-backup-name>} --allow-partially-failed

  Workaround 2: Reconfigure Velero to v.1.11.1 before you perform the backup by running the following commands:

  ose args set -k oss.k8s.velero.tag -v v1.11.1

  ose service restart

• New - The Kubernetes cluster protection status remains as Restoring

  After performing a restore task in the target Kubernetes cluster, the cluster protection status remains as Restoring. The problem might occur when VMware Cloud Director Object Storage Extension continues to monitor the status of the restoring task and the state of the task remains in an InProgress state.

  Workaround: Manually delete the restoring task.

  1. Get the name of the "InProgress" restore task by running the following command:

     velero -n velero-09ad8e66-1841-4933-ad50-162170ed0ae7 restore describe

  2. Delete the restore task by name by running the following command:

     velero -n velero-09ad8e66-1841-4933-ad50-162170ed0ae7 restore delete {restoreName}

     It takes a few moments for the cluster protection status to return to its normal state.

• You cannot log in to VMware Cloud Director Object Storage Extension as a service administrator

  When you try to log in to VMware Cloud Director Object Storage Extension as a service administrator, the login is unsuccessful. The problem is observed on a random basis and it occurs if you have multiple groups which are mapped to multiple VMware Cloud Director roles.

  Workaround: Log in to VMware Cloud Director Object Storage Extension with a user that has a single group, mapped to their VMware Cloud Director role.

• Bucket synchronization does not synchronize all objects when object paths contain special symbols

  VMware Cloud Director Object Storage Extension synchronizes the first 1,000 objects to the database. Special symbols include characters that might be encoded in the URL.

  Workaround: None.

• You cannot push images to the VMware Harbor Registry

  When you try to push an image to the VMware Harbor Registry and the VMware Harbor Registry server uses VMware Cloud Director Object Storage Extension S3 as a backend storage, the process fails. The problem is observed when VMware Harbor Registry is configured with VMware Cloud Director Object Storage Extension S3 endpoint and credentials.

  Workaround: Request a hotfix from VMware.

• You cannot create a bucket policy with a UI-generated service instance ID

  When you try to create a bucket policy and use the UI to generate the service instance ID, the process fails.

  Workaround: Create the service instance ID manually and ensure it does not include the hyphen symbol (-) .

• Synchronization tasks of buckets with more than 60,000 objects fail with a timeout error

  When you try to sync a bucket that contains more than 60,000 objects, the synchronization task expires after 24 hours.

  Workaround: Click the Sync Now button repeatedly until the synchronization task completes successfully.

• The S3 service of VMware Cloud Object Storage Extension is unavailable

  When you start or view VMware Cloud Object Storage Extension, the VMware Cloud Object Storage Extension service is active, but the S3 service is unavailable, with the following error message in the log file:

  S3_TOKEN_AUTH_ERROR

  The issue is observed if the time gap between the S3 client and the VMware Cloud Object Storage Extension VM is over 20 seconds. 

  Workaround 1: Change the gap time between the S3 client and the VMware Cloud Object Storage Extension VM to less than 20 seconds, for example, NTP for the VMware Cloud Object Storage Extension VM.

  Workaround 2: Run the command oss.s3.request-expire-time=3600 and restart the VMware Cloud Object Storage Extension service.

• Region metrics on the provider portal's tenant onboarding page does not distinguish region specific metrics data

  With multi-region deployment, when multiple regions are activated for a tenant organization, active region cards show the global consumption metrics, not region-specific data. The problem is observed, because region specific metrics is not supported yet.

  Workaround: None.