VMware vSphere Replication 8.2.0.2 | OCT 3 2019 | Build 14761900 | Download VMware vSphere Replication 8.2.0.1 | AUG 13 2019 | Build 14338525 | Download VMware vSphere Replication 8.2 | MAY 9 2019 | Build 13480246 | Download Check for additions and updates to these release notes. |
For information about the vSphere Replication 8.2.x.x patch releases, see the corresponding section.
What's in the Release Notes
These release notes cover the following topics:
- Localization
- What's New
- Product Documentation
- Product Support Notices
- Installation
- Upgrading vSphere Replication
- Operational Limits of vSphere Replication
- Open Source Components
- Caveats and Limitations
- Resolved Issues
- Known Issues
Localization
VMware vSphere Replication 8.2 is available in the following languages:
- English
- French
- German
- Japanese
- Korean
- Spanish
- Simplified Chinese
- Traditional Chinese
What's New
- VMware vSphere Replication 8.2 adds compatibility with VMware vSphere 6.7 Update 2.
- Support for VM encryption: You can replicate encrypted virtual machines if you are using VMware vSphere 6.7 Update 1 or later.
- Minimize security risks by enabling network encryption: You can enable encryption of replication data transfer in VMware vSphere Replication 8.2, if you are using VMware vSphere 6.0 or later.
- Enhancements to the Site Recovery user interface: With vSphere Replication 8.2, you can monitor target datastores in the replication details pane of the Site Recovery user interface, and use vSphere Replication reports that display transferred bytes per virtual machine and per hour.
- vRealize Operations Management Pack: The new vRealize Operations Management Pack for vSphere Replication 8.2 provides increased visibility into the status of your vSphere Replication environment, RPO violations and other details. For more information, see VMware vRealize Operations Management Pack for vSphere Replication 8.2 Release Notes.
- vRealize Orchestrator Plug-In for vSphere Replication 8.2. For information about new workflows, see VMware vRealize Orchestrator Plug-In for vSphere Replication 8.2 Release Notes.
Note: For information about the features of vSphere 6.7 Update 2, see the vSphere 6.7 Update 2 documentation.
For interoperability with the releases of VMware vSphere, see the Compatibility Matrices for vSphere Replication 8.2.x.
Product Documentation
In addition to the current release notes, you can use the documentation set for vSphere Replication 8.2 that includes the following deliverables.
Product Support Notices
vSphere Replication 8.2 is the final release that supports VMware vCloud Availability for vCloud Director. For more information, see VMware vCloud Availability product page.
vSphere Replication 8.2 is the final release that supports external databases.
Installation
Download the vSphere Replication .iso
image and mount it. You can deploy the vSphere Replication appliance by using the Deploy OVF wizard in the vSphere Web Client or the vSphere Client. Navigate to the \bin
directory in the .iso
image and use the corresponding OVF file:
vSphere_Replication_OVF10.ovf
: Use this file to install all vSphere Replication components, including the vSphere Replication Management Server and a vSphere Replication Server.vSphere_Replication_AddOn_OVF10.ovf
: Use this file to install an optional additional vSphere Replication Server.
For more information on installation, see section Installing vSphere Replication in the vSphere Replication Documentation Center.
For vCenter Server to vCenter Server replications, the version of the vSphere Replication Management server on the source and the target site can be either 8.1 or 8.2.
Upgrading vSphere Replication
The downloadable ISO image is the only means of upgrading from vSphere Replication 6.5.x, 8.1.1 and 8.1.2 to vSphere Replication 8.2.
You cannot upgrade vSphere Replication from version 6.5.1 to version 8.2 by using the official VMware Update Repository from the VAMI of the vSphere Replication appliance. See the compatibility matrices for further information on supported versions.
Important: Before you initiate an upgrade, verify that the vSphere Replication appliance has an OVF environment, or context. See Checking and Restoring the OVF Context of the vSphere Replication Appliance (2106709).
Verify that you read the Upgrade and General sections under Known Issues.
Мake sure that you deploy the new vSphere Replication 8.2 OVF first. In the console of the newly deployed vSphere Replication appliance, make sure that you select the Upgrade Option.
See Upgrade Additional vSphere Replication Servers and Upgrade the vSphere Replication Appliance for the procedures on upgrading to vSphere Replication 8.2.
Operational Limits for vSphere Replication
The operational limits of vSphere Replication 8.2 are documented in the VMware Knowledge Base. See Operational Limits for vSphere Replication 6.x and 8.x (KB 2102453).
Note: vSphere Replication requires additional configuration to support more than 500 replications per a vSphere Replication Management server. See Operational Limits for vSphere Replication 6.x and 8.х and Configuring Upgraded vSphere Replication Appliances to Support up to 2000 Replications.
Open Source Components
The copyright statements and licenses applicable to the open source software components distributed in vSphere Replication 8.2 are available at the vSphere Replication Open Source Disclosure page.
Caveats and Limitations of vSphere Replication 8.2
To ensure successful virtual machine replication, you must verify that your virtual infrastructure respects certain limits before you start the replication.
- vSphere Replication 8.2 does not support the virtual NVMe (vNVMe) capability in VMware vSphere.
- You cannot configure the vSphere Replication appliance when the Platform Services Controller is installed with a custom port.
- The 5 minute RPO scales to a maximum supported limit of 50 VMs on a provisional VVOL datastore.
- vSphere Replication does not support VSS quiescing on Virtual Volumes.
- vSphere Replication cannot replicate virtual machines that share vmdk files.
- vSphere Replication does not support vSphere APIs for IO Filtering on both the source and the target sites. You cannot replicate a virtual machine that is assigned a VM Storage Policy that contains IOFilters, nor can you assign such a policy to the replication target VM. Before configuring a virtual machine for replication, verify that the VM Storage Policy that is assigned to it does not contain IOFilters. Do not assign VM Storage policies with IOFilters to virtual machines that are already configured for replication.
- Deploying more than one vSphere Replication appliance produces a warning on the boot screen. This requires user confirmation to either continue and configure all replications again or shut down the new appliance so that it does not interfere with the old one. This situation does not occur when deploying more than one vSphere Replication servers.
- Each vSphere Replication Management Server can manage a maximum of 2000 replicated virtual machines. See Configuring Upgraded vSphere Replication Appliances to Support up to 2000 Replications (KB 2102463) and Requirements to the Environment... (KB 2107869).
- vSphere Replication supports a maximum disk size of 62TB. If you attempt to enable replication on a virtual machine with a disk larger than 62TB, the virtual machine will not perform any replication operation and will not power on.
- vSphere Replication tracks larger blocks on disks over 2TB. Replication performance on a disk over 2TB might be different than replication performance on a disk under 2TB for the same workload depending on how much of the disk goes over the network for a particular set of changed blocks.
- vSphere Replication does not support upgrading the VMware Tools package in the vSphere Replication appliance.
- vSphere Replication supports replicating RDMs in Virtual Compatibility Mode. RDMs in Physical Compatibility Mode cannot be configured for replication.
- vSphere Replication does not replicate virtual machine snapshot hierarchy at the target site.
- You can configure virtual machines that are powered off for replication. However, actual replication traffic begins when the virtual machine is powered on.
- When using Storage DRS at a replication site, ensure that you have homogeneous host and datastore connectivity to prevent Storage DRS from performing resource consuming cross-host moves (changing both the host and the datastore) of replica disks.
- For VMware vCloud Availability for vCloud Director, a seed vApp can be used for only one replication.
- The 5 minute RPO requires the source host to be ESXi 6.0 Update 3 or later for VSAN, and ESXi 6.5 for other supported datastores.
- To use the network isolation feature, vSphere Replication requires the host to be ESXi 6.0 or later.
Supported Browser Versions
For supported browser versions for the vSphere Web Client and vSphere Client, see the documentation of the vSphere Web Client or vSphere Client that you use.
Available Patch Releases
vSphere Replication 8.2.0.2 Express Patch Release
Released 3 OCT 2019 | Build 14761900 |
- The vSphere Replication 8.2.0.2 Express Patch Release provides bug fixes.
Installation Notes
The downloadable ISO image is the only means of upgrading from vSphere Replication 8.2 to vSphere Replication 8.2.0.2 See the Upgrading vSphere Replication in vSphere Replication Administration for instructions about updating vSphere Replication 8.2.x.x.
You upgrade vSphere Replication by using a downloadable ISO image.
vSphere Replication 8.2.0.1 Express Patch Release
Released 13 AUG 2019 | Build 14338525 |
- The vSphere Replication 8.2.0.1 Express Patch Release provides bug fixes.
Installation Notes
The downloadable ISO image is the only means of upgrading from vSphere Replication 8.2 to vSphere Replication 8.2.0.1 See the Upgrading vSphere Replication in vSphere Replication Administration for instructions about updating vSphere Replication 8.2.x.x.
You upgrade vSphere Replication by using a downloadable ISO image.
Resolved Issues
- NEW You cannot break the vSphere Replication site pairing
You cannot force break the vSphere Replication site pairs if the remote site is powered off or unaccessible.
This issue is resolved in vSphere Replication 8.2.0.2.
- NEW Replication fails with an error
When you try to synchronise an ongoing replication or create a new one, the replication fails with an error message:
Failed storing configuration state and error
The replications might fall into an error state if vSphere Replication has a connection issue with the the Customer Experience Improvement Program.
This issue is resolved in vSphere Replication 8.2.0.2.
- NEW The default datastore storage policy is not applied on the target Virtual Volumes datastore or vSAN datastore
When you configure a new replication or reconfigure an existing replication to a vSAN datastore or a Virtual Volumes datastore from the Site Recovery user interface and select Datastore Default storage policy, the policy is not applied on the replication files at the target datastore. The datastore policy is correctly applied if it is explicitly selected from the drop-down menu.
This issue is resolved in vSphere Replication 8.2.0.2.
- Reversing a replication fails with an error
When you try to reverse the direction of a virtual machine's replication, the process fails with the following error:
Unable to reverse replication for the virtual machine 'VM-APP-1'. A runtime error occurred in the vSphere Replication Management Server. Exception details: 'Could not find datastore: prod-hostid01 in datacenter:...
This issue is resolved in vSphere Replication 8.2.0.1
- The vSphere Replication Management Server does not start
When you try to start the vSphere Replication Management Server, it fails with the following error:
"Bad Exit code" - HMS SERVER ERROR
This issue is resolved in vSphere Replication 8.2.0.1.
- Virtual machine replications stop randomly
A content ID mismatch might lock the vSphere Replication server disks causing the virtual machine replications to stop.
This issue is resolved in vSphere Replication 8.2.0.1.
- You cannot see the progress of the initial sync operation when configuring replications within the same vCenter Server
If you configure replications within the same vCenter Server, when you expand the settings of the replication, you see Sync operation in progress: LOGIN REQUIRED instead of the actual progress of the initial sync operation. For replications within the same vCenter Server, login is not required, and this is unexpected behavior.
This issue is resolved in vSphere Replication 8.2.0.1.
- If you reconfigure a replication that you initially configured with default seeds to a vSAN target datastore, replica disks might be deleted and replication status changes to Error
If you configure a replication by using the default, not manually selected, seeds to a vSAN datastore, if you later reconfigure the replication, the replica base disks might be deleted regardless of the settings that you change. Also the replication status changes to Error immediately or within one RPO cycle. The deletion of the replica base disks leads to data loss upon recovery. The issue affects only replications configured to a vSAN datastore with seeds that are automatically suggested by the Site Recovery UI and not when seeds are manually selected by browsing datastores.
This issue is resolved in this release. If you have upgraded from earlier versions of vSphere Replication with replications in Error state because of this problem, the replications are not automatically fixed and must be stopped and configured again.
- An encrypted recovered virtual machine in a vCenter Server 6.7 Update 1 system does not power on
In a vCenter Server 6.7 Update 1 system, you can configure replication of encrypted virtual machines and even run a recovery, but the recovered virtual machines might not power on if the virtual machine configuration file (.vmx) encryption key is not available at the Key Management Server (KMS) in the target site. In rare cases, the encryption keys for the
.vmx
file and for some disks in an encrypted virtual machine might be different, and the vSphere Replication Management server can only validate keys for new replication placeholder disks.This issue is resolved in vCenter Server 6.7 Update 2. For vCenter Server 6.7 Update 1 systems, make sure all the encryption keys are available at the destination site. The issue does not affect environments using the same KMS.
- When you configure an encrypted virtual machine for replication in a vCenter Server 6.7 Update 1 system, the replication might enter an error state
When you configure an encrypted virtual machine for replication in a vCenter Server 6.7 Update 1 system, although the workflow succeeds, the replication might enter an error state, because encryption keys for existing seed disks might not be available at the Key Management Server (KMS) in the target site. When the vSphere Replication server tries to open a seed disk to read or write, the operation fails, because the server is not able to validate the seed disk encryption key.
This issue is resolved in vCenter Server 6.7 Update 2. For vCenter Server 6.7 Update 1 systems, make sure all the encryption keys are available at the destination site. The issue does not affect environments using the same KMS.
- If you deploy a vSphere Replication virtual appliance by using the OVF deployment wizard in the vSphere Client, the appliance fails to start
If you deploy a vSphere Replication virtual appliance by using the OVF deployment wizard in the vSphere Client on a vCenter Server system with a version earlier than 6.7 Update 1, the deployment succeeds, but the OVF virtual appliance fails to start. You might see an error similar to:
The virtual machine 'vSphere_Replication' has a required vService dependency 'vCenter Extension Installation' which is not bound to a provider.
The vSphere Client in versions earlier than vCenter Server 6.7 Update 1 does not support selecting vService extensions in the Deploy OVF Template wizard, which causes the issue.This issue is resolved in the vSphere Client in vCenter Server 6.7 Update 1 and later. If you already face the issue in a vCenter Server system with a version earlier than 6.7 Update 1, use the vSphere Web Client, which supports vService extensions for the deployment of OVF virtual appliances.
Known Issues
The known issues are grouped as follows.
Upgrade- Upgrade from vSphere Replication version 6.5 to vSphere Replication 8.2 fails with an error
When you try to upgrade vSphere Replication version 6.5 to vSphere Replication 8.2, the upgrade fails with the following error:
Failure during upgrade procedure at Upgrade Services phase:
java.io
.Exception: inputstream is closed
The failure might be due to an error in the
sshd_config
file.Workaround:
1. Establish an SSH connection to the vSphere Replication 6.5 Appliance.
2. Navigate to/etc/ssh/
.
3. Opensshd_config
in a text editor, change the value/usr/lib64/ssh/
to/usr/lib/ssh/
and save the file.
4. Restart the SSH service by using theservice sshd restart
command.
5. Retry the upgrade. - You cannot use network encryption for vSphere Replication
After upgrading from vSphere Replication 8.1 to version 8.2, you cannot use the network encryption feature.
Workaround 1:
1. Establish an SSH connection to the vSphere Replication Appliance.
2. Navigate to/opt/vmware/hms/conf/
.
3. Openhms-configuration.xml
in a text editor and set the<hms-auto-install-hbragent-vib>
value to true.
4. Restart the HMS service.Workaround 2:
See vSphere replication 8.2. ESXi unable to download hbr-agent.vib "VibDownloadError" (KB 75321).
- The upgrade of the vSphere Replication Management Server to version 8.2 in an IPv6 environment fails when upgrading VCTA
The user interface of the console upgrade displays the following error:
Failure during upgrade procedure at Upgrade Services phase: No such file
Workaround: Select the Ignore option to continue with the upgrade.
- After upgrading vSphere Replication Management Server to version 8.2 with an automatically generated certificate, a certificate problem occurs when you log in the Site Recovery UI
The Site Recovery UI log file contains the following information:
Caused by: javax.net.ssl.SSLException: Certificate for <HMS.FQDN> doesn't match common name of the certificate subject: <HMS.IP>
. vSphere Replication Management Server is registered with the lookup service by using FQDN, but the SSL certificate uses the IP address, which causes a mismatch.Workaround:
- After the vSphere Replication Management Server upgrade, log in to the VAMI of the vSphere Replication Management Server and generate and install a new SSL certificate.
- Reconfigure the connection between the vSphere Replication servers.
- After a fresh installation or upgrade to vSphere Replication 8.2, a vSphere Replication appliance configured with trust by certificate policy fails to connect to vCloud Availability
After a fresh installation or upgrade to vSphere Replication 8.2, if you use trust by certificate instead of the default trust by thumbprint certificate policy, the vSphere Replication appliance might fail to connect to vCloud Availability. The connection fails, because the vSphere Replication Management Server does not find any certification authority (CA) in
/opt/vmware/hms/security/hms-truststore.jks
.Workaround: You must manually import additional trusted CA certificates in
/opt/vmware/hms/security/hms-truststore.jks
on the vSphere Replication appliance. -
The vSphere Replication Management service does not start after the upgrade
After you upgrade vSphere Replication, the vSphere Replication Management (VRM) service appears as stopped in the VAMI, and the
/opt/vmware/hms/logs/hms-configtool.log file in the virtual appliance contains java.net.ConnectException: Connection refused
error messages.This problem is observed if the upgrade procedure of the embedded DB schema fails because the vPostgreSQL service was not fully started.
Workaround:
- In the virtual appliance console, log in as the root user.
- Run the following command:
$ /opt/vmware/hms/bin/hms-configtool -cmd upgrade -configfile /opt/vmware/hms/conf/hms-configuration.xml
The DB schema upgrade starts.
- Wait for the DB upgrade procedure to complete.
- In the vSphere Replication VAMI, navigate to the Configuration tab, and complete the SSO registration of the appliance.
-
Missing vSphere Replication permissions after upgrading the vSphere Replication appliance, certificate or IP change
If you upgrade the vSphere Replication appliance, or if for some other reason the certificate or the IP address of the vSphere Replication appliance changes, the permissions that are assigned to the default VRM user roles are deleted.
This problem is observed every time the vSphere Replication extension is unregistered and registered with the vCenter Server extension manager.Workaround: Clone the predefined VRM roles and create your custom roles before upgrading the vSphere Replication appliance, or changing its certificate or IP address. The permissions that are assigned to custom roles are not removed.
-
The vSphere Replication Virtual Appliance Management Interface (VAMI) becomes inaccessible after upgrade
After the upgrade, the vSphere Replication VAMI changes and you cannot access it in the same browser window that you used before the upgrade.
Workaround: Do one of the following.
- Change the browser that you use to open the VAMI.
- Close the entire browser and open a new browser window to connect to the VAMI.
- Clear the cache of your browser.
- Open an incognito tab in your browser.
-
The vSphere Replication appliance changes to a vSphere Replication server after the upgrade
If you do not check the OVF context of the vSphere Replication appliance before you perform an upgrade, and the upgrade operation does not fail, the upgraded vSphere Replication appliance appears as a vSphere Replication Server. The data about replications that were configured before the upgrade is lost.
Workaround:
- If you have a pre-upgrade snapshot of the vSphere Replication appliance, revert to that snapshot and see Checking and Restoring the OVF Environment of the vSphere Replication Appliance (2106709).
- If you do not have a pre-upgrade snapshot of the vSphere Replication appliance, uninstall the upgraded vSphere Replication instance and perform a fresh deployment. See Installing and Uninstalling vSphere Replication.
-
After an upgrade of the vCenter Server and vSphere Replication, configuring the SSO in the vSphere Replication VAMI fails with error Bad exit code: 1
After you upgrade the vCenter Server to version 6.5 and vSphere Replication to version 6.5, you must register the appliance with vCenter Single Sign-on. On the Configuration tab of the vSphere Replication VAMI, you enter the LookupService address and the credentials of an SSO administrator, and click Save and Restart Service. The following error message appears:
Bad exit code: 1
.
This problem is observed because the upgraded vCenter Server changes its IP address or certificate, but the vSphere Replication Management server preserves the old IP address and certificate of the vCenter Server in its OVF environment. As a result, the validation of the vCenter Server fails.Workaround: In the vSphere Web Client, right-click the vSphere Replication Management server VM and power it off and on. This operation forces the update of the OVF environment on the vSphere Replication Management server VM.
-
Site Recovery Manager cannot be upgraded after upgrading vSphere Replication
On upgrade of vSphere Replication to version 6.5, Site Recovery Manager cannot be upgraded as the vSphere Replication versions is detected as incompatible. Under solutions manager in vCenter, the vSphere Replication version appears not to have been upgraded though the appliance reports the upgrade is successful.
Workaround: Register the vSphere Replication appliance with vCenter Single Sign-On.
- Connect to the VAMI interface of the vSphere Replication appliance by using a supported browser.
- On the Configuration tab, enter the user name and password of an SSO administrator.
Note: The text boxes for the SSO credentials will not be visible if you are using an unsupported browser.
- Reconfiguring a replication fails after changing the Virtual Device Node on the source VM
If you change the Virtual Device Node settings on a replicated disk and then you attempt to reconfigure the replication, the process fails with the following error:
Unable to complete the reconfiguration task at remote site for replication group '<VM_ID>' (managed object ID: 'GID-<group-ID>'): task 'HTID-<hms-task-ID>'. Details: 'A runtime error occurred in the vSphere Replication Management Server. Exception details: 'VR Server error: 'Error for (diskId: "RDID-<replica-disk-ID>"): SQLite error 19: UNIQUE constraint failed: ReplicatedDisk.diskID; Returned error message: UNIQUE constraint failed: ReplicatedDisk.diskID; Code set to: A disk with the given ID already exists.; Disk ID already in database!; Adding replica disk RDID-<replica-disk-ID> (groupID=GID-<group-ID>) to database; Adding disk RDID-<replica-disk-ID>; Adding disk info to database.'.'.'.
Workaround:
- Reconfigure the replication in the Site Recovery UI and exclude the disk, whose Virtual Device Node settings you changed.
- Reconfigure the replication again and include the previously excluded disk.
- Reconfiguring a replication fails with an error
If you exclude a disk from a replication and then try to include it again, the reconfiguration fails with the following error:
Unable to complete the reconfiguration task at remote site for replication group '<vm_id>' (managed object ID: 'GID-<group-id>'): task 'HTID-<hms-task-id>'. Details: 'Disk file name '<vm_id2>.vmdk' already exists.
Workaround: Wait for a time period, which is greater than the RPO, after you exclude the disk, to be able to include it again. If you have enabled MPIT, you must wait until all replication instances, which contain references to the excluded disk, expire. Alternatively, you can manually delete the vmdk file from the destination datastore (you can see the name of the vmdk file in the error message.)
- You cannot remove or force stop a replication
If you are logged only in the target site and not on the source site, you cannot remove or force stop a replication.
Workaround: Use the vSphere Replication Management Server Managed Object Browser by opening
https://vrms_address:8043/mob/?vmodl=1
.1. For incoming replications, navigate to content > replica-manager > getIncomingReplications. For outgoing replications, navigate to content > replication-manager > getOutgoingReplications.
2. Change the parametres as follows:
start: 0
count: 2000
Clear sorters and filter and leave them blank, then click Invoke Method.
3. Find the replication that you need to remove by looking for the VM name and copy the replication ID (GID-<uuid> value).
4. Click on the replication ID value > destroy > Invoke method.
5. Click on val > info and ensure that the state value is success and the error value is Unset.If the task is still in progress, refresh the info window and wait until it completes.
- Planned migration fails at the synchronization step
When you run a planned migration to Cloud, vSphere Replication runs a virtual machine data synchronization as part of the process. The migration might fail at the synchronization step, if you attempt to remove a snapshot at the same time.
Workaround: Re-run the planned migration process, after the snapshot is removed.
- Replications change state to Not Active if you try to configure a replication to use both the network encryption and network traffic isolation features
If you try to configure a replication to use both the network encryption and network traffic isolation features, the replication changes state to Not Active. For example, if you try to use network traffic isolation on the replication of encrypted virtual machines, where network encryption is not optional.
Workaround: Until a future vSphere Replication release to enable the full use of both features, you can only partially combine network encryption and traffic isolation. For example, if you go to the settings of the VMkernel network adapters on the source host and switch off the vSphere Replication tags, the replication state changes to OK, and only traffic isolation of the outgoing traffic from the source site is disabled.
- If the source VM for a replication runs on ESXi 6.7 or 6.7 Update 1, an initial or full synchronization might stop progress before completion
The synchronization of replications for which the source VM is running on ESXi 6.7 or 6.7 Update 1 remains in progress, but the checksum bytes value in the replication details information does not progress. Operations such as powering off, taking a snapshot, reverting to a snapshot, and migrations fail with a timeout or
Task in progress
errors.Workaround:
1. In the ESXi Advanced settings, disable the checksum for vSphere Replication by setting
HBR.ChecksumUseChecksumInfo = 0
.2. Migrate all VMs and power off the ones that cannot be migrated on the ESXi host.
3. Place the host in maintenance mode.
4. Reboot the ESXi host.
With these steps, you disable the checksum part of the sync process and all of the allocated blocks are sent to the remote site, regardless of whether they are different or not. Also, you cannot use seeds.
- Generation of self-signed certificates of the vSphere Replication Appliance fails if a domain name starts with a number
Generation of self-signed certificates of the vSphere Replication Appliance might fail if a domain name starts with a number, such as 888xxx.
Workaround: To generate a Certificate Authority certificate for fresh deployments, follow the steps in VMware knowledge base article 2080395.
- You cannot recover or reconfigure virtual machines with names containing special characters
You cannot recover or reconfigure virtual machines with names containing the special characters
{
or}
. Although the use of special characters is allowed for file system names that are derived from the virtual machine name, the vSphere Replication Management Server fails to create disk files at the destination folder. If the recovery fails, you cannot rename the virtual machine.Workaround: None
- If the source VM for a replication runs on ESXi 6.7 or 6.7 Update 1, replication synchronization seems to be in progress, but the replication instance never completes successfully
In ESXi 6.7 and 6.7 Update 1, it is possible that more demand log chunks be scheduled for parallel transfer than the actual number that can be transmitted. If you are replicating a VM that is running on such a host and this coincides with a slow target host or temporary network errors, this might result in replication failure with
DiskQueue is full
errors.Workaround:
- Move all the VMs to another ESXi host.
- Edit the value of the
HBR.DemandlogTransferMaxNetwork
ESXi Advanced setting to 63 instead of the default 64. - Place the ESXi host in maintenance mode.
- Reboot the ESXi host.
- The Virtual Appliance fails to start after an OVF deployment from the HTML5-based vSphere Client
The HTML5-based vSphere Client does not support selecting vService extensions in the Deploy OVF Template wizard. As a result, if a virtual appliance uses vService extensions and you use the vSphere Client to deploy it from an OVF file, the deployment succeeds, but the virtual appliance fails to start with an error:
"The virtual machine 'vSphere_Replication' has a required vService dependency 'vCenter Extension Installation' which is not bound to a provider."
Workaround: Use the vSphere Web Client for OVF deployments that use vService extensions.
- When you right-click on a replicated VM and select Reconfigure Replication in the vSphere UI, the pop-up window for the Site Recovey UI is blocked without notification in Mozilla Firefox browser
By default the Site Recovey UI opens in a new tab. When you right-click on a replicated VM and select Reconfigure Replication in the vSphere UI, the pop-up window for the Site Recovey UI is blocked without notification in Mozilla Firefox browser.
Workaround: From the Options menu in Mozilla Firefox, select the Content tab and add the URL of the vCenter Server to the Pop-ups exception list.
- The Site Recovery UI becomes unusable showing a constant stream of 403 - OK error message
The Site Recovery UI shows no data and an error 403 - OK.
Workaround:
1. Log out from Site Recovery UI and log in again.
2. Disable the browser's 'Restore last session' checkbox. For Chrome disable the 'Continue where you left off' option.
- Reconfigure replication fails when a seed is selected for a newly configured VM disk
When a virtual machine disk is enabled for replication during the reconfigure replication operation, and a seed is selected for that disk, the reconfigure replication task fails.
Workaround: When enabling a new disk for replication do not select a seed file. If you must use a seed file, stop the replication for that virtual machine and make a new configuration by enabling all required disks for replication.
- vSphere Replication shows inconsistent information about status and number of replications or Site Recovery Manager test failover fails
When vSphere Replication service reaches a limitation of the threads it can start and cannot create threads to process new events, one of the following can occur:
- vSphere Replication shows mismatching information about the status and number of replications on the source and target sites.
- In Site Recovery Manager, test failover fails with the following record in the system log:
VR synchronization failed for VRM group ifulgv002a. A generic error occurred in the vSphere Replication Management Server. Exception details: 'com.vmware.hms.replication.sync.DeltaAbortedException.
Workaround: Change the vSphere Replication configuration to remove the maximum thread limitation.
- Log into the vSphere Replication appliance as root.
- Open the file
/etc/systemd/system.conf
in an editor. - Uncomment the line
DefaultTasksMax=512
and set its value toDefaultTasksMax=infinity
. - Save the file.
- Reboot the vSphere Replication appliance.
-
Configuring a replication that uses seeds on a VVOL target datastore succeeds, but the replication is in
Error
stateIf you configure a replication to use as a seed a VM that has snapshots, the configure operation succeeds, but the replication goes into the
Error
state at the end of theInitial Full Sync
. An issue with a similar error description appears:
"A replication error occurred at the vSphere Replication Server for replication 'vmname'. Details: 'Error for (datastoreUUID: "vvol:9148a6192d0349de-94149524b5f52bc4"), (diskId: "RDID-fd3ed4de-2356-43c7-a0e2-7bc07a7da012"), (hostId: "host-33"), (pathname: "vmname/vmname.vmdk"), (flags: retriable): Class: NFC Code: 10; NFC error: NFC_DISKLIB_ERROR (Input/output error); Set error flag: retriable; Can't write (multiEx) to remote disk; Can't write (multi) to remote disk'."
Workaround: Delete the snapshots from the seed VM.
-
Some vSphere Replication operations fail or become unresponsive
If you initiate dozens of vSphere Replication operations simultaneously, some of them might hang. This might also cause the VRMS site to become unresponsive as a result of the vSphere Replication appliance requiring more memory to perform many operations in parallel.
Workaround:
- SSH to the vSphere Replication appliance.
- Stop the
vcta
service:
service vcta stop
Note: Stopping this service interrupts the replications to and from the cloud.
-
During full synchronization vSphere Replication fails with error: A replication error occurred at the vSphere Replication Server
During full synchronization vSphere Replication might fail with the following error.
A replication error occurred at the vSphere Replication Server for replication <group_name>. Details: 'Error for (datastoreUUID: "..."), (diskId: "..."), (hostId: "..."), (pathname: "..."), (flags: retriable, pick-new-host, nfc-no-memory): Class: NFC Code: 5; NFC error: NFC_NO_MEMORY; Set error flag: nfc-no-memory; Code set to: Host unable to process request.; Set error flag: retriable; Set error flag: pick-new-host; Can't write (single) to remote disk'.
Usually, this error is transient and the operation succeeds after some time. -
Replacing the SSL certificate of vCenter Server causes certificate validation errors in vSphere Replication
If you replace the SSL certificate on the vCenter Server system, a connection error occurs when vSphere Replication attempts to connect to vCenter Server.
Workaround: For information about how to update vCenter Server certificates and allow solutions such as vSphere Replication to continue to function, see http://kb.vmware.com/kb/2109074.
-
Data synchronization fails and the log file of the source vSphere Replication Management Server contains error
DeltaAbortedException
If your environment experiences connectivity issues during data synchronization, you might observe the following problems.
- Replication group synchronizations fail and the
hms<n>.log
file in the vSphere Replication Management server at the source site contains the following error message:
DeltaAbortedException
. - In Site Recovery Manager, replication group synchronizations fail with the following error message:
VR synchronization failed for VRM group <group_name>. A generic error occurred in the vSphere Replication Management Server. Exception details: 'com.vmware.hms.replication.sync.DeltaAbortedException'
.
Workaround: Resolve the connectivity issues in your environment before you proceed.
- Replication group synchronizations fail and the
-
The initial configuration task fails with error
InvalidArgument
If you configure a replication for a virtual machine that contains disks without UUID, vSphere Replication assigns UUIDs for these disks during the initial configuration. However, if these disks have parent disks, for example preceding snapshots, vSphere Replication cannot assign UUIDs for them and the initial configuration task fails with error
InvalidArgument
.Workaround: Consolidate the disks on the source virtual machine and try configuring a replication again.
-
Failover with "Sync latest changes" might fail with SocketTimeoutException when multiple replications are recovered concurrently and there is a huge accumulated delta since the latest synchronization
The vSphere Replication Management server might not receive due responses through the vCenter reverse proxy when there is heavy replication traffic at the same network. Some replication management or monitoring operations might fail with the following error message:
'com.vmware.vim.vmomi.client.exception.ConnectionException: java.net.SocketTimeoutException: Read timed out'
Workaround: Configure network traffic isolation for vSphere Replication traffic, so that the management communication between vCenter and the vSphere Replication Management server is not affected by the heavy replication traffic. See Isolating the Network Traffic of vSphere Replication.
-
Virtual machines that are located in the target folder are overwritten during recovery
If the target folder contains a registered virtual machine with the same name as the replicated virtual machine, the registered virtual machine is overwritten during the recovery. When you start the Recovery wizard, vSphere Replication checks the target folder and displays a dialog box for you to confirm the overwrite operation. On rare occasions, after the target check is complete, and while the wizard is still open, a virtual machine might be registered to the target folder. On these occasions, the virtual machine that was copied to the target folder will be overwritten without further notice.
Workaround: None.
-
Replications appear in Not Active (RPO violation) status after changing the IP address of the vSphere Replication server at the target site
If the IP address of the vSphere Replication server at the target site changes, the status of all replications to this site turns to Not Active (RPO violation). This problem is observed because replications on the source site are not reconfigured automatically when the IP address changes.
Workaround: Reconfigure all replications, so that the source hosts use the new IP address of the target vSphere Replication server.
-
Transient Error state during the initial full synchronization
During the initial synchronization, you might observe that the state of the synchronization changes temporarily to
Error
and back to normal multiple times. The error state might indicate resource deficiency at the target site. If the IO workload caused by the sync operation is higher than the load that target hosts can handle, the state of the replication will turn toError
. When the IO workload decreases, the error disappears.Workaround: Reduce the value of the host configuration option called
HBR.TransferMaxContExtents
on each ESXi host where replication source VMs are running. The default value is 8, and a lower value decreases the size of data blocks that are sent during one sync update, but increases the duration of the initial full sync. After the initial full sync, change the value back to its default (8
) to achieve maximum RPO performance. If transient errors continue to appear during delta synchronizations, it might mean that a lot of changed blocks are transferred during each delta, and the hosts at the target site cannot accommodate the incurred IO workload. In such cases, keep the value of theHBR.TransferMaxContExtents
configuration option low.
Alternatively, you can add more hosts to the secondary site. -
Users that are assigned the VRM administrator or VRM virtual machine replication role cannot access the Configure Replication wizard
The Configure Replication wizard is not launched if a user that is assigned the predefined VRM administrator or VRM virtual machine replication role logs in the Site Recovery user interface and attempts to configure a replication.
Workaround: Clone the default role to add the Profile-driven storage -> Profile-driven storage view privilege to it, and assign the cloned role to the user.
-
The option to enable quiescing is disabled in Configure Replication wizard for a powered off replication source VM, though the guest OS supports quiescing
For both Linux and Windows sources, the Enable Quiescing option is enabled based on the information about the guest OS. If a virtual machine has never been powered on, ESXi hosts always report no support for quiescing, because the guest OS information is not available.
Workaround: Verify that replication source VMs have been powered on at least once before you configure replications.
-
vSphere Replication service is inaccessible after vCenter Server certificate changes
If the vCenter Server certificate changes, vSphere Replication becomes inaccessible.
Workaround: See vSphere Replication is Inaccessible After Changing vCenter Server Certificate.
-
vSphere Replication Management Server (VRMS) might leak a partially recovered virtual machine in the target vCenter Server after a failed recovery
In rare cases VRMS might stop during recovery immediately after registering the recovered virtual machine in the target vCenter Server. The last recovery error in the replication details panel says
VRM Server was unable to complete the operation.
When VRMS restarts, it cleans up the files for the partially recovered virtual machine. In some cases, it fails to unregister the virtual machine from the target vCenter Server. Subsequent recovery attempts show an error in the recovery wizard that the selected virtual machine folder already contains an entity with the same name.Workaround: Manually remove the virtual machine from the target vCenter Server, but keep its disks as they point to the replica placeholder files.
-
During replication of multiple virtual machines, a vSphere Replication server might enter a state where it does not accept any further VRMS connections but continues to replicate virtual machines
Workaround: Reboot the vSphere Replication server.
-
vSphere Replication operations fail with a Not Authenticated error
If you start an operation on one site, for example configuring vSphere Replication on a virtual machine, and then restart vCenter Server and the vSphere Replication appliance on the other site, vSphere Replication operations can fail with the error
VRM Server generic error. Please check the documentation for any troubleshooting information. The detailed exception is: 'com.vmware.vim.binding.vim.fault.NotAuthenticated'
. This problem is caused by the fact that the vSphere Replication server retains in its cache the connection session from before you restarted vCenter Server and the vSphere Replication appliance.Workaround: Clear the vSphere Replication connection cache by logging out of the vSphere Web Client and logging back in again.
-
Operation in vSphere Replication Management Server fails with error "... UnmarshallException"
When the vSphere Replication Management Server experiences high load or transient network errors, operations can fail with UnmarshallException due to errors in the communication layer.
Workaround: Try the failed operation again.
-
The VAMI might not respond when you install an update
When you upgrade vSphere Replication, a status message 'Installing Updates' might not disappear even after the updates install successfully because the VAMI is not responding.
Workaround: Refresh the VAMI UI in the browser or open it in a new tab.
-
A virtual machine recovered in vSphere Replication does not power on in vCenter Server
When you use vSphere Replication to run a recovery on a virtual machine, it fails, and the status of the replication is not 'Recovered'. The virtual machine is registered in the vCenter inventory, but when you try to power it on, it fails with error:
File [datastorename] path/vmname.vmx was not found.
The virtual machine registration as part of the vSphere Replication recovery workflow can succeed in vCenter Server, but the response might not reach the vSphere Replication Management Server due to a transient network error. vSphere Replication reverts the replication image and reports a failed recovery task due to virtual machine registration error. If you initiate another recovery, it fails with a message that a virtual machine with the same name is already registered in vCenter Server.Workaround: Remove the partially recovered virtual machine from the vCenter Server inventory. Do not delete the files from disk. Try the recovery again.
-
vSphere Replication operations fail when there is heavy replication traffic
vSphere Replication operations might fail with error
java.net.UnknownHostException
. These errors occur because DNS requests are dropped due to network congestion.Workaround: Configure your network to ensure that management traffic is not dropped, by configuring traffic shaping, quality of service, or DNS on the vSphere Replication appliance. One possible solution is to modify the network address caching policy for the vSphere Replication appliance.
- Log into the vSphere Replication appliance as root.
- Open the file
/usr/java/jre1.7.0_72/lib/security/java.security
in an editor. - Uncomment the line
networkaddress.cache.ttl
and set its value to at least 86400 seconds (24 hours) or to the longest time that is required for an initial full sync to complete. - Save the file and reboot the vSphere Replication appliance.
- Repeat the procedure for all remaining vSphere Replication appliances.
- You cannot encrypt an unencrypted source VM in an active replication
If you try to encrypt an unencrypted virtual machine in an active replication configuration, the encryption fails.
Workaround: Recover the unencrypted virtual machine and configure a new replication with encrypted seed disks.
1. Recover the VM on the remote site, but do not power the VM on.
2. Edit the settings of the VM on the target site and change the VM storage policy to VM Encryption Policy.
3. Edit the settings of the source VM on the source site and change the VM storage policy to VM Encryption Policy.
4. Remove the replication of the source VM.
5. Configure a new replication and select the disks of the recovered VM on the target site as seeds. - If you reconfigure a replication to assign a new vSAN storage policy to some virtual machine disks, the policy is not applied to the replicas at the target site
vSAN storage policy is applied to replicas at the recovery site at the time you first configure or recover a replication. If you reconfigure the replication with a new storage policy, the change is not automatically reflected to the pair site.
Workaround:
- Recover the virtual machines with reconfigured replication.
- By using the vSphere Client, change the storage policy of the recovered virtual machines to the new policy.
- Unregister the recovered virtual machines from the vCenter Server inventory.
- Configure replication again by using seeds and with the new storage policy.
- Reconfiguring a replication fails if a Storage DRS cluster is selected as destination for the replication
If you try to reconfigure a replication and move the replication to a datastore part from a Storage DRS cluster, the reconfiguration fails.
Workaround: Remove the replication and configure a new replication to the desired datastore.
-
You cannot use custom defined users and roles with vSphere Replication
You are unable to configure a replication with a custom user, even if that custom user is assigned all required VRM privileges on both sites. The error message
Permission to perform this operation is denied
appears on the Target Location page in Configure Replication wizards.Workaround: None. All vSphere Replication operations must be performed with the SSO administrator user on both sites.
-
A recovered virtual machine with multiple point-in-time instances enabled can lose the attached disks to the latest snapshot when you revert to a previous snapshot and then revert to latest snapshot again
When you recover a virtual machine for which you enabled point-in-time instances and attach a disk for unresolved disks, if any, the disks attach to the latest snapshot. If you revert to a previous snapshot and then revert to the latest one, the attached disks are not available.
Workaround: Edit settings of the virtual machine and add the required disks as existing hard disks.
-
When a target vSphere Replication server is not available, vSphere Replication does not show an error in the vSphere Web Client
If the target vSphere Replication server is not available because it is powered off or has network connectivity issues, and a replication is in an initial full-sync state, vSphere Replication does not report an issue in the Web Client of the target vCenter Server. Instead, you see an event on the vCenter Server and a disconnected status.
Workaround: Check if a target vSphere Replication server is currently available. Alternatively, set an alarm for "VR Server disconnected" event on the target vCenter Server.
-
Cannot reconfigure replication after switching from embedded database to existing external database
If you configure vSphere Replication with an external database and configure replication within the same site, then switch to the embedded database, the replication is not available, which is as designed. If you switch back to the external database, the replication is in an error state. Reconfiguring the replication fails with the following error:
ManagedObjectNotFound
Workaround: When restoring the vSphere Replication database to the previous external or embedded database, you must reset its contents.
-
Cannot configure a virtual machine with physical mode RDM disk even if the disk is excluded from replication
If you configure a replication for a virtual machine with physical mode, you might see the following error:
VRM Server generic error. Check the documentation for any troubleshooting information. The detailed exception is: HMS can not set disk UUID for disks of VM : MoRef: type = VirtualMachine, value =
, serverGuid = null'. Workaround: None.
-
Recovering a virtual machine using the "Recover with latest available data" option is possible when the source virtual machine is powered on
Before you start a recovery operation on the target site, you must power off the replication source virtual machine. However, if you select the option Recover with latest available data when recovering a virtual machine, it is possible to perform the recovery while the source virtual machine is powered on. This causes the following problem.
-
The network cards of the recovered virtual machine are disconnected when it powers on.
Workaround: Ensure that the source virtual machine is powered off before you connect the recovered virtual machine to the network.
If you select Recover with recent changes when you recover a virtual machine, it is not possible to complete the recovery if the source virtual machine is powered on.
-
The network cards of the recovered virtual machine are disconnected when it powers on.
-
Recovering a virtual machine with vSphere Replication 8.2 fails to power on the recovered virtual machine
If a replicated virtual machine is attached to a distributed virtual switch and you attempt to perform a recovery in an automated DRS cluster, the recovery operation succeeds but the resulting virtual machine cannot be powered on.
Workaround: Edit the recovered virtual machine settings to attach it to the correct network.
-
Registering additional vSphere Replication servers takes a long time
If vCenter Server manages several hundred ESXi Server hosts, registering an additional vSphere Replication server with the vSphere Replication appliance can take several minutes.
This is because the vSphere Replication server must register with each ESXi Server host.
-
org.hibernate.exception in the VCTA log file
In the
vcta-info.log.<n>
file or thevcta-debug.log
file, you might observe the following message:
org.hibernate.exception.ConstraintViolationException: Could not execute JDBC batch update
Workaround: You can ignore this message because it does not affect the operation of vCloud Air - Disaster Recovery.
-
The vApp in the cloud organization is not powered off after a recovery on premise
When you recover a replication from cloud at the tenant site and, in the Recovery wizard, you select to recover the VM by using the option Use latest available data, vSphere Replication does not power off the source vApp in the cloud.
This is because the option Use latest available data assumes that there is no connection to the replication source site.Workaround: You can connect to the cloud site to manually power off the source vApp.
-
Replications from cloud turn into Error state
If you use the vCloud Air web user interface to add a new disk to a virtual machine that serves as a replication source, vSphere Replication at your local site automatically pauses the incoming replication for that machine, and moves the replication group into
Error
state.Workaround: Stop the replication from cloud that indicates
Error
state, and configure a new replication. -
Hardware changes on the replication source VM might not be automatically copied to the placeholder vApp in the cloud
Changes to the protected virtual machine on the source site, such as changes to memory, CPU, networks, and so on, might not be replicated to the placeholder vApp in your cloud organization if you apply them while vSphere Replication is running a workflow, for example, a test recovery.
Workaround: Edit the hardware of the replication source VM again to trigger a full synchronization.
- In the vSphere Web Client inventory tree, right-click the source VM.
- From the drop-down menu, select Edit Settings, and apply a change to the virtual hardware.
Note: Opening and closing the Edit Setting dialog box is not enough. You must apply some change to the hardware. - Click OK.
-
Disks are not automatically consolidated during recovery at the cloud site
If you configure a replication to cloud that has the MPIT functionality enabled, and you recover the replicated virtual machine at the cloud site, its retained instances are not consolidated during the recovery. By design, replication instances are not consolidated to speed up the recovery process.
The unconsolidated disks in the recovered virtual machine might cause performance problems as follows.- The recovered virtual machine runs slower than expected.
- The recovered virtual machine requires more storage resources.
Workaround: Use the vCloud Air interface to manually consolidate the disks on the recovered virtual machine.
-
Outgoing replications to cloud remain in Not Active state
By default, when you power on the vSphere Replication appliance, a vSphere Installation Bundle (VIB) is installed on all supported ESXi hosts in the vCenter Server inventory where the appliance is deployed. The VIB creates a firewall rule, Replication-to-Cloud Traffic, that opens TCP ports 10000 to 10010 for outgoing traffic. However, the automatic installation of the VIB file might fail due to network issues in your environment. When the firewall rule is missing on the source ESXi hosts, outgoing replications to cloud remain in
Not Active
state.
Workaround: Install the vSphere Replication VIB file on each ESXi instance that hosts a cloud replication source VM.
- Temporarily disable the firewall on the ESXi host.
- Establish an SSH connection to the ESXi Server.
- Run the following command:
$ esxcli software vib install -v https://VR_APPLIANCE_IP:8043/vib/vr2c-firewall.vib
- Enable the firewall on the ESXi host.
-
Planned migration or synchronization fails with error: A replication error occurred at the vSphere Replication Server
If, during planned migration, the infrastructure (hosts, network, or storage) is under heavy load, running a planned migration might fail with one of the following errors.
A replication error occurred at the vSphere Replication Server for replication <group_name>. Details: 'Error for (datastoreUUID: "..."), (diskId: "..."), (hostId: "..."), (pathname: "..."), (flags: retriable): Class: NFC Code: 10; NFC error: The operation completed successfully; Set error flag: retriable; ...'
A replication error occurred at the vSphere Replication Server for replication <group_name>. Details: 'Error for (datastoreUUID: "..."), (diskId: "..."), (hostId: "..."), (pathname: "..."), (flags: retriable, pick-new-host, nfc-no-memory): Class: NFC Code: 5; NFC error: NFC_NO_MEMORY; Set error flag: nfc-no-memory; Code set to: Host unable to process request.; Set error flag: retriable; Set error flag: pick-new-host; Can't write (single) to remote disk'.
Usually, these errors are transient and the operation succeeds if you retry running it.Workaround: If the error occurs frequently in your environment, you can increase the toleration period for replication synchronizations on the vSphere Replication Management Server (VRMS).
- Log in to the VRMS appliance as the root user and navigate to
/opt/vmware/hms/conf/
. - Open the
hms-configuration.xml
file for editing and set the value of thehms-sync-replication-error-toleration-period
property to300000
. - Try running the planned migration task again.
-
All operations on a seed vApp in vCloud Air are disabled
If you configure a replication to cloud and select a vApp from the vCloud Air inventory to be used as a replication seed, all operations on the seed vApp are disabled.
Workaround: None. Replication seeds cannot operate as virtual machines. A seed vApp can be used for only one replication.