You upgrade the vSphere Replication appliance and any additional vSphere Replication servers by using a downloaded ISO image and the VRMS Appliance Management Interface.
When you upgrade to vSphere Replication 9.0, vSphere Replication with enhanced replication is automatically activated. Any replications that you configured before the upgrade are still active and you can reconfigure them to use vSphere Replication with enhanced replication capabilities. See How do I reconfigure existing replications to use vSphere Replication with enhanced replication capabilities.
vSphere Replication Upgrade Scenarios
You use the ISO file and the VRMS Appliance Management Interface to upgrade from vSphere Replication 8.7.x or 8.8.x to vSphere Replication 9.0.
For the full list of supported upgrade paths, see the Compatibility Matrices for vSphere Replication 9.0.x at https://docs.vmware.com/en/vSphere-Replication/9.0/rn/compatibility-matrices-for-vsphere-replication-90/index.html.