Prerequisites

  • NSX version is 4.0.1 or later.
  • VMware NSX-T Tile version is 4.1.2.x or later.
  • NSX network is used by only TAS Foundations.
  • All foundations deployed on the same NSX manager cluster must be migrated in the same maintenance window.
  • NCP must be paused in all the foundations (running in Manager mode or Policy mode) deployed on the same NSX Manager cluster during the maintenance window. This implies that all create, update, and delete TAS operations such as creating task or apps to drain event will not work during this time

Migration Steps in Brief

NSX-T tile provides a post-deployment errand named "migrate-mp2p" to migrate TAS foundations to NSX Policy. The errand runs on a dedicated VM in the deployment created for the VMware NSX-t tile. Follow these steps to migrate your TAS foundations to NSX Policy mode:
  1. Refactor manually-created DFW sections, if needed. See Dealing with DFW Sections Created by NSX Admin.
  2. Do the following for all foundations (running in Manager or Policy mode) that use the same NSX Managers:
    1. Pause NCP in the foundation. Refer section "How to Pause NCP in foundation" for detailed steps. This should be done in parallel in all foundations
  3. Create an NSX manager backup.
  4. Do the following for each foundation (running in Manager mode only) identified in step 2, one foundation at time:
    1. Migrate the foundation using migrate-mp2p errand. Refer to section "How to migrate TAS foundation to NSX Policy" for more details.
  5. Do the following for each foundation (running in Manager or Policy mode) identified in step 2:
    1. If there are Gateway Firewall rules on the top Tier-0 that can block traffic from containers, setting the NAT Firewall Match for SNAT Rules property to BYPASS will avoid firewall enforcement on NCP-created SNAT rules. Otherwise, gateway firewall rules will need to be reviewed to ensure that traffic from containers is not accidentally blocked.
    2. Start NCP in Policy mode in the foundation. Refer to section "How to start TAS foundations in NSX Policy mode" for detailed steps. This should be done in parallel in all foundations.

How to Pause NCP in Foundation

NCP process runs on diego_database VMs in "VMware Tanzu Application Service" deployment. As it modifies NSX state, it must be paused during the migration process. For the same reason NCP must be paused before taking a NSX backup. Follow these steps to pause NCP in the foundation. These should be done in parallel in all TAS foundations that share the same NSX managers.
  1. Click the VMware NSX-T tile.Screen showing NSX-T tile
  2. Click the MP to Policy Migration configuration pane. Select the Pause NCP in the foundation radio button. Click Save.Screen showing the MP to Policy migration configurationScreen showing the MP to Policy migration configuration
  3. Click Apply Changes.Screen showing the pending changes

How to migrate TAS foundation to NSX Policy

NSX resource types for migration

Migration of a TAS foundation involves migration of two kinds of NSX resources which primarily differ on the basis of who created them. They are defined as follow:
  • Shared NSX resources: These NSX resources are explicitly created by the user either manually or using some automation like terraform. They can be shared by multiple TAS foundations or with other products like TKGi. Shared resources encompass the NSX resources created using https://docs.vmware.com/en/VMware-Tanzu-Application-Service/2.11/tas-for-vms/vsphere-nsx-t.html#nsx-t-setup, specified as annotations on the TAS objects (for example SNAT IP Pools for apps or orgs), or any explicitly created NSX resource like DFW Sections and rules that are used in the foundation. These resources are required to be migrated so that NCP can operate in Policy mode, but must be provided by the user as they are unknown to NCP. Hence, these NSX resources need to be specified in the MP to Policy Migration config pane.
  • NCP-created NSX resources: These NSX resources are created by NCP in response to TAS workload. These resources are automatically inferred during migration of a foundation so they do not need to be specified by the user.

Modes of operation of the migrate-mp2p errand

The migrate-mp2p errand can be run in two modes:
  1. To migrate both shared and NCP-created NSX resources. This is the default choice. During the migration process, shared NSX resources are migrated before the migration of any NCP-created NSX resource.
  2. To migrate only Shared NSX resources. This is helpful in following cases:
    1. some NSX resources need to be migrated after migration of NCP-created NSX resources, or
    2. the user simply forgot to provide specify some Shared NSX resources in the first attempt, or
    3. the user simply wants to become familiarized with the migration process.

      Step 3 under "Migration Steps" below show how to run the errand in this mode.

Migration Steps

The following steps summarize the actions associated with migration of foundation to Policy. Step 4 must be done in one foundation at a time.
  1. Find and fill the ID or display name of all the shared NSX resources that need to be migrated to Policy in the MP to Policy Migration configuration pane. An example is shown below.Screen showing the MP to Policy migration configuration
  2. Select the Perform Migration to NSX Policy radio button. Click Save.Screen showing the Perform Migration to NSX Policy button
  3. (Optional) If you only want to migrate the Shared NSX resources, then enable the checkbox Migrate only manually created NSX resources. If this is enabled, NCP-created NSX resources for this foundation are not migrated.
  4. Click Installation Dashboard and then Review Pending Changes. Enable the errand Perform MP2P Migration. Click Apply Changes.Screen showing the pending changes
  5. If the migration fails and the errand logs state that it needs to be retried, please repeat the process from step 1. If NSX Backup needs to be used, go to section "How to use NSX Backup to restore TAS Foundations". Otherwise, migration is successful. Migration state can be inferred with tags on the foundation Tier-0. Refer to section "How to check migration status of a foundation" for more details.

How to check migration status of a foundation

The migration status of a foundation can be read from tag with scope "ncp/mp2p_status_<foundation name>" on the foundation Tier0 in NSX, if the Tier-0 has capacity to add this tag. This tag is always added on the NSX Policy Domain created for the foundation. The ID of the domain is same as the foundation name and this domain is not visible on the NSX UI. There is no impact on the data plane in any of the statuses. The status can be one of the following:
  1. "in_progress": Indicates that the migration is in progress in this foundation.
  2. "success": Indicates that foundation has been migrated to Policy and NCP can be started in Policy mode
  3. "rollback_succeeded": Indicates that migration to Policy failed but all NSX resources that were migrated as part of this foundation were rollbacked, that is, restored to their original state in Manager mode before migration. The recommended next steps are:
    1. run the migrate-mp2p errand again. It may succeed if it failed due to a temporary reason. We recommend to retry upon failure at least 3 times with a time gap of 15 minutes between each attempt before moving to step 3b.
    2. restart NCP in Manager mode in this foundation and keep it running at least 30 minutes to allow NCP to reconcile TAS state with the NSX backend before resuming TAS operations. Not doing so might cause some temporary failures in TAS operations. Reattempt to migrate this foundation when possible. We recommend to do this at least 3 times before moving to step 3c.
    3. Restore the TAS foundations and NSX managers using the Backup that was created before any foundation was migrated. Refer to section "How to use NSX Backup to restore TAS Foundations" for more details. Contact VMware support.
  4. "rollback_failed": Indicates that migration to Policy failed and all NSX resources that were migrated as part of this foundation could not be rollbacked. If NCP is run in Manager or Policy mode in this case, it will fail to start. This condition is likely due to an unexpected failure in NSX manager. The recommended next steps are
    1. run the migrate-mp2p errand again. It will attempt to rollback the NSX resources again. This can be done multiple times. It should be done until rollback can be successfully completed which is indicated by the errand logs as well as the tags on the foundation Tier0. We recommend to retry upon failure at least 5 times with a time gap of 15 minutes between each attempt before moving to step 4b.
    2. Restore the TAS foundations and NSX managers using the Backup that was created before any foundation was migrated. Refer to section "How to use NSX Backup to restore TAS Foundations" for more details. Contact VMware support.
  5. "commit_failed": Indicates that migration to Policy failed because after a successful migration of NSX resources to Policy mode, a failure occurred while updating these resources for NCP consumption (see the section "Phase 4" in Migrating NCP Clusters and TAS Foundations to Policy). It is recommended to re-run the migrate-mp2p errand in this case. NCP should not be run in Manager or Policy mode in this state. This can be done multiple times. It should be done until migration can be successfully completed which is indicated by the errand logs as well as the tags on the foundation Tier0. We recommend to retry upon failure at least 5 times with a time gap of 15 minutes between each attempt and contact VMware support if the issue persists.
  6. "payload_creation_failed": Indicates that migration errand could not create the payload required to migrate the NSX resources. It is recommended to re-run the errand in this case. If the issue persists,
    1. Restart NCP in Manager mode in this foundation, keep it running at least 30 minutes to allow NCP to reconcile TAS state with the NSX backend before resuming TAS operations, and then reattempt to migrate this foundation when possible. We recommend to do this at least 3 times before moving to step 6b.
    2. Restore the TAS foundations and NSX managers using the Backup that was created before any foundation was migrated. Refer to section "How to use NSX Backup to restore TAS Foundations" for more details. Contact VMware support.

How to start TAS foundations in NSX Policy mode

Follow these steps to run a TAS foundation in Policy mode on the Opsmanager UI
  1. Go to the NCP configuration pane in the VMware NSX-T tile. Select the checkbox Enable NSX-T Policy API. Click Save.Screen showing the NCP configuration
  2. Go to the MP to Policy Migration configuration pane in the VMware NSX-T tile. Select the radio button Run NCP. Click Save.Screen showing the MP to Policy migration configuration
  3. Click Installation Dashboard and then Review Pending Changes. Click Apply Changes.

How to use NSX Backup to restore TAS Foundations

The sections below provide the steps to restore TAS foundations to Manager or Policy mode after restoring all the NSX Manager nodes using their NSX Backup.

Steps to restore TAS foundations to Manager mode

These steps should be taken to restore the foundations that were migrated to Policy successfully or unsuccessfully and that share the same NSX managers:
  1. On the Ops Manager UI of the foundation,
    1. Go to the NCP configuration pane in the VMware NSX-T tile. Deselect the checkbox Enable NSX-T Policy API. Click Save.
    2. Go to the MP to Policy Migration configuration pane in the VMware NSX-T tile. Select the radio button Run NCP. Click Save.
  2. Click Installation Dashboard and then Review Pending Changes. Click Apply Changes. After the changes are applied successfully, the foundation will be working again in Manager mode.

Steps to restore TAS foundations to Policy mode

These steps should be taken to restore the foundations that were already Policy owned (i.e. in which no migration was done) and that share the same NSX managers:
  1. On the Ops Manager UI of the foundation,
    1. Go to the MP to Policy Migration configuration pane in the VMware NSX-T tile. Select the radio button Run NCP. Click Save.
  2. Click Installation Dashboard and then Review Pending Changes. Click Apply Changes. After the changes are applied successfully, the foundation will be working again in Policy mode.

Troubleshooting

Migration Logs

When an NSX resource is migrated to Policy, mp2p script caches its NSX MP ID locally (i.e. on the VM). All such IDs together constitute what is referred to as migration records. See Migrating NCP Clusters and TAS Foundations to Policy for more details.

We store the migration logs and records to a persistent storage at path /var/vcap/store/migrate-mp2p which is of size 2048MB by default. We create a new directory, called target directory, in this path with name like "mp2p-records-<time in epoch>". This directory contains all the logs and migration records. This directory is accessible on the VM "migrate-mp2p" deployed in the "VMware NSX-T" deployment. The migration logs of last execution of the errand are also present at path /var/vcap/sys/log/migrate-mp2p/ on this VM.
  • If migration is successful, then the migration records are empty.
  • If migration fails, the errand will attempt an automatic rollback using the migration records. If the rollback fails, it stores the migration records in the target directory. When the errand is run again, it will load the migration records from the last run and automatically attempt to either rollback the NSX resources or complete the migration.

Things to Note

The following facts should be kept in mind when migrating to Policy mode:
  1. The Policy ID of all Shared NSX resources in TAS is the same as their MP ID.
  2. Once a foundation is migrated to Policy mode, NCP must be started in Policy mode in it. It cannot be used in Manager mode unless NSX Backup is used to restore the migrated NSX resources to a state recorded before the migration was performed.
  3. Once a foundation is migrated to Policy mode and NCP has been started in Policy mode, NSX Backup may no longer be used to restore the foundation to Manager mode.
  4. Migration logs are available only up to 5 days after NCP is started in Policy mode. It is highly recommended to store these logs in a long term persistent storage in case an issue happens at a later point in time.
  5. It may be that the "ncp/mp2p_status_<foundation name>" tag cannot be added on the foundation Tier-0 Gateway after migration of a foundation because the Tier-0 does not have spare capacity. In such cases, an existing tag from the previous migration will be replaced with the new tag, if possible.
  6. If NCP is run in Manager mode after migration, it will be unable to change the state of any NSX resource, as they are now owned by NSX policy.
  7. Once all the foundations are migrated to Policy and only if there is no other product running on NSX, it is recommended to migrate all the remaining NSX resources in Manager mode using the NSX UI by navigating to System > Migrate on any of the NSX Managers
  8. If DFW rules have source and destination as ANY then such DFW sections should be migrated to the "default" NSX Policy domain.