Migrating the NSX Management Plane API to NSX Policy API

This topic describes how to migrate VMware Tanzu Kubernetes Grid Integrated Edition (TKGI) from NSX Management Plane API to NSX Policy API.

For an overview of NSX Management Plane API to NSX Policy API Migration, see Migrating the NSX Management Plane API to NSX Policy API - Overview.

Overview

The NSX Management Plane API to NSX Policy API (MP2P) Migration feature switches environments using the NSX Management Plane API TKGI to the NSX Policy API.

VMware recommends that your MP2P Migration adheres to the following Prepare -> Migrate -> Clean Up workflow:

	Procedure	Description	Outcomes
Prepare	Enable Migration	Configure your environment for MP2P Migration in TKGI, NSX, and BOSH.
	Re-Create Top DFW Firewall Rules	Re-create your high-priority user-defined DFW rules.
	Activate Policy API in TKGI	To create and configure NSX Policy API objects: Create and promote a single test cluster. Configure the TKGI tile with the IDs for the created NSX Policy API objects.	Migrates shared resources, including Tier-0, FIP block, Pod IP block, and Node IP block to NSX Policy API. Newly created clusters will use NSX Policy API.

Migrate	Migrate a Cluster	Steps: Back up your cluster’s critical workloads. Promote the cluster. Verify the migration of the cluster.	Use `tkgi promote-cluster-to-policy` to promote the cluster.
	Migrate All Production Clusters	Repeat the Migrate a Cluster procedure above for each production cluster.	Notes: Only promote clusters one at a time. Always verify the migration of your cluster before migrating another cluster.
	Re-Create Bottom DFW Firewall Rules	Steps: Re-create your low-priority user-defined DFW rules. Remove original Management Plane section user-defined DFW rules.

Clean Up	Remove NSX Management Plane API-related configurations	Clean up: Clean up Network Profiles.
	Promote Remaining NSX Objects	Use the NSX Promoter to convert the NSX Objects that are configured outside of TKGI.	Notes: The NSX Promoter updates all objects in the environment to NSX Policy API. Use to promote the Deployment Router, and IP blocks, etc. Use with caution if TKGI shares NSX with other products.
	Migrate BOSH to NSX Policy API mode	Deactivate BOSH migration mode to support NSX Policy API clusters only.
	Verification	Verify the MP2P Migration.

To complete an MP2P Migration in your TKGI environment:

Before migrating your TKGI environment to NSX Policy API, review the warnings and considerations in Migrating the NSX Management Plane API to NSX Policy API - Overview.
Confirm your environment meets the Prerequisites.
Prepare for MP2P Migration.
Migrate TKGI Clusters from the NSX Management Plane API to NSX Policy API.
Post-Migration Cleanup.

Warning: Limit upgrading NSX and TKGI to only resolving critical issues while your environment is in MP2P Migration mixed-mode.

Prerequisites

Before migrating TKGI from NSX Management Plane API to NSX Policy API, verify your TKGI environment meets the following requirements:

TKGI Version Requirements:
- TKGI Tile and CLI: v1.17.0 or later.
- TKGI clusters: v1.17.0 or later.
Environment Version Requirements:
- NSX:
  - NSX v4.0.1.1 or later.
  - NSX environment is a dedicated, single TKGI foundation environment. For example, an environment with one TKGI foundation and without VMware Tanzu Application Service or other installations in production.
- VMware Tanzu Operations Manager:
  - Ops Manager v2.10.59 or later or v3.0.13 or later.
  - Ops Manager CLI on the latest version.
Cluster Requirements:
- TKGI MP2P Migration does not support clusters configured with:
  - NSGroups, including Bootstrap Security Group and BOSH VM Extensions NSGroup configurations.
  - The ncp.nsx_v3.k8s_np_use_ip_sets Network Profile parameter set to false. Before migrating a cluster, you must restore the parameter to true, the default value for NSX Management Plane API. For more information, see Migrate a TKGI Cluster to NSX Policy API below.
  - NSX Policy API Mode resources, such as the default-balanced-client-ssl-profile, default-high-compatibility-client-ssl-profile or default-high-security-client-ssl-profile default ssl profiles. MP2P Migration supports only clusters that reference only NSX Management Plane API resources.
Other Requirements:
- Administrator access to NSX, Ops Manager, BOSH, and TKGI.
- For more information about TKGI MP2P Migration limitations, see TKGI MP2P Migration Configurations and TKGI MP2P Migration Operational Limitations in Migrating the NSX Management Plane API to NSX Policy API - Overview.

Prepare for MP2P Migration

To prepare your TKGI environment for MP2P Migration:

Enable Migration
Migrate DFW Top Firewall Rules
Activate NSX Policy API in TKGI
Configure the Environment for NSX Policy API

Enable Migration

You must enable support for MP2P Migration in NSX before promoting clusters to NSX Policy API. Additionally, Ops Manager and BOSH must be configured to support a mixed environment of NSX Management Plane API and NSX Policy API clusters before promoting clusters.

Note: After activating NSX Policy API, existing NSX backups created while using NSX Management Plane API cannot be used to restore your environment or your clusters.

To prepare TKGI for MP2P Migration:

Disable the TKGI Upgrade All Clusters errand.

To prepare NSX for MP2P Migration:

To activate the NSX Migration Coordinator Service on all NSX managers:
1. SSH to the NSX Manager with administrative privileges:
2. At the nsxmanager> prompt, run the following:
```
nsxmanager> start service migration-coordinator
```
If your NSX Manager cluster is configured with VIP, configure the Source IP Persistence Profile for LB or use a Source IP LB algorithm during TKGI foundation migration. Make all migration requests on a single NSX Manager.

To prepare Ops Manager and BOSH for MP2P Migration:

If you are using Ops Manager v3.0.0 or later:
1. Open your Ops Manager BOSH Director for vSphere tile to the vCenter Config pane.
2. Activate Use NSX-T Policy API Migration Mode.
  
  Note: Do not activate Use NSX-T Policy API at this time.
3. Click Save.
4. On the Ops Manager Installation Dashboard, select Review Pending Changes, review the changes, and select Apply Changes.
If you are using Ops Manager v2.10.45 or later:
1. Download bosh_migration_mode.sh, the BOSH Migration Mode script, from the VMware Tanzu Kubernetes Grid Integrated Edition documentation GitHub repository.
2. Select a virtual machine that is able to reach Ops Manager, and can run the Ops Manager CLI, BOSH CLI, and yq CLI.
3. Copy the the BOSH Migration Mode script to the virtual machine.
4. Export the BOSH_CLIENT, BOSH_CLIENT_SECRET, BOSH_ENVIRONMENT, and BOSH_CA_CERT environment variables. For more information, see Set the BOSH Environment Variables on the Ops Manager VM in Advanced Troubleshooting with the BOSH CLI.
  
  For example:
```
export BOSH_CLIENT=ops_manager  BOSH_CLIENT_SECRET=FHoiRmt3qq1LfbPncF4vAyxZWUSpqbZ-  BOSH_ENVIRONMENT=88.0.0.3 bosh  BOSH_CA_CERT=/var/tempest/workspaces/default/root_ca_certificate  
```
5. To activate BOSH Migration Mode run the BOSH Migration Mode script:
```
./bosh_migration_mode enable OPSMAN-IP USERNAME PASSWORD 
```
  Where:
  - OPSMAN-IP is the IP address for the Ops Manager.
  - USERNAME is the account to use to run Ops Manager API commands.
  - PASSWORD is the password for the account.

Migrate DFW Top Firewall Rules

If your clusters are customized with DFW rules or use Network Profiles configured with DFW section markers, migrate the DFW rules:

Re-create your existing top section DFW rules from above your NSX Manager top_firewall_section_marker in the NSX Policy Environment section. Include the top_firewall_section_marker in the new copy.

Warning: If you do not configure your DFW Rules correctly, your workloads will lose network connectivity. Both the original copy of the top_firewall_section_marker section and the re-created copy in the NSX Policy Environment section must be in your DFW rules after you complete this step.

For more information on configuring DFW rules, see DFW Migration in Migrating the NSX Management Plane API to NSX Policy API - Overview.

Activate NSX Policy API in TKGI

To migrate NSX Management Plane API shared network resources to NSX Policy API objects:

Create a simple test cluster.
Migrate the cluster to the NSX Policy API:
```
pks promote-cluster-to-policy CLUSTER-NAME
```
Where CLUSTER-NAME is the name of the promoted cluster.
Verify successful cluster migration to the NSX Policy API:
```
pks cluster CLUSTER-NAME
```
Where CLUSTER-NAME is the name of the promoted cluster.

To collect the newly created Resource Policy IDs:

Retrieve the existing NSX Management Plane API IDs:
1. Open the TKGI tile UI.
2. Open the Networking tab.
3. Retain the following NSX Management Plane API IDs:
  - Pods IP Block ID
  - Nodes IP Block ID
  - T0 Router ID
  - Floating IP Pool ID
  For more information on Networking tab settings, see Networking in Installing TKGI on vSphere with VMware NSX.
Retrieve the new Resource Policy IDs:
1. Log in to the NSX Web UI using an account with with admin privileges.
2. Go to the homepage.
3. Enter one of the NSX Management Plane API IDs retrieved in the last step into the search bar and search.
4. Retrieve the corresponding Resource Policy ID From the ID field. Confirm that the indicated Creation Time matches the time when you promoted your test cluster.
5. Retrieve and retain the Resource Policy ID for each NSX Management Plane API ID you collected in the last step.

Configure the Environment for NSX Policy API

To reconfigure TKGI with Resource Policy IDs:

Open the TKGI tile UI.
Open the Networking tab.
Activate Policy API mode.
Replace the Management Plane API IDs with the retained Resource Policy IDs:
- Pods IP Block ID
- Nodes IP Block ID
- T0 Router ID
- Floating IP Pool ID
For more information on Networking tab settings, see Networking in Installing TKGI on vSphere with VMware NSX.
Select Apply Changes.

Note: After configuring the TKGI tile, newly created clusters use the NSX Policy API.

Migrate TKGI Clusters from the NSX Management Plane API to NSX Policy API

To migrate TKGI to NSX Policy API, serially promote all of your TKGI clusters individually:

Migrate a TKGI Cluster to NSX Policy API

After migrating all clusters:

Migrate DFW Bottom Firewall Rules

Note: Do not intentionally run TKGI in mixed mode for an extended period of time. Promote all TKGI clusters to NSX Policy API as quickly as possible.

Migrate a TKGI Cluster to NSX Policy API

To migrate an individual cluster from the NSX Management Plane API to NSX Policy API:

If your cluster is custom configured with the ncp.nsx_v3.k8s_np_use_ip_sets Network Profile parameter set to false, restore the default "ncp.nsx_v3.k8s_np_use_ip_sets": true configuration by updating the cluster with a modified Network Profile.
Promote the cluster using the TKGI CLI:
```
tkgi promote-cluster-to-policy CLUSTER-NAME
```
Where CLUSTER-NAME is the name of the promoted cluster.

Note: Do not attempt to promote an additional TKGI cluster to NSX Policy API before completing the promotion of the current clusters.
Validate successful migration to NSX Policy API topology for the cluster before promoting subsequent clusters:
1. Run the following:
```
pks cluster CLUSTER-NAME
```
  Where CLUSTER-NAME is the name of the promoted cluster.
2. Confirm successful cluster promotion by reviewing the migration log on the Kubernetes control plane VM.
  
  NCP returns one of three exit codes after promoting a cluster: 0, 1, or 2:
  - 0: The cluster has successfully promoted to NSX Policy API.
  - 1: Cluster promotion has failed, and its NSX resources are a mixture of NSX Management Plane API and NSX Policy API resources. Restart cluster promotion for this cluster from the same master VM.
    
    Note: If you restart cluster promotion from a different VM, mp_to_policy_importer will attempt to recover the cluster by rolling back the successfully promoted NSX resources. Do not attempt to promote a different cluster until after this cluster has successfully promoted.
  - 2: Cluster promotion has failed, and the cluster’s NSX resources remain Management Plane API resources. Do not restart cluster promotion for this cluster. Review the mp_to_policy_importer logs and confirm NCP can be manually restarted on all master VMs in Management Plane API mode.

For more information on cluster limitations while promoting a cluster, see During Cluster Promotion to NSX Policy API in Migrating the NSX Management Plane API to NSX Policy API - Overview.

Note: Do not attempt to promote an additional TKGI cluster to NSX Policy API before completing the promotion of the current clusters.

Migrate DFW Bottom Firewall Rules

If your clusters are customized with DFW rules or use Network Profiles configured with DFW section markers, finish migrating the DFW rules:

Re-create your existing bottom section DFW rules from below your NSX Manager bottom_firewall_section_marker in the NSX Policy Application section BELOW all migrated NCP rules. Include the bottom_firewall_section_marker.
Remove all of the original NSX Management Plane customer-defined DFW rules. When done, confirm there is only one copy of the top firewall rules, the one in the NSX Policy Environment section, and only one copy of the bottom firewall rules, the one in the NSX Policy Application section.

Warning: If you do not configure your DFW Rules correctly, your workloads will lose network connectivity. You must remove all NSX Management Plane user-defined DFW rules before starting post-migration cleanup.

For more information on configuring DFW rules, see DFW Migration in Migrating the NSX Management Plane API to NSX Policy API - Overview.

Post-Migration Cleanup

After all TKGI clusters have been promoted to NSX Policy API, remaining NSX resources must be promoted and the TKGI configurations for NSX Management Plane API objects removed.

To clean up after promoting all clusters:

Remove NSX Management Plane API-related configurations:
1. Remove the original NSX Management Plane API-configured Network Profiles.
Use the NSX Promoter to promote the remaining NSX resources:
1. Log in to the NSX Web UI using an account with with admin privileges.
2. Go to the NSX > System > General Settings.
3. Select Start Objects Promotion.
  
  The following are reconfigured by the NSX Promoter:
  - NSX resources in TKGI tile Resource Config.
  - NSX resources in cluster vm_extension configurations.
  - Custom infra-level resources out of TKGI scope. For example, NAT, IP allocations, subnet allocations, and the load balancers that you have created.
  Note: When using NCP v4.1.1, the deployment network and router are automatically reconfigured during MP2P Migration and are not migrated by the NSX Promoter.
For information about the NSX Promoter, see Promote Manager Objects to Policy Objects.
(Optional) To deactivate the NSX Migration Coordinator Service on all NSX managers:
1. SSH to the NSX Manager with administrative privileges.
2. At the nsxmanager> prompt, run the following:
```
nsxmanager> stop service migration-coordinator
```
If your NSX Manager cluster was configured with VIP before you started MP2P Migration, restore your VIP configuration.
To switch BOSH to Policy API mode:
- If you are using Ops Manager v3.0.0 or later:
  1. Open your Ops Manager BOSH Director for vSphere tile to the vCenter Config pane.
  2. Deactivate Use NSX-T Policy API Migration Mode.
  3. Activate Use NSX-T Policy API.
  4. Click Save.
  5. On the Ops Manager Installation Dashboard, select Review Pending Changes.
  6. Ensure that BOSH Director is the only product selected.
  7. Select Apply Changes.
- If you are using Ops Manager v2.10.45 or later:
  1. Access the virtual machine where you ran the BOSH Migration Mode script when you prepared Ops Manager and BOSH for MP2P Migration.
  2. Export the BOSH_CLIENT, BOSH_CLIENT_SECRET, BOSH_ENVIRONMENT, and BOSH_CA_CERT environment variables. For more information, see Set the BOSH Environment Variables on the Ops Manager VM in Advanced Troubleshooting with the BOSH CLI.
    
    For example:
```
export BOSH_CLIENT=ops_manager  BOSH_CLIENT_SECRET=FHoiRmt3qq1LfbPncF4vAyxZWUSpqbZ-  BOSH_ENVIRONMENT=88.0.0.3 bosh  BOSH_CA_CERT=/var/tempest/workspaces/default/root_ca_certificate  
```
  3. Run the following BOSH Migration Mode script command:
```
./bosh_migration_mode disable OPSMAN-IP USERNAME PASSWORD
```
    Where:
    - OPSMAN-IP is the IP address for the Ops Manager.
    - USERNAME is the account to use to run Ops Manager API commands.
    - PASSWORD is the password for the account.