Importing a Kubernetes cluster can take some time and requires locking down the cluster (no create, update, or delete operations allowed).

NSX can be shared by multiple Kubernetes clusters or products, such as TKGi, TAS, and Vanilla Kubernetes clusters. Only one cluster can be imported to Policy at a time among these clusters. To migrate all these clusters to Policy, there are two strategies:
  1. (Recommended) Schedule a control plane downtime on all clusters simultaneously. The downtime will last until all clusters are imported to Policy.
  2. Schedule a control plane downtime on one cluster at a time. After a cluster is imported, NCP is started in Policy mode on that cluster.

Strategy 1 requires control plane downtime for all clusters simultaneously. The benefit is that it allows you to use the NSX Backup and Restore feature. This strategy is recommended if the Kubernetes cluster must be restored to its previous state in the unlikely event that both migration and rollback fail.

Strategy 2 requires control plane downtime only to the cluster being migrated. However, NSX Backup and Restore cannot be used with this strategy because other clusters may create new workloads on NSX while the current cluster is being migrated. This mode is recommended if it is acceptable to discard the cluster if both migration and rollback fail.

The Python3 script that performs the import, mp_to_policy_importer.py, is located in the directory scripts/mp_to_policy_import. It is also referred as mp_to_policy_importer in this document. mp_to_policy_importer must be run on one Kubernetes master node at a time. If it needs to be run again, it must be done on the same master node. mp_to_policy_importer can terminate with the following exit codes:
  • 0 - Successful migration to Policy
  • 1 - Unsuccessful migration to Policy. NCP should not be run in MP mode
  • 2 - Unsuccessful migration to Policy. NCP should be restarted in MP mode

For limitations and workarounds, see Limitations and Workarounds. For information on how to handle errors, see Failure and Recovery.

Requirements for running mp_to_policy_importer:
  • The NSX version is 3.2.0 or later.
  • The NCP version is 4.0 or later.
  • Python3 is installed on the master VM.
  • pyyaml is installed on the master VM.
  • The script must have execution access to "kubectl".
  • NCP must not be running. This will result in control-plane downtime. You must stop NCP manually. The script will not do it.
  • All manually-created DFW sections that are present between any NCP-created sections must be moved outside the range of NCP-created DFW sections. See Dealing with DFW Sections Created by NSX Admin.
  • All user-created rules inside NCP-created sections must be moved out of the NCP-created sections. See Dealing with DFW Sections Created by NSX Admin.
  • NCP-created LoadBalancer Virtual Servers in Manager mode must not have more than 255 rules. This means that Kubernetes must not have more than 255 Ingress rules on the default LoadBalancer. If there are more than 255 Ingress rules, you must split Ingress into multiple LoadBalancer CRDs.
After the requirements are met, the cluster can be imported to Policy. The steps are:
  1. In the NCP config file ncp.ini, make sure that policy_nsxapi is set to false.
  2. Start the migration coordinator on any of the NSX Managers with the following command:

    /etc/init.d/migration-coordinator start

    Verify that it is successfully running with the following command:

    /etc/init.d/migration-coordinator status

    Be sure to use this Manager's IP address when importing Manager objects to Policy.

  3. Refactor manually-created DFW sections, if needed. See Dealing with DFW Sections Created by NSX Admin.
  4. Do the following for each cluster to be imported:

    Stop NCP. The Kubernetes API server should be in read-only mode. Do not perform any create, update, or delete operations on Kubernetes sources. Wait at least 10 minutes before proceeding to the next step.

  5. Create an NSX backup.
  6. Do the following for each cluster to be imported:
    1. Import shared resources. See Importing Shared Resources. Verify that mp_to_policy_importer terminated with exit code 0. If not, see Failure and Recovery.
    2. Import the Kubernetes cluster related resources. See Importing a Kubernetes Cluster. Verify that mp_to_policy_importer terminated with exit code 0. If not, see Failure and Recovery.
    3. Update ncp.ini if required for this cluster. See Importing Shared Resources.
    4. In the NCP YAML file, make sure that policy_nsxapi is set to true.
  7. Do the following for each cluster to be imported:

    Start NCP. You can now perform create, update, or delete operations on Kubernetes resources.

  8. Switch TKGI automation and BOSH CPI to use the Policy API once all clusters in the deployment are imported.