Before setting up OpenShift and NCP, take note of the following information.

  • A pod must have no more than 11 labels and a namespace must have no more than 12 labels.

  • Labels added for OpenShift internal usage, for example, a label with prefix openshift.io in its key, will be disregarded by NCP and thus user won't see the corresponding tags created on the related NSX resources. Here is a list of label prefixes used by OpenShift, and you should avoid using a label key starting with any of the following:

        openshift.io
        pod-template
  • The nodes will need access to the pods, for example, for Kubelet health-checks. Make sure the host management interface is able to access the pod network.

  • Linux capabilities NET_ADMIN and NET_RAW can be exploited by attackers to compromise the pod network. You should disable these two capabilities of untrusted containers. By default, with restricted and anyuid SCC, NET_ADMIN is not granted. Be wary of any SCC that enables NET_ADMIN explicitly, or enables the pod to run in privileged mode. In addition, for untrusted containers, create a separate SCC based on, for example, anyuid SCC, with NET_RAW capability removed. This can be done by adding NET_RAW to `requiredDropCapabilities` list in the SCC definition.

  • Allow root access in PODs/Containers (only for testing). Commands below will require root access in all PODs of the oc project you are currently logged in to.

        oc new-project test-project
        oc project test-project
        oc adm policy add-scc-to-user anyuid -z default
  • Configure (add) the OpenShift Registry.

        oc login -u system:admin -n default
        oc adm registry --service-account=registry --config=/etc/origin/master/admin.kubeconfig
  • Delete the OpenShift Registry

        oc login -u system:admin -n default
        oc delete svc/docker-registry dc/docker-registry
  • There is a missing IPtables firewall rule to allow DNS requests from the Docker default bridge containers to the dnsmasq process on the Node. It needs to be opened manually. Edit /etc/sysconfig/iptables and add the following Rules at the bottom of the file before COMMIT:

        -A OS_FIREWALL_ALLOW -p tcp -m state --state NEW -m tcp --dport 53 -j ACCEPT
        -A OS_FIREWALL_ALLOW -p udp -m state --state NEW -m udp --dport 53 -j ACCEPT
        COMMIT
  • Restart iptables, docker and origin-node (restarts kube-proxy and kubelet).

        systemctl restart iptables
        systemctl restart docker
        systemctl restart origin-node
  • The internal docker registry of OpenShift needs to be allowed to use non-TLS for OpenShift to work. Normally this should be added automatically by the OpenShift Ansible installer, but it seems that this is currently not working. Edit /etc/sysconfig/docker and add:

        INSECURE_REGISTRY='--insecure-registry 172.30.0.0/16'
  • Restart Docker.

    systemctl restart docker
  • NCP's support for network policies is the same as the support provided by Kubernetes and depends on the Kubernetes version used by OpenShift.

    • OpenShift 3.9 - The rule clauses in the network policy may contain at most one selector from namespaceSelector, podSelector and ipBlock.

  • The Kubernetes API server does not perform validation of a network policy specification. It is possible to create a network policy that is invalid. NCP will reject such a network policy. If you update the network policy to make it valid, NCP will still not process the network policy. You must delete the network policy and recreate one with a valid specification.

  • Certain versions of Kubernetes has a subPath-related issue (see https://github.com/kubernetes/kubernetes/issues/61076). If the OpenShift version does not contain a fix for this issue, the creation of the NCP pod will fail with the error CreateContainerConfigError: failed to prepare subPath for volumeMount. You can work around this problem by removing the use of subPath from the NCP yaml. Specifically, remove the line containing subPath: ncp.ini and replace the configuration for volumes with the following:

        volumes:
          - name: config-volume
            # ConfigMap nsx-ncp-config is expected to supply ncp.ini
            configMap:
              name: nsx-ncp-config
              items:
                - key: ncp.ini
                  path: ncp.ini

    A side effect of this change is that the entire /etc/nsx-ujo directory becomes read-only. As a result, connecting with NSX-T using certificate and private key will not work because NCP will not be able to create a temporary file under /etc/nsx-ujo to move both certificate and private key into a single file.

  • If you are running or upgrading to OpenShift 3.10 cluster, note the following:

    • You must specify configuration of Node groups specific to OpenShift 3.10 cluster. Node config map configuration must be provided in inventory hosts file.

    • All hosts defined in the [nodes] group in the inventory hosts file must be assigned to a node group name.

    • Upgrading OpenShift cluster from an Ansible playbook might cause network loss. Make sure to add the patch (https://github.com/openshift/openshift-ansible/pull/8016/files#diff-2386e21861da3f95091dbb27d72ca366) on the openshift-ansible repository to remove the stop/uninstallation of Open vSwitch packages.

  • Starting with OpenShift 3.10, kube-proxy was moved from the openshift-node service to a DaemonSet. It is no longer started by default. Perform the following steps to start kube-proxy manually (assuming that the openshift-ansible repo has been cloned):

    • Go to the openshift-ansible directory, under [defaults] set the following:

          library = roles/lib_utils/library/
    • Create a create_proxy.yaml file in the playbooks directory with the following entries:

          - import_playbook: byo/openshift_facts.yml
          - hosts: masters
            run_once: True
            roles:
              - kube_proxy_and_dns
    • Run the playbook:

          ansible-playbook -i hosts playbooks/create_proxy.yaml

      You will see error messages indicating the failure of some operations. These messages can be ignored. You can verify the result by running the command oc get po --all-namespaces.