Before setting up OpenShift and NCP, take note of the following information.
A pod must have no more than 11 labels and a namespace must have no more than 12 labels.
Labels added for OpenShift internal usage, for example, a label with prefix openshift.io in its key, will be disregarded by NCP and thus user won't see the corresponding tags created on the related NSX resources. Here is a list of label prefixes used by OpenShift, and you should avoid using a label key starting with any of the following:
The nodes will need access to the pods, for example, for Kubelet health-checks. Make sure the host management interface is able to access the pod network.
Linux capabilities NET_ADMIN and NET_RAW can be exploited by attackers to compromise the pod network. You should disable these two capabilities of untrusted containers. By default, with restricted and anyuid SCC, NET_ADMIN is not granted. Be wary of any SCC that enables NET_ADMIN explicitly, or enables the pod to run in privileged mode. In addition, for untrusted containers, create a separate SCC based on, for example, anyuid SCC, with NET_RAW capability removed. This can be done by adding NET_RAW to `requiredDropCapabilities` list in the SCC definition.
Allow root access in PODs/Containers (only for testing). Commands below will require root access in all PODs of the oc project you are currently logged in to.
oc new-project test-project oc project test-project oc adm policy add-scc-to-user anyuid -z default
Configure (add) the OpenShift Registry.
oc login -u system:admin -n default oc adm registry --service-account=registry --config=/etc/origin/master/admin.kubeconfig
Delete the OpenShift Registry
oc login -u system:admin -n default oc delete svc/docker-registry dc/docker-registry
There is a missing IPtables firewall rule to allow DNS requests from the Docker default bridge containers to the dnsmasq process on the Node. It needs to be opened manually. Edit /etc/sysconfig/iptables and add the following Rules at the bottom of the file before COMMIT:
-A OS_FIREWALL_ALLOW -p tcp -m state --state NEW -m tcp --dport 53 -j ACCEPT -A OS_FIREWALL_ALLOW -p udp -m state --state NEW -m udp --dport 53 -j ACCEPT COMMIT
Restart iptables, docker and origin-node (restarts kube-proxy and kubelet).
systemctl restart iptables systemctl restart docker systemctl restart origin-node
The internal docker registry of OpenShift needs to be allowed to use non-TLS for OpenShift to work. Normally this should be added automatically by the OpenShift Ansible installer, but it seems that this is currently not working. Edit /etc/sysconfig/docker and add:
systemctl restart docker
NCP's support for network policies is the same as the support provided by Kubernetes and depends on the Kubernetes version used by OpenShift.
OpenShift 3.9 - The rule clauses in the network policy may contain at most one selector from namespaceSelector, podSelector and ipBlock.
The Kubernetes API server does not perform validation of a network policy specification. It is possible to create a network policy that is invalid. NCP will reject such a network policy. If you update the network policy to make it valid, NCP will still not process the network policy. You must delete the network policy and recreate one with a valid specification.
Certain versions of Kubernetes has a subPath-related issue (see https://github.com/kubernetes/kubernetes/issues/61076). If the OpenShift version does not contain a fix for this issue, the creation of the NCP pod will fail with the error CreateContainerConfigError: failed to prepare subPath for volumeMount. You can work around this problem by removing the use of subPath from the NCP yaml. Specifically, remove the line containing subPath: ncp.ini and replace the configuration for volumes with the following:
volumes: - name: config-volume # ConfigMap nsx-ncp-config is expected to supply ncp.ini configMap: name: nsx-ncp-config items: - key: ncp.ini path: ncp.ini
A side effect of this change is that the entire /etc/nsx-ujo directory becomes read-only. As a result, connecting with NSX-T using certificate and private key will not work because NCP will not be able to create a temporary file under /etc/nsx-ujo to move both certificate and private key into a single file.
If you are running or upgrading to OpenShift 3.10 cluster, note the following:
You must specify configuration of Node groups specific to OpenShift 3.10 cluster. Node config map configuration must be provided in inventory hosts file.
All hosts defined in the [nodes] group in the inventory hosts file must be assigned to a node group name.
Upgrading OpenShift cluster from an Ansible playbook might cause network loss. Make sure to add the patch (https://github.com/openshift/openshift-ansible/pull/8016/files#diff-2386e21861da3f95091dbb27d72ca366) on the openshift-ansible repository to remove the stop/uninstallation of Open vSwitch packages.
Starting with OpenShift 3.10, kube-proxy was moved from the openshift-node service to a DaemonSet. It is no longer started by default. Perform the following steps to start kube-proxy manually (assuming that the openshift-ansible repo has been cloned):
Go to the openshift-ansible directory, under [defaults] set the following:
library = roles/lib_utils/library/
Create a create_proxy.yaml file in the playbooks directory with the following entries:
- import_playbook: byo/openshift_facts.yml - hosts: masters run_once: True roles: - kube_proxy_and_dns
Run the playbook:
ansible-playbook -i hosts playbooks/create_proxy.yaml
You will see error messages indicating the failure of some operations. These messages can be ignored. You can verify the result by running the command oc get po --all-namespaces.