Before setting up OpenShift and NCP, take note of the following information.

  • The names of OpenShift Namespace, Pod and Network Policy resource, OpenShift Cluster (as specified in ncp.ini) and Node, must not be longer than 40 characters.

  • Label key and value must not be longer than 20 and 40 characters, respectively.

  • A pod must have no more than 11 labels and a namespace must have no more than 12 labels.

  • Labels added for OpenShift internal usage, for example, a label with prefix openshift.io in its key, will be disregarded by NCP and thus user won't see the corresponding tags created on the related NSX resources. Here is a list of label prefixes used by OpenShift, and you should avoid using a label key starting with any of the following:

        openshift.io
        pod-template
  • The nodes will need access to the pods, for example, for Kubelet health-checks. Make sure the host management interface is able to access the pod network.

  • Linux capabilities NET_ADMIN and NET_RAW can be exploited by attackers to compromise the pod network. You should disable these two capabilities of untrusted containers. By default, with restricted and anyuid SCC, NET_ADMIN is not granted. Be wary of any SCC that enables NET_ADMIN explicitly, or enables the pod to run in privileged mode. In addition, for untrusted containers, create a separate SCC based on, for example, anyuid SCC, with NET_RAW capability removed. This can be done by adding NET_RAW to `requiredDropCapabilities` list in the SCC definition.

  • Allow root access in PODs/Containers (only for testing). Commands below will require root access in all PODs of the oc project you are currently logged in to.

        oc new-project test-project
        oc project test-project
        oadm policy add-scc-to-user anyuid -z default
  • Configure (add) the OpenShift Registry.

        oc login -u system:admin -n default
        oadm registry --service-account=registry --config=/etc/origin/master/admin.kubeconfig
  • Delete the OpenShift Registry

        oc login -u system:admin -n default
        oc delete svc/docker-registry dc/docker-registry
  • There's a missing IPtables firewall rule to allow DNS requests from the Docker default bridge containers to the dnsmasq process on the Node. It needs to be opened manually. Edit /etc/sysconfig/iptables and add the following Rules at the bottom of the file before COMMIT:

        -A OS_FIREWALL_ALLOW -p tcp -m state --state NEW -m tcp --dport 53 -j ACCEPT
        -A OS_FIREWALL_ALLOW -p udp -m state --state NEW -m udp --dport 53 -j ACCEPT
        COMMIT
  • Restart iptables, docker and origin-node (restarts kube-proxy and kubelet).

        systemctl restart iptables
        systemctl restart docker
        systemctl restart origin-node
  • The internal docker registry of OpenShift needs to be allowed to use non-TLS for OpenShift to work. Normally this should be added automatically by the OpenShift Ansible installer, but it seems that this is currently not working. Edit /etc/sysconfig/docker and add:

        INSECURE_REGISTRY='--insecure-registry 172.30.0.0/16'
  • Restart Docker.

    systemctl restart docker
  • Configure (add) the OpenShift routers (HA-Proxy N/S LBs).

        oc login -u system:admin -n default
        oadm router router --replicas=2 --service-account=router
  • Delete the created routers.

        oc login -u system:admin -n default
        oc delete svc/router dc/router
  • Create a sample Ruby based 2 tier apps.

        oc login -u system:admin -n default
        oc oc new-project nsx
        oc process -n openshift mysql-ephemeral -v DATABASE_SERVICE_NAME=database | oc create -f -
        oc new-app centos/ruby-22-centos7~https://github.com/openshift/ruby-hello-world.git
        oc expose service ruby-hello-world
        oc env dc database --list | oc env dc ruby-hello-world -e -