In this section, the VMware solutions that have specific impacts on the NERC CIP standard requirements are elaborated, as indicated in the preceding matrix. Controls might only have partial coverage of the associated requirement by the technology mentioned.
Therefore, it is suggested in the summary at the end of this document that companion products, in conjunction with the policies, procedures, methods and best practices of each NERC identified ‘Responsible Entity’ (that is, power service provider), are used to create a comprehensive compliance status.
CIP-005-8: Electronic Security Perimeter (ESP)
Requirement R1: The ESP
The primary focus of these controls is to secure the Responsible Entity networks and specifically firewall and router-based security controls that are provided by several elements of the ESXi hypervisor and VeloCloud SD-WAN products. The software-defined networking features of ESXi deliver the essential basic networking used to create segments that are potential boundaries for security zones, or the basic network that might be enclosed by an ESP. The VMware VeloCloud Edge (VCE) product, by virtue of its firewall object performing the duties of an Electronic Access Point (EAP), is combined with the basic ESXi networking to constrain the access that a network segment might have with a rule-based, access-list driven technology, satisfying the R1.1 through R1.3 control requirements.
Time-sensitive protection systems (vPR) are exempted from these requirements.
The VMware VCE network toolset has further capacity to construct the required ESP/EAP control via the use of segmentation. Running as a built-in resource, VCE adds a toolkit of routing, VPN, VLAN, and security components that may be engaged on VMs which are centrally managed and monitored by vCenter and Aria Suite administration. The segmentation component is effectively a “firewall wrapper” that, once applied to a VM, enables packet inspection at Layers 2 to 7, and firewall access rules to apply to incoming and outgoing traffic to/from the VM. The unique aspect of VCE segmentation is that it requires no host-based software to deliver this service and is entirely agnostic and transparent with respect to the operating system. It may be applied to insecure network architectures which may not have appropriate conventional segmentation to create the EAP, and is a terrific tool to achieve compliance without re-engineering BES networks.
VCE features like IDP/IDR, can easily exceed the requirements for R1.5 and R1.6. Malicious activity is not just detected, but blocked and isolated, allowing for immediate visibility and analysis to begin. And, detection is provided into the compute layers of CPU and memory, resulting in fewer false positives. The data encryption requested by R1.6 can be provided by vSphere itself, as explained in the previous sections of this document.
Requirement R2: Remote Access Management
Access to equipment identified as a NERC CIP Protected Cyber Asset (PCA), which includes virtual cyber assets, must be accessed through an intermediate system. ESXi may be used to construct this as a VM, and to logically connect it to the appropriate network within the ESP, protected by an EAP as outlined in the section above. This use-case is typically referred to as a “jump host” and is a mainstay of virtualization architectures. Complete management of this host is supported by additional components in Edge Compute Stack 3.0 (ECS 3.0), specifically: vCenter and Aria Operations. In this scenario, the creation and support of jump host inventories, including rapid deploy and re-configuration technologies, are supplied by vCenter through use of templates and other features that are built-in.
A companion product supporting this method of implementing ESXi would be VMware’s Horizon/Microsoft RDP server, which can scale much more easily and provide end-to-end security from a “golden image” used to manage virtual desktops and applications for accessing PCAs. When paired with SD-WAN, Velo Cloud Solution, a zero trust model of deployment can be built with highly enhanced security, easily satisfying requirement R2.1 and remaining compliant with R2.6, which prescribes that no CPU or memory resources be shared between the intermediate system and a BCS.
R2.2 controls may be satisfied by use of VMware VCE VPN encryption, as applied within Horizon or directly to one or more jump host VMs. In conjunction with firewall rules for Layer 2-7 access in and out, strong access encryption is assured. If external firewall devices are desired, they may be supplied by partners, and can be deployed in either physical or virtual appliance forms.
The requirement R2.3 for multi-factor authentication (MFA) is supported throughout any of VMware’s aforementioned products, and can even be further enhanced through partner offerings. And the detection and control of vendor access, provided in the administrative capabilities of vSphere and/or Horizon, can satisfy R2.4 and R2.5.
Requirement R3: Vendor Remote Access Management
Electronic Access Controller Monitoring Systems (EACMS) and Physical Access Control Systems (PACSs), along with any supporting Shared Cyber Infrastructure (SCI), must be monitored for vendor remote access sessions with the ability to terminate them. Overall, VMware solutions are dependent on access control capacities provided by integration with a third party when using an application provided by them.
CIP-007-7: Systems Security Management
Requirement R1: System Hardening
Unnecessary routable protocol communications to applicable BCSs is to be disabled and/or prevented. VMware is a participant in control and allocation of network ports and will be working in parallel with physical network switches in the overall system architecture. Network switch integration to vSphere is via connectivity to the VMware ESXi hypervisors, where the ports on the ESXi server are connected directly to dedicated ports on the switch(es), which reside on one or more VLANs. The total footprint of physical ports is likely to be far less with software-defined infrastructure, when compared to traditional, microprocessor devices. VMware vCenter and Aria Operations Management have authoritative control over all hypervisor and VM usage of network ports, and therefore can easily meet R1.1 in terms of documentation and management. R1.3 is similar to CIP-005-8, R2.6 in prescribing no shared CPU or memory resources between Virtual Cyber Assets (VCAs) of varying impact classes. This can be avoided with proper planning and mitigated automatically with virtualization affinity rules within vSphere.
R1.2 requires physical port protection, which must be managed with physical security provided by the end user. Similarly, the ports, services, configurations, and settings of the physical networking switches must be managed outside of vCenter and Aria.
When VeloCloud VCG/VCE has been deployed within a control site environment, a higher resolution of control of the ESXi hypervisor switch ports groups, as they are consumed by the VMs, is added to the basic Layer 2 delivery provided by VMware Distributed vSwitches. By virtue of the VCE components (firewall, router, VLAN management, VPN, segmentation, etc.) and the pervasive and transparent presence of VeloCloud Gateway (VCG) on a hypervisor, virtual networking may receive the architectural benefits of those additions to basic virtual switching. Please note that the firewall rules are bound to the VM and not a specific hypervisor/host, and any vMotion activities will keep the rules connected to the VM, regardless of where it is hosted.
Requirement R2: Security Patch Management
Every responsible entity shall have a documented process to not only evaluate and install available cyber security patches in a timely manner, but also to track the sources of such software to ensure receipt of any newly available corrections and to develop their own schedule for implementation.
Management of patching, as required in R2.1 through R2.4 is supported by both VMware vCenter Update Manager and partner products for the hypervisor system images, as well as optionally for the underlying operating systems. The Update Manager is merely a vehicle for administering the program defined in R2.1 and sustained in R2.2, while it is the actual agent used to apply patches to VMware-implemented software. It is customary that third-party partner products are relied upon to supply patches for network equipment, servers, and other critical infrastructure.
Requirement R3: Malicious Code Prevention
The deterrence, detection, and prevention of malicious code is principally a function provided by VMware partner products. However, the mitigation required by R3.2 is easily facilitated by VMware’s own VCE 6.0 and vSphere activities, which coordinate the containment process and are used to automate (or manually perform) the shutting down and quarantining of the infected VM. VCE will inherit the signatures from the NSX cloud database and update its own signature database records for the detection of suspicious activities. Subsequent usage of VMware snapshot technologies can then re-deploy a pre-infection instance of a decommissioned VM. Rapid recovery, in this manner, is a key strength of software-defined workloads versus traditional system deployments.
It should be noted here as well, that VMware’s core product, the ESXi hypervisor, uses a combination of on-host security and best practices (mentioned at the end of this NERC CIP section) to protect management interfaces and the underlying hypervisor. Antivirus and Malware Detection are needed on General Purpose (GP) computing environments, to mitigate risks incurred when a user or a process loads arbitrary executables from indeterminate sources. These risks are typical to GP computing environments, where users can execute code with minimal policy. However, ESXi is not a GP computing environment. Only binaries from VMware and third parties that are delivered in signed packaging (VIB) and validated by strong controls are executable. There is no facility to interpret code at runtime and the compiled modules are subject to both the controls for execution and a default-deny policy (for unsigned code), integral to the kernel. Also, VM secure booting is a way to make sure that only allowed code/OS should boot.
Antivirus software is not required with the vSphere Hypervisor and the use of such software is not supported. Customers may ensure the integrity of software they install into vSphere by downloading the software only from the HTTPS-protected VMware site. The installation kits published there are signed, and the integrity and signatures are checked by vSphere during the installation process. When updates are necessary, VMware publishes the necessary files to the same download site. In the case of security updates, VMware announces updates as part of its security advisory process and necessary patches are also placed on the secure download site.
Requirement R4: Security Event Monitoring
Responsible entities must log, alert for, and retain security events, along with periodically reviewing them, to identify cyber security incidents, in accordance with R4.1 through R4.4. These security events would include both successful and failed logins, which are supported across the entire suite of VMware products. Via traditional syslog services and integrated internal logging, all actions, including malicious code mitigation actions, are treated with the intended safety of this requirement. They would also support a third-party Security Information and Event Management (SIEM) product to provide a more comprehensive view of events recorded. VMware's VCE Intrusion Protection Systems (IPS) and Intrusion Detection Systems (IDS) can offer further network traffic analysis, intrusion detection and prevention, and advanced malware analysis with comprehensive network detection and response capabilities. VCE has a capability to pull the latest and greatest records from the NSX cloud via VCO. These records can be pulled once and updated in the VCE database to ensure the security of the code. The regular pull can be discussed along with the customer for the up to date records.
Requirement R5: System Access Control
Requirements R5.1 through R5.4 and R5.7 include describing how interactive user access gets authenticated, inventorying any/all accounts and individual users with access to any shared accounts, as well as prohibiting usage of default passwords, and providing alerts and lockout for a defined quantity of unsuccessful access attempts. These are all supported natively by each VMware product, but third-party applications residing on the infrastructure would have to be evaluated and tracked externally. Integration across the entire platform is possible with a comprehensive authentication and access control component such as VMware’s Workspace ONE.
Password-only authenticated systems (something that is known), which do not support a second-factor or multi-factor authentication (additionally requiring personal identification and/or personal possession artifacts), are additionally subject to requirements R5.5 and R5.6. These prescribe rules for password complexity (or maximum system capability) and regular rotation. VMware products support MFA, and therefore would not be subject to these rules. However, third-party applications within the deployment may need individual evaluation, if not wrapped into a holistic, automated, authentication management system such as Workspace ONE.
CIP-009-7: Recovery Plans for BES Cyber Systems
Requirement R1: Recovery Plan Specifications
The first two parts of requirement R1 are policy-related and organizational. Conditions must be determined for the activation of recovery plans, and roles/assignments made. R1.3 through R1.5 then include the documented plan for backup and storage of information necessary to fully recover BCSs, as well as the methods for verification and preservation of the data.
- Systems disaster recovery plan
- Critical information data backup and recovery plan
- Cyber system asset valuation and identification of critical systems
- Risk analysis and ranking.
- Processes for restoring services.
With these components in place, emphasis should then be placed on the restoration of service, versus just trying to meet the “basic requirements” of backing up data. In other words, disaster recovery should be recognized as more than just backups. This is a stance necessary for all information-reliant entities, regardless of the type of industry.
Several of VMware vSphere core capabilities provide tolerance for faults and mitigation of disaster scenarios. Firstly, the built-in snapshot feature offers fault prevention by functioning to preserve the state of a VM before making changes (e.g. updates, upgrades, patches, etc.). This encompasses the entirety of an applied system, including the data, context, and operational state of a VM. This brings more utility to both testing and actual recovery, by packaging data plus machine details into a “moment in time” instance of the VM.
Additionally, there are High Availability (HA) and Fault Tolerance (FT) functions that can provide quick recovery of an application in the event of a failure. HA quickly restarts applications on nearby, available hardware, should the original host fail. And, FT provides similar functionality, but occurs instantaneously with the standby application always ready to be inserted at the momentary onset of an active device’s failure. And, these can both be applied with affinity rules to ensure that reapplied applications are started in an environment that is both capable and secured to meet its requirements.
The addition of VMware’s vSAN (virtualized storage area network) to ESXi offers many benefits regarding resource flexibility and security. And, in terms of disaster recovery, it can be designed tolerant to multiple levels of failure. Even with a minimal amount of physical hardware, a simple “stretched cluster” can provide continuous operation without loss of functionality or data in an (n-1) server scenario where only two local servers are installed.
Finally, VMware Site Recovery Manager (SRM) is an additional product which can supply a unique bundle of enhancement to the standard VMware vCenter / ESXi core. It provides for a comprehensive set of disaster recovery tools that allow backup and redundant systems to be developed and deployed quickly, and automatically. By replicating the VM context from one vCenter data center to a backup, and then remaining in constant readiness to “cut over” to the redundant location, SRM positions the responsible entity to be prompt and effective in the event of a failure.
Requirement R2: Recovery Plan Implementation and Testing
Requirements found in R2 all call for testing of the recovery plans developed for applicable systems of varying impact levels. The aforementioned functionality found within vSphere (snapshots, HA, and FT) can be tested and verified VMware SRM provides complete, non-disruptive simulation of the fail-over and fail-back processes, built-in as part of its basic functionality. And, even within a non-SRM environment, manual actions coupled with VMware Aria Automation may be used to construct test scenarios, clone critical assets for testing, etc. So either by using basic tools or those including SRM, these CIP requirements can easily be met.
Requirement R3: Recovery Plan Review, Update, and Communication
R3.1 and R3.2 require the documentation of lessons learned, updates to plans, and notification to teams to communicate any changes in plans or in roles/responsibilities. Therefore, these organizational-related tasks are not directly applicable to VMware products.
Overall, CIP-009 remains somewhat weak on the comprehensive nature of system recovery. The specific emphasis on the recovery of information is not in alignment with the more holistic approach that has evolved in other regulatory frameworks (e.g. NIST), which are also used by infrastructure planners. However, if we were to adopt the notion that all digital systems are simply “data”, then in that perspective, preservation and rapid recovery of data that constitutes a virtual application, network, etc., is well supported by VMware’s vSphere, vSAN, and SRM technologies, with the ability to meet these regulations in a cost-effective and efficient manner.
CIP-010-5: Configuration Change Management and Vulnerability Assessments
Requirement R1: Security Configuration Change Management
The first CIP-010 requirement, R1.1, stipulates that a documented process exists for any changes to be made to security controls implemented for CIP-005 and -007. This includes the installation, update, or removal of operating systems, applications, firmware, or patches. This also includes configuration changes to network access, SCI or ESPs between systems of different impact ratings, or to parent images from which child images are derived. This is to be accompanied by change management control (e.g. dated requests and implementation, along with output of vulnerability scans). At least two mechanisms are available in VMware vSphere to accomplish this:
- The template features of vCenter, specifically the Customization Specification Manager tool
- Aria Operations deployment tools
Metadata can be attached to the VM images, templates, and to the Customization Specification, which can serve as part of the required documentation base to support the change control process, as images are iterated during their lifetime. These tools actually go beyond the minimal requirement of a control process, and can be used to create, maintain, and deploy a large collection of system images.
Network access control is also mentioned within R1.1. Therefore, VMware VCO or any host system firewall software would also play into this requirement. VCO of course offers logging for any changes made, but also has a management functionality that can monitor and analyze traffic prior to updates being tested or otherwise administered.
As mentioned within the CIP-009 section, VMware also provides a snapshot function, which can capture a VM system image at a particular instant in time. Its transient state information is then tracked from that point on, satisfying R1.2, which calls for any changes to be tested prior to implementation. And this can be performed within the production environment while minimizing any possible adverse effects.
R1.3 requires that any intended software changes are verified with their acquisition source prior to deployment. VMware offers security advisories for all products through one subscription. The MD5/SHA1/SHA256 sum value is posted at the download location for each file, and any VMware software acquired should be manually verified, according to that published value, before installation.
Requirement R2: Security Configuration Monitoring
A method to monitor for unauthorized changes in any applicable system’s software and configurations is required (at a minimum interval of 35 days), with dated records and results of investigation(s) serving as evidence. With the previous VMware products mentioned here (vSphere, VCE, vSAN, Aria Operations and Automation, etc.), there is a holistic tool available to collect and analyze data from them all, in VMware Aria Operations for Logs. This offering has the capability to not only log any environmental changes, meeting the requirement’s need for the software definition layer, but can also provide manual searching functionality alongside automated analysis (using machine learning) to create alerts based on security or performance policies. The additional SaltStack plug-in can also continuously monitor environments for adherence to compliant configurations.
The monitoring for the third-party application layers can be performed by partner software which facilitates the centralized deployment and ongoing configuration management of virtual appliances.
Requirement R3: Vulnerability Assessments
Responsible entities are to perform vulnerability assessments of all applicable systems. This includes periodic testing of existing systems, per R3.1 and R3.2, as well as any new system, per R3.3, prior to placing in-service. Additionally, a plan of action for mitigating or remediating vulnerabilities is required per R3.4.
Symantec Endpoint Application Control relies on a robust discovery engine that uncovers all the applications operating in the environment. It discovers every instaIled application and its associated files on the endpoint. It also automatically assesses each application’s vulnerabilities, reputation, and prevalence to generate a risk score. And with this score, Symantec Endpoint Application Control delivers a risk assessment, actionable insight, and smart recommendations for blocking or allowing the application.
Outside of using Symantec, vSphere provides a flexible environment which can aid in the vulnerability assessment of individual applications. VMs could be duplicated, or cloned, which would allow manual assessments to be conducted of each type, while it is active in the production environment and the original, primary application is left unhindered. This process could also be scripted within VMware’s Aria Operations CSM tool to generate automated reports when coupled with partner software offering port/services scanning products.
Requirement R4: Plans for Transient Cyber Assets (TCAs) and Removable Media
TCAs and removable media must both meet security requirements before being used with any applicable systems. The management and authorization sections, found in R4.1, R4.2, and R4.3, are processes authorizing individuals and/or groups to use these devices only as needed for business purposes, which must be defined and enforced by the responsible entity.
VMware then has solutions to support the areas of prevention, isolation, and mitigation of software vulnerabilities, malicious code, and unauthorized use of these devices. These are outlined in requirements R4.1.3 through R4.1.5, R4.2.1 through R4.2.3, and R4.3.2. As mentioned in the CIP-005 section, VMware’s Horizon virtual desktop infrastructure offers a method of managing as many TCAs as a responsible entity requires. A single “golden” image can be used to control the known good state of executed software and applications. Patch management and anti-virus programs would be controls implemented across all devices to mitigate vulnerabilities and malicious code. And unauthorized access can be prevented with encrypted credentials and MFA login support.
The prevention of unauthorized usage of removable media is mainly a policy-based process, which may leverage physical protections required by the responsible entity. VMware products can support the mitigation actions necessary, however, for removable media that is used on applicable systems or TCAs. The methods of detection described for R3 can also be leveraged for R4.3.2.
CIP-012-1: Communications between Control Centers
Requirement R1
There is only a single requirement found in CIP-012, with three sub-sections. The first section, R1.1, stipulates the protection of real-time assessment and monitoring data passing between operational control centers. vSAN’s data-in-transit encryption can easily facilitate this, as it seamlessly hardens all data being transmitted between physical hardware. Therefore, as long as both control centers were using vSAN storage solutions, they can easily meet this compliance objective.
The remaining two sections of CIP-012, R1.2 and R1.3, rely on policy and procedures formed and enforced by the responsible entity to identify where protection has been applied and who is responsible for implementation of the security.
CIP-013-3: Supply Chain Risk Management (SCRM)
Requirement R1: Supply Chain Cyber Security Risk Management Plan
R1.1 requires the responsible entity to enforce a process to assess the risk in procurement and deployment of products, including vendor-to-vendor transactions. R1.2 then outlines portions of the process to include:
- Notification of any vendor incidents
- Coordination of vendor incident response
- Notice to terminate any applicable vendor local or remote access has passed need
- Receipt of disclosure of any known vulnerabilities
- Verification of software integrity and authenticity
- Coordination of control for any vendor remote access
VMware follows a very comprehensive Security Development Lifecycle (SDL) that is outlined on the Security Development Lifecycle landing page. Individuals concerned about Supply Chain Risk Management (SCRM) should visit the VMware Security Response Center landing page and sign up for security advisories. A Trust and Assurance site provides additional details covering a framework, with content around SCRM. They also support the North American Transmission Forum (NATF) Cyber Security Criteria for Suppliers process, which provides a utility-approved approach for vendors to provide organizational information necessary for individual assessment.
Several procedures are necessary for VMware to maintain security when distributing the product to a customer’s site. For a valid delivery, the product received must correspond precisely to the product master copy, without tampering, or substitution of a false version. The delivery procedures ensure that the integrity and authenticity of the product are maintained and that they are verifiable by the customer and by VMware after delivery has been completed. The product is delivered via Broadcom Customer Support Portal websites by electronic distribution only. The end user is supplied with the product, product documentation, and product license.
This is inclusive of patches, which are cryptographically signed to ensure authenticity. Before a patch is installed or updated, the system verifies the signature. This signature enforces the end-to-end protection of the patch itself and can also address any concerns about patch download, whether installing the patch(es) manually or through an automated tool such as vSphere Lifecycle Manager or Aria Lifecycle Manager. Cryptographically signed patches extend to vSphere and other product & solution patches which are securely downloaded from Broadcom Customer Support Portal website and/or content distribution network.
Requirements R2, which speaks to the implementation of the plan outlined in R1, and R3, which mandates the appropriate approval of that overall plan, are policy-based functions of a responsible entity. Therefore, VMware does not have a viable product offering.
