If your environment uses vSphere Virtual Machine Encryption, and if an error occurs on the ESXi host, the resulting core dump is encrypted to protect customer data. Core dumps that are included in the vm-support package are also encrypted.

Note: Core dumps can contain sensitive information. Follow your organization's data security and privacy policy when handling core dumps.

Core Dumps on ESXi Hosts

When an ESXi host, a user world, or a virtual machine fails, a core dump is generated, and the host reboots. If the ESXi host has encryption mode enabled, the core dump is encrypted using a key that is in the ESXi key cache. (Depending on the key provider in use, the key comes from an external key server, the Key Provider Service, or vCenter Server). See How vSphere Virtual Machine Encryption Protects Your Environment for background information.

When an ESXi host is cryptographically "safe," and a core dump is generated, an event is created. The event indicates that a core dump occurred along with the following information: world name, occurring times, keyID of the key used to encrypt the core dump, and core dump filename. You can view the event in the Events viewer under Tasks and Events for the vCenter Server.

The following table shows encryption keys used for each core dump type, by vSphere release.

Table 1. Core Dump Encryption Keys
Core Dump Type Encryption Key (ESXi 6.5) Encryption Key (ESXi 6.7 and Later)
ESXi Kernel Host Key Host Key
User World (hostd) Host Key Host Key
Encrypted Virtual Machine (VM) Host Key Virtual Machine Key
What you can do after an ESXi host reboot depends on several factors.
  • In most cases, the key provider attempts to push the key to the ESXi host after reboot. If the operation is successful, you can generate the vm-support package and you can decrypt or re-encrypt the core dump. See Decrypt or Re-Encrypt an Encrypted Core Dump.
  • If vCenter Server cannot connect to the ESXi host, you might be able to retrieve the key. See Resolve Missing Encryption Key Issues.
  • If the host used a custom key, and that key differs from the key that vCenter Server pushes to the host, you cannot manipulate the core dump. Avoid using custom keys.

Core Dumps and vm-support Packages

When you contact VMware Technical Support because of a serious error, your support representative usually asks you to generate a vm-support package. The package includes log files and other information, including core dumps. If your support representatives cannot resolve the issues by looking at log files and other information, they might ask you to decrypt the core dumps and make relevant information available. To protect sensitive information such as keys, follow your organization's security and privacy policy. See Collect a vm-support Package for an ESXi Host That Uses Encryption.

Core Dumps on vCenter Server Systems

A core dump on a vCenter Server system is not encrypted. vCenter Server already contains potentially sensitive information. At the minimum, ensure that the vCenter Server is protected. See Securing vCenter Server Systems. You might also consider turning off core dumps for the vCenter Server system. Other information in log files can help determine the problem.

Collect a vm-support Package for an ESXi Host That Uses Encryption

If host encryption mode is enabled for the ESXi host, any core dumps in the vm-support package are encrypted. You can collect the package from the vSphere Client, and you can specify a password if you expect to decrypt the core dump later.

The vm-support package includes log files, core dump files, and more.

Prerequisites

Inform your support representative that host encryption mode is enabled for the ESXi host. Your support representative might ask you to decrypt core dumps and extract relevant information.

Note: Core dumps can contain sensitive information. Follow your organization's security and privacy policy to protect sensitive information such as host keys.

Procedure

  1. Log in to the vCenter Server system with the vSphere Client.
  2. Click Hosts & Clusters, and right-click the ESXi host.
  3. Select Export System Logs.
  4. In the dialog box, select Password for encrypted core dumps, and specify and confirm a password.
  5. Leave the defaults for other options or make changes if requested by VMware Technical Support, and click Export Logs.
    If you have not configured your browser to ask where to save files before downloading, the download starts. If you have configured your browser to ask where to save files, specify a location for the file.
  6. If your support representative asked you to decrypt the core dump in the vm-support package, log in to any ESXi host and follow these steps.
    1. Log in to the ESXi host and connect to the directory where the vm-support package is located.
      The filename follows the pattern esx.date_and_time.tgz.
    2. Make sure that the directory has enough space for the package, the uncompressed package, and the recompressed package, or move the package.
    3. Extract the package to the local directory. 
      vm-support -x *.tgz .
      The resulting file hierarchy might contain core dump files for the ESXi host, usually in /var/core, and might contain multiple core dump files for virtual machines.
    4. Decrypt each encrypted core dump file separately. 
      crypto-util envelope extract --offset 4096 --keyfile vm-support-incident-key-file 
--password encryptedZdump decryptedZdump
      vm-support-incident-key-file is the incident key file that you find at the top level in the directory.

      encryptedZdump is the name of the encrypted core dump file.

      decryptedZdump is the name for the file that the command generates. Make the name similar to the encryptedZdump name.

    5. Provide the password that you specified when you created the vm-support package.
    6. Remove the encrypted core dumps, and compress the package again. 
      vm-support --reconstruct
  7. Remove any files that contain confidential information.

Decrypt or Re-Encrypt an Encrypted Core Dump

You can decrypt or re-encrypt an encrypted core dump on your ESXi host by using the crypto-util CLI.

You can decrypt and examine the core dumps in the vm-support package yourself. Core dumps might contain sensitive information. Follow the security and privacy policy for your organization to protect sensitive information such as keys.

For details about re-encrypting a core dump and other features of crypto-util, see the command-line help.
Note: crypto-util is for advanced users.

Prerequisites

The key that was used to encrypt the core dump must be available on the ESXi host that generated the core dump.

Procedure

  1. Log directly in to the ESXi host on which the core dump happened.
    If the ESXi host is in lockdown mode, or if SSH access is deactivated, you might have to activate access first.
  2. Determine whether the core dump is encrypted.
    Option Description
    Monitor core dump 
    crypto-util envelope describe vmmcores.ve
    zdump file 
    crypto-util envelope describe --offset 4096 zdumpFile
  3. Decrypt the core dump, depending on its type.
    Option Description
    Monitor core dump 
    crypto-util envelope extract vmmcores.ve vmmcores
    zdump file 
    crypto-util envelope extract --offset 4096 zdumpEncrypted zdumpUnencrypted