Use vSphere On-disk Metadata Analyzer (VOMA) to identify incidents of metadata corruption that affect file systems or underlying logical volumes.

Problem

You can check metadata consistency when you experience problems with a VMFS datastore or a virtual flash resource. For example, perform a metadata check if one of the following occurs:
  • You experience storage outages.
  • After you rebuild RAID or perform a disk replacement.
  • You see metadata errors in the vmkernel.log file similar to the following: 
    cpu11:268057)WARNING: HBX: 599: Volume 50fd60a3-3aae1ae2-3347-0017a4770402 ("<Datastore_name>") may be damaged on disk. Corrupt heartbeat detected at offset 3305472: [HB state 0 offset 6052837899185946624 gen 15439450 stampUS 5 $
  • You are unable to access files on a VMFS.
  • You see corruption being reported for a datastore in events tabs of vCenter Server.

Solution

To check metadata consistency, run VOMA from the CLI of an ESXi host. VOMA can be used to check and fix minor inconsistency issues for a VMFS datastore or a virtual flash resource. To resolve errors reported by VOMA, consult VMware Support.

Follow these guidelines when you use the VOMA tool:
  • Make sure that the VMFS datastore you analyze does not span multiple extents. You can run VOMA only against a single-extent datastore.
  • Power off any virtual machines that are running or migrate them to a different datastore.
The following example demonstrates how to use VOMA to check VMFS metadata consistency.
  1. Obtain the name and partition number of the device that backs the VMFS datastore that you want to check.

    #esxcli storage vmfs extent list

    The Device Name and Partition columns in the output identify the device. For example: 
    Volume Name  XXXXXXXX  Device Name                             Partition  
1TB_VMFS5    XXXXXXXX  naa.00000000000000000000000000000703    3
  2. Check for VMFS errors.

    Provide the absolute path to the device partition that backs the VMFS datastore, and provide a partition number with the device name. For example:

    # voma -m vmfs -f check -d /vmfs/devices/disks/naa.00000000000000000000000000000703:3

    The output lists possible errors. For example, the following output indicates that the heartbeat address is invalid. 

    XXXXXXXXXXXXXXXXXXXXXXX
Phase 2: Checking VMFS heartbeat region
 ON-DISK ERROR: Invalid HB address
Phase 3: Checking all file descriptors.
Phase 4: Checking pathname and connectivity.
Phase 5: Checking resource reference counts.

Total Errors Found:           1

Command options that the VOMA tool takes include the following.

Table 1. VOMA Command Options
Command Option Description
-m|--module The modules to run include the following:
vmfs If you do not specify the name of the module, this option is used by default.

You can check the VMFS file systems and the file systems that back virtual flash resources. If you specify this module, minimal checks are performed for LVM as well.

lvm Check logical volumes that back the VMFS datastores.
ptbl Check and validate VMFS partitions, such as MBR or GPT. If no partition exists, determine whether partitions should exist.
-f|--func Functions to be performed include the following:
query List functions supported by module.
check Check for errors.
-d|--device Device or disk to be inspected. Make sure to provide the absolute path to the device partition backing the VMFS datastore. For example, /vmfs/devices/disks/naa.00000000000000000000000000:1.
-s|--logfile Specify the log file to output the results.
-v|--version Display the version of VOMA.
-h|--help Display the help message for the VOMA command.

For more details, see the VMware Knowledge Base article 2036767.