Use the vSphere On-disk Metadata Analyzer (VOMA) to check metadata consistency and identify and fix incidents of metadata corruption on the VMFS datastores or logical volumes that back the VMFS datastores.

Problem

You can check metadata consistency when you experience problems with a VMFS datastore or a virtual flash resource. For example, perform a metadata check if one of the following occurs:

  • You experience storage outages.
  • After you rebuild RAID or perform a disk replacement.
  • You see metadata errors in the vmkernel.log file similar to the following:
    cpu11:268057)WARNING: HBX: 599: Volume 50fd60a3-3aae1ae2-3347-0017a4770402 ("<Datastore_name>") may be damaged on disk. Corrupt heartbeat detected at offset 3305472: [HB state 0 offset 6052837899185946624 gen 15439450 stampUS 5 $
  • You are unable to access files on a VMFS.
  • You see corruption being reported for a datastore in events tabs of vCenter Server.

Solution

To check metadata consistency, run VOMA from the CLI of an ESXi host. VOMA can be used to check and fix minor inconsistency issues for a VMFS datastore or logical volumes that back the VMFS datastore.
VOMA can check and fix the following items.
Table 1. VOMA Functions

VOMA Functions

Description

Metadata check and fix

Examples of metadata check and fix include, but are not limited to, the following:

  • Validation of VMFS volume header for basic metadata consistency.

  • Checking consistency of VMFS resource files (system file).

  • Checking the pathname and connectivity of all files.

Affinity metadata check and fix

To enable the affinity check for VMFS6, use the -a|--affinityChk option.

Several examples of affinity metadata check and fix include the following:

  • Affinity flags in resource types and FS3_ResFileMetadata.

  • Validation of affinity flags in SFB RC meta (FS3_ResourceClusterMDVMFS6).

  • Validation of all entries in the affinityInfo entries in rcMeta of RC, including the overflow key, to make sure that no invalid entries exist. Checking for missing entries.

Directory validation

VOMA can detect and correct the following errors:

  • Directory hash block corruption.

  • Alloc map corruption.

  • Link blocks corruptions.

  • Directory entry block corruptions.

Based on the nature of the corruption, VOMA can either fix only the corrupted entries or fully reconstruct the hash block, alloc map blocks and link blocks.

Lost and found files

During a filesystem check, VOMA can find files that are not referenced anywhere in the filesystem. These orphaned files are valid and complete, but do not have a name or directory entry on the system.

If VOMA encounters orphaned files during scanning, it creates a directory named lost+found at the root of the volume to store the orphaned files. The names of the files use the Filesequence-number format.

Command options that the VOMA tool takes include the following.

Table 2. VOMA Command Options

Command Option

Description

-m|--module

The modules to run include the following:

vmfs

If you do not specify the name of the module, this option is used by default.

You can check the VMFS file systems and the file systems that back virtual flash resources. If you specify this module, minimal checks are performed for LVM as well.

lvm

Check logical volumes that back the VMFS datastores.

ptbl

Check and validate VMFS partitions, such as MBR or GPT. If no partition exists, determine whether partitions should exist.

-f|--func

Functions to be performed include the following:

query

List functions supported by module.

check

Check for errors.

fix

Check and fix errors.

dump

Collect metadata dump.

-a|--affinityChk

Include affinity-related check and fix for VMFS6.

-d|--device

Indicate the device or disk to inspect. Make sure to provide the absolute path to the device partition backing the VMFS datastore. If the datastore spans multiple devices, provide the UUID of the head extent.

For example, voma -m vmfs -f check -d /vmfs/devices/disks/naa.xxxx:x

If you use the -x|--extractDump command, enter multiple device paths, with a partition qualifier, separated with a comma. The number of device paths you enter equals the number of spanned devices.

-b|--blockSize

Indicate the disk block size.

-s|--logfile

Specify the path to log file to output the results.

-x|--extractDump

Extract the collected dump using VOMA.

-D|--dumpfile

Indicate the dump file to save the collected metadata dump.

-v|--version

Display the version of VOMA.

-h|--help

Display the help message for the VOMA command.

-Y

Indicate that you run VOMA without using PE tables for address resolution.

-Z| --file

Indicate that you run VOMA on extracted device files.

Example

Collect metadata dump from a spanned volume:

voma -m vmfs -f dump -d head_extent -D dump_filename

Extract collected dump back to devices of a spanned volume:

voma -x dump_filename -d head_extent,extent_2,extent_3...extent_n

Use VOMA to Check Metadata Consistency

The task demonstrates how to use VOMA to check VMFS metadata consistency. VOMA can be used to check and fix minor inconsistency issues for a VMFS datastore or a virtual flash resource. Run VOMA from the CLI of an ESXi host.

Prerequisites

Power off any virtual machines that are running or migrate them to a different datastore.

Procedure

  1. Obtain the name and partition number of the device that backs the VMFS datastore that you want to check.
    #esxcli storage vmfs extent list
    The Device Name and Partition columns in the output identify the device. For example: 
    Volume Name  .....  Device Name                             Partition  
1TB_VMFS6    .....  naa.xxxx                                   3
  2. Check for VMFS errors.

    Provide the absolute path to the device partition that backs the VMFS datastore, and provide a partition number with the device name. For example:

    # voma -m vmfs -f check -d /vmfs/devices/disks/naa.xxxx:x

    The output lists possible errors. For example, the following output indicates that the heartbeat address is invalid. 

    XXXXXXXXXXXXXXXXXXXXXXX
Phase 2: Checking VMFS heartbeat region
 ON-DISK ERROR: Invalid HB address
Phase 3: Checking all file descriptors.
Phase 4: Checking pathname and connectivity.
Phase 5: Checking resource reference counts.

Total Errors Found:           1