VMware Skyline Health Diagnostics Release Notes

VMware Skyline Health Diagnostics | 20 FEB 2024 Check for additions and updates to these release notes.

About VMware Skyline Health Diagnostics

VMware Skyline Health Diagnostics is VMware's Self-Service diagnostic and health platform. It helps you with the following:

• Diagnose any failures or known issues and provide recommendation in the form of Knowledge Base or remediation steps

• Run health checks

• Understand applicability of VMware Security Advisories and related resolutions

• Identify issues which might impact the update or upgrade of the product

It uses product logs, configuration information, and various other information to detect problems and provide recommendations in the form of KB articles or steps to remediate them.

A vSphere Administrator can use this tool to troubleshoot the issue before contacting the VMware Global Support Service. This platform detects and provides remediation to problems in the vSphere product line. The platform provides recommendations in the form of Knowledge Base articles or steps to remediate the issues. It can work in offline mode or a disconnected environment. It uses product logs to detect issues. vSphere Administrator can use this tool for troubleshooting, before contacting the VMware Global Support Service. Your operation staff or support engineers can save a significant amount of time to detect the problem, identifying a cause, and getting a quick resolution, for VMware vSphere, using the Skyline Health Diagnostics.

For more information, see VMware Skyline Health Diagnostics Installation, Configuration and Operations Guide.

What's New

• Support for VMware Cloud Gateway Diagnostics for vSphere+

  • Added diagnostics capability for VMware Cloud Gateway for vSphere+ customers using direct connect and Log bundle upload.

  • You can perform diagnostics when vSphere+ registered vCenters or VMware Cloud Gateway is disconnected from Cloud or run health checks periodically to ensure the system is in healthy state.

  • Added diagnostics capability for agents running in the VMware Cloud Gateway appliance for vSphere+

• Support for offline (non-internet connected) patching the Skyline Health Diagnostics appliance.

  • Upgrade using the ISO media file has been deprecated in 4.0.0. You can use offline patching feature to update the environment.

  • Offline instances must upgrade to 4.0.0 release to use this capability.

• Enhanced Health & Diagnostics support for VMware Cloud Foundation 4.x, 5.x

  • Added health checks to enhance the support for Aria Product suite like vRealize Automation, vRealize Log Insight, vRealize Operations, Workspace ONE and vRSLCM in SDDC environment.

  • DNS, NTP, SSH, Certs & Ping checks add capability to monitor the health of product if deployed in SDDC stack.

• Use VMXNET3 adapter for default deployment of VMware Skyline Health Diagnostics appliance.

• Added description and helper text for password in VMware Skyline Health Diagnostics appliance OVA deployment.

• New Plugins Added

  • vSphere: 41

  • vSAN: 3

  • VMware Cloud Foundation : 10

  • VMware NSX Health Checks : 4 (Technical Preview Mode)

• ISO Deprecation

  Starting version 4.0.0, Offline ISO based installation or patching or upgrade is deprecated.

  All existing ISO based installations are recommended to be migrated to freshly deployed Skyline Health Diagnostics Appliance. Migration will copy the data and configuration from existing Skyline Health Diagnostics Virtual Machine. Future patches for VMware Skyline Health Diagnostics will be delivered through bundles that can be imported into VMware Skyline Health Diagnostics.

Supported VMware Products and Browser Compatibility

Supported VMware vSphere Releases

• VMware ESXi version 6.5, 6.7 and 7.0 and Update releases, 8.0, 80 U1 and onwards

• VMWare vCenter version 6.5, 6.7 and 7.0 and Update releases, 8.0, 80 U1 and onwards

Supported VMware vSAN Releases

• VMware vSAN version 6.5, 6.7 and 7.0 and Update releases, 8.0, 80 U1 and onwards

Supported VMware Cloud Foundation Releases

• VMware Cloud Foundation version 4.0, 4.1, 4.2, 4.3, 4.4, 4.5 and 5.0

Supported VMware Horizon Releases

• VMware Horizon version 7.0 and 8.0

Supported VMware SD-WAN Releases

• VMware SD-WAN version 3.4, 4.0, 4.2, 4.3, 4.5, 5.0, 5.1 and 5.2.

Supported Browsers

Microsoft Window 10

Browser Version
Microsoft Internet Explorer 11 and later
Mozilla Firefox version 56 and later
Mozilla Firefox version 56 and later
Google Chrome version 62 and later
Microsoft Edge: 44.18362.449.0
Safari version 12.1 and later

MAC OS

Browser Version
Mozilla Firefox version 56 and later
Google Chrome version 62 and later
Safari version 12.1 and later

Feature list for version 4.0.3, Feb 2024 Release

​VMware Certified remote plugin for vSphere Client

vSphere 8.0 update 1 and onwards, supports the VMware Skyline Health Diagnostics as a remote plugin.

Profile based Analysis

Simplified user experience to connect and analyse the VMware products for issues. Added capability to connect to a VMware product, detect the issues and get recommendation to remediate the issues. Select the Diagnostics or Security analysis plug-in or Health Checks. Name the profile to save the input provided and use it later. Tag the specific analysis run for a faster search in the reports afterward. Schedule the analysis to detect the issue and get report automatically.  Finally, get the report of the identified issues and remediation.

Encryption of the profile data

The information in the profile can be encrypted using the encryption key feature.

Collect logs for Disconnected Hosts

The VMware Skyline Health Diagnostics provides a log collection from the disconnected hosts.

Upload and Analyze logs

Upload the logs from the VMware vCenter Server or VMware ESXi host or VMware Cloud Foundation SDDC Manager workload domain or VMware Horizon or VMware SD-WAN and analyze the log bundle for issues. Get the report of the identified issues and remediation.

Save the analysis inputs as Profile

Save the inputs provided for the analysis like VMware product to analyse, FQDN or IP address, user credentials, tag name and notification settings as a profile. These details can be encrypted using the key to keep the information secured.

VMware Security Advisory Scanning

Scan your infrastructure to know whether deployed VMware products have any security threat and to know the version of the product that has the fix for this threat, using VMware Security Advisory recommendations. This involves lightweight data collection for analysis, without the need to collect the entire log bundle.

VMware Cloud Foundation Upgrade Assessment supported on VMware Cloud Foundation release versions 4.5 or later only

You can perform the pre-upgrade assessment for VMware Cloud Foundation version 4.5 or later versions to the latest version before performing the upgrading, to know the issues impacting the upgrade and get recommendation to remediate it.

API Support

Added APIs support so that Administrator can use them in the automation script or solution. APIs are fully documented using OpenAPI 3.0 specifications. All operations except Profile Management are supported through API.

Show Reports

See the list of the last reports, download, and view it.

Scheduler

Schedule periodic health check to monitor the health status and ensure healthy and reliable infrastructure. 

Settings

Upgrade and Download History

• Upgrade History: See that the last five upgrades carried out with Source, Target version, start and end time, time is taken.

• Download History: See the last five download version, start and end time, time is taken.

Customer Experience Improvement Program

• Enable or disable the CEIP to share the data about the success or failure of the plug-ins and usage details with VMware.

• View the sample data collected.

• See Internet connectivity.

User Management

• You can add new users with the Operator role.

• As an shd-admin, you can reset the password or update user details like the name and the email ID.

Configuration

• You can now improve the performance of the analyze operation by modifying the timeout and indexer settings from Skyline Health Diagnostics user interface.

• You can now change the password expiration, lockout and allowed authentication failures count by modifying the respective settings from Skyline Health Diagnostics user interface.

• You can now limit the time duration of logs to analyze to the last few days from the log collection date, by using the configuration settings.

Email Notification Support

• You can now receive the report of Scheduled Health Check run over email using Notification Settings. Configure notification setting with SMTP server details to use this feature.

• Create distribution list by adding one or more users to get notified.

• Select one or more distribution list while adding the new run with notification enabled option.

Reset Password

• You can update the existing password before it expires, provide a current password and a new password.

Proxy Settings

If VMware Skyline Health Diagnostics deployment is behind the proxy, provide the proxy details to enable Internet access. You need to be connected to the Internet to download the latest signatures.

Help and Support

• Tools usage: Allow you to see the data Skyline Health Diagnostics collects under the CEIP program to improve the tool, you can see the data. If you are satisfied, either you can enable the CEIP data sharing with VMware or share the report through email with VMware.

​About

• Product version information

• Release date

Signatures Added for Detecting Issues 

vCenter Server Diagnostics

• vCenter server un-availability due to service failure

• vCenter Services un-availability 

• Certificate validation including PSC, Machine, STS, and Service

• Appliance space-related issues

• Password expiry

• VMware Compatibility validation for driver, firmware, and IO devices

• IO Device compatibility check for vSAN

• vCenter server Install/Upgrade/Migrate failures - partial

ESXi Diagnostics

• ESXi PSOD

• ESXi storage

• Hostd issues

• Networking related issues

• Path trashing

• Storage latency

• File system corruption

• CPU Microcode checks that require updates

• Deprecation check - Driver Compatibility on ESXi 6.x with respect to 7.0

• upgrade failure checks

• time sync related anomalies

• Host disconnect coverage – partial

• Host Upgrade failure – partial

vSAN Health Checks.

VMware Cloud Foundation Health Checks, Diagnostics and VMware Security Advisor issues using SDDC Manager.

VMware Horizon Diagnostics.

VMware SD-WAN Diagnostics.

Report on VMware Compatibility Guide

When the ESXi logs are analyzed the VCG plug-ins automatically detect the hardware and driver versions. Based on this version information it runs the analysis. The analysis report provides details about the existing hardware & driver versions, the latest available versions, and support status.

Note: The disk compatibility analysis is not covered in this version.

Report on VMware Security Guide

The new report layout now has a section "VMSA Finding" to provide a summary of security issues and recommendations using VMware Security Advisor.

Report on Diagnostics

The new report layout now has a section "Diagnostics Finding" to provide a summary of diagnostics problems and recommendations.

Report on Health Checks

The new report layout now has a section "Health Check Findings" to provide a summary of health issues.

Performance Improvements

The Skyline Health Diagnostics has improvements in user interface and performance optimisation.

Other Improvements

• Improved error messages and display on the UI for the failed downloads from external sites (Updates, VCG Data).

• Reports will display the summary section even if there were no findings, conveying the same.

• Support to force reset the shd-admin user password.

• Administrator can be extending the Administrative privileges to other users.

• Thumbprint based SSL Verification for all VMware products supported for analysis and diagnostics.

Unsupported Scenarios

VMware Skyline Health Diagnostics will address the following scenarios in future releases:

1. To collect the log bundle for disconnected encrypted ESXi hosts.

2. Analyse the crash dump for the encrypted ESXi hosts. 

3. Firmware validation in VMware Compatibility Guide Report.

4. VMware vCenter Servers in enhanced linked mode.

Known Limitations

1.  Only shd-admin can download the tools update.

2. Concurrent "Collect Logs & Analyze" the operation are limited to four.

3. The maximum number for targets (VMware ESXi hosts and VMware vCenter server) for each "Collect Logs & Analyze" operation is sixty-five (sixty-four ESXi hosts and one vCenter Server).

4. Uninstallation of the tool removes all historical data.

5. The maximum no. of reports shown is the last fifty.

VMware Compatibility Guide Report

VMware compatibility guide report is available as a fully functional feature. Manual verification is suggested. The following are known limitations:

1. To download the VCG and Tool Update, Internet connectivity is a must. No offline patches.

2. vSAN HCL checks are limited only to storage I/O device used with VMware vSAN

3. Firmware versions are not validated for VCG.

Known Issues

• Longer Analysis Time

  The host having non-default log configurations (with respect to size and number of files) mi.ght see a longer analysis time due to the number of logs.

  Workaround: None

• Analyzing vCenter Server as a target might not progress beyond the Generate Log Bundle stage.

  Collect Logs and Analyze --> Run Diagnostics with vCenter Server as a target might not progress beyond the Generate Log Bundle stage. The issue occurs due to the corresponding "Generate Log Bundles" task on the vCenter server is not completing.

  Workaround:

  (1) You can restart the Skyline Health Diagnostics appliance to cancel this pending analysis task and retry after some time.

  (2) You can use the timeout setting under VMware Skyline Health Diagnostics UI --> Settings Tab -->Configuration-->Log Generation Timeout; so that you can control the time duration of log collection.

• For some Internet Explorer browser versions, the header content might not be aligned properly.

  VMware Skyline Health Diagnostics supports the limited browsers and versions, please refer the section in release notes for supported browsers.

  Workaround: None

• Skyline Health Diagnostics user interface might show a generic error [Object Object]

  VMware Skyline Health Diagnostics user interface might show a generic error [Object Object] while downloading the online update bundle for few cases while updating from 2.5.0 to 2.5.1, user can ignore it and proceed.

  Workaround: None. You can ignore it and proceed.

Resolved Issues

• Fixed Critical Security vulnerabilities issues for OpenJDK.

  The OpenJDK version is updated to 1.8.0_382

  This issue is fixed in 4.0.1 release and onward.

• Running Analysis workflow with only 'vSAN Health Check Plug-in' selected may show report with no VSAN Summary section.

  Running Analysis workflow with only 'vSAN Health Check Plug-in' selected may show a report with no VSAN Summary section in case no plug-ins matched. There is a functional issue caused by this issue.

  This issue is fixed in 2.5.1 release and onwards.

• Old reports may not be seen categorized

  Once you click on SHow Reports -> Filter to Products or components may not work on old reports. This is due to the new filtering changes that can not be applied to old reports.

  This issues is fixed in 2.5.1 release and onwards.

• Auto update may fail with the error.

  This is seen in scenarios where the update check is done again before the downloaded update package is applied.

  Fix : This issue is fixed in 2.5.1 release and onwards.

  Workaround: Download the updates again and reboot the VM immediately.

• You might want to run multiple logs analysis in parallel, against the same or multiple vCenter using the same SHD instance. This operation might fail sometimes. 

  This issue is fixed for 2.5.1 release and onwards.

• Log bundle collecting from direct ESXi connect might fail in some scenarios.

  In some scenarios where you are connecting to ESXi for diagnostics, the workflow can fail at the first stage of the log bundle generation. This issue is fixed for the 2.0.2 release and onwards.

• Proxy server configuration with authentications mat not work as expected.

  Navigate to Proxy Setting under the Settings tab. Provide Proxy authentication details. Try to download VCG signature,  the operation will fail.

  The log files under /opt/vmware-shd/vmware-shd/log/vmware-shd-support* , may show following error.

• Using FQDN to connect to vCenter Server or ESXi host failed with the "Lookup timed out" error.

  To collect log bundle for analysis of the issue, you may provide a vCenter Server or ESXi host FQDN. But the operation will fail as the DNS name resolution was timing out. This will fail the  FQDN based connection to vCenter Server or ESXi host. This issue also caused failures for checking and downloading updates. This issue is fixed for the 2.0.2 release and onwards.

• The operation to update VMware Compatibility Guide (VCG) might fail.

  VCG update was failing due to the high number of parallel requests to download the VCG information.  This issue is fixed for the 2.0.2 release and onwards.

• VMware Compatibility Guide(VCG) /Software Updates failed in case of the proxy server with MITM configuration is used.

  VCG and Software updates failed in cases proxy server with MITM (SSL termination at Proxy). This issue is resolved in the 2.0.4 version and onwards.

• VMware Compatibility Guide Update resulted in failure of compatibility checks

  The VCG Update resulted in invalid data being updated to the database due to server-side issues. This issue is resolved in version 2.0.5 and onwards.

• The VMware Compatibility Guide Update failures did not correctly propagate the error to the user.

  The VCG update failures with error descriptions were not getting propagated to the user. This issue has been fixed in version 2.0.5 and onwards.

• When you are Changing the password age (account or passage), it does not correctly reset the expiry date for the existing passwords

  When you are Changing the password age (account or passage). The change in password age was not correctly updated in the user record. This issue has been fixed in version 2.0.5 and later.

• You might observe the Analysis workflow stalled when multiple log bundles are downloaded from ESXi or vCenter.

  This issue is resolved in version 2.0.5 and later.

• While collecting the log bundle from the ESXi host over SSH, the Connections to ESXi might not timeout in case of failures.

  This issue has been fixed in the release 2.0.5 and later.

• The Product names appearing in the report were inconsistent.

  Product name in the report appeared differently depending on the data source available. The issue is fixed for the 2.50 release and onwards.

• vCenter Server build/release detection could fail in a few cases thus raising false match.

  vCenter release detection was incorrect for some vCenter builds. The issue is fixed in the 2.5.0 release and onwards.

• VCG Check incorrectly tagged NVMe of FC devices.

  NVMe Over FC devices were not correctly validated. The issue is resolved in the 2.5.0 release and onwards.

• The vSAN HCL check failed in cases due to case sensitive search.

  A case-sensitive search for a device on the HCL could fail to match the device. The issue is resolved in the 2.5.0 release and onwards.

• Online update of SHD failed with "Read Manifest failed" error

  In some cases, the online update of SHD failed due to a conflict between check and update. The issue is fixed in the 2.5.0 release and onwards.

• Parallel vCenter Server Appliance analysis could fail.

  Running more than one vCenter Server Appliance diagnostics resulted in failure of all or one workflow. The issue fixed in the  2.5.0  release and onwards.

• Running Diagnostics on some ESXi servers fails either at Log Collection or Analysis phase

  On servers running ESXi Server 6.5, diagnostics workflow may fail. These servers did no product unique filenames for log bundle which caused the failure. This issue has been fixed with release. This issue has been fixed in version 3.0 and above.

• VCG Compatibility check might show supported hardware as “Not Supported” for ESXi Server 7.0 Update 3

  Due to inconsistency in VCG data, a supported server/IO device  would have shown as “Not Supported” for ESXi Server 7.0 Update 3. This issue has been resolved in version 3.0 and above.

• Certain devices might have been appearing as not supported/not checked.

  Certain devices might have been appearing as not supported/not checked due to error in VCG Data. This issue has been fixed in version 3.0.1 and above.

• SSH Server/Client Hardening settings were missing from 3.0.0 release when online/offline patching was used.

  SSH Server was configured to accept only strong Ciphers with release of version 3.0.0. Using offline/online updates did not update the settings on the existing installation. TThis issue has been fixed in version 3.0.1 and above.

• Generating Analysis report at Log Directory/Bundle level failed

  Generating analysis report at Log Directory/Bundle level failed due to incorrect handling of input parameters. This issue has been resolved in 3.0.1 and above.

• Skyline Health Diagnostics user interface does not display the buttons on Diagnostic Run Wizard and user can not schedule the diagnostics run

  When you update to Skyline Health Diagnostics to version 3.0.1 and try to execute the diagnostics or heath run. The wizard does not display the button to submit the run. This is happening due to the changes in the User interface style sheet ( CSS ) in Clarity Design Package V 13.0.3 released on March 18, 2022.Skyline Health Diagnostics consumes this version and hence got impacted. This issue is resolved in Skyline Health Diagnostics version 3.0.2. You are advised to update to version 3.0.2

• VMware Compatibility Guide Validation for certain Multi-Function IO devices were incorrect.

  When validating a Multi-Function I/O device, incorrect device id was used for validating other functions provided by same adapter. For example a device with both Network and iSCSI Function might not be correctly validated. This release has fix to address this. This issue has been resolved in version 3.5.1 and above.

• Nginx updated to Version nginx-1.22.0-1.ph3 (PHSA-2022-0481)

  Nginx has been updated to Version 1.22.0-1.ph3. Please read https://github.com/vmware/photon/wiki/Security-Update-3.0-481to find more details. This issue has been resolved in version 3.5.1 and above.

• For VCF version 4.4 and 4.5, Skyline Health Diagnostics report shows critical warning if SSH service is disabled.

  On VCF version 4.4 and 4.5, VMware has disabled the SSH service by default. Skyline health diagnostics was flagging , SSH disablement as critical warning. This issue has been fixed in Skyline Health Diagnostics version 3.5.2 and now SSH disablement will be informed to user as info level issue. This is fixed in version 3.5.2 and onwards.

• Scheduler can not be activated as the passphrase save button remains grayed out.

  When you are trying to activate the scheduler and inputs the passphrase with alphanumeric and special characters, SAVE button remain greyed out. This issue fixed in Skyline Health Diagnostics version 3.5.2 and onwards.

• VMSA Summary table has empty KB column.

  In the report, VMSA section displays the security vulnerability for the product, but it does not display the KBs helping to remediate it. This issue fixed in version 3.5.2 and onwards.

• SHD VC plugin view, selecting the dark theme Next button remains disabled after selecting inventory

  When you perform following steps,

  1. login into vc plugin view

  2. switch to dark theme

  3. select vsan or diagnostic category

  4. select inventory

  The Next button remains disabled after inventory selection.

  This issue is fixed in version 4.0.3 and onwards.

  None

• During the download of upgrade payload the user error displayed for users having session with old password or deleted user.

  Once user password is changed or user account is deleted, and if such user is still have the active session, during the download of upgrade payload the user error is displayed. Issue is such user session is still valid after password is changed or user account deleted.

  This issue is fixed in version 4.0.1 and onward.

• SMTP Configuration was not handled correctly on upgraded instances.

  When you try to configure the SMTP server Skyline Health Diagnostics could not determine if its a administrator user or operator user. This leads to less privileged users, like operator, modifying the SMTP configuration that would not be desirable.

  This issue is fixed in version 4.0.1 and onward.