Check Failed Error during Service Engine Upgrade

The NSX Advanced Load Balancer SE upgrade can fail with the error Check failed. This topic covers the troubleshooting and resolution steps for this error.

The following errors were observed in the SE log:

[127.1.0.3] Executing task 'se_upgrade_check'
cat: /host/prev/bootstrap/VERSION: No such file or directory
se_upgrade_check error exception: Error reading SSH protocol banner

The following error was observed in the secure_channel.log (in /var/log/upstart in SE CLI):

AVI-SE:/var/log/upstart
Authenticated to 10.1.1.1 ([10.1.1.1]:22).^M

connect_to localhost: unknown host (Name or service not known)^M

connect_to localhost: unknown host (Name or service not known)^M

connect_to localhost: unknown host (Name or service not known)^M

connect_to localhost: unknown host (Name or service not known)^M

connect_to localhost: unknown host (Name or service not known)^M

Resolution

The errors observed in the SE log file suggest that the entry for the localhost mapping to 127.0.0.1 is missing in /etc/hosts. This issue occurs when name resolution for the local host from SE is not successful as shown in the following snippet.

root@app-node3:/var/log/upstart# ping localhost

ping: unknown host localhost

Check /etc/hosts file by using cat command. Note that the localhost entry was missing as shown below.

app-node3.avi-systest.local-avitag-1root@app-node3:/# cat /etc/hosts

127.0.0.1 app-node3.avi-systest.local--nameless-abc-xyz

#used by abc_servers: node1.controller.local

{127.0.0.9, 10.140.88.197}

127.0.0.9 node1.controller.local

#used by abc_servers: node2.controller.local {127.0.0.8, 10.140.88.199}

127.0.0.8 node2.controller.local

#used by abc_servers: node3.controller.local {127.0.0.7, 10.140.88.198}

127.0.0.7 node3.controller.local

root@app-node3:/#

To solve this issue, add the localhost entry 127.0.0.1 localhost to the localhost file. After adding the localhost entry, try again upgrading the SEs.

Additional Troubleshooting Tips

Usually, /etc/hosts content on the root1 folder has this entry. From the NSX Advanced Load Balancer Controller, run the following command:

ssh -i /etc/ssh/id_se [email protected]

If ssh is successful, no issue is observed with the localhost entry.

Run the following command:

ssh -i /etc/ssh/id_se -p 5097 [email protected]

If ssh fails, add an entry to /etc/hosts file with the localhost 127.0.0.1 and retry ssh.

If ssh is successful, retry upgrading NSX Advanced Load Balancer SEs.