The 470 errors are observed in NSX Advanced Load Balancer logs even though the pools and servers are in UP state during the specific duration. This topic details the steps to troubleshoot and resolve this error.
The following 470 error can be observed:
Request ended abnormally: no available servers
This occurs only for the virtual services which have content-switching policies attached to them. Virtual service logs exhibit the following behavior:
The failed POST requests hit the HTTP Request policy and also the network security policy, but the NSX Advanced Load Balancer cannot send that traffic to back-end servers as specified in the content switch pool.
The request data is similar in failed and working scenarios as seen from the virtual service logs and also from the Wireshark capture.
Resolution
This issue occurs when the content-switching is used, but connection multiplexing is disabled for the virtual service.
To resolve the issue, enable connection multiplexing for the associated content-switching virtual service.
Reason for Error
Content-switching (with connection multiplexing off) essentially means that the front-end and back-end connections are fixed. When the first request on the front-end connection arrives, a server is picked based on applicable policy or other settings and a connection is established to the server. Once the back-end connection is established, the FE and BE connections are bound. If at a later point, another request comes in and due to a policy picks another pool, the request cannot go on the bound back-end connection and needs a new connection to be established, which breaks the notion of connection switching. As a result, content-switching will not work with connection-multiplexing off.
On some instances, the request is the first one on that front-end connection, without any existing back-end connection and so might work even when content-switching is used.
Observations
Following are few points that were observed during troubleshooting:
From safari browser, it is observed that when you first go to your website hosted on the NSX Advanced Load Balancer and try to login, it fails with error login failed. The following errors were observed in the virtual service logs:
Req Policy Rule: redirect_apiNetwork Security Rule: permit1Significance:Request ended abnormally: no available serversRequest ended abnormally: response code 4xx.,
but if the browser is refreshed once or twice, it works.
From the Chrome browser, the same pattern is observed but the initial 2 POSTs fail. The one which works comes automatically and you do not have to refresh the chrome browser.
The back-end servers are stable.
From the packet capture, it is observed that the NSX Advanced Load Balancer is receiving the POST request, but closes the connection and does not initiate the connection to the back-end server.
When the browser is refreshed, it works. When another similar POST request comes, the NSX Advanced Load Balancer opens the connection to the back-end server.
