504 errors can occur while upgrading pools on Avi Load Balancer using the patch API.
Reason
Timeout errors are observed when the logs are checked after the upgrade failure. When the timeout value is increased for the pools, 504 errors are not observed.
When more than 20 patch upgrade requests per min are made to the same API object, it can result in increased processing time for the patch calls. This eventually exceeds the default timeouts set for pools and virtual services and results in the 504 status being sent to the client. Timeout errors are observed as the number of requests was beyond the limit that the Avi Load Balancer patch API can handle.
Resolutions
Use the following steps to resolve 504 errors while upgrading using Patch:
The rate of update API per minute on a particular pool must not exceed 20.
Increase timeout values for virtual services and pools. Check logs on the Avi Load Balancer to determine the new timeout values for the virtual services and the pools.