This section contains the list of new features, issues resolved, key changes, and known issues for 22.1.5 release.

Patch Release Notes for 22.1.5

22.1.5-2p7
Release Date: 10 September 2024

  • AV-211473: The default TACACS login timeout was too short, causing multi-factor authentication failures. The default TACACS login timeout has now been increased.

22.1.5-2p6
Release Date: 03 June 2024

  • AV-206581: Using a variable in avi.pool.select() fails to identify the pool during a virtual service update.

  • AV-206573: When creating pools and adding servers, the UI automatically includes all configuration fields, while the API allows pools to be created without certain values, such as the PRST key. This inconsistency can result in pools where some servers have the PRST key and others do not.

22.1.5-2p5
Release Date: 29 March 2024

  • AV-198913: Using HTTP Response Policy or HTTP Response DataScript to replace the Content-Type header with "charset" directive present leads to an incomplete rewrite of the header.

  • AV-198989: When both WAF and Thales HSM are yenabled, `se_dp` processes can fail.

  • AV-192778: In Docker-based Controllers, during reboot upgrade workflows, DNS configuration is not being persisted because the system configuration script was referring to a dynamically generated configuration file instead of `/etc/systemd/resolved.conf`.

22.1.5-2p4
Release Date: 13 March 2024

  • AV-194438: When health monitor sharding is enabled for GSLB services, an unsolicited shard state update from the control plane to the service engine performing data plane health monitoring may incorrectly mark the GSLB service member as down.

  • AV-195157: DNS resolution is affected due to incorrect GSLB status being synced across all the sites.

22.1.5-2p3
Release Date: 13 February 2024

  • AV-197046: IPAM allocation for A records with multiple subnets will fail when the first subnet is exhausted.

  • AV-197350: When the connection gets reset by rsyslog server, the SE log agent buffer becomes full and the logs are not streamed even after the connection is restored.

  • AV-198972: Missing multi-queue support for UDP

    Tx and NAT flows.

22.1.5-2p2
Release Date: 19 January 2024

  • AV-185059: CSR certificates managed through the certificate management profile get stuck in a renewal loop, leading to repeated renewal attempts every few seconds and resulting in anticipated failures.

  • AV-185882: Unable to update secure channel root certificate when the cloud is not No Orchestrator or SEs are running in the system.

  • AV-192901: Updating passwords in vCenter can transition the Avi vCenter cloud to failed state.

  • AV-193663: Metrics Manager’s database connections with Postgres are unclosed, causing a connection leak.

  • AV-195223: Name resolution on the Controller fails.

  • AV-195595: External log streaming to a servers or load balancers which erroneously responds to simplex log stream causes Service Engine memory growth, eventually leading to SE crash.

  • AV-195716: Although licenses are available in Pulse, changing the Bandwidth Type of SE Group in the Cloud Services tier failed.

  • AV-196914: VSVIP objects having same the IP address may cause SE to fail.

22.1.5-2p1
Release Date: 06 December 2023

  • AV-188363: In LSC hosts, when configuring Mellanox devices in combination with Broadcom components, the ring size computation logic can cause initialization errors and stall the SE during connection to the Controller.

  • AV-190126: Using Broadcom NIC as management with Mellanox Nic for datapath causes issues in bringing up the NIC.

  • AV-190461: Frequent updates to string groups attached to a DataScript, that also makes repeated calls to avi.stringgroup functions may result in failures in StringGroup lookups.

  • AV-190475: se_dp crash happens with GRO code signature in stack trace, in a rare case scenario.

  • AV-191149: Objsync may cause memory build-up and might lead to Out-of-Memory eventually on the SE caused by objsync peer connection failures either due to port 9001 or 4001 not being open in DFW in nsx-t or no management plane connectivity between SEs.

  • AV-192083: Failure in Objsync connection over management interfaces between SEs might lead to memory exhaustion.

  • AV-192951: Unable to use Infoblox DNS and Infoblox IPAM profiles when they are handled by different Infoblox instances.

  • AV-192508: When an IPv4 address is added while the IP address type is set to IPv6 for Pool-Servers, it leads to se_agent crash.

  • AV-193221: Missing support for Outbound NAT with source port preserved for UDP flows.

What’s New in 22.1.5

To refer to the upgrade checklist, see Checklist for Upgrade to NSX Advanced Load Balancer Version 22.1.5.

  • An application level timer is introduced to ensure DNS requests close within a defined time limit, thus preventing any kind of Slowloris attacks on the DNS virtual service.

Issues Resolved in 22.1.5

  • AV-127214: SE failure due to incompatibility in hardware versions for LSC deployments on VMware ESXi VMs.

  • AV-171793: Intermittently, virtual service logs may not load or exhibit delay in loading.

  • AV-179018: Service Engines might not get placed in configured datastore in Service Engine Group if content lib is enabled in cloud configuration.

  • AV-179167: False alerts stating, 100% of total licensed Service Engine service cores used., displayed when license consumption is greater than the license capacity of the recently added license unit.

  • AV-179869: When a GSLB service is configured to return all the records if it is down, if this GSLB service has multiple CNAME records in it, only one of CNAME records will be included in the ‘down’ response.

  • AV-179893: A discrepancy between the timeline of federated queue and the timeline used during the subscribe operation, triggers the repetitive cycle of Sync and Subscribe operations, resulting in high bandwidth utilization.

  • AV-179916: Replication from the leader site to follower site stalls when a file fails to download even if the subsequent downloads are successful.

  • AV-180062: The IP Address/ FQDN field under Client Logs in Analytics Profile does not accept hostnames as valid input through the UI.

  • AV-180173: When HTTP Cookie Persistence is used, and there are longstanding connections, and if the virtual service configuration is changed, then for the subsequent requests over the connection, the persistent cookies are not honored, and a different backend server can get selected.

  • AV-180535: In virtual service logs, the location of origin of the Client IP address is unavailable through the UI and DataScripts.

  • AV-180654: WAF PSM duplicate Rule ID generated owing to the number of URI params restricted to 10,000.

  • AV-181710: If a virtual service is in a fault state due to issues with a WAF policy, and if this WAF policy has Positive Security Model (PSM) groups configured, and if these groups were updated after the WAF policy entered the fault state, then deleting the WAF policy can cause SE failure.

  • AV-181723: Unable to assign an SNAT IP to an SNI parent virtual service that is attached to a content switching rule pool.

  • AV-181805: Issue with accounting related to memory management in the Controller for memory held in buffers and caches.

  • AV-181840: Security Manager failure when DNS servers are either not configured or not reachable.

  • AV-182114: When the SEs are created with insufficient licenses, the NSX Advanced Load Balancer UI shows that the SE is enabled, when the SE is in the disabled state. On clicking -DISABLE, the SE is stuck and displays the error message, Cannot change state since disable operation is in progress.

  • AV-182499: In the DPDK mode, NSX Advanced Load Balancer does not support the NIC model used by the host. As a result, the traffic for the VLAN interface configured with the Mellanox interface fails to work.

  • AV-182702: The Prometheus-metrics API endpoint intermittently provides empty responses within a one-hour timeframe.

  • AV-182827: Updating credentials in vCenter Cloud through the UI fails.

  • AV-182830: L4 SSL DataScripts with collect API in the request or response events may cause SE failure.

  • AV-183138: Long requests with SAML authentication can cause SE failure.

  • AV-183400: HTTP request header size greater than 4K with ICAP deployment enabled can cause Service Engine failure.

  • AV-183885: If an HTTP/1.0 header arrives without a host header, which is NULL (permissible in HTTP/1.0), and this header is internally processed for comparison with GS domain names, it results in SE failure.

  • AV-184154: vNICs in a No-Access setup had is_avi_internal set to True, causing VNIC IP updates to not persist.

  • AV-184189: The cloud name changes to “TRUSTED” or “UNTRUSTED” in the GUI instead of the actual name.

  • AV-184284: Duplicated network names in the UI cause inability to uniquely identify a network.

  • AV-184734: NSX Advanced Load Balancer AWS S3 backup failing when using only IAM roles with S3 bucket permissions.

  • AV-184809: In the NSX Advanced Load Balancer UI, the message “No pools configured” is displayed although the pool has pool groups configured under it.

  • AV-184853: Disabling the virtual service having VIP as SNAT configuration when two virtual services are sharing the same VIP, can leave the other VS non-functional.

  • AV-185279: Unable to edit a Cloud of type GCP in the UI if the optional Routes field is missing.

  • AV-185506: If an NXDomain DoS attack is detected, the Service Engine may experience memory leakage.

  • AV-180910: When a GSLB service member is configured as FQDN and GSLB service is using external health monitors, member FQDN will be passed as an environmental variable to external health monitor script.

  • AV-181918: After upgrading to 22.1.3-2p4, the user-defined cloud name changes to a default vCenter cloud name.

  • AV-182892: AWS cloud-specific information is not displayed in the Clouds page (Infrastructure > Clouds) in the NSX Advanced Load Balancer UI.

  • AV-185604: When configuring a TCP request for a health monitor of type TCP with user-defined settings including get or post strings, the system automatically appends HTTP/1.0 and \r\n\r\n to the TCP request.

  • AV-186355: DNS resolution for pool FQDN may result in failure when the response is big enough (greater than 512 bytes) to trigger the resolution to happen through TCP transport.

  • AV-186806: Service Engine fails during a pool update followed by the deletion and reconfiguration of a child virtual service.

  • AV-186925: Service Engine might fail when it receives traffic in which the Ethernet header and the other headers arrive in separate packets.

  • AV-187052: CRS overrides are not being added when the CRS version is updated in the WAF Policy modal.

    Workaround: Save the uncommitted WAF Policy settings before updating.

  • AV-187301: If a virtual service is in a fault state due to issues with a WAF policy, and if this WAF policy has Positive Security Model (PSM) groups configured, and if these groups were updated after the WAF policy entered the fault state, then deleting the WAF policy can cause SE failure.

  • AV-187301: When incorrect credentials are provided to the Avi Terraform provider, it initiates the creation of resources that have already been created.

  • AV-187523: Configuration replication does not work for uncommon federated objects between leader and follower if they belong to different versions.

  • AV-187919: SE failure when client sends an invalid HTTP/2 header.

  • AV-188363: In LSC hosts, when configuring Mellanox devices in combination with Broadcom components, the ring size computation logic can cause initialization errors and stall the SE during connection to the Controller.

  • AV-188419: In application logs the location of origin of the Client IP address is unavailable through the UI and DataScripts.

  • AV-188464: Modifying the pool configuration through the GUI on an NSX-T cloud with Security Groups as Server definitions can lead to the removal of pool members until the next discovery sync occurs. This issue occurs even when the existing pool configuration is not modified, but just saved via UI.

  • AV-188919: If the vm_uuid file is edited or saved manually, it can result in the generation of an extra newline at the end, which may lead to image upload failures. These failures can potentially be attributed to host resolution issues.

  • AV-189340: The se_log_agent fails with the error message, “SE crashed with fatal error” for external log streaming over TCP/TLS.

Key Changes in 22.1.5

  • In case of LSC deployments on VMware ESXi VMs, the hardware compatibility version is 11 or earlier.

  • Support for TCP Proxy Protocol (Enable Proxy Protocol is selected) when an L4 application profile is used as an override application profile.

  • If a GSLB service member is monitored by multiple sites through a health monitor proxy, in the sites that rely on remote status from the health monitor proxy sites, the member will be marked UP if at least one health monitor proxy site reports the status as UP.

  • When a GSLB service member is configured as FQDN and GSLB service is using external health monitors, member FQDN will be passed as an environmental variable to external health monitor script.

  • Starting 08 May 2023, some NSX editions include NSX Advanced Load Balancer Enterprise with a ratio of 1 NSX Advanced Load Balancer unit per 250 NSX CPU cores. Starting with version 22.1.5, the license keys generated as part of the specified entitlements will be recognized and decoded natively by the Controller. To learn more about the specific editions in which these entitlements are included, see NSX Editions and Feature Guide.

  • Version 22.1.5 supports Enhanced Datapath mode for NSX. Enhanced Datapath mode must be selected while preparing the ESXi hosts as Transport Nodes. NSX Advanced Load Balancer seamlessly adapts to the Enhanced Datapath mode in NSX. It is recommended to use the ENS interrupt mode for better performance. See NSX documentation for different modes and pre-requisites for more information.

Known Issues in 22.1.5

  • AV-187931: When System-SCTP-Proxy TCP/UDP Profile is selected as network profile for virtual services, a port range cannot be specified under Service Ports. If a port range is configured, only the first port within the specified range handles traffic.

  • AV-190003: High CPU utilization may be observed in NSX-T based cloud environments (check using the show cpuusage controller command).

    • Workaround: For optimal CPU utilization by cloud connector routines/processes on the system, use the configuration shown below:

configure cloud <cloud_name>
autoscale_polling_interval 600
configure controller properties
cloud_reconcile_interval 10
cloud_discovery_interval 15
attach_ip_retry_limit 1
vs_se_attach_ip_fail 240
detach_ip_retry_limit 1

Checklist for Upgrade to NSX Advanced Load Balancer Version 22.1.5

Refer to this section before initiating upgrade.

  • Upgrade to NSX Advanced Load Balancer to 22.1.5 is only supported from the following versions:

    • Version 20.1.1 through 20.1.9

    • Version 21.1.1 through 21.1.6

    • Version 22.1.1 through 22.1.4

    • Starting with NSX Advanced Load Balancer version 22.1.3, the minimum memory requirement for Service Engines is increased to 2GB. Before upgrading to any version in the 22.1.x release, ensure the Service Engines are configured to a capacity greater than 2 GB. The current considerations for memory sizing as listed under Sizing Service Engines in the VMware NSX Advanced Load BalancerConfiguration Guide continue to apply.

      For more information on flexible upgrades, see Upgrade Overview in the VMware NSX Advanced Load BalancerAdministration Guide.

  • Before upgrading to version 22.1.2 and higher, export the Avi metrics database. In case of rolling back from NSX Advanced Load Balancer 22.1.2 to an earlier version, import the metrics database to prevent loss of metrics data.

    For more information, see FAQs on Controller Cluster in the VMware NSX Advanced Load BalancerAdministration Guide.

  • Starting with NSX Advanced Load Balancer version 22.1.3, the minimum memory recommended for an Essentials Controller is 24G. Ensure that the memory of an Essentials Controller is at least 24G before upgrade.

  • The ControlScripts framework has been updated. This requires the ControlScripts to be modified prior to upgrade or on upgrade.

    For more information, see the Scripts topic in the VMware NSX Advanced Load BalancerConfiguration Guide.

  • As mentioned in the Key Changes, starting with NSX Advanced Load Balancer version 22.1.1, there is an enforcement on the string length in the name field for all objects. Use the script available here to identify all the objects that exceed the name length threshold. Ensure that the object names are modified before upgrading.