VMware's Tanzu RabbitMQ Intra-cluster Compression feature is where RabbitMQ nodes form clusters and then try to use compression.
Note: This feature is only supported in Tanzu RabbitMQ. It is not supported in the RabbitMQ open-source product.
If you want to find out more information about clustering, go to the Clustering Guide.
RabbitMQ nodes communicate with their peers and CLI tools using dedicated TCP connections, optionally protected with TLS.
In heavily loaded system, inter-node traffic flows can be substantial, approaching or even saturating the bandwidth provided by network links. Compression of this traffic helps reduce the load on bandwidth available, up to 96%, depending on the nature of the workload.
How to use Intra-cluster Compression
Intra-cluster compression is turned off by default in Tanzu RabbitMQ. To turn on Intra-cluster compression, set the environment variable RABBITMQ_SERVER_START_ARGS="-proto_dist inet_tcp_compress". When RabbitMQ nodes that have this environment variable set form a cluster, these nodes will try to use compression.
For the data to be compressed, the following conditions MUST be met:
Both RabbitMQ nodes must support intra-cluster compression. In other words, both nodes must run Tanzu RabbitMQ. The open source edition does not support this feature.
Both nodes must share at least one compression algorithm in common.
If the conditions are not met, nodes will communicate as they otherwise would without traffic compression. This means that Tanzu RabbitMQ remains compatible with the open source RabbitMQ edition. It is therefore possible to switch from one edition to the other without stopping the entire cluster.
How Intra-cluster Compression Works
The first time a RabbitMQ node tries to contact another node, the following happens:
After the TCP connection is open, compression is not used at first.
The node initiating the connection detects if the remote peer has Erlang distribution compression support. If it does not, then the connection remains uncompressed and the remaining steps are skipped.
When the RabbitMQ node knows the remote node supports compression, it negotiates the compression algorithm to use. To start this process, it sends a message to the remote node and specifies the list of algorithms it supports.
The remote node compares the received list of algorithms to its own list. The remote node's list is ordered by preference. The selected algorithm is the first one in the remote node's list which is also supported by the initiating node. If there is no algorithm in common, the connection remains uncompressed and following steps are skipped.
Once an algorithm is selected, the remote node sends a message back to the initiating node to inform it of its decision.
The two nodes synchronize to start compression on the existing TCP connection.
How to get a License
To get a license for Tanzu RabbitMQ products, fill out the Tanzu RabbitMQ support contact form and we will get back to you with a tailored quote.
Limitations
Intra-cluster compression and TLS can't be used at the same time: they are mutually exclusive at the moment.
Why can't Intra-cluster compression and TLS be used at the same time? because the RabbitMQ node is configured with a specific distribution module provided with VMware's RabbitMQ, plus a small add-on to deal with the algorithm negotiation. The distribution module is a replacement to the default module (
inet_tcp_dist) or the TLS-enabled one (inet_tls_dist). It is impossible to use two modules simultaneously.In Tanzu RabbitMQ, the compression code relies on native libraries. Only Linux/amd64 is supported in the existing packaging. We might compile for more platforms in the future.
