Running RabbitMQ on Amazon EC2

Overview

This guide assumes familiarity with the general clustering guide as well the guide on cluster peer discovery.

Using RabbitMQ on EC2 is quite similar to running it on other platforms. However, there are certain minor aspects to EC2 that need to be accounted for. They primarily have to do with hostnames and their resolution.

This guide demonstrates manual (CLI-driven) RabbitMQ clustering. Peer discovery plugin for AWS (RabbitMQ 3.7.0 or later) is an option more suitable for automation.

AMIs

RabbitMQ works well on up-to-date Ubuntu, Debian and CentOS AMIs as long as a compatible version of Erlang/OTP is installed.

Choosing an Instance Type

RabbitMQ will work on every instance type, but there are a few considerations worth bearing in mind:

Use 64-bit instances.
Depending on the workload and settings, RabbitMQ can require substantial amounts of memory. Make sure that your host does have an appropriate amount of RAM and always have at least a few gigabytes of swap space enabled. Workloads can be simulated using PerfTest. A separate guide on reasoning about node memory usage is available.
RabbitMQ generally will take advantage of all the CPU cores in the system provided the workload uses multiple queues. Other factors should be taken into account (e.g. disk and network I/O throughput).
Monitoring RabbitMQ nodes as well as applications that use it on real or simulated workloads will help assess how suitable a particular instance type is.

Operating Systems

Although RabbitMQ is tested with most major Linux distributions, Ubuntu support for Amazon EC2 seems to be strongest, so that's the distribution this guide will use.

Ubuntu Cloud Images provides access to Ubuntu images (builds) specifically designed to be used in public clouds.

Installation

Please consult the installation guides for

A wide variety of deployment tools can be used to automate RabbitMQ deployment.

are some popular options.

Durable Storage on EBS Volumes

On Linux, RabbitMQ will use the following directories for its node data directory:

/var/lib/rabbitmq/ to store persistent data like the messages or queues
/var/log/rabbitmq/ to store logs

See the File and Directory Locations for details.

Those directories can be symlinks to a dedicated storage volume. The node must be stopped before symlinking is performed:

sudo service rabbitmq-server stop

We recommend performing symlinking and other storage preparation steps before installing RabbitMQ when possible.

Note that EBS volumes have an IOPS limits, which can impact throughput and RabbitMQ operations. If an EBS volume hits the limit, disk writes will worsen. It is also possible that the RabbitMQ message store compaction (garbage collection of on-disk data) can fall behind disk writes, which means the disk will be filled up quicker than disk space can be reclaimed after messages were consumed and acknowledged. This will eventually lead to resource alarms and publisher throttling. Please make sure the limit is high and set up I/O operation rate monitoring.