This guide provides an overview of queues in RabbitMQ. Since many features in a messaging system are related to queues, it is not meant to be an exhaustive guide but rather an overview that provides links to other guides.
This guide covers queues primarily in the context of AMQP 0-9-1, however, much of the content is applicable to other supported protocols.
Some protocols (e.g. STOMP and MQTT) are based around the idea of topics. For them, queues act as data accumulation buffer for consumers. However, it is still important to understand the role queues play because many features still operate at the queue level, even for those protocols.
Streams is an alternative messaging data structure available in RabbitMQ. Streams provide different features from queues.
A queue is a sequential data structure with two primary operations: an item can be enqueued (added) at the tail and dequeued (consumed) from the head. Queues play a prominent role in the messaging technology space: many messaging protocols and tools assume that publishers and consumers communicate using a queue-like storage mechanism.
Queues have names so that applications can reference them.
Applications may pick queue names or ask the broker to generate a name for them. Queue names may be up to 255 bytes of UTF-8 characters.
Queue names starting with "amq." are reserved for internal use by the broker. Attempts to declare a queue with a name that violates this rule will result in a channel-level exception with reply code 403 (
In AMQP 0-9-1, the broker can generate a unique queue name on behalf of an app. To use this feature, pass an empty string as the queue name argument: The same generated name may be obtained by subsequent methods in the same channel by using the empty string where a queue name is expected. This works because the channel remembers the last server-generated queue name.
Server-named queues are meant to be used for state that is transient in nature and specific to a particular consumer (application instance). Applications can share such names in message metadata to let other applications respond to them (as demonstrated in tutorial six). Otherwise, the names of server-named queues should be known and used only by the declaring application instance. The instance should also set up appropriate bindings (routing) for the queue, so that publishers can use well-known exchanges instead of the server-generated queue name directly.
Queues have properties that define how they behave. There is a set of mandatory properties and a map of optional ones:
Note that not all property combination make sense in practice. For example, auto-delete and exclusive queues should be server-named. Such queues are supposed to be used for client-specific or connection (session)-specific data.
When auto-delete or exclusive queues use well-known (static) names, in case of client disconnection and immediate reconnection there will be a natural race condition between RabbitMQ nodes that will delete such queues and recovering clients that will try to re-declare them. This can result in client-side connection recovery failure or exceptions, and create unnecessary confusion or affect application availability.
Before a queue can be used it has to be declared. Declaring a queue will cause it to be created if it does not already exist. The declaration will have no effect if the queue does already exist and its attributes are the same as those in the declaration. When the existing queue attributes are not the same as those in the declaration a channel-level exception with code 406 (
PRECONDITION_FAILED) will be raised.
Optional queue arguments, also known as "x-arguments" because of their field name in the AMQP 0-9-1 protocol, is a map (dictionary) of arbitrary key/value pairs that can be provided by clients when a queue is declared.
The map is used by various features and plugins such as
and so on.
Most optional arguments can be dynamically changed after queue declaration but there are exceptions. For example, queue type (
x-queue-type) and max number of queue priorities (
x-max-priority) must be set at queue declaration time and cannot be changed after that.
Optional queue arguments can be set in a couple of ways:
The former option is more flexible, non-intrusive, does not require application modifications and redeployments. Therefore it is highly recommended for most users. Note that some optional arguments such as queue type or max number of priorities can only be provided by clients because they cannot be dynamically changed and must be known at declaration time.
The way optional arguments are provided by clients varies from client library to client library but is usually an argument next to the
auto_delete and other arguments of the function (method) that declares queues.
Queues in RabbitMQ are ordered collections of messages. Messages are enqueued and dequeued (delivered to consumers) in the FIFO manner.
Ordering also can be affected by the presence of multiple competing consumers, consumer priorities, message redeliveries. This applies to redeliveries of any kind: automatic after channel closure and negative consumer acknowledgements.
Applications can assume messages published on a single channel will be enqueued in publishing order in all the queues they get routed to. When publishing happens on multiple connections or channels, their sequences of messages will be routed concurrently and interleaved.
Consuming applications can assume that initial deliveries (those where the
redelivered property is set to
false) to a single consumer are performed in the same FIFO order as they were enqueued. For repeated deliveries (the
redelivered property is set to
true), original ordering can be affected by the timing of consumer acknowledgements and redeliveries, and thus not guaranteed.
In case of multiple consumers, messages will be dequeued for delivery in the FIFO order but actual delivery will happen to multiple consumers. If all of the consumers have equal priorities, they will be picked on a round-robin basis. Only consumers on channels that have not exceeded their prefetch value (the number of outstanding unacknowledged deliveries) will be considered.
Queues can be durable or transient. Metadata of a durable queue is stored on disk, while metadata of a transient queue is stored in memory when possible. The same distinction is made for messages at publishing time in some protocols, e.g. AMQP 0-9-1 and MQTT.
In environments and use cases where durability is important, applications must use durable queues and make sure that publish mark published messages as persisted.
Transient queues will be deleted on node boot. They therefore will not survive a node restart, by design. Messages in transient queues will also be discarded.
Durable queues will be recovered on node boot, including messages in them published as persistent. Messages published as transient will be discarded during recovery, even if they were stored in durable queues.
In most other cases, durable queues are the recommended option. For replicated queues, the only reasonable option is to use durable queues.
Throughput and latency of a queue is not affected by whether a queue is durable or not in most cases. Only environments with very high queue or binding churn — that is, where queues are deleted and re-declared hundreds or more times a second — will see latency improvements for some operations, namely on bindings. The choice between durable and transient queues therefore comes down to the semantics of the use case.
Temporary queues can be a reasonable choice for workloads with transient clients, for example, temporary WebSocket connections in user interfaces, mobile applications and devices that are expected to go offline or use switch identities. Such clients usually have inherently transient state that should be replaced when the client reconnects.
Some queue types do not support transient queues. Quorum queues must be durable due to the assumptions and requirements of the underlying replication protocol, for example.
With some workloads queues are supposed to be short lived. While clients can delete the queues they declare before disconnection, this is not always convenient. On top of that, client connections can fail, potentially leaving unused resources (queues) behind.
There are three ways to make queue deleted automatically:
An auto-delete queue will be deleted when its last consumer is cancelled (e.g. using the
basic.cancel in AMQP 0-9-1) or gone (closed channel or connection, or lost TCP connection with the server).
If a queue never had any consumers, for instance, when all consumption happens using the
basic.get method (the "pull" API), it won't be automatically deleted. For such cases, use exclusive queues or queue TTL.
An exclusive queue can only be used (consumed from, purged, deleted, etc) by its declaring connection. An attempt to use an exclusive queue from a different connection will result in a channel-level exception
RESOURCE_LOCKED with an error message that says
cannot obtain exclusive access to locked queue.
Exclusive queues are deleted when their declaring connection is closed or gone (e.g. due to underlying TCP connection loss). They therefore are only suitable for client-specific transient state.
It is common to make exclusive queues server-named.
In RabbitMQ versions up to and including 3.9, exclusive queues are subject to the leader location selection process. To make sure it will be located on the same cluster node that the connection is established to, set
x-queue-master-locator="client-local" when declaring the queue.
Queues can be replicated to multiple cluster nodes and federated across loosely coupled nodes or clusters. There are two replicated queue types provided:
The difference between them is covered in the Quorum queues guide. Quorum queues is the recommended option for most workloads and use cases.
Note that intra-cluster replication and federation are orthogonal features and should not be considered direct alternatives.
Both features can be used for data expiration and as a way of limiting how many resources (RAM, disk space) a queue can use at most, e.g. when consumers go offline or their throughput falls behind publishers.
Queues keep messages in RAM and/or on disk. In some protocols (e.g. AMQP 0-9-1) this is in part controlled by the client. In AMQP 0-9-1, this is done via a message property (
delivery_mode or, in some clients,
Publishing messages as transient suggests that RabbitMQ should keep as many messages as possible in RAM. Queues will, however, page even transient messages to disk when they find themselves under memory pressure.
Persistent messages routed to durable queues are persisted in batches or when a certain amount of time passes (fraction of a second).
Lazy queues page messages out to disk more aggressively regardless of their persistence property.
Queues can have 0 or more priorities. This feature is opt-in: only queues that have maximum number of priorities configured via an optional argument (see above) will do prioritisation.
Publishers specify message priority using the
priority field in message properties.
If priority queues are desired, we recommend using between 1 and 10. Currently using more priorities will consume more resources (Erlang processes).
Currently a single queue replica (whether leader or follower) is limited to a single CPU core on its hot code path. This design therefore assumes that most systems use multiple queues in practice. A single queue is generally considered to be an anti-pattern (and not just for resource utilisation reasons).
In case when it is desirable to trade off message ordering for parallelism (better CPU core utilisation), rabbitmq-sharding provides an opinionated way of doing so transparently to the clients.
RabbitMQ collects multiple metrics about queues. Most of them are available via RabbitMQ HTTP API and management UI, which is designed for monitoring. This includes queue length, ingress and egress rates, number of consumers, number of messages in various states (e.g. ready for delivery or unacknowledged), number of messages in RAM vs. on disk, and so on.
rabbitmqctl can list queues and some basic metrics.
Runtime metrics such as VM scheduler usage, queue (Erlang) process GC activity, amount of RAM used by the queue process, queue process mailbox length can be accessed using the rabbitmq-top plugin and individual queue pages in the management UI.
Messages can be consumed by registering a consumer (subscription), which means RabbitMQ will push messages to the client, or fetched individually for protocols that support this (e.g. the
basic.get AMQP 0-9-1 method), similarly to HTTP GET.
Delivered messages can be acknowledged by consumer explicitly or automatically as soon as a delivery is written to connection socket.
Automatic acknowledgement mode generally will provide higher throughput rate and uses less network bandwidth. However, it offers the least number of guarantees when it comes to failures. As a rule of thumb, consider using manual acknowledgement mode first.
Automatic acknowledgement mode can also overwhelm consumers which cannot process messages as quickly as they are delivered. This can result in permanently growing memory usage and/or OS swapping for the consumer process.
Manual acknowledgement mode provides a way to set a limit on the number of outstanding (unconfirmed) deliveries: channel QoS (prefetch).
Consumers using higher (several thousands or more) prefetch levels can experience the same overload problem as consumers using automatic acknowledgements.
High number of unacknowledged messages will lead to higher memory usage by the broker.
Enqueued messages therefore can be in one of two states:
Message breakdown by state can be found in the management UI.
It is possible to determine queue length in a number of ways:
queue.declaremethod response (
queue.declare-ok). The field name is
message_count. How it is accessed varies from client library to client library.
Queue length is defined as the number of messages ready for delivery.