As a RabbitMQ consumer, GPSS manages the progress of each load operation using RabbitMQ's broker-based offset management. This management involves identifying how, when (before commit, after commit, or never), and where (history table, broker, both, nowhere) the offset is stored, and is directed by a combination of RabbitMQ client and GPSS load configuration properties.
RabbitMQ Properties
When loading from a RabbitMQ queue or stream, GPSS uses a default consumer.name of gpss_rabbitmq_consumer_<job_id>. You can set the consumer name by specifying this RabbitMQ client property in the PROPERTIES (version 2) or properties (version 3 (beta)) load configuration file block. For example:
PROPERTIES:
consumer.name: my_rmq_consumer
When loading from a RabbitMQ queue, you may also choose to set the consumer.exclusive and qos.prefetch.count client properties, which specify the number of queue consumers and prefetch limits per consumer. GPSS sets the consumer.exclusive property to false by default.
GPSS Properties
GPSS uses the CONSISTENCY (version 2) or consistency (version 3 (Beta)) load configuration file property setting to govern how it manages offsets.
GPSS supports the following CONSISTENCY settings for RabbitMQ:
CONSISTENCY: { strong | at-least | at-most | none }
NoteGPSS supports
strongconsistency for RabbitMQ streams only.
Summary
The following table summarizes the offset commit behaviour for load jobs for a GPSS RabbitMQ consumer:
| Consistency Value | Behaviour |
|---|---|
| strong (available for RabbitMQ streams only) | GPSS stores offsets in a history table. |
| at-least [or empty] | GPSS stores in the offset in the RabbitMQ broker before commit. Note that this could result in duplicate data written to Greenplum Database when the connection to RabbitMQ closes after GPSS writes, but before it sends the response. |
| at-most | GPSS stores the offset in the RabbitMQ broker after commit. |
| none | GPSS does not store offsets anywhere. |
