How Persistence and Overflow Work

To use Tanzu GemFire persistence and overflow, you should understand how they work with your data.

Tanzu GemFire persists and overflows several types of data. You can persist or overflow the application data in your regions. In addition, Tanzu GemFire persists and overflows messaging queues, to manage memory consumption and provide high availability.

Persistent data outlives the member where the region resides and can be used to initialize the region at creation. Overflow acts only as an extension of the region in memory.

The data is written to disk according to the configuration of Tanzu GemFire disk stores. For any disk option, you can specify the name of the disk store to use or use the Tanzu GemFire default disk store. See Disk Storage.

How Data Is Persisted and Overflowed

For persistence, the entry keys and values are copied to disk. For overflow, only the entry values are copied. Other data, such as statistics and user attributes, are retained in memory only.

Data regions are overflowed to disk by least recently used (LRU) entries because those entries are deemed of least interest to the application and therefore less likely to be accessed.
Server subscription queues overflow most recently used (MRU) entries. These are the messages that are at the end of the queue and so are last in line to be sent to the client.

Persistence

Persistence provides a disk backup of region entry data. The keys and values of all entries are saved to disk, like having a replica of the region on disk. Region entry operations such as put and destroy are carried out in memory and on disk.

Member with Region A containing X, Y, and Z. Disk Files (keys and values for all entries). Arrows labeled 'persist values from create' and 'update/invalidate/destroy' from Member to Disk Files.

When the member stops for any reason, the region data on disk remains. In partitioned regions, where data buckets are divided among members, this can result in some data only on disk and some on disk and in memory. The disk data can be used at member startup to populate the same region.

Overflow

Overflow limits region size in memory by moving the values of least recently used (LRU) entries to disk. Overflow basically uses disk as a swap space for entry values. If an entry is requested whose value is only on disk, the value is copied back up into memory, possibly causing the value of a different LRU entry to be moved to disk. As with persisted entries, overflowed entries are maintained on disk just as they are in memory.

In this figure, the value of entry X has been moved to disk to make space in memory. The key for X remains in memory. From the distributed system perspective, the value on disk is as much a part of the region as the data in memory.

Member with Region B containing Y and Z. Disk Files (keys and values for all entries) with X. Arrows labeled 'offload overflow' and 'update/invalidate/destroy' from Member to Disk Files. Arrow from Disk FIles to Member labeled 'satisfy get'.

Persistence and Overflow Together

Used together, persistence and overflow keep all entry keys and values on disk and only the most active entry values in memory. The removal of an entry value from memory due to overflow has no effect on the disk copy as all entries are already on disk.

Member with Region C containing Y and Z. Disk Files (keys and values for all entries) with X, Y, and Z. Arrows labeled 'persist values from create' and 'update/invalidate/destroy' from Member to Disk Files. Arrow from Disk FIles to Member labeled 'satisfy get'.

Persistence and Multi-Site Configurations

Multi-site gateway sender queues overflow most recently used (MRU) entries. These are the messages that are at the end of the queue and so are last in line to be sent to the remote site. You can also configure gateway sender queues to persist for high availability.