Tanzu GemFire performs different consistency checks depending on the type of region you have configured.
Partitioned Region Consistency
For a partitioned region, Tanzu GemFire maintains consistency by routing all updates on a given key to the Tanzu GemFire member that holds the primary copy of that key. That member holds a lock on the key while distributing updates to other members that host a copy of the key. Because all updates to a partitioned region are serialized on the primary Tanzu GemFire member, all members apply the updates in the same order and consistency is maintained at all times. See Understanding Partitioning.
Replicated Region Consistency
For a replicated region, any member that hosts the region can update a key and distribute that update to other members without locking the key. It is possible that two members can update the same key at the same time (a concurrent update). It is also possible that, due to network latency, an update in one member is distributed to other members at a later time, after those members have already applied more recent updates to the key (an out-of-order update). By default, Tanzu GemFire members perform conflict checking before applying region updates in order to detect and consistently resolve concurrent and out-of-order updates. Conflict checking ensures that region data eventually becomes consistent on all members that host the region. The conflict checking behavior for replicated regions is summarized as follows:
- If two members update the same key at the same time, conflict checking ensures that all members eventually apply the same value, which is the value of one of the two concurrent updates.
- If a member receives an out-of-order update (an update that is received after one or more recent updates were applied), conflict checking ensures that the out-of-order update is discarded and not applied to the cache.
How Consistency Checking Works for Replicated Regions and How Destroy and Clear Operations Are Resolved provide more details about how Tanzu GemFire performs conflict checking when applying an update.
Non-Replicated Regions and Client Cache Consistency
When a member receives an update for an entry in a non-replicated region and applies an update, it performs conflict checking in the same way as for a replicated region. However, if the member initiates an operation on an entry that is not present in the region, it first passes that operation to a member that hosts a replicate. The member that hosts the replica generates and provides the version information necessary for subsequent conflict checking. See How Consistency Checking Works for Replicated Regions.
Client caches also perform consistency checking in the same way when they receive an update for a region entry. However, all region operations that originate in the client cache are first passed onto an available Tanzu GemFire server, which generates the version information necessary for subsequent conflict checking.
Configuring Consistency Checking
Tanzu GemFire enables consistency checking by default. You cannot deactivate consistency checking for persistent regions. For all other regions, you can explicitly activate or deactivate consistency checking by setting the concurrency-checks-enabled region attribute in cache.xml to “true” or “false.”
All Tanzu GemFire members that host a region must use the same concurrency-checks-enabled setting for that region.
A client cache can deactivate consistency checking for a region even if server caches activate consistency checking for the same region. This configuration ensures that the client sees all events for the region, but it does not prevent the client cache region from becoming out-of-sync with the server cache.
See <region-attributes>.
Note: Regions that do not enable consistency checking remain subject to race conditions. Concurrent updates may result in one or more members having different values for the same key. Network latency can result in older updates being applied to a key after more recent updates have occurred.
Overhead for Consistency Checks
Consistency checking requires additional overhead for storing and distributing version and timestamp information, as well as for maintaining destroyed entries for a period of time to meet consistency requirements.
To provide consistency checking, each region entry uses an additional 16 bytes. When an entry is deleted, a tombstone entry of approximately 13 bytes is created and maintained until the tombstone expires or is garbage-collected in the member. (When an entry is destroyed, the member temporarily retains the entry with its current version stamp to detect possible conflicts with operations that have occurred. The retained entry is referred to as a tombstone.) See How Destroy and Clear Operations Are Resolved.
If you cannot support the additional overhead in your deployment, you can deactivate consistency checks by setting concurrency-checks-enabled to “false” for each region. See Consistency for Region Updates.
How Consistency Checking Works for Replicated Regions
Each region stores version and timestamp information for use in conflict detection. Tanzu GemFire members use the recorded information to detect and resolve conflicts consistently before applying a distributed update.
By default, each entry in a region stores the ID of the Tanzu GemFire member that last updated the entry, as well as a version stamp for the entry that is incremented each time an update occurs. The version information is stored in each local entry, and the version stamp is distributed to other Tanzu GemFire members when the local entry is updated.
A Tanzu GemFire member or client that receives an update message first compares the update version stamp with the version stamp recorded in its local cache. If the update version stamp is larger, it represents a newer version of the entry, so the receiving member applies the update locally and updates the version information. A smaller update version stamp indicates an out-of-order update, which is discarded.
An identical version stamp indicates that multiple Tanzu GemFire members updated the same entry at the same time. To resolve a concurrent update, a Tanzu GemFire member always applies (or keeps) the region entry that has the highest membership ID; the region entry having the lower membership ID is discarded.
Note: When a Tanzu GemFire member discards an update message (either for an out-of-order update or when resolving a concurrent update), it does not pass the discarded event to an event listener for the region. You can track the number of discarded updates for each member using the conflatedEvents statistic. See Tanzu GemFire Statistics List. Some members may discard an update while other members apply the update, depending on the order in which each member receives the update. For this reason, the conflatedEvents statistic differs for each Tanzu GemFire member. The example below describes this behavior in more detail.
The following example shows how a concurrent update is handled in a cluster of three Tanzu GemFire members. Assume that Members A, B, and C have membership IDs of 1, 2, and 3, respectively. Each member currently stores an entry, X, in their caches at version C2 (the entry was last updated by member C):
Step 1: An application updates entry X on Tanzu GemFire member A at the same time another application updates entry X on member C. Each member increments the version stamp for the entry and records the version stamp with their member ID in their local caches. In this case the entry was originally at version C2, so each member updates the version to 3 (A3 and C3, respectively) in their local caches.
Step 2: Member A distributes its update message to members B and C.
Member B compares the update version stamp (3) to its recorded version stamp (2) and applies the update to its local cache as version A3. In this member, the update is applied for the time being, and passed on to configured event listeners.
Member C compares the update version stamp (3) to its recorded version stamp (3) and identifies a concurrent update. To resolve the conflict, member C next compares the membership ID of the update to the membership ID stored in its local cache. Because the distributed system ID the update (A3) is lower than the ID stored in the cache (C3), member C discards the update (and increments the conflatedEvents statistic).
Step 3: Member C distributes the update message to members A and B.
Members A and B compare the update version stamp (3) to their recorded version stamps (3) and identify the concurrent update. To resolve the conflict, both members compare the membership ID of the update with the membership ID stored in their local caches. Because the distributed system ID of A in the cache value is less than the ID of C in the update, both members record the update C3 in their local caches, overwriting the previous value.
At this point, all members that host the region have achieved a consistent state for the concurrent updates on members A and C.
How Destroy and Clear Operations Are Resolved
When consistency checking is enabled for a region, a Tanzu GemFire member does not immediately remove an entry from the region when an application destroys the entry. Instead, the member retains the entry with its current version stamp for a period of time in order to detect possible conflicts with operations that have occurred. The retained entry is referred to as a tombstone. Tanzu GemFire retains tombstones for partitioned regions and non-replicated regions as well as for replicated regions, in order to provide consistency.
A tombstone in a client cache or a non-replicated region expires after 8 minutes, at which point the tombstone is immediately removed from the cache.
A tombstone for a replicated or partitioned region expires after 10 minutes. Expired tombstones are eligible for garbage collection by the Tanzu GemFire member. Garbage collection is automatically triggered after 100,000 tombstones of any type have timed out in the local Tanzu GemFire member. You can optionally set the gemfire.tombstone-gc-threshold property to a value smaller than 100000 to perform garbage collection more frequently.
Note: To avoid out-of-memory errors, a Tanzu GemFire member also initiates garbage collection for tombstones when the amount of free memory drops below 30 percent of total memory.
You can monitor the total number of tombstones in a cache using the tombstoneCount statistic in CachePerfStats. The tombstoneGCCount statistic records the total number of tombstone garbage collection cycles that a member has performed. replicatedTombstonesSize and nonReplicatedTombstonesSize show the approximate number of bytes that are currently consumed by tombstones in replicated or partitioned regions, and in non-replicated regions, respectively. See Tanzu GemFire Statistics List.
About Region.clear() Operations
Region entry version stamps and tombstones ensure consistency only when individual entries are destroyed. A Region.clear() operation, however, operates on all entries in a region at once. To provide consistency for Region.clear() operations, Tanzu GemFire obtains a distributed read/write lock for the region, which blocks all concurrent updates to the region. Any updates that were initiated before the clear operation are allowed to complete before the region is cleared.
Transactions with Consistent Regions
A transaction that modifies a region having consistency checking enabled generates all necessary version information for region updates when the transaction commits.
If a transaction modifies a normal, preloaded or empty region, the transaction is first delegated to a Tanzu GemFire member that holds a replicate for the region. This behavior is similar to the transactional behavior for partitioned regions, where the partitioned region transaction is forwarded to a member that hosts the primary for the partitioned region update.
The limitation for transactions on normal, preloaded or or empty regions is that, when consistency checking is enabled, a transaction cannot perform a localDestroy or localInvalidate operation against the region. Tanzu GemFire throws an UnsupportedOperationInTransactionException exception in such cases. An application should use a Destroy or Invalidate operation in place of a localDestroy or localInvalidate when consistency checks are enabled.
