Highly-Available Client/Server Event Messaging

This topic explains highly-available client/server event messaging in VMware Tanzu GemFire.

With server redundancy, each pool has a primary server and some number of secondaries. The primaries and secondaries are assigned on a per-pool basis and are generally spread out for load balancing, so a single client with multiple pools may have primary queues in more than one server.

The primary server pushes events to clients and the secondaries maintain queue backups. If the primary server fails, one of the secondaries becomes primary to provide uninterrupted event messaging.

For example, if there are six servers running and subscription-redundancy is set to two, one server is the primary, two servers are secondary, and the remaining three do not actively participate in HA for the client. If the primary server fails, the system assigns one of the secondaries as the new primary and attempts to add another server to the secondary pool to retain the initial redundancy level. If no new secondary server is found, then the redundancy level is not satisfied but the failover procedure completes successfully. As soon as another secondary is available, it is added.

When high availability is enabled:

• The primary server sends event messages to the clients.
• Periodically, the clients send received messages to the server and the server removes the sent messages from its queues.
• Periodically, the primary server synchronizes with its secondaries, notifying them of messages that can be discarded because they have already been sent and received. There is a lag in notification, so the secondary servers remain only roughly synchronized with the primary. Secondary queues contain all messages that are contained in the primary queue plus possibly a few messages that have already been sent to clients.
• In the case of primary server failure, one of the secondaries becomes the primary and begins sending event messages from its queues to the clients. Immediately after failover, the new primary usually resends some messages that were already sent by the old primary. The client recognizes these as duplicates and discards them.

In stage 1 of this figure, the primary sends an event message to the client and a synchronization message to its secondary. By stage 2, the secondary and client have updated their queue and message tracking information. If the primary failed at stage two, the secondary would start sending event messages from its queue beginning with message A10. The client would discard the resend of message A10 and then process subsequent messages as usual.

Change Server Queue Synchronization Frequency

By default, the primary server sends queue synchronization messages to the secondaries every second. You can change this interval with the gfsh alterruntime command

Set the interval for queue synchronization messages as follows:

• gfsh:

  gfsh>alter runtime --message-sync-interval=2
  

• XML:

  <!-- Set synchronization interval to 2 seconds --> 
  <cache ... message-sync-interval="2" />
  

• Java:

  cache = CacheFactory.create();
  cache.setMessageSyncInterval(2);  
  

The ideal setting for this interval depends in large part on your application behavior. These are the benefits of shorter and longer interval settings:

• A shorter interval requires less memory in the secondary servers because it reduces queue buildup between synchronizations. In addition, fewer old messages in the secondary queues means reduced message re-sends after a failover. These considerations are most important for systems with high data update rates.
• A longer interval requires fewer distribution messages between the primary and secondary, which benefits overall system performance.

Set Frequency of Orphan Removal from the Secondary Queues

Usually, all event messages are removed from secondary subscription queues based on the primary’s synchronization messages. Occasionally, however, some messages are orphaned in the secondary queues. For example, if a primary fails in the middle of sending a synchronization message to its secondaries, some secondaries might receive the message and some might not. If the failover goes to a secondary that did receive the message, the system will have secondary queues holding messages that are no longer in the primary queue. The new primary will never synchronize on these messages, leaving them orphaned in the secondary queues.

To make sure these messages are eventually removed, the secondaries expire all messages that have been enqueued longer than the time indicated by the servers’ message-time-to-live.

Set the time-to-live as follows:

• XML:

  <!-- Set message ttl to 5 minutes --> 
  <cache-server port="41414" message-time-to-live="300" />
  

• Java:

  Cache cache = ...;
  CacheServer cacheServer = cache.addCacheServer();
  cacheServer.setPort(41414);
  cacheServer.setMessageTimeToLive(200);
  cacheServer.start();