Configuring Client Single-Hop Access to Partitioned Regions

This topic explains how to configure client single-hop access to partitioned regions on a VMware Tanzu GemFire server.

Client connection pools support direct, single-hop access to partitioned region data in the servers by means of a parameter, pr-single-hop-enabled, which is true by default for any newly-created connection pool.

Understanding Client Single-Hop Access to Partitioned Regions

When enabled, single-hop access is used for the following operations when they are applied to data in partitioned regions:

put
get
destroy
invalidate
putAll
getAll
removeAll
onRegion function execution

To perform single-hop data access, clients use metadata from the servers about where the entry buckets are hosted. The metadata is cached locally and maintained ‘lazily’, that is, it is updated only after a single-hop operation attempt requires multiple hops to succeed, an indicator of stale metadata in the client.

How Single-Hop and Connection Limit Settings Interact

The connection pool supports two sets of connection limits, a maximum and minimum limit per server, used when single-hop mode is in effect, and a maximum and minimum limit for the pool as a whole, used when single-hop mode is disabled:

Parameter	Default Value	Notes
`max-connections-per-server`	-1 (no maximum)	Effective when `pr-single-hop-enabled=true`
`min-connections-per-server`	1	Effective when `pr-single-hop-enabled=true`
`max-connections`	-1 (no maximum)	Effective when `pr-single-hop-enabled=false`
`min-connections`	1	Effective when `pr-single-hop-enabled=false`

When pr-single-hop-enabled is true,

max-connections-per-server and min-connections-per-server are used to configure the client pool.
In this case, max-connections and min-connections are customarily unused (that is, set to their default values), and are ignored.
If max-connections and min-connections are set to non-default values (for example, if you are updating a pre-v10.1 client application), their values are converted to max-connections-per-server and min-connections-per-server values using the Connection Limit Adaptation Formula.
- This conversion is provided as a courtesy; it is not intended as a long-term solution.
- For production systems, set max-connections-per-server and min-connections-per-server values explicitly and reset max-connections and min-connections to their default values.

When pr-single-hop-enabled is false,

max-connections and min-connections are used to configure the client pool.
Default settings of max-connections-per-server and min-connections-per-server are ignored.
If your application attempts to set max-connections-per-server or min-connections-per-server in this case, an exception is thrown.

How Single-Hop and Connection Timeout Settings Interact

The connection pool supports two timeout parameters:

Parameter	Default Value
`server-connection-timeout`	0
`free-connection-timeout`	10000 ms (10 seconds)

When pr-single-hop-enabled is true

If the max-connections-per-server parameter is set to its default value of -1, meaning “no maximum”, then timeouts are ignored. The connection request results immediately in a connection or in a “NoAvailableServerException”.
If the max-connections-per-server parameter is set to a value other than its default, then:
- The single-hop operations listed above first try to get a direct connection to the server hosting the data. The single-hop request is subject to both timeout intervals:
- The operation tries to get a direct connection to the desired server until server-connection-timeout (default 0) has elapsed.
- If no direct connection is obtained in time, the client then tries to get a connection to any server until free-connection-timeout (default 10 seconds) has elapsed.
- If no connection is established when free-connection-timout expires, an “AllConnectionsInUseException” is thrown.
- You can tune these two timeout periods to suit your application’s needs, which may be especially useful if your application has a low max-connections-per-server count.
- The two timeout periods are additive. This means that the maximum time that a single-hop operation will spend trying to get a connection is the sum of the two.

When pr-single-hop-enabled is false

If the max-connections parameter is set to its default value of -1, meaning “no maximum”, then timeouts are ignored. The connection request results immediately in a connection or in a “NoAvailableServerException”.
If the max-connections parameter is set to a value other than its default, then:
- The client tries to get a connection to any server until free-connection-timeout (default 10 seconds) has elapsed.
- If no connection is established when free-connection-timout expires, an “AllConnectionsInUseException” is thrown.
- The server-connection-timeout parameter does not apply – when single-hop is not in effect, the server-specific timeout is ignored.

Optimizing Single-Hop Access

To make best use of single-hop capabilities:

Verify that the client’s pool attribute, pr-single-hop-enabled is not set or is set to true. pr-single-hop-enabled is set to true by default.
If possible, use a custom data resolver to partition your server region data according to your clients’ data use patterns. See Custom-Partition Your Region Data. Include the server’s partition resolver implementation in the client’s CLASSPATH. The server passes the name of the resolver for each custom partitioned region, so the client uses the proper one. If the server does not use a partition resolver, the default partitioning between server and client matches, so single hop works.
Add single-hop considerations to your overall server load balancing plan. Single-hop uses data location rather than least loaded server to pick the servers for single-key operations. Poorly balanced single-hop data access can affect overall client/server load balancing. Some counterbalancing is done automatically because the servers with more single-key operations become more loaded and are less likely to be picked for other operations.

Balancing Single-Hop Server Connection Use

Single-hop provides the greatest benefits when data access is well balanced across your servers. In particular, the loads for client/server connections can get out of balance if you have these in combination:

Servers that are empty data accessors or that do not host the data the clients access through single-key operations
Many single-key operations from the clients

If data access is greatly out of balance, clients can thrash trying to get to the data servers. In this case, it might be faster to deactivate single hop and go through servers that do not host the data.

Performance Tuning Considerations

With single-hop access the client connects to every server, so more connections are generally used. This works fine for smaller installations, but is a barrier to scaling. If you have a large installation with many clients, you may want to deactivate single hop by setting the pool attribute pr-single-hop-enabled to false in your pool declarations.

Without single-hop, the client uses the first server connection available. The server that receives the request determines the data location and contacts the host, which might be a different server, so additional requests are made to the server system.

Setting no limit on connections can result in too many connections to your servers, possibly causing you to run up against your system’s file handle limits. Review your anticipated connection use and make sure your servers are able to accommodate it.

Connection Limit Adaptation Formula

The max-connections-per-server and min-connections-per-server parameters were introduced in VMware GemFire version 10.1. They take effect when pr-single-hop-enabled is true, which is the default. This means that client applications developed before these per-server connection limits existed (VMware GemFire versions 10.0 and earlier) must be updated to use the new per-server limits.

If your client connection pool runs in single-hop mode (pr-single-hop-enabled is true, which is the default), then you should:

Set max-connections and min-connections to their default values of -1 and 1, respectively.
Use max-connections-per-server and min-connections-per-server parameters to configure your connection pool.

If GemFire v10.1 (or later) detects that:

pr-single-hop-enabled=true, and
max-connections-per-server and min-connections-per-server are set to their default values of -1 and 1, respectively, and
max-connections and min-connections are NOT set to their default values, then
GemFire assumes that you are running a legacy application that has not yet been updated to use the new per-server parameters and applies an adaptation formula to convert max-connections and min-connections, respectively, to max-connections-per-server and min-connections-per-server.

This conversion is provided as a courtesy to allow older applications to continue functioning until you reset the connection limits. It is not intended as a long-term solution.

When GemFire converts max-connections to max-connections-per-server (or convert min-connections to min-connections-per-server), it uses the ceiling of max-connections (or min-connections) divided by the current number of servers known to the client. For example,

If current number of servers is 3, and
max-connections is 10, then
the resulted max-connections-per-server is 4, so
each server will have max-connections-per-server=4, and
the total maximum number of connections for the client pool is 12.

If a fourth server joins later:

The max-connections-per-server of the fourth server is set as ceilingOf(10/4)=3, so
the total max number of connections for the client pool becomes 15.

Note
The conditions described above are most likely to result from running a pre-10.1 client application, but can also result from mis-configuring the client pool connection limits. In either case, you should update your client application to use the new per-server connection limits as soon as possible when pr-single-hop-enabled is true.