Load-balance adapters with source affinity

Source affinity is an alternative, deterministic load-balancing algorithm that distributes channel subscriptions evenly across a set of adapters while maintaining the integrity of user sessions.

Contents:

Overview
System requirements
Enabling source affinity
- Configuration syntax
- Configuration checklist
Behaviour in the event of a peer returning NODATA
Behaviour in the event of a peer failing
- Peer failure in a load-balanced set
- Peer failure in a load-balanced set with failover
Worked example
- Why the standard load-balancing algorithm is inappropriate for this example
- Modifying this example to use source affinity
Logging

Overview

When a DataSource routes a new subscription (see Data services) to a priority group of adapter instances, the DataSource selects an adapter instance to serve the subscription based on one of two algorithms:

Adapter workload (default): the DataSource selects the adapter instance that has the least number of subscriptions.
Source affinity: the DataSource selects an adapter instance mathematically based on a numeric hash of a composite of the source group's affinity key (an arbitrary constant) and a substring extracted from the subscription’s subject (usually a username).

The primary use case for source affinity is to ensure that all of a user’s channel subscriptions to an adapter are served by the same load-balanced instance of that adapter. In this use case, the substring extracted from channel subjects is a username. Other use cases may arise in the future that use some other substring, such as a currency pair or asset class.

Service rebalancing (see service-rebalance-enable and rebalance-enable) is automatically disabled for any subscriptions subject to source affinity.

Examples of scenarios when all of a user’s channel subscriptions to an adapter should be served by the same instance of the adapter include, but are not limited to:

Adapters that maintain a cache
Adapters that maintain session state
Adapters that publish data to a channel based on events in a messaging workflow on another channel. See Worked example, below.

System requirements

Source affinity is available in the following versions of Liberator and Transformer:

Support for source affinity in Liberator and Transformer
	Liberator	Transformer
Definition at data-service level^†	6.2.2+	6.2.2+
Definition at source-group level	6.2.6+	6.2.5+

† Deprecated

Enabling source affinity

Source affinity is disabled by default. To enable source affinity, use the affinity option of the add-source-group configuration item.

Prior to Liberator 6.2.6 and Transformer 6.2.5, affinity was a configuration option of add-data-service. This usage is deprecated and is supported for backwards compatibility only. Don’t include the affinity configuration option at both data service and source-group level within the same data service.

Configuration syntax

The source group affinity option has the following syntax:

affinity <affinity_key> <regular_expression>

<affinity_key>

An arbitrary string identifier for the peers in this source group. You can choose any string to identify the peers in the source group, provided that you use the string consistently when identifying the same peers when they are referenced in other source groups. Common examples of affinity keys include 'transformers', 'trading-adapters', and 'order-adapters'.

The affinity key always identifies the peers in this source group, even if the peers in this source group are not the ultimate destination of the subscription.

The source affinity algorithm prepends the affinity key to the value extracted by the second parameter (<regular_expression>), hashes the composite string, and uses the resulting numeric value to determine which peer in the source group is used to serve a subscription.

If you need multiple data services to route their subscriptions to the same instance of a load-balanced DataSource, ensure that the source groups that reference the DataSource instances are assigned the same affinity key. For the full list of requirements, see Configuration checklist below.

<regular_expression>

A POSIX Extended Regular Expression (ERE) containing a single capture group that extracts a value from the subscription subject.

You can define multiple affinity options for a source group. They are tried in the order they are defined, and the first affinity option with a regular expression that matches the subscription subject is applied. If none of affinity options have a matching regular expression, the standard load balancing algorithm determines which peer serves the subscription.

The most common use of source affinity is to establish an affinity between a user and an adapter instance, and so the value extracted is usually a username.

Example: regular expression to extract the username from subject /PRIVATE/<session-name>/FXTRADE

/PRIVATE/([^/]+)-[0-9]+/FXTRADE

Example: regular expression to extract the username from subject /PRIVATE/<username>/BLOTTER/TRADE

/PRIVATE/([^/]+)/BLOTTER/TRADE

The source affinity algorithm prepends the affinity key to the extracted value, hashes the composite string, and uses the resulting numeric value to determine which peer in the source group is used to serve a subscription.

If you need multiple data services to route their subscriptions to the same instance of a DataSource, ensure the source groups that reference the DataSource instances are configured with affinity regular expressions that extract values in the same format. For the full list of requirements, see Configuration checklist below.

Do not include forward slash (/) delimiters in the regular expression and don’t escape forward slash (/) characters.

Include only one capture group in the regular expression.

Do not chain multiple regular expressions together with a regular expression pipe (|) operator; instead, specify multiple affinity options for the source group. For example:

add-source-group
  affinity trading-adapters "^/PRIVATE/([^/]+)/BLOTTER/TRADE"
  affinity trading-adapters "^/PRIVATE/([^/]+)/BLOTTER/EXECUTION"
  ...
add-source-group

You can test POSIX extended regular expressions using the sed command with the -r option. The example below tests extracting the username from a trade channel subject:

$ sed -r 's|/PRIVATE/([^/]+)-[0-9]+/FXTRADE|\1|' <<< '/PRIVATE/alice-0/FXTRADE'
alice

When source affinity (by username) is enabled for a source group of DataSource instances, all of a user’s subscriptions served by that source group, and by any other source group with the same affinity key, show an affinity for a specific source (DataSource instance). If that DataSource instance fails, then a new affinity is formed between the user’s subscriptions and one of the remaining active DataSource instances. This new affinity continues to be respected when the failed DataSource comes back online.

Source affinity has precedence over service rebalancing (see service-rebalance-enable and rebalance-enable). Service rebalancing does not rebalance subscriptions subject to source affinity.

The source affinity algorithm has two important characteristics:

Determinism: the algorithm distributes new subscriptions to DataSource instances based on a numeric hash of a composite of the source group’s affinity key and a substring extracted from the subject (usually a username). Given the same affinity key, extracted substring, and available DataSource instances, the source affinity algorithm always selects the same adapter instance.
Stickiness: the algorithm remembers the DataSource instance chosen for an affinity key and subject substring, and reuses it for all future subscriptions with the same affinity key and substring. Stickiness ensures that a chosen DataSource instance is used repeatedly, even when siblings of the chosen instance fail and the algorithm would, were it given the opportunity, route a new subscription to a different instance of the DataSource.

Affinities associated with an affinity key are remembered until the DataSource application is restarted, or until all subscriptions associated with the affinity key have been discarded.

DataSource applications calculate and remember affinities independently.

Configuration checklist

When configuring source affinity, follow the guidance below:

Name your affinity keys after the set of peers they are associated with. For example: transformers, trading-adapters, …
When writing multiple affinity options for the same affinity key, ensure that the regular expression extracts the same type of value (usually a username) in every case.
When writing a regular expression to capture a username from a mapped subject, the method you use to extract the username depends on whether the subject contains a username or a session name (<username>-<session-number>).
Example: extracting a username from a subject that contains a username
```
affinity trading-adapters ^/PRIVATE/([^/]+)/BLOTTER/TRADE
```
Example: extracting a username from a subject that contains a session name
```
affinity trading-adapters ^/PRIVATE/([^/]+)-[0-9]+/FXTRADE
```

If you are configuring source affinity for a subject that is sorted and filtered by Transformer’s Refiner module, then configure source affinity for the Refiner subject (/FILTER/…) too:

Example: setting the affinity key for Refiner subject requests

add-data-service
  include-pattern '^/PRIVATE/[^/]+/BLOTTER/TRADE'
  include-pattern '^/FILTER/PRIVATE/[^/]+/BLOTTER/TRADE'
  ...
  add-source-group
    affinity transformers '^/PRIVATE/([^/]+)/BLOTTER/TRADE'
    affinity transformers '^/FILTER/PRIVATE/([^/]+)/BLOTTER/TRADE'
    ...
    add-priority
      label transformer1
      label transformer2
    end-priority
    ...
  end-source-group
  ...
end-data-service

Source groups that use the same affinity key must contain identical priority group configuration.
In two-leg deployments, don’t use the variables THIS_LEG and OTHER_LEG in peer labels within add-priority blocks. Source affinity requires that labels are listed in an identical order on both legs, so if you use THIS_LEG and OTHER_LEG in peer labels, then you must manually reverse the label order on one of the legs. Use literal labels instead to make it clearer that label order is identical on both legs:
Example: in two-leg deployments, use literal peer labels in add-priority blocks
```
add-priority
  remote-label transformer1
  remote-label transformer2
end-priority
```

Behaviour in the event of a peer returning NODATA

From C DataSource API 7.1.5, Liberator 7.1.4, and Transformer 7.1.4, if a subscription request managed by source affinity receives a NODATA response with the Unavailable flag or NotFound flag set, then the NODATA response is deemed authoritative for all peers in the source group. The request is not routed to other peers in the source group.

Behaviour in the event of a peer failing

This section describes how various source-affinity configurations behave in the event of an adapter instance failure.

Peer failure in a load-balanced set

Two adapters, FooAdapter1 and FooAdapter2, arranged in a load-balancing formation with source affinity:

add-source-group
    affinity foo-adapters '/PRIVATE/([^/]+)-[0-9]+/FOO' (1)
    add-priority
        label FooAdapter1
        label FooAdapter2
    end-priority
end-source-group

1	If the regular expression matches the subject /PRIVATE/<username>-<session-number>/FOO, capture the username from the subject and set the affinity key to 'foo-adapters'.

This configuration is compatible with permissioning adapters.

How the load-balancer selects an adapter instance for a user’s subscription depends on whether the user already has a prior affinity for an adapter under the affinity key 'foo-adapters'.

If the user has a prior affinity for an adapter instance, the adapter will be selected to serve the new subscription too.
If the user does not have a prior affinity for an adapter, an adapter will be selected by the source affinity algorithm based on the user’s username, the affinity key and the number of adapters in the priority group. The algorithm will create an affinity between the user and the chosen adapter, stored under the affinity key 'foo-adapters'.

If FooAdapter1 or FooAdapter2 fails, then all existing subscriptions served by the failed adapter, and all affinities for the failed adapter, are reallocated to the surviving adapter. New subscriptions, and their associated users' affinities, are allocated to the surviving adapter.

When the failed adapter recovers, subscriptions and affinities are not automatically redistributed across both adapters. Existing affinities are unaffected by the recovery of a failed adapter, and are retained until one of the events below occurs:

The adapter instance associated with the affinity fails
The DataSource application (Liberator or Transformer) is restarted
All users with affinities under affinity key 'foo-adapters' log out.

Peer failure in a load-balanced set with failover

Four adapters, arranged into two load-balanced sets of two adapters:

add-source-group
    affinity foo-adapters '/PRIVATE/([^/]+)-[0-9]+/FOO' (1)
    add-priority
        label FooAdapter1
        label FooAdapter2
    end-priority
    add-priority
        label FooAdapter3
        label FooAdapter4
    end-priority
end-source-group

1	If the regular expression matches the subject /PRIVATE/username-session_number/FOO, capture the username from the subject and set the affinity key to 'foo-adapters'.

This configuration is not compatible with permissioning adapters. Use the configuration in Load balancing instead.

While at least one adapter is available in the first priority group, the configuration will behave as scenario 1 does.

If both adapters in the first priority group fail, then existing subscriptions and affinities are redistributed across the adapters of the second priority group.

When an adapter in the first priority group recovers, new subscriptions with no existing affinity are routed to the first priority group, but existing subscriptions and affinities are not transferred back to the first priority group until one of the events below occurs:

The adapter instance associated with an affinity fails
The DataSource application (Liberator or Transformer) is restarted
All users with affinities under affinity key 'foo-adapters' log out.

Worked example

This example describes how to load-balance a trade adapter that serves both trade channels and trade blotter containers.

A container is a record that contains a list of references to other records (in this case, blotter items). The blotter’s container and items can be served by different adapters. For more information on writing adapters that serve blotter data, see Serving blotter data with the Java Blotter API.

In this example, the trade channel is served solely by the trade adapter, and the trade blotter is served by both the trade adapter and an historical adapter:

Subscription

Trade Adapter

Historical Adapter

Trade channel

Blotter container

Blotter items

This arrangement is commonly used where it is not practical for the historical adapter to receive a live feed of a user’s trades from a trading system API. The historical adapter populates the blotter container with an initial list of trades, and the trading adapter supplements the container with details of new trades made over the user’s trade channel.

Why the standard load-balancing algorithm is inappropriate for this example

This section shows how the standard load-balancing algorithm is inappropriate for a trade adapter that serves trade blotter containers.

Do not use the configuration in this section to load-balance trade adapters that serve blotter containers. Use the configuration in Modifying this example to use source affinity.

Configuration for Liberator and Transformer:

Liberator config: data service for trade blotter container

add-data-service
  service-name trade-blotter-container
  include-pattern "^/PRIVATE/([^/]+)/BLOTTER/TRADE"

  add-source-group
    add-priority
      remote-label Transformer
    end-priority
  end-source-group

end-data-service

Liberator config: data service for trade blotter items

add-data-service
  service-name trade-blotter-items
  include-pattern "^/PRIVATE/([^/]+)/ITEM"

  add-source-group
    add-priority
      remote-label Transformer
    end-priority
  end-source-group

end-data-service

Liberator config: data service for trade channel

add-data-service
  service-name trade-messaging-channel
  include-pattern "^/PRIVATE/([^/]+)/FXTRADE"

  add-source-group
    add-priority
      remote-label TradeAdapterA
      remote-label TradeAdapterB
    end-priority
  end-source-group

end-data-service

Transformer config: data service for trade blotter container

add-data-service
  service-name trade-blotter-container
  include-pattern "^/PRIVATE/([^/]+)/BLOTTER/TRADE"

  # Live feed of new trades
  add-source-group
    add-priority
      remote-label TradeAdapterA
      remote-label TradeAdapterB
    end-priority
  end-source-group

  # Initial blotter contents
  add-source-group
    add-priority
      remote-label HistoricalAdapter
    end-priority
  end-source-group

end-data-service

Transformer config: data service for trade blotter items

add-data-service
  service-name trade-blotter-items
  include-pattern "^/PRIVATE/([^/]+)/ITEM"

  add-source-group
    add-priority
      remote-label HistoricalAdapter
    end-priority
  end-source-group

end-data-service

The design of the trading adapter depends on a user’s trade channel and blotter container being served by the same trade adapter instance. However, without source affinity enabled, Liberator and Transformer will distribute subscriptions to the trading adapter instances based on adapter workload. There is no guarantee that a user’s trade channel and blotter container will be served by the same trade adapter instance.

The diagram below illustrates what happens when the standard load-balancing algorithm distributes a user’s trade channel and blotter container to the same trade adapter instance:

The diagram below illustrates what happens when the standard load-balancing algorithm distributes a user’s trade channel and blotter container to different adapter instances:

In the scenario illustrated above, TradeAdapterA attempts to publish new content to Alice’s blotter following the successful execution of a trade by Alice. Unfortunately, TradeAdapterA does not serve the subscription for the Alice’s blotter; the standard load-balancing algorithm routed it to TradeAdapterB instead. TradeAdapterB, which does not serve Alice’s trade channel, is unaware that that Alice has placed a new trade, so the new trade is not added to Alice’s blotter.

In the next subsection, we look at how we can enable source affinity to ensure that a user’s trade channel and blotter container are served by the same trade adapter instance.

Modifying this example to use source affinity

To prevent a user’s trade channel and blotter channel from being served by two different instances of the trade adapter, enable source affinity on the two source groups that load balance the subscriptions to the trade adapter instances.

The configuration for each source group should be identical in the following ways:

The affinity key must be identical
The affinity regular expression must capture the username
Adapter instances must be identical and in an identical order

The modified configuration is shown below, with changes annotated:

Liberator config: data service for trade blotter container

add-data-service
  service-name trade-blotter-container
  include-pattern "^/PRIVATE/([^/]+)/BLOTTER/TRADE"

  add-source-group
    add-priority
      remote-label Transformer
    end-priority
  end-source-group

end-data-service

Liberator config: data service for trade blotter items

add-data-service
  service-name trade-blotter-items
  include-pattern "^/PRIVATE/([^/]+)/ITEM"

  add-source-group
    add-priority
      remote-label Transformer
    end-priority
  end-source-group

end-data-service

Liberator config: data service for trade channel

add-data-service
  service-name trade-messaging-channel
  include-pattern "^/PRIVATE/([^/]+)/FXTRADE"

  add-source-group
    affinity trading-adapters "^/PRIVATE/([^/]+)-[0-9]+/FXTRADE" (1)
    add-priority (2)
      remote-label TradeAdapterA
      remote-label TradeAdapterB
    end-priority
  end-source-group

end-data-service

1	If the regular expression matches the subject /PRIVATE/username-session_number/FXTRADE, capture the username from the subject and set the affinity key to 'trading-adapters' (identical to the affinity key used for the blotter container).
2	Adapter instances must be in the same order as the instances listed for the blotter container.

Transformer config: data service for trade blotter container

add-data-service
  service-name trade-blotter-container
  include-pattern "^/PRIVATE/([^/]+)/BLOTTER/TRADE"

  # Live feed of new trades
  add-source-group
    affinity trading-adapters "^/PRIVATE/([^/]+)/BLOTTER/TRADE" (1)
    add-priority (2)
      remote-label TradeAdapterA
      remote-label TradeAdapterB
    end-priority
  end-source-group

  # Initial blotter contents
  add-source-group
    add-priority
      remote-label HistoricalAdapter
    end-priority
  end-source-group

end-data-service

1	If the regular expression matches the subject /PRIVATE/username/BLOTTER/TRADE, capture the username from the subject and set the affinity key to 'trading-adapters' (identical to the affinity key used for the trade channel).
2	Adapter instances must be in the same order as the instances listed for the trade channel.

Transformer config: data service for trade blotter items

add-data-service
  service-name trade-blotter-items
  include-pattern "^/PRIVATE/([^/]+)/ITEM"
  add-source-group
    add-priority
      remote-label HistoricalAdapter
    end-priority
  end-source-group
end-data-service

The diagram below illustrates what happens when the source affinity algorithm distributes a user’s trade channel and blotter container:

In the scenario illustrated above, the source affinity algorithm on Liberator and on Transformer both hash the same string, ‘trading-adapters:alice’, with the result that Liberator and Transformer choose the same trade adapter instance to serve Alice’s trade channel and blotter container.

Logging

When a DataSource uses the source affinity algorithm to select an adapter instance, the DataSource writes a message in the following format to its event log:

INFO: Object <subject> is bound to affinity <affinity_key:captured_value>

Continuing the Worked example above, Liberator would write the following messages to its event log regarding Alice’s trade channel:

INFO: Object </PRIVATE/alice/FXTRADE> is bound to affinity <trading-adapters:alice> (1)
...
INFO: We can request object </PRIVATE/alice/FXTRADE> from peer 2 (TradeAdapterA) (2)

1	The numeric hash of 'trading-adapters:alice' determines which adapter instance serves /PRIVATE/alice/FXTRADE
2	'TradeAdapterA' is the adapter instance selected to serve /PRIVATE/alice/FXTRADE

See also: