Clustering configuration

The following configuration items define how Transformer participates in a cluster of Transformers.

Transformer’s clustering configuration must be kept in a separate configuration file called cluster.conf

If you’re using the Deployment Framework, put changes and additions to this configuration in <Framework-root>/global_config/overrides/servers/Transformer/etc/cluster.conf, where <Framework-root> is the Deployment Framework’s topmost directory.

The clustering configuration must be the same for every Transformer in the cluster, in order for the cluster to operate correctly, except (where used) the configuration in the file defined by primary-config-file.
"this Transformer" means the Transformer for which you are defining the configuration. "this node" means the cluster node that this Transformer represents.

add-cluster-node

add-cluster-node identifies a Transformer (a "node") in the cluster.

Ensure that every Transformer in the cluster has the same set of add-cluster-node configuration items, specified in the same order.
For more about how to use add-cluster-node, see Clustering Transformers without the Caplin Deployment Framework in How can I.. Set up a cluster of Transformers.

Syntax:

add-cluster-node
   cluster-addr    [string]
   cluster-port    [string]
end-cluster-node

The options for add-cluster-node are:

Name Type Default Description

cluster-addr

string

127.0.0.1

Network interface address or hostname of the cluster node.

In the Caplin Platform Deployment Framework, the configuration macros TRANSFORMER${THIS_LEG}_HOST and TRANSFORMER${OTHER_LEG}_HOST specify the cluster-addr values. (Clustering is used in the Deployment Framework to support failover capability.) You normally use the ./dfw hosts command to set up the hostnames in these macros. For more about this, see Configuring server hostnames in How can I…​ Set up server failover capability, and Change server hostnames in How can I …​ Change server-specific configuration. Also see Deployment Framework Configuration macros and items.

cluster-port

string

[none]

Network port number of the cluster node. This port is used for cluster-related communication between the Transformer and the other Transformers in the cluster.

In the Caplin Platform Deployment Framework, the configuration macros TRANSFORMER${THIS_LEG}_CLUSTER_PORT and TRANSFORMER${OTHER_LEG}_CLUSTER_PORT specify the cluster-port values. For more about this, see Clustering Transformers in the Caplin Deployment Framework in How can I …​ Set up a cluster of Transformers. Also see Deployment Framework Configuration macros and items.

always-auto-distribute

always-auto-distribute specifies a space-separated list of subject names for which this Transformer automatically distributes updates to all other Transformers in the cluster. A subject name can be a Linux shell wildcard pattern, in which case updates for all subject names that match the pattern are forwarded. The pattern can contain any of the following wildcards in an appropriate combination:

  • * (matches zero, one or many characters)

  • ? (matches just one character)

  • [ ] (matches any of the characters inside the brackets)

To prevent updates for a subject from being repeatedly "bounced" between Transformers, make sure that none of the subjects specified in the always-auto-distribute list for a Transformer (either explicitly specified or selected through a wildcard pattern) are also selected by the always-auto-distribute lists of any other Transformers in the cluster.

Also see always-auto-distribute-ignore

Syntax: always-auto-distribute <subject-1> <subject-2> …​ <subject-n>

Type: array of strings

Default value: (none, but this configuration item is optional)

always-auto-distribute-ignore

always-auto-distribute-ignore specifies a space-separated list of subject names that are excluded from the patterns specified in always-auto-distribute. Updates are for subjects specified in always-auto-distribute-ignore are therefore not automatically distributed to any other Transformers in the cluster.

A subject name in the list can be a Linux shell wildcard pattern, as specified in always-auto-distribute

Syntax: always-auto-distribute-ignore <subject-1> <subject-2> …​ <subject-n>

Type: array of strings

Default value: (none, but this configuration item is optional)

Example:

always-auto-distribute        /I/*.L
always-auto-distribute-ignore /I/VOD.L

In this example, all updates whose subjects have names of the form /I/<NAME>.L are automatically distributed to the other Transformers in the cluster, except updates for the subject /I/VOD.L

auto-distribute

auto-distribute specifies a space-separated list of subject names for which all updates are automatically distributed to all other Transformers in the cluster, provided this Transformer is the primary node in the cluster.

A subject name in the list can be a Linux shell wildcard pattern, as specified in always-auto-distribute

Also see auto-distribute-ignore

Syntax: auto-distribute <subject-1> <subject-2> …​ <subject-n>

Type: array of strings

Default value: (none, but this configuration item is optional)

auto-distribute-ignore

auto-distribute-ignore specifies a space-separated list of subject names that are excluded from the patterns specified in auto-distribute and for which updates are therefore not automatically distributed to any other Transformers in the cluster when this Transformer is the primary node of the cluster.

A subject name in the list can be a Linux shell wildcard pattern, as specified in always-auto-distribute

Syntax: auto-distribute-ignore <subject-1> <subject-2> …​ <subject-n>

Type: array of strings

Default value: (none, but this configuration item is optional)

Example:

auto-distribute        /I/*.L
auto-distribute-ignore /I/VOD.L

In this example, all updates whose subjects have names of the form /I/<NAME>.L are automatically distributed to the other Transformers in the cluster when this Transformer is the primary node of the cluster, except updates for the subject /I/VOD.L

check-period

check-period specifies the interval in seconds at which the status of the cluster’s primary node is checked. The status is exposed in the Transformer’s JMX monitoring interface.

Also, at the end of every check-period this Transformer marks any auto-distributed data (see auto-distribute) that has changed since the last check-period, so that the updated data is saved to the object cache file at the end of the current write-cache-period. (The data’s only saved if this Transformer is the primary cluster node, or if it’s a secondary node and cluster-nosave-on-secondary is FALSE.)

Syntax: check-period <interval-in-seconds>

Type: float

Default value: 1.0 second

cluster-connect-timeout

cluster-connect-timeout specifies this Transformer’s timeout on an attempt to connect to another Transformer in the cluster; for example, when this Transformer starts up. When the timeout expires, this Transformer waits one second and then attempts to reconnect. After ten attempts at reconnecting, the Transformer doubles the wait time before trying again. It doubles the delay every ten attempts up to a maximum delay of 32 seconds. after which it stops attempting to connect to the other Transformer.

Syntax: cluster-connect-timeout <timeout-in-seconds>

Type: float

Default value: 10 seconds

cluster-index

cluster-index specifies the index number of this Transformer cluster node. The index number uniquely identifies this Transformer within the cluster. It states which of the add-cluster-node items this node represents. Index numbers start at 0 and correspond to the order of the add-cluster-node items in the configuration file.

In the Caplin Platform Deployment Framework, you use a configuration macro TRANSFORMER_CLUSTER_NODE_INDEX to specify Transformer’s cluster-index. See Clustering Transformers in the Caplin Deployment Framework in How can I …​ Set up a cluster of Transformers, and Deployment Framework Configuration macros and items.

Syntax: cluster-index <index-number>

Type: integer

Default value: -1 (This transformer is not part of a cluster)

cluster-nosave-on-secondary

cluster-nosave-on-secondary specifies when TRUE that if this Transformer is a secondary cluster node (that is, not the primary node), it won’t save updates for it’s auto-distributed data (see auto-distribute) to the object cache file (see check-period and write-cache-period).

If this Transformer is the primary cluster node, it always saves these updates to the object cache file, regardless of the setting of this configuration item.

Syntax: cluster-nosave-on-secondary <boolean>

Type: boolean

Default value: FALSE (this Transformer’s auto-distributed data is always saved to the object cache file)

cluster-packet-logfile

cluster-packet-logfile specifies the name of the log file in which message packets passed between this Transformer and other nodes in the Transformer cluster are recorded.

The filename can contain the parameter %a, which is replaced at run-time by this Transformer’s DataSource application-name. The log file is located in Transformer’s var directory; in the Deployment Framework, the full directory path is <Framework-root>/servers/Transformer/var where <Framework-root> is the Deployment Framework’s topmost directory.

The log file’s (binary) format is the same as that of the standard DataSource packet log (see datasrc-pkt-log in DataSource peers configuration part 2), so you need to use the logcat utility to read it.

Only configure cluster-packet-logfile when you need to troubleshoot your Transformer cluster. Cluster packet logs can quite quickly become very large, so you should only enable logging of this information for debugging purposes.

Syntax: cluster-packet-logfile <log-file-name>

Type: string

Default value: (none, but this configuration item is optional - if it’s not present then cluster packets aren’t logged)

connect-sync

connect-sync specifies a space-separated list of subject names for which a snapshot of the each subject’s current data held by this Transformer is sent on to another Transformer in the cluster when that Transformer connects to this one. The data is only sent to a connecting node when either this Transformer is the primary node of the cluster, or connect-sync-always is specified as TRUE.

A subject name in the list can be a Linux shell wildcard pattern, as specified in always-auto-distribute

Also see connect-sync-ignore

Syntax: connect-sync <subject-1> <subject-2> …​ <subject-n>

Type: array of strings

Default value: (none, but this configuration item is optional)

connect-sync-always

connect-sync-always specifies that this Transformer must always send to another Transformer in the cluster that connects to it a snapshot of the current data for each subject specified by connect-sync (and filtered by connect-sync-ignore), even if this Transformer is not the primary node in the cluster.

Syntax: connect-sync-always <boolean>

Type: boolean

Default value: FALSE (data is only sent to a connecting Transformer if this Transformer is the primary node of the cluster)

connect-sync-ignore

connect-sync-ignore specifies a space-separated list of subject names that are excluded from the patterns specified in connect-sync Therefore, a snapshot of the current data for these subjects held by this Transformer is not sent on to any other Transformer in the cluster when that Transformer that connects to this one.

A subject name in the list can be a Linux shell wildcard pattern, as specified in always-auto-distribute

Syntax: connect-sync-ignore <subject-1> <subject-2> …​ <subject-n>

Type: array of strings

Default value: (none, but this configuration item is optional)

Example:

connect-sync        /I/*.L
connect-sync-ignore /I/VOD.L

In this example, the current data for all subjects with names of the form /I/<NAME>.L is automatically distributed to another connecting Transformer in the cluster, except the data for the subject /I/VOD.L

force-primary

force-primary specifies that this Transformer is forced to be the default primary node in the cluster. This configuration item only applies when primary-method has the value forced.

Only set this configuration item in the file defined by the configuration item primary-config-file.

Once set to TRUE, the value of force-primary can subsequently be changed to FALSE if a JMX monitoring command is used to force a different Transformer in the cluster to become the primary; for more about this see primary-config-file.

Syntax: force-primary <boolean>

Type: boolean

Default value: FALSE (the cluster’s primary node is either selected at run time by the method specified by primary-method, or, when primary-method is set to forced, by the configuration for some other Transformer in the cluster having force-primary set to TRUE)

heartbeat-slack-time

heartbeat-slack-time When this Transformer doesn’t receive an expected cluster heartbeat (see heartbeat-time) from another Transformer in the cluster, it waits heartbeat-slack-time seconds before disconnecting from the cluster and trying to reconnect to it.

When all the Transformers in the cluster have initially connected, they compare their heartbeat-slack-time values and use the lowest.

Syntax: heartbeat-slack-time <slack-time-in-seconds>

Type: integer

Default value: 5 seconds

heartbeat-time

heartbeat-time specifies the time in seconds between each cluster heartbeat. The Transformers forming a cluster exchange heartbeat messages at regular intervals, allowing each of them to check that all the others are still present. If the cluster heartbeat times out, this Transformer waits heartbeat-slack-time and then disconnects from the cluster and tries to reconnect to it.

When all the Transformers in the cluster have initially connected, they compare their heartbeat-time values and use the lowest.

In the Caplin Platform Deployment Framework, you use a configuration macro TRANSFORMER_CLUSTER_HEARTBEAT_TIME to specify Transformer’s cluster-heartbeat-time. See Clustering Transformers in the Caplin Deployment Framework in How can I …​ Set up a cluster of Transformers, and Deployment Framework Configuration macros and items.

Syntax: heartbeat-time <heartbeat-time-in-seconds>

Type: integer

Default value: 15 seconds

The minimum allowed value is 1 second.

logfile

logfile specifies the name of the file to which cluster-related events are logged.

The filename can contain the parameters %a and %h At run time, %a is replaced by the Transformer’s application name (see the DataSource configuration item application-name), and %h is replaced by the host name of the machine on which the Transformer is running.

Syntax: logfile <log-file-name>

Type: string

Default value: cluster.log

log-level

log-level specifies the severity of the errors and events that this Transformer reports to the cluster-related events log file (see logfile).

Syntax: log-level <log-level-name>

Type: string

Default value: INFO

Values accepted:

log-level-name Description

DEBUG

Reports all errors and events.

INFO

Reports events and information regarding normal operation, and all errors included in the WARN, NOTIFY, ERROR and CRIT log levels.

WARN

Reports minor errors and all errors included in the NOTIFY, ERROR and CRIT log levels.

NOTIFY

Reports errors regarding data corruptions and all errors included in the ERROR and CRIT log levels.

ERROR

Reports serious errors and all errors included in the CRIT log level.

CRIT

Reports critical errors that prevent this Transformer from participating in the cluster.

primary-config-file

primary-config-file specifies the name, and optionally the directory path of the Transformer configuration file that records that this Transformer has been forced to become the primary node.

When this Transformer becomes the primary node in the cluster, it sends a message to all the other Transformers in the cluster to inform them of this. If you force this Transformer to become the primary node by sending it a JMX monitoring command (see the forced method in primary-method), the Transformer creates a file defined by primary-config-file that contains the following configuration:

force-primary TRUE

When this Transformer is restarted, this configuration file ensures that the Transformer resumes the role of primary node. When another Transformer in the cluster receives this Transformer’s "I am now primary" message, it marks itself as a secondary node and clears out its own primary-config-file.

This procedure ensures that once the administrators of a Transformer cluster have established a particular node as the primary, when the cluster is restarted, that node will continue to be the primary one.

You can manually configure one Transformer of the cluster to be the primary, but still allow the primary to subsequently be changed through JMX. To do this:

  1. In all the Transformers of the cluster, set primary-method to the value forced

  2. For the primary Transformer, create a configuration file with the name as defined by primary-config-file

  3. Put the configuration line force-primary TRUE in the file.

  4. Include the file in the Transformer’s cluster.conf file. (In the Deployment Framework, add the include-file statement to the cluster.conf file located in

    <Framework-root>/global_config/overrides/servers/Transformer/etc/)

  5. Repeat steps 2 and 4 only for all the secondary Transformers in the cluster (so these Transformers have empty primary-config-files).

The primary-config-file filename can contain the parameters %a and %h At run time, %a is replaced by the Transformer’s application name (see the DataSource configuration item application-name), and %h is replaced by the host name of the machine on which the Transformer is running. The directory path can contain the parameter %r, which is replaced at run time by the root directory (application-root) under which the DataSource application runs,

Syntax: primary-config-file <filename-and-path>

Type: string

Default value: %r/etc/cluster-primary.conf

primary-method

primary-method specifies how to determine which Transformer ("node") in the cluster takes on the role of the primary.

Syntax: primary-method <method-name>

Type: string

Default value: order

Values accepted:

<method-name> Meaning

forced

The primary node is defined either:

This setting ensures that once the administrators of a Transformer cluster have established a particular node as the primary, when the cluster is restarted, that node will continue to be the primary one.

For more about how this works, see primary-config-file

order

The primary node is the one with the lowest non-negative cluster-index value in its configuration.

persisted

Same as for forced.

uptime

The node that has been running for the longest time becomes the primary.
Make sure that primary-method is set to the same value for every Transformer within the cluster.

stable-time

stable-time specifies the minimum time that this Transformer has to have been running in order to be considered for promotion to the cluster’s primary node.

Before attempting to change stable-time from its default value, please seek advice from Caplin Support.

Syntax: stable-time <time-interval-in-seconds>

Type: integer

Default value: 5 seconds

See also: