DataSource peers configuration (part 2)

Introduced in platform 8, DataSource supports using an API key to authenticate and authorise adapters. This allows the Liberator or Transformer to control whether to accept a peer connection and which dynamic services to allow it access to.

The adapter uses the add-peer:apikey configuration option to specify an API key for a connection, and the Liberator or Transformer uses add-apikey to assign roles to that API key.

The first value provided to add-apikey is the SHA-256 hash of the API key. A value for this can be generated using the sha256sum command-line tool available on Linux machines. The second value is a comma-separated list of security roles to grant to the API key.

The following security roles are available:

Since: DataSource 8

Use in: C

Syntax: add-apikey <SHA256_hash> "<role>[,…​]"

Type: string array

Default value: none

Example:

datasrc-batch-time specifies the time period in seconds for this DataSource application to batch together messages sent to a peer.

As an example, if datasrc-batch-time time is 0.2 seconds, when the DataSource application needs to send a message to the peer and there are no outgoing messages already in the queue, it queues the message and waits 0.2 seconds. Any subsequent outgoing messages are also queued until 0.2 seconds have elapsed, and then all the queued messages are sent to the peer in one batch. Batching messages in this way can improve performance when there is a high rate of data updates being sent to the peer.

If datasrc-batch-time is set to 0.0, messages are not batched.

Use in: C

Syntax: datasrc-batch-time <time-period>

Type: float

Default value: 0.01 (seconds)

Values accepted: 0.0 — 10.0

datasrc-default-obj-hash-size specifies the default number of entries in the hashtable that holds information about this DataSource application’s active subscriptions.

Use in: C

Syntax: datasrc-default-obj-hash-size <number-of-objects>

Type: integer

Default value: 16384 objects

When this DataSource application doesn’t receive an expected DataSource heartbeat from a peer, it waits datasrc-heartbeat-slack-time seconds before disconnecting from the peer and trying to reconnect to it.

datasrc-heartbeat-slack-time applies to all dynamic peers, and static peers that do not have heartbeat-slack-time configured. If add-peer::heartbeat-slack-time is set, it takes precedence over datasrc-heartbeat-slack-time.

After modifying any of the heartbeat configuration items, restart both the DataSource application and its peers for the change to take effect.

Available from: DSDK 7.1.16 (Liberator 7.1.15, Transformer 7.1.10), DS.NET 7.1.16, DataSource for Java 7.1.13

Use in: C, Java

Syntax: datasrc-heartbeat-slack-time <time_in_seconds>

Type: float

Default value: 5.0

The two peers involved in a DataSource connection exchange heartbeat messages at regular intervals so that each of the DataSource applications can check that the other is still present, and take appropriate action if it is not (such as attempt to reconnect to the peer, or fail over to another peer).

datasrc-heartbeat-time defines the time in seconds between each DataSource heartbeat. When two DataSource applications connect, they compare their configured heartbeat times and use the lowest.

datasrc-heartbeat-time applies to all dynamic peers, and static peers that do not have heartbeat-time configured. If add-peer::heartbeat-time is set, it takes precedence over datasrc-heartbeat-time.

After modifying any of the heartbeat configuration items, restart both the DataSource application and its peers for the changes to take effect.

Available from: DSDK 7.1.16 (Liberator 7.1.15, Transformer 7.1.10), DS.NET 7.1.16, DataSource for Java 7.1.13

Use in: C, Java

Syntax: datasrc-heartbeat-time <time_in_seconds>

Type: integer

Default value: 15

datasrc-id specifies the ID number of this DataSource application. You can override this ID for a particular peer connection by adding a local-id option to the add-peer configuration item relating to the connection.

Use in: C, Java

Syntax: datasrc-id <ID-number>

Type: integer

Default value: 0

datasrc-interface binds the DataSource server socket to a single IP address or to a single IP address wildcard.

If no value is specified (default), then the DataSource server socket is available on all IPv4 and IPv6 addresses on the host. Some system settings affect this functionality; see the two tables below.

The behaviour of datasrc-interface differs slightly between C-based and Java-based DataSource applications:

Use in: C, Java

Syntax: datasrc-interface <address_or_hostname>

Type: string

Default value: "" (empty string)

Specifies flags determining the behaviour of this DataSource application when it starts up and establishes connections to peers, or when it reconnects to peers.

Overridden by add-peer:local-flags for specific peer connections.

Syntax: datasrc-local-flags flag[|flag …​]

Example: datasrc-local-flags nofields

datasrc-local-label sets a name that uniquely defines this DataSource application within the Caplin Platform installation. Together with the remote-label option of add-peer, it replaces datasrc-id and the remote-id option of add-peer.

You can override this label for a particular peer connection by adding a local-label option to the add-peer configuration item relating to the connection (this allows you to implement multiple connections for the same peer).

Use in: C, Java

Syntax: datasrc-local-label <label-name>

Type: string

Default value: [none]

The name of this DataSource application, which the application’s DataSource peers use to identify it.

You can override this name by adding a local-name option to an add-peer entry - the local-name is then the name that the DataSource peer defined by the add-peer entry uses to identify this application.

The name can contain the parameters %a and %h. At run time, %a is replaced by the DataSource application-name, and %h is replaced by the host name of the machine on which the DataSource application is running.

Use in: C, Java

Syntax: datasrc-name <name-of-this-DataSource-application>

Type: string

Default value: %a-%h

datasrc-pkt-log specifies the file path of this DataSource application’s packet log file. It can also be used to turn off packet logging.

The file path can be relative or absolute. If relative, the file path is relative to the value of log-dir.

To disable the log, set the file path to /dev/null.

Use in: C, Java

Syntax: datasrc-pkt-log <file_path>

Type: string

Default value: packet-%a.log

datasrc-port specifies the network port to listen on for non-WebSocket connections from DataSource peers. The default of 0 means that no connections can be made to this DataSource application (in which case, you’d configure the application so that it initiates the connections to its peers; see the addr and port options of add-peer).

To listen for secure (SSL) connections from DataSource peers, you must specify the configuration item datasrc-ssl-port.

Use in: C, Java

Syntax: datasrc-port <port-number>

Type: integer

Default value: 0 (No connections can be made.)

datasrc-public-addresses specifies a list of network addresses at which this DataSource component’s DataSource port (datasrc-port) is available to peers and Discovery. These addresses, together with the DataSource component’s hostname as reported by the OS or as set by datasrc-interface, are registered by the DataSource with Discovery during the Discovery-DataSource handshake. Discovery and peers attempting to connect to this DataSource try addresses in datasrc-public-addresses first before trying the DataSource component’s hostname.

Network addresses may be specified as hostnames, fully-qualified domain names, or IP addresses.

Use in: C (DSDK 7.1.27+, Liberator 7.1.27+, Transformer 7.1.17+), Java (DataSource for Java 7.1.20+)

Compatibility: Discovery 7.1.4

Syntax: datasrc-public-addresses address …​

Type: string array

Default value: none

Example: datasrc-public-addresses liberator1 liberator1.example.com

datasrc-ws-port specifies the network port to listen on for WebSocket connections from DataSource peers. The default of 0 means that no WebSocket connections can be made to this DataSource application (in which case, you’d configure the application so that it initiates the WebSocket connections to its peers; see the addr, port, websocket and websocket-path options of add-peer).

Unlike datasrc-port there is no separate port for secure connections. This port listens for secure (SSL) connections from DataSource peers if datasrc-ssl-enable is TRUE.

Use in: C, Java

Since: DSDK 7.1.30 (Liberator 7.1.30, Transformer 7.1.18), DataSource for Java 7.1.22, Discovery 7.1.5

Syntax: datasrc-ws-port <port-number>

Type: integer

Default value: 0 (No connections can be made.)

datasrc-ws-path specifies the path to listen on for WebSocket connections from DataSource peers. See the websocket-path option of add-peer when configuring applications initiating the connection.

Use in: C, Java

Since: DSDK 7.1.30 (Liberator 7.1.30, Transformer 7.1.18), DataSource for Java 7.1.22, Discovery 7.1.5

Syntax: datasrc-ws-path <path>

Type: string

Default value: /

datasrc-reject-new-peers when set to TRUE, prevents a peer from connecting to this DataSource application when there’s already a peer connected that has the same label (or the same ID if there’s no label defined for the peer).

Setting datasrc-reject-new-peers prevents abnormal behaviour if the system is wrongly configured and two peers are inadvertently deployed with the same labels or IDs. By default, if a DataSource peer tries to connect to this DataSource application but there is already one connected with the same label or ID, the current peer is disconnected and the new one is allowed to connect. The first peer could then attempt to reconnect, which would cause the second peer to be disconnected. The second peer would then in turn attempt to reconnect, and so on.

This configuration item turns off this default behaviour, so the second DataSource peer is never allowed to connect.

The peer’s label is defined in the remote-label option of this DataSource application’s add-peer configuration entry. The peer’s ID is defined in the remote-id option of the add-peer.

Use in: C, Java

Syntax: datasrc-reject-new-peers <boolean>

Type: boolean

Default value: FALSE (a new peer with the same label or ID as an existing one is allowed to connect)

datasrc-tcp-nodelay-off specifies whether the DataSource application’s peer connection sockets should have the TCP_NODELAY feature turned off. The default is FALSE, which means TCP_NODELAY is enabled. Setting this configuration item to TRUE disables TCP_NODELAY.

Use in: C, Java

Syntax: datasrc-tcp-nodelay-off <boolean>

Type: boolean

Default value: FALSE (TCP_NODELAY is disabled)

peer-accept-wait-time specifies the time in seconds to wait for an information packet when accepting a connection from a peer.

Use in: C

Syntax: peer-accept-wait-time <time-interval>

Type: float

Default value: 10.0 seconds

peer-debug-metrics-interval specifies the interval in seconds between updates of the following Prometheus metrics:

Use in: Java

Syntax: peer-debug-metrics-interval <time-interval>

Type: int

Default value: 10 seconds

peer-monitor-interval specifies the interval in seconds at which statistics about this DataSource application’s connection to its DataSource peers are read and transferred to the application’s JMX monitoring module.

You can override this item for a specific peer by adding a monitor-interval option to this DataSource application’s add-peer configuration item for the peer.

Use in: C

Syntax: peer-monitor-interval <time-interval>

Type: float

Default value: 30.0 seconds

peer-request-timeout is a higher-precedence alias of source-request-timeout.

Peer-request timeout precedence (highest first): add-peer::request-timeout > peer-request-timeout > source-request-timeout.

Use in: C

Syntax: peer-request-timeout <timeout-in-seconds>

Type: float

Default value: -1.0 (no timeout)

peer-thread-pool-size specifies the maximum number of pooled connection threads allocated to handle the DataSource application’s peer connections that don’t have an explicitly named thread.

If you configure more than 30 peers, you should define peer-thread-pool-size. For more about peer-thread-pool-size and the thread-name option of add-peer, see Configure threads for peer communication.

If this item is configured, the DataSource application creates a pool of threads that are used to handle peer connections. The item specifies the maximum number of threads in the pool.

When a peer connection doesn’t have a named thread configured for it (see the thread-name option of the add-peer item), communication with the peer is handled on one of the pool threads. However, if peer-thread-pool-size is 0 or not defined, an unnamed thread is created and dedicated to that peer. If peer-thread-pool-size is less than the number of peer connections with no explicitly named thread, these connections share threads from the pool.

Use in: C

Syntax: peer-thread-pool-size <max-number-of-threads>

Type: integer

Default value: -1 (no pool configured - see explanation below)

source-request-timeout specifies a timeout in seconds for a DataSource peer’s response to a request (for example, a subscription request) sent to it by this DataSource application. If the peer doesn’t respond within this time, the requested object is assumed to be not available and the application sends a discard message for the object to the peer.

Peer-request timeout precedence (highest first): add-peer::request-timeout > peer-request-timeout > source-request-timeout.

A request to a peer in the context of a data-service is also subject to the data-service’s request timeout. For more information, see add-data-service::request-timeout and service-request-timeout.

Use in: C

Syntax: source-request-timeout <timeout-in-seconds>

Type: float

Default value: -1.0 (no timeout)

See also: