Data services configuration

These DataSource configuration items define a DataSource application’s data services.

When the DataSource application requests an object (such as subscribing to a subject), the data services system determines, through the configuration, which peers the request goes to instead of you having to specify this explicitly in the application’s code. It also prevents a request from going to any peer more than once.

Of course, if your DataSource application is an Integration Adapter that only sends data to its peers, you don’t need to define any data services for it.

Data services can’t be configured in Java-based DataSource applications.
"this DataSource application" means the DataSource application for which you are defining the configuration. "DataSource peer" or "peer" means a DataSource application that this DataSource application communicates with.

For some examples of how data services are configured, see the data services features and concepts page.

add-data-service

add-data-service specifies a data service that this DataSource application can use.Through the add-source-group option and the add-priority option within add-source-group, it allows you to specify multiple peers that provide data for the service, load balance the provision of data across multiple peers, and configure alternate peers that your DataSource application can failover to. For examples of these setups, see the data services features and concepts page.

Also see the note below about default behaviour.

Use in: C

Syntax
add-data-service

  add-source-group
    …
  end-source-group

  affinity                        [array of strings]
  disable-auto-status             [boolean]
  discard-timeout                 [float]
  exclude-pattern                 [array of strings]
  include-pattern                 [array of strings]
  min-source-groups               [integer]
  nodata-ignore-request-timeout   [boolean]
  rebalance-enable                [boolean]
  rebalance-time                  [float]
  request-timeout                 [float]
  required-state                  [integer/string]
  service-name                    [string]
  service-type                    [integer/string]
end-data-service
Options
add-source-group

Type: group item | Default: see Default behaviour

The add-source-group group item specifies, via the add-priority option, one or more DataSource peers that provide the data for the data service.

  • By using multiple add-source-group items, you can specify multiple sources of data for the data service.

  • By using multiple add-priority options within an add-source-group item, you can specify alternative sets of peers to failover to.

For examples of these setups, see the data services features and concepts page

affinity

Type: array of strings | Default: null

Deprecated from version 6.2.6+ of Liberator and version 6.2.5+ of Transformer; use the affinity option within the add-source-group group item instead.

disable-auto-status

Type: boolean | Default: FALSE

If you don’t want this DataSource application to send subject status changes for this data service to subscribing peers, set disable-auto-status to TRUE You’d typically do this in a Transformer, to prevent it from sending subject status changes unnecessarily on to subscribing Liberators.

discard-timeout

Type: float | Default (Liberator): value of active-discard-timeout | Default (all others): -1.0

Specifies the time in seconds for which this DataSource application holds on to the data for a subscribed subject once the last client application has unsubscribed from that data. After this time, the object is deleted from the DataSource application’s cache, and the DataSource application sends a discard instruction to the data service’s peer(s) to cancel the subscription.

For Liberator only:

  • This option overrides the setting of the global time out for active objects, active-discard-timeout.

  • If an object obtained by this data service was specified through the Liberator’s add-object configuration item, the discard-timeout option of the add-object (if any) overrides this discard-timeout.

exclude-pattern

Type: array of strings | Default: null

One or more regular expressions that determine subjects for which this data service must not provide data.

This option is optional. Use it in conjunction with the include-pattern option.

For example, include-pattern ^/FX/MAJOR/ specifies that this data service is to provide data for subjects that begin with /FX/MAJOR/

Then specifying exclude-pattern ^/FX/MAJOR/AUDUSD ^/FX/MAJOR/AUDGBP excludes the currency pairs AUDUSD and AUDGBP from the prices returned by the data service.

You can have multiple occurrences of exclude-pattern in the same add-data-service item, so the above example could be written as:

exclude-pattern ^/FX/MAJOR/AUDUSD
exclude-pattern ^/FX/MAJOR/AUDGBP
include-pattern

Type: array of strings | Default: ^/

One or more regular expressions that determine the subjects for which this data service provides data.

For example: include-pattern ^/FX/MAJOR/ ^/FX/MINOR/ specifies that this data service is to provide data for subjects that begin with /FX/MAJOR/ or /FX/MINOR/

You can have multiple occurrences of include-pattern in the same add-data-service item, so the above example could be written as:

include-pattern ^/FX/MAJOR/
include-pattern ^/FX/MINOR/

You should normally specify an include-pattern option, otherwise the data service will return updates for all subjects (that is, subjects starting with /) that the service’s DataSource peers can provide.

Also see the exclude-pattern option.

min-source-groups

Type: integer | Default: 1 | Since: C DataSource API 7.1.6 (Liberator 7.1.5, Transformer 7.1.5)

The minimum number of source groups within this data service that must have a status of 'up' for this data service to have a status of 'up'.

nodata-ignore-request-timeout

Type: boolean | Default: FALSE

A DataSource peer that provides a data service can send back a "no data" response if there is no data available for a requested subject. The data service then discards any outstanding requests for the subject.

The nodata-ignore-request-timeout option applies when multiple peers can supply requested data for a service.

When this option is TRUE, if any peer providing the service sends back a "no data" response, this response is immediately propagated from the DataSource API to the custom code of the DataSource application - the request-timeouts (or source-request-timeouts) for the remaining peers are ignored.

When this option is FALSE, if a "no data" response is received from a peer providing the service, the data service continues wait for the data from any of the other peers, applying the request-timeouts (or source-request-timeouts), until either the data is returned by at least one peer, or all the peers have timed out or returned a "no data" response.

request-timeout

Type: float | Default: value of service-request-timeout

Specifies the timeout in seconds for all subscription requests sent to this data service. If the DataSource peers providing the service don’t respond within this time, the requested subject is assumed to not be available and the application sends a discard message for the subject to the peers.

A value of 0 means no timeout is set and this DataSource application then waits indefinitely for this data service to respond to a request. Use this setting in Liberator if you want to allow all Liberator’s subscription requests for this data service to be open subscriptions (see How can I…​ Keep subscriptions open when the peer is down).

request-timeout overrides the global service-request-timeout configuration item.

rebalance-enable

Type: boolean | Default: null | Since: DataSource for C (DSDK) 7.1.21

Enable or disable automatic rebalancing of workload in source groups that don’t have source affinity enabled.

When rebalancing is enabled and a peer in a priority group changes its status to UP, after a set time (rebalance-time) the data service discards subscriptions to peers in the priority group and re-requests them, balancing the requests evenly over peers in the priority group.

Overrides the global item service-rebalance-enable.

rebalance-time

Type: float | Default: null | Since: DataSource for C (DSDK) 7.1.21

Set the time to wait (in seconds) after a peer in a priority group (see add-priority) sets its status to UP before rebalancing workload across all peers in the priority group.

Overrides global item service-rebalance-time.

To enable service rebalancing, see service-rebalance-enable and rebalance-enable.

required-state

Type: integer/string | Default: 0

This option only applies to Liberator.

The minimum state this data service must be in for Liberator to begin accepting client connections.

Before version 7.1.19, Liberator ignores required-state after it has begun accepting connections.

From version 7.1.19, Liberator can be configured to continue to enforce required-state after Liberator has begun accepting connections. If session-requires-services is set to true, then required-state is the minimum state this data service must be in for Liberator to accept and retain client connections.

Valid values for required-state

  • 0, down, optional: Down (no source groups are up, or one or more required source groups are down). Default

  • 1, limited: Limited (at least one source group is up, and all required source groups are up)

  • 2, ok, up: Up (all source groups are up)

See also:

service-name

Type: string | Default: null

The name of the data service.

In deployments of the Caplin Platform with Discovery, providing DataSources that are configured with a matching service name in discovery-provide-service automatically attempt peer connections to this DataSource. For more information, see Peer discovery.

service-type

Type: integer/string | Default: 3

Specifies whether the DataSource peers that supply data for the service are of type active, broadcast, or both.

Possible values are:

  • active or 1

    This data service is provided by active DataSource peers. An active DataSource keeps track of which subjects have been requested and sends updates for those subjects only (see Active subscription model).

  • broadcast or 2

    This data service is provided by broadcast DataSource peers.

  • active|broadcast or 3

    This data service is provided by both active and broadcast peers.

You can use this option to override the local-type option of a peer’s add-peer configuration. For example, assume a peer that supplies data for a data service is configured as an active peer and also is configured to broadcast data (that is, its add-peer's local-type option is set to broadcast|active). If you don’t want your data service to receive the peer’s broadcast data, set the add-data-service's service-type option to active.

add-source-group

add-source-group specifies, via the add-priority option one or more DataSource peers that provide the data for the data service defined by add-data-service. By using multiple add-source-group items, you can specify multiple sources of data for the data service. By using multiple add-priority options within an add-source-group item, you can specify alternative sets of peers to failover to. If you specify more than one peer within an add-priority item, subscriptions are load-balanced across them. For examples of these setups, see the data services features and concepts page.

Use in: C

Syntax
add-data-service
   …
   add-source-group
      add-priority
        …
      end-priority
      …
      affinity         [array of strings]
      contrib          [boolean]
      min-priorities   [integer]
      required         [boolean]
      retry-time       [float]
   end-source-group
   …
end-data-service
Options
add-priority

Type: group item | Default: see Default behaviour

Specifies one or more DataSource peers that this DataSource application can use to provide data for a data service. If you specify more than one peer in an add-priority option, subscriptions are load-balanced across those peers.

You can also use multiple instances of add-priority to implement failover to alternative peers.

Because the remote-label option of add-priority points to an add-peer configuration item, this DataSource application will first try to fail over to any alternative hosts that you’ve defined in the addr option of the add-peer item.

For examples of load balancing and failover configurations, see the data services features and concepts page.

affinity

Type: array of strings | Default: null

Enables the alternative source affinity load-balancing algorithm for load-balanced adapters in this source group.

Syntax: affinity <affinity_key> <regular_expression>

For a full description of the syntax, guidelines on use, and a worked example, see Load balancing adapters using source affinity.

contrib

Type: boolean | Default: FALSE

When set to TRUE, this option declares that peers listed in this source group provide contributions only (they don’t accept subscription requests).

A source group containing contribution-only peers is a required addition to a data service that acts as a conduit for a messaging channel from one DataSource to another.

For example, consider a data service on Transformer that routes a trade channel from Liberator to a trade adapter. Liberator is included in its own source group in the data service so that Transformer accepts Liberator as a valid source of data for the channel, and the contrib option of the source group is enabled to stop Transformer from routing subscription requests for the trade channel to Liberator.

A contribution-only source group in a Transformer data service
add-data-service
  service name trade-channel
  include-pattern '^/PRIVATE/FX'

  add-source-group
    add-priority
      remote-label trade-adapter
    end-priority
  end source group

  add-source-group
    contrib
    add-priority
      remote-label liberator
    end-priority
  end-source-group
end-data-service
min-priorities

Type: integer | Default: 1 | Since: C DataSource API 7.1.6, Liberator 7.1.5, and Transformer 7.1.5

Minimum number of priority groups in this source group that must have a status of 'up' for this source group to have a status of 'up'.

required

Type: boolean | Default: FALSE

When set to TRUE:

  • If one of the peers in this source group is down, then updates from the data service are marked as stale and the status of the data service is changed to stale.

  • If only one peer in this source group responds and that peer responds with NODATA, then the DataSource API propagates the NODATA message to application code.

retry-time

Type: float | Default: 30.0

After finding that none of the peers defined by add-priority options in the source group have responded to a request, this DataSource application waits retry-time seconds before reissuing the request to the source group.

Valid values

  • DSDK < 7.1.29: Minimum value 1.0. Any value < 1.0 is rounded up to 1.0.

  • DSDK >= 7.1.29: Minimum positive value 1.0. Any positive value < 1.0 is rounded up to 1.0. Any value <= 0 disables the retry feature.

The DataSource application issues the request to each of the peers in the source group in turn. Each request is timed out according to the setting of add-peer::request-timeout or source-request-timeout. If none of the peers in the source group replies, the DataSource application waits retry-time and then tries each connection again in turn. It repeats this sequence until the master timeout for the service, defined by service-request-timeout or add-data-service::request-timeout, expires. If add-data-service::request-timeout is set to -1, the DataSource reissues the request indefinitely.

add-priority

add-priority specifies one or more DataSource peers that this DataSource application can use to provide data for a data service. By using multiple add-priority options within an add-source-group item, you can specify alternative sets of peers to failover to. If you specify more than one peer within an add-priority item, subscriptions are load-balanced across them; for an example of this, see the data services features and concepts page.

Use in: C

Syntax
add-data-service
  ...
  add-source-group
    ...
    add-priority
      label                   [array of strings]
      min-peers               [integer]
      remote-label            [array of strings]
      remote-label-regex      [string]
      ...
    end-priority
  end-source-group
end-data-service
Options
min-peers

Type: integer | Default: 1 | Since: C DataSource API 7.1.6, Liberator 7.1.5, and Transformer 7.1.5

The minimum number of peers in this priority group that must be connected for this priority group to have a status of 'up'.

label

Type: array of strings | Default: see Default behaviour | Deprecated: use remote-label instead

One or more labels identifying DataSource peers that supply data for this DataService. Each label must match the label option of an add-peer configuration item that defines a DataSource peer. If you specify more than one peer, either through multiple values of a label option or by multiple label options, subscriptions to the referenced peers are load-balanced across the peers.

Also see Note 1.

remote-label

Type: array of strings | Default: see Default behaviour

One or more labels identifying DataSource peers that supply data for this DataService. Each label must match the remote-label option of an add-peer configuration item that defines a DataSource peer.

If you specify more than one peer, either through multiple values of a remote-label option or by multiple remote-label options, subscriptions to the referenced peers are load-balanced across the peers.

Also see Note 1 below, and the example in DataSource peers configuration (part 1).

remote-label-regex

Type: string | Default: null | Since: DataSource for C (DSDK) 7.1.21, Liberator 7.1.21, Transformer 7.1.13

A single regular expression (POSIX ERE) that describes the labels (datasource-local-label) of DataSource peers that supply data for this DataService.

This option is available only for DataSources connected to Discovery (see discovery-addr).

Only one remote-label-regex option may be specified per priority group. If multiple remote-label-regex options are specified for a priority group, only the final option applies.

Subject requests to the peers defined by remote-label-regex are load balanced using the standard workload-based algorithm.

Note 1: Platform blades, and Liberator and Transformer, that run under the Caplin Platform Deployment Framework use the deprecated label option in their configuration, rather than remote-label. This will change in a future release of the Framework.

broadcast-cleanup-age

broadcast-cleanup-age specifies how old in minutes broadcast objects provided by this DataSource application need to be before they’re removed from the DataSource application’s cache. Every broadcast-cleanup-period minutes past broadcast-cleanup-time, the application checks the cache and deletes any broadcast objects that are older than broadcast-cleanup-age.

This configuration item is ignored if broadcast-cleanup-time isn’t set.

Use in: C

Syntax: broadcast-cleanup-age <age-in-minutes>

Type: integer

Default value: -1 (no timeout set, so broadcast items are never removed from the cache)

broadcast-cleanup-period

broadcast-cleanup-period specifies how often in minutes, relative to broadcast-cleanup-time, broadcast objects provided by this DataSource application are removed from the DataSource application’s cache. Every broadcast-cleanup-period minutes past broadcast-cleanup-time, the application checks the cache and deletes any broadcast objects that are older than broadcast-cleanup-age.

This configuration item is ignored if broadcast-cleanup-time isn’t set.

Use in: C

Syntax: broadcast-cleanup-period <time-interval-in-minutes>

Type: integer

Default value: 1440 minutes (= 24 hours; that is, clean up every day at the time set by broadcast-cleanup-time.)

broadcast-cleanup-time

broadcast-cleanup-time specifies when broadcast objects provided by this DataSource application are first removed from the DataSource application’s cache. This time is the number of minutes after midnight (local time) on the day the DataSource was started, or if it’s greater than or equal to 1440 (24 hours), it’s the number of minutes past midnight on a Sunday. At broadcast-cleanup-time, and subsequently every broadcast-cleanup-period minutes past broadcast-cleanup-time, the application checks the cache and deletes any broadcast objects that are older than broadcast-cleanup-age.

By default, the DataSource application’s cache is not cleaned up on a timed basis.

Use in: C

Syntax: broadcast-cleanup-time <time-past-midnight-in-minutes>

Default value: -1 (Don’t clean up the DataSource application’s cache on a timed basis, so broadcast-cleanup-age and broadcast-cleanup-period are ignored.)

We recommend that you configure a daily cleanup of broadcast objects. For example, set broadcast-cleanup-time to 0 and broadcast-cleanup-period to 1440 mins (= 24 hrs), so that broadcast objects are cleaned up at midnight every day. Use broadcast-cleanup-age to control how old the broadcast objects should be before they’re actually removed from the cache.

cleanup-stale-timeout

cleanup-stale-timeout specifies when to clean up stale objects (such as subscriptions). When all the DataSource peers in a data service have been down for cleanup-stale-timeout seconds, all the objects that can no longer be updated by the service are deleted from this DataSource application’s cache.

Use in: C

Syntax: cleanup-stale-timeout <timeout-in-seconds>

Type: float

Default value: -1.0 (no timeout)

service-global-peer-balance

service-global-peer-balance determines how the workload-based load-balancing algorithm calculates the workload of peers in an add-priority set.

  • When service-global-peer-balance is set to false (the default), peer workload is defined as the number of subscriptions the peer is serving to the data service.

  • When service-global-peer-balance is set to true, peer workload is defined as the total number of subscriptions the peer is serving to all data services on the DataSource.

For deployments in which peers serve more than one data service, set service-global-peer-balance to true to provide the load-balancing algorithm with a more accurate determination of peer workload.

This configuration item has no affect on the source-affinity load-balancing algorithm (add-source-group:affinity).

Example: load balancing with and without service-global-peer-balance

Consider two data services that have a peer in common (PeerA):

service1 (/SERVICE1)
add-data-service
  service-name service1
  include-pattern /SERVICE1
  add-source-group
    add-priority
      remote-label PeerA
      remove-label PeerB
    end-priority
  end-source-group
end-data-service
service2 (/SERVICE2)
add-data-service
  service-name service2
  include-pattern /SERVICE2
  add-source-group
    add-priority
      remote-label PeerA
      remove-label PeerC
    end-priority
  end-source-group
end-data-service

Assume the following requests have been made to service1 (in this order):

  1. /SERVICE1/A

  2. /SERVICE1/B

  3. /SERVICE1/C

  4. /SERVICE1/D

  5. /SERVICE1/E

  6. /SERVICE1/F

  7. /SERVICE1/G

  8. /SERVICE1/H

…resulting in the following distribution of subscriptions:

Data Service Peer Subscriptions Total Subscriptions

service1

PeerA

4

4

PeerB

4

4

service2

PeerA

0

4

PeerC

0

0

Now we request /SERVICE2/A. Because each peer in service2 has 0 service2 requests and PeerA is the 1st peer, PeerA gets the request even though it already has 4 service1 requests and PeerC has no requests at all.

If service-global-peer-balance were set to true, then peer C would get the new requests and the next 3 service2 requests until it had a total of 4 before service2 requests went to peer A.

Use in: C

Since: DataSource for C (DSDK) 7.1.28, Liberator 7.1.28

Syntax: service-global-peer-balance

Type: boolean

Default value: false

service-rebalance-enable

service-rebalance-enable enables automatic rebalancing of workload in all data-service source groups that don’t have source affinity enabled.

To enable or disable rebalancing for a specific data service, use rebalance-enable, which overrides service-rebalance-enable.

When rebalancing is enabled and a peer in a priority group changes its status to UP, after a set time (service-rebalance-time) the data service discards subscriptions to peers in the priority group and re-requests them, balancing the requests evenly over peers in the priority group.

Service rebalancing is disabled by default and must be enabled globally (using service-rebalance-enable) or for specific data services (using rebalance-enable) where required. The one exception to this is when no data services have been configured: in this one case, service-rebalance-enable defaults to true.

Use in: C

Since: DataSource for C (DSDK) 7.1.21

Syntax: service-rebalance-enable

Type: boolean

Default value:

  • No configured data services: true

  • Configured data services: false

service-rebalance-time

service-rebalance-time sets the time to wait (in seconds) before rebalancing workload after a peer in priority group changes its status to UP.

Service rebalancing is disabled by default. To enable service rebalancing, see service-rebalance-enable and the add-data-service option rebalance-enable.

service-rebalance-time is overridden by the add-data-service option rebalance-time

Use in: C

Since: DataSource for C (DSDK) 7.1.21

Syntax: service-rebalance-time <seconds>

Type: float

Default value: 5.0

service-request-timeout

service-request-timeout specifies the global request timeout in seconds for all data services.

This is the time that this DataSource application waits for a data service to respond to a request (for example, a subscription request). If the DataSource peers providing the service don’t respond within this time, the requested object is assumed to not be available and the application sends a discard message for the object to the peers.

A value of 0 or -1 means no timeout is set and this DataSource application then waits indefinitely for a data service to respond to a request. Use this setting in Liberator if you want to allow all of the Liberator’s subscription requests to be open subscriptions (see How can I…​ Keep subscriptions open when the peer is down).

service-request-timeout is a global setting applied to all data services. You can override this timeout for a particular data service by setting the request-timeout option of add-data-service.

Use in: C

Syntax: service-request-timeout <timeout-in-seconds>

Type: float

Default value: 10.0 seconds

Data service default behaviour

If you don’t define at least one data service for your DataSource application, then the DataSource will use the following default data service and request each subject request from all known peers of the DataSource:

Default data service
add-data-service
   service-name         default
   required-state       down
   include-pattern      ^/

   add-source-group
      required         FALSE
      add-priority
         remote-label  source1
      end-priority
   end-source-group

   add-source-group
      required         FALSE
      add-priority
         remote-label  source2
      end-priority
   end-source-group
   ...
   add-source-group
      required         FALSE
      add-priority
         remote-label  sourceN
      end-priority
   end-source-group
end-data-service

See also: