Measuring latency in record updates

You can configure your DataSource applications so they record the latency of data updates to each record as the record passes from one DataSource application to the next.

Overview

A record update typically passes through a number of DataSource applications running on the Caplin Platform. For example, a price update record that originates in a pricing adapter may pass through a Transformer (perhaps to be filtered), followed by a Liberator, before reaching a client application such as Caplin FX Professional.

The update takes a finite time to pass from the Pricing Adapter to Transformer and on to Liberator. Each of these components adds a processing delay, and the update also takes time to travel through the network connecting one component to another. This is illustrated in the following diagram:

Latency chain example: FX Pricing Adapter to Transformer to Liberator to StreamLink in client

Here a pricing adapter receives an FX price notification from a pricing system at time T1. The FX Pricing Adapter creates a record, which leaves the adapter at time T2. The record enters a Transformer at time T3. The Transformer sends the record on to a Liberator at time T4, which the Liberator receives at time T5. The record exits Liberator at time T6, when it is sent by RTTP message to a StreamLink client.

You can configure each of the DataSource applications (adapter, Transformer, and Liberator) to record time elapsed since an initial timestamp at each of the data points in the diagram above. The initial timestamp is normally, but doesn’t have to be, recorded at time T1.

How latency data is recorded

Latency data is recorded to three fields, which by default are LTY_INIT_TS, LTY_LIST_EVENT, and LTY_LIST_TS:

Example record containing latency chain data (other fields omitted)
Field Name Field Description Field Value

LTY_INIT_TS

Initial timestamp

1125062541880

LTY_LIST_EVENT

List of event names

fxadapter1_X,transformer1_E,transformer1_X,rttpd1_E,rttpd1_X

LTY_LIST_TS

List of event values

0.000642066,​0.001400478,​0.001432103,​0.001542571,​0.001624482

The values recorded in the fields LTY_LIST_EVENT and LTY_LIST_TS record a chain of latency data:

Events from LTY_LIST_EVENT correlated with values from LTY_LIST_TS
LTY_LIST_EVENT event LTY_LIST_TS delta Description

fxadapter1_X

0.000642066

Time elapsed between the initial timestamp and the DataSource packet exiting (X) fxadapter1

transformer1_E

0.001400478

Time elapsed between the initial timestamp and the DataSource packet entering (E) transformer1

transformer1_X

0.001432103

Time elapsed between the initial timestamp and the DataSource packet exiting (X) transformer1

rttpd1_E

0.001542571

Time elapsed between the initial timestamp and the DataSource packet entering (E) liberator1 (the default name for a Liberator is 'rttpd')

rttpd1_X

0.001624482

Time elapsed between the initial timestamp and the DataSource packet exiting (X) liberator1 (the default name for a Liberator is 'rttpd')

The resolutions of the initial timestamp and each data point are version dependent:

Resolution of the initial timestamp
Platform version Timestamp resolution Example

>= 7.1

Seconds since 1 Jan 1970, with a nanosecond fractional component

1125062541.880462579

< 7.1

Milliseconds since 1 Jan 1970

1125062541880
Resolution of individual data events
Platform version Event resolution Example

>= 7.1

Seconds, with a nanosecond fractional component

0.000587903

< 7.1

Milliseconds

0.587903

Requirements

To measure latency over the journey of a DataSource record from origin to exit, you require the following:

  • Configuration changes to all DataSource applications through which the record travels

  • The record must have an initial latency timestamp added at origin. The originating adapter or module may provide for this already by providing a set of test subjects you can use to measure latency; if it does not, further software development of the adapter or module is required.

  • The following minimum software versions for measuring latency in Type 1 Records and Generic Objects:

    Type 1 Records Generic Objects

    C DataSource API

    6+

    6.2.8+

    Java DataSource API

    6+

    6.2.7+

    .NET DataSource API

    6+

    6.2.8+

    Liberator

    6+

    6.2.10+

    Transformer

    6+

    6.2.7+

Enabling latency measurement

Latency measurement is disabled by default. To enable latency measurement for records served by an adapter, follow the steps below:

  1. Synchronise the clocks of all your Caplin Platform servers with a common time source.

  2. Add the following configuration to the main configuration override file of Liberator, Transformer, and the adapter (usually rttpd.conf, transformer.conf, and datasource.conf respectively):

    # Enable latency measurement
latency-chain-enable TRUE

# Field numbers for latency fields (1)
add-field  LTY_INIT_TS     -10302
add-field  LTY_LIST_EVENT  -10303
add-field  LTY_LIST_TS     -10304
    1 The field numbers are arbitrary, and you can change the field numbers if they clash with existing field numbers in your deployment. Any changes you make must be consistent across all DataSources.

  3. Customise the adapter to publish a set of test subjects that include an initial timestamp field. For guidance on how to do this, see Adding the initial timestamp

    Measuring latency incurs a small processing overhead, so for production systems we recommend that an adapter add the initial timestamp only to updates for test subjects.

  4. Restart all DataSources to enable the configuration changes.

  5. Using the Liberator Explorer web application, connect to your Liberator and request the latency test subjects published by the adapter. Note the values in the three latency fields.

    The Liberator Explorer web application is included in the default Liberator website, but you can use it to connect to any Liberator if you have the associated KeyMaster URL and have previously logged into the web application server hosting KeyMaster.

Since latency recording is really only meant to be used as a tool for tuning your system, a flexible way to implement it is to implement the Integration Adapter so that you can dynamically control whether or not it creates records with an initial timestamp. You can then permanently turn on latency recording for all your DataSource applications by setting latency-chain-enable. At run-time, latency records are only sent through the Platform components if you’ve turned on their generation at the Integration Adapter. When latency record generation is turned off, the rest of the components in the chain will still check every record for the presence of the timestamp field, but the overhead of doing this is trivial.

When you deploy a Liberator in the Caplin Deployment Framework, the Liberator fields configuration item requested-fields-only is set to TRUE by default. The effect of this is that, even when latency recording is turned on, Liberator doesn’t send the latency chain data back to clients unless they explicitly request the latency timestamp fields. If you want to always pass the latency data through to clients, you must set requested-fields-only to FALSE; however, this means that all other record updates sent to clients may contain fields that the client is not interested in, increasing the Liberator to client traffic unnecessarily.

Regardless of whether the latency data is sent to clients, you can always view it using the Caplin Management Console — see Monitor latency statistics using the CMC.

Adding the initial timestamp

The initial timestamp in a latency chain must be added manually. When the initial timestamp is present on a record, latency data points relative to the initial timestamp are recorded automatically by all DataSource components that have latency-chain-enable set.

To avoid unnecessary overhead when serving 'live' data, we recommend that the initial timestamp be added only to dummy subjects that are served by your adapter specifically for the purpose of measuring latency.

Adding the initial timestamp using a DataSource API

To start off the chain of latency data you must record an initial timestamp in the relevant data records within the first DataSource application in the chain. This would usually be done in the Integration Adapter from where the data is sent into the Caplin Platform.

You create the timestamp by calling a DataSource API function or method on the record, which records the time in the record field whose name is defined by the configuration item latency-chain-init-ts-field, as explained in How does latency recording work? above.

The method you call depends on the language in which you’ve implemented your DataSource application:

C DataSource API

ds_add_data(ds_data_t *data, int32_t fieldnum, const char *value)

See Creating Data Objects in the C DataSource API documentation.

Java DataSource API

RecordMessage.setInitialLatencyChainTime()

.NET DataSource API

IRecordMessage.InitialLatencyChainTime()

IGenericMessage.InitialLatencyChainTime()

Adding the initial timestamp using Transformer’s Lua pipeline API

If you want to start recording the latency chain from Transformer (for example, because the data whose latency you want to track has been created in Transformer), you can set the initial timestamp from within a Transformer pipeline.

To do this in Lua, follow the steps below:

  1. Create the record:

    local myrecord = createrecord("MYSUBJECT")

  2. Set the timestamp field LTY_INIT_TS to a millisecond value (pre Transformer 7.1) or a nanosecond timestamp in the format 'S.sssssssss', where 'S' is the number of seconds from 1 January 1970 and 'ssssssssss' is a fractional component that runs to 9 digits (Transformer 7.1 and higher):

    myrecord:setfield("LTY_INIT_TS", timestamp)

The example uses the default name of the initial timestamp field (LTY_INIT_TS).

Adding custom data points to the latency chain

You can add additional data points to a latency chain. For example, you may want to record how long it takes the DataSource application to perform a specific calculation.

Before you write additional instrumentation points to a record, check the following is true:

  • Latency chaining is enabled (check the value of the latency-chain-enable configuration item).

  • The initial latency-chain timestamp has been set (indicating that this is a record for which latency is being measured).

To add a custom data point to a latency chain, use one of the API calls below:

C DataSource API

void ds_add_latency_chain

void ds_add_latency_chain2

Java DataSource API

RecordMessage.addLatencyChainPoint()

.NET DataSource API

IRecordMessage.AddLatencyChainPoint()

IGenericMessage.AddLatencyChainPoint()

Changing the names of the latency chain fields

To change the names of the latency chain fields from their defaults (LTY_INIT_TS, and so on), define the following configuration items in each of the DataSource applications in the chain. Each application must have the same definitions.

Configuration item Default value

Initial timestamp

latency-chain-init-ts-field

LTY_INIT_TS

List of event names

latency-chain-list-event-field

LTY_LIST_EVENT

List of event values

latency-chain-list-ts-field

LTY_LIST_TS

Comparison with Liberator’s timestamp field

You can configure Liberator to add a timestamp to records just before Liberator streams the records to clients over RTTP. The timestamp provides a simple way for client applications to calculate the time taken for records to reach them from Liberator.

Liberator’s timestamp field is independent of the gathering of latency chain statistics, and it is not affected by the value of the latency-chain-enable configuration item. Liberator’s timestamp can be enabled in addition to, or instead of, the latency chain statistics described on this page.

To enable the Liberator timestamp, set a name for the timestamp field using the global timestamp-field configuration item or the timestamp-field option of add-object, which overrides the global timestamp-field configuration item.

Liberator only writes the timestamp to records that don’t already contain a field of that name.

For information on the precision and format of the timestamp, see the documentation for the global timestamp-field configuration item.

Liberator throttling and bursting

Latency measurements are affected by some Liberator configuration settings. The two main features that can delay messages are throttling and bursting.

Object throttling by default is set to 1 second, which means that an update could be delayed by up to 1 second as a result. Bursting on client session output is set by default to 0.5 seconds, so an update could get delayed by a further 0.5 seconds on top of the throttling delay.

Both throttling and bursting have their benefits. Throttling prevents sending out multiple updates to the same object in a short space of time, while bursting can improve overall performance in a system with a large number of clients by batching together small messages when output to a client.

You can turn throttling off if it has a detrimental effect on latency, but you should always have a burst setting, even if it’s a small value such as 0.1 seconds.

See also: