Configuration: Data Services

Data Services are a new feature in the 4.0 Caplin platform and offer improved source mapping when compared to previous versions with the following key features:

• More intuitive configuraiton
• More clearly defined handling of objects receiving data from multiple sources at the same time
• Ability to define both failover and round-robin style mappings.

A Source Mapping definition is called a Service. A service comprises a name, one or more subject patterns to match and one or more source groups.

Service name

This is an indentifier which can be used in status messages. RTTP objects have a field called SID which is the service name.

Note:When picking a service for an object, the first defined match takes priority. As such you should ensure that each object is associated with one and only one service.

The subject patterns

These are regular expression strings to accept or deny for this service. A service will allow multiple patterns including patterns to deny. Exclude patterns can help to define the namespace used.

Note: When checking pattern matches within a service definition, the first match takes priority whether it is an include or an exclude.

The source groups

The main part of the service definition is the source groups. This is one or more sets of sources, plus certain attributes which define the behaviour of the group. In most cases only a single group is defined. When multiple groups are defined for a service it means that a request will attempt to get the object from a source from each group. Multiple groups allow an object to have different sets of fields coming from different sources, for example.

Priorities

Priorities are defined within each source group and are taken in the order in which they are defined. Multiple labels can be defined within each priority. Within a priority, the peer with the lowest number of subscriptions will be used.

Timeouts

There are various timeouts associated with Data Services. By default when an object is requested if after 10 seconds there has been no response, the request will be cancelled and the user informed. This is the service-request-timeout and applies to the whole object request. It is also possible to timeout requests to individual sources, to allow the request to move on to another source. This is the source-request-timeout, which by default is not turned on. The Source timeout is only useful when using a priority group with multiple sources within it as it allows the system to try more than one source before giving up. All these timeouts are only relevant when a source does not respond to a request, when the timeout occurs a discard message is sent to the source so it is aware that the request is cancelled.

Default Behaviour

If no data-service is defined, then the application will act as if the following was defined:

add-data-service
    service-name         default
    include-pattern      ^/
    add-source-group
        required         false
        add-priority
            label        source1
        end-priority
    end-source-group
    add-source-group
        required         false
        add-priority
            label        source2
        end-priority
    end-source-group
    .
    .
    add-source-group
        required         false
        add-priority
            label        sourceN
        end-priority
    end-source-group
end-data-service

This means a request will be sent to all active Data Sources at once.

When using the DataSource SDK to request objects programmatically, the Data Services system will prevent requests going to any peer that has already been sent a request. It also handles which peers requests go to rather than the programmer explicitly specifying the peers.

Conversion

Pre-version 4.0 source mapping should be converted to data services in the following manner:

All peers should have a label defined in the add-peer configuration section, for these example conversion, the label is 'src' appended with the the remote-id

add-source-mapping /A/* 1 should be converted to:

include-pattern      ^/A/
add-source-group
    required         true
    add-priority
        label        src1
    end-priority
end-source-group

add-source-mapping /A/* 1,2 should be converted to:

include-pattern      ^/A/
add-source-group
    required         true
    add-priority
        label        src1
        label        src2
    end-priority
end-source-group

add-source-mapping /A/* 1 2 should be converted to:

include-pattern      ^/A/
add-source-group
    required         true
    add-priority
        label        src1
    end-priority
end-source-group
add-source-group
    required         true
    add-priority
        label        src2
    end-priority
end-source-group

add-source-mapping /A/* 1,2 3,4 should be converted to:

include-pattern      ^/A/
add-source-group
    required         true
    add-priority
        label        src1
        label        src2
    end-priority
end-source-group
add-source-group
    required         true
    add-priority
        label        src3
        label        src4
    end-priority
end-source-group

service-request-timeout

Type: Floating Point Value
Default: 10.000000

Global request timeout for all services

source-request-timeout

Type: Floating Point Value
Default: -1.000000

Global request timeout for all sources

Configuration group add-data-service

Starts the definition of data service.

Format:

add-data-service
    service-name      [value]
    request-timeout   [value]
    exclude-pattern   [value]
    include-pattern   [value]
    add-source-group
        required      [boolean]
        retry-time    [value]
        add-priority
            label     [value]
        end-priority
    end-source-group
end-data-service

Example:

add-data-service 
    service-name       FX 
    exclude-pattern    ^/I/X 
    include-pattern    ^/I/
    include-pattern    ^/B/
    request-timeout    20
    add-source-group
        required       true 
        retry-time     45 
        add-priority 
            label      sourceA 
        end-priority 
        add-priority 
            label      sourceB 
            label      sourceC 
        end-priority 
    end-source-group 

    add-source-group 
        required       false
        add-priority 
            label      source1 
            label      source2 
        end-priority 
    end-source-group 
end-data-service

In this example, the service FX is defined which matches the patterns ^/I/ and ^/B/ but not the symbols ^/B/. There are two service groups, one providing required data and another providing optional data.

For the required group, there are two priorities defined, requests will be preferentially sent to sourceA, however should it not be available then sourceB and sourceC will be asked in a round-robin fashion.

For the non-required group, two sources are defined, requests will be sent to source1 and source2 in a round-robin fashion.

service-name

Type: String
Default: None

Name of the service group

affinity

Type: Function
Default: None

Affinity configuration

service-type

Type: Integer
Default: 3
Acceptable Values:

Name	Value	Desc
active	1
broadcast	2

Type of the service group

request-timeout

Type: Floating Point Value
Default: -1.000000

This option configures the timeout for all requests for this service. Should no response be received from peers within this time, the object will be assumed to be not available.

The default value of -1 means that requests will never timeout.

discard-timeout

Type: Floating Point Value
Default: -1.000000

Timeout for all discards for this service

exclude-pattern

Type: Function
Default: None

Patterns to exclude

include-pattern

Type: Function
Default: None

Patterns to include

required-state

Type: Integer
Default: 0
Acceptable Values:

Name	Value	Desc
down	0
optional	0
limited	1
ok	2
up	2

Required state for this service

cache-contrib

Type: Boolean
Default: FALSE

Whether contrib symbols from this group are cached

nodata-ignore-request-timeout

Type: Boolean
Default: FALSE

This option configures the service so that all nodatas received are propagated to the user code and any outstanding requests for the nodata subject are discarded.

The default value is for the legacy behaviour.

disable-auto-status

Type: Boolean
Default: FALSE

Disable the automatic generation of object status

Configuration group add-source-group

Add a source group

affinity

Type: Function
Default: None

Affinity configuration

required

Type: Boolean
Default: FALSE

Whether this source group is required

retry-time

Type: Floating Point Value
Default: 30.000000
Minimum: 1.000000

Time to retry when we've looped through all priorities

Configuration group add-priority

Start a priority group

remote-label

Type: String Array
Default: None

Peer labels

label

Type: String Array
Default: None

Peer labels

datasrc-default-obj-hash-size

Type: Integer
Default: 16384

Default number of entries in the active object hashtable. This size can be overridden by putting a value in the obj-hash-size option of the add-peer entry.

cleanup-stale-timeout

Type: Floating Point Value
Default: -1.000000

Default delay in seconds before cleaning STALE objects from the cache

broadcast-cleanup-time

Type: Integer
Default: -1

Time in minutes past midnight to clean up broadcast objects

broadcast-cleanup-period

Type: Integer
Default: 1440

How often to cleanup broadcast objects

broadcast-cleanup-age

Type: Integer
Default: -1

How old broadcast objects need to be for cleanup