Configuration: Data Services

Data Services are a new feature in the 4.0 Caplin platform and offer improved source mapping when compared to previous versions with the following key features:

• More intuitive configuraiton
• More clearly defined handling of objects receiving data from multiple sources at the same time
• Ability to define both failover and round-robin style mappings.

A Source Mapping definition is called a Service. A service comprises a name, one or more subject patterns to match and one or more source groups.

Service name

This is an indentifier which can be used in status messages. RTTP objects have a field called SID which is the service name.

Note:When picking a service for an object, the first defined match takes priority. As such you should ensure that each object is associated with one and only one service.

The subject patterns

These are regular expression strings to accept or deny for this service. A service will allow multiple patterns including patterns to deny. Exclude patterns can help to define the namespace used.

Note: When checking pattern matches within a service definition, the first match takes priority whether it is an include or an exclude.

The source groups

The main part of the service definition is the source groups. This is one or more sets of sources, plus certain attributes which define the behaviour of the group. In most cases only a single group is defined. When multiple groups are defined for a service it means that a request will attempt to get the object from a source from each group. Multiple groups allow an object to have different sets of fields coming from different sources, for example.

Priorities

Priorities are defined within each source group and are taken in the order in which they are defined. Multiple labels can be defined within each priority. Within a priority, the peer with the lowest number of subscriptions will be used.

Timeouts

There are various timeouts associated with Data Services. By default when an object is requested if after 10 seconds there has been no response, the request will be cancelled and the user informed. This is the service-request-timeout and applies to the whole object request. It is also possible to timeout requests to individual sources, to allow the request to move on to another source. This is the source-request-timeout, which by default is not turned on. The Source timeout is only useful when using a priority group with multiple sources within it as it allows the system to try more than one source before giving up. All these timeouts are only relevant when a source does not respond to a request, when the timeout occurs a discard message is sent to the source so it is aware that the request is cancelled.

Default Behaviour

If no data-service is defined, then the application will act as if the following was defined:

add-data-service
    service-name         default
    include-pattern      ^/
    add-source-group
        required         false
        add-priority
            label        source1
        end-priority
    end-source-group
    add-source-group
        required         false
        add-priority
            label        source2
        end-priority
    end-source-group
    .
    .
    add-source-group
        required         false
        add-priority
            label        sourceN
        end-priority
    end-source-group
end-data-service

This means a request will be sent to all active Data Sources at once.

When using the DataSource SDK to request objects programmatically, the Data Services system will prevent requests going to any peer that has already been sent a request. It also handles which peers requests go to rather than the programmer explicitly specifying the peers.

Conversion

Pre-version 4.0 source mapping should be converted to data services in the following manner:

All peers should have a label defined in the add-peer configuration section, for these example conversion, the label is 'src' appended with the the remote-id

add-source-mapping /A/* 1 should be converted to:

include-pattern      ^/A/
add-source-group
    required         true
    add-priority
        label        src1
    end-priority
end-source-group

add-source-mapping /A/* 1,2 should be converted to:

include-pattern      ^/A/
add-source-group
    required         true
    add-priority
        label        src1
        label        src2
    end-priority
end-source-group

add-source-mapping /A/* 1 2 should be converted to:

include-pattern      ^/A/
add-source-group
    required         true
    add-priority
        label        src1
    end-priority
end-source-group
add-source-group
    required         true
    add-priority
        label        src2
    end-priority
end-source-group

add-source-mapping /A/* 1,2 3,4 should be converted to:

include-pattern      ^/A/
add-source-group
    required         true
    add-priority
        label        src1
        label        src2
    end-priority
end-source-group
add-source-group
    required         true
    add-priority
        label        src3
        label        src4
    end-priority
end-source-group

Minimum configuration

Data services can be configured to have a hierarchy of minimum requirements met and take action depending on whether these requirements are met. This works by using the following configuration items

Configuration	Description
min-peers	If at least this number of peers are not up in a priority then the priority is regarded as down.
min-priorities	If at least this number of priorities are not up in a source group then the source group is regarded as down.
min-source-groups	If at least this number of source groups are not up in a Data service then the Data service is regarded as down.

• If a priority is down, no requests will be issued to any of its peers.
• If a source group is down, no requests will be issued to any of its priorities.
• If a Data service is down, no requests will be issued to any of its source groups and the service status will be set be down.

Minimum configuration interaction with required groups

This configuration interacts with the required flag of the source group.

• If all of the peers of a required source group are down, then the Data service state will be down - even if the minimum number of source groups are up.
• If the minimum number of source groups are not up, then the Data service state will be down - even if all the required source groups have peers that are up.

---

disable-default-data-service

Type: Boolean
Default: FALSE

Disable creation of the default service

---

service-request-timeout

Type: Floating Point Value
Default: 10.000000

Global request timeout for all services

---

source-request-timeout

Type: Floating Point Value
Default: -1.000000

Global request timeout for all sources

---

service-rebalance-enable

Type: Boolean
Default: FALSE

Enable service rebalancing

---

service-rebalance-time

Type: Floating Point Value
Default: 5.000000

Time to wait after a peer comes up before rebalancing

---

service-global-peer-balance

Type: Boolean
Default: FALSE

Balance subscriptions based on the peer with the smallest number of subscriptons across all services

---

add-data-service

Configuration group add-data-service

Starts the definition of data service.

Format:

add-data-service
    service-name      [value]
    request-timeout   [value]
    exclude-pattern   [value]
    include-pattern   [value]
    add-source-group
        required      [boolean]
        retry-time    [value]
        add-priority
            label     [value]
        end-priority
    end-source-group
end-data-service

Example:

add-data-service 
    service-name       FX 
    exclude-pattern    ^/I/X 
    include-pattern    ^/I/
    include-pattern    ^/B/
    request-timeout    20
    add-source-group
        required       true 
        retry-time     45 
        add-priority 
            label      sourceA 
        end-priority 
        add-priority 
            label      sourceB 
            label      sourceC 
        end-priority 
    end-source-group 

    add-source-group 
        required       false
        add-priority 
            label      source1 
            label      source2 
        end-priority 
    end-source-group 
end-data-service

In this example, the service FX is defined which matches the patterns ^/I/ and ^/B/ but not the symbols ^/B/. There are two service groups, one providing required data and another providing optional data.

For the required group, there are two priorities defined, requests will be preferentially sent to sourceA, however should it not be available then sourceB and sourceC will be asked in a round-robin fashion.

For the non-required group, two sources are defined, requests will be sent to source1 and source2 in a round-robin fashion.

object-map

Type: String Array
Default: None

Add an object name mapping (valid for Liberator only)

---

min-source-groups

Type: Integer
Default: 1
Minimum: 0

Minimum number of up source groups required

---

service-name

Type: String
Default: None

Name of the service group

---

affinity

Type: Function
Default: None

Affinity configuration

---

service-type

Type: Integer
Default: 3
Acceptable Values:

Name	Value	Desc
active	1
broadcast	2
rest	13
contrib	9

Type of the service group

---

request-timeout

Type: Floating Point Value
Default: -1.000000

This option configures the timeout for all requests for this service. Should no response be received from peers within this time, the object will be assumed to be not available.

The default value of -1 means that requests will never timeout.

---

discard-timeout

Type: Floating Point Value
Default: -1.000000

Timeout for all discards for this service

---

exclude-pattern

Type: Function
Default: None

Patterns to exclude

---

include-pattern

Type: Function
Default: None

Patterns to include

---

required-state

Type: Integer
Default: 0
Acceptable Values:

Name	Value	Desc
down	0
optional	0
limited	1
ok	2
up	2
once	3

Required state for this service

---

nodata-ignore-request-timeout

Type: Boolean
Default: FALSE

This option configures the service so that all nodatas received are propagated to the user code and any outstanding requests for the nodata subject are discarded.

The default value is for the legacy behaviour.

---

disable-auto-status

Type: Boolean
Default: FALSE

Disable the automatic generation of object status

---

rebalance-enable

Type: Boolean
Default: FALSE

Enable service rebalancing

---

rebalance-time

Type: Floating Point Value
Default: -1.000000

Time to wait after a peer comes up before rebalancing

---

throttle-time

Type: Floating Point Value
Default: -1.000000

Throttle time for subjects within the namespace of this service

---

---

add-source-group

Configuration group add-source-group

Add a source group

min-priorities

Type: Integer
Default: 1
Minimum: 0

Minimum number of up priorities

---

affinity

Type: Function
Default: None

Affinity configuration

---

required

Type: Boolean
Default: FALSE

Whether this source group is required

---

contrib

Type: Boolean
Default: FALSE

Whether this source-group is a source of contributions

---

retry-time

Type: Floating Point Value
Default: 30.000000

Time to retry when we've looped through all priorities

---

---

add-priority

Configuration group add-priority

Start a priority group

remote-label

Type: String Array
Default: None

Remote peer label

---

label

Type: String Array
Deprecated: Yes
Default: None

Use remote-label

---

min-peers

Type: Integer
Default: 1
Minimum: 0

Minimum number of up peers required

---

remote-label-regex

Type: String
Default: None

Regex to capture remote peer labels

---

---

datasrc-default-obj-hash-size

Type: Integer
Default: 16384

Default number of entries in the active object hashtable. This size can be overridden by putting a value in the obj-hash-size option of the add-peer entry.

---

broadcast-cleanup-time

Type: Integer
Default: -1

Time in minutes past midnight to clean up broadcast objects

---

broadcast-cleanup-stale-timeout

Type: Floating Point Value
Default: -1.000000

Default delay in seconds before cleaning STALE broadcast objects from the cache

---

broadcast-cleanup-period

Type: Integer
Default: 1440

How often to cleanup broadcast objects

---

broadcast-cleanup-age

Type: Integer
Default: -1

How old broadcast objects need to be for cleanup

---

add-provided-data-service

Configuration group add-provided-data-service

Add a service group

object-map

Type: String Array
Default: None

Add an object name mapping (valid for Liberator only)

---

apply-if-label

Type: String Array
Default: None

If specified, only peers matching the label regex should add the service

---

service-name

Type: String
Default: None

Name of the service group

---

service-type

Type: Integer
Default: 3
Acceptable Values:

Name	Value	Desc
active	1
broadcast	2
rest	13
contrib	9

Type of the service group

---

exclude-pattern

Type: Function
Default: None

Patterns to exclude

---

include-pattern

Type: Function
Default: None

Patterns to include

---

remote-label-regex

Type: String
Default: None

Regex to capture remote peer labels

---

throttle-time

Type: Floating Point Value
Default: -1.000000

Throttle time for subjects serviced by this namespace

---

discard-timeout

Type: Floating Point Value
Default: -1.000000

Discard timeout for subjects serviced by this namespace

---

required-state

Type: Integer
Default: 0
Acceptable Values:

Name	Value	Desc
down	0
optional	0
limited	1
ok	2
up	2
once	3

Required state for this service

---

open-subscriptions

Type: Boolean
Default: FALSE

Whether open subscriptions behaviour should be applied to this service

---

---