|
DataSource for C SDK
7.1.28.98838-0467ac22
|
 |
|
DataSource for C SDK
7.1.28.98838-0467ac22
|
|
Data Services are a new feature in the 4.0 Caplin platform and offer improved source mapping when compared to previous versions with the following key features:
A Source Mapping definition is called a Service. A service comprises a name, one or more subject patterns to match and one or more source groups.
This is an indentifier which can be used in status messages. RTTP objects have a field called SID which is the service name.
Note:When picking a service for an object, the first defined match takes priority. As such you should ensure that each object is associated with one and only one service.
These are regular expression strings to accept or deny for this service. A service will allow multiple patterns including patterns to deny. Exclude patterns can help to define the namespace used.
Note: When checking pattern matches within a service definition, the first match takes priority whether it is an include or an exclude.
The main part of the service definition is the source groups. This is one or more sets of sources, plus certain attributes which define the behaviour of the group. In most cases only a single group is defined. When multiple groups are defined for a service it means that a request will attempt to get the object from a source from each group. Multiple groups allow an object to have different sets of fields coming from different sources, for example.
Priorities are defined within each source group and are taken in the order in which they are defined. Multiple labels can be defined within each priority. Within a priority, the peer with the lowest number of subscriptions will be used.
There are various timeouts associated with Data Services. By default when an object is requested if after 10 seconds there has been no response, the request will be cancelled and the user informed. This is the service-request-timeout and applies to the whole object request. It is also possible to timeout requests to individual sources, to allow the request to move on to another source. This is the source-request-timeout, which by default is not turned on. The Source timeout is only useful when using a priority group with multiple sources within it as it allows the system to try more than one source before giving up. All these timeouts are only relevant when a source does not respond to a request, when the timeout occurs a discard message is sent to the source so it is aware that the request is cancelled.
If no data-service is defined, then the application will act as if the following was defined:
add-data-service
service-name default
include-pattern ^/
add-source-group
required false
add-priority
label source1
end-priority
end-source-group
add-source-group
required false
add-priority
label source2
end-priority
end-source-group
.
.
add-source-group
required false
add-priority
label sourceN
end-priority
end-source-group
end-data-service
This means a request will be sent to all active Data Sources at once.
When using the DataSource SDK to request objects programmatically, the Data Services system will prevent requests going to any peer that has already been sent a request. It also handles which peers requests go to rather than the programmer explicitly specifying the peers.
Pre-version 4.0 source mapping should be converted to data services in the following manner:
All peers should have a label defined in the add-peer configuration section, for these example conversion, the label is 'src' appended with the the remote-id
add-source-mapping /A/* 1 should be converted to:
include-pattern ^/A/
add-source-group
required true
add-priority
label src1
end-priority
end-source-group
add-source-mapping /A/* 1,2 should be converted to:
include-pattern ^/A/
add-source-group
required true
add-priority
label src1
label src2
end-priority
end-source-group
add-source-mapping /A/* 1 2 should be converted to:
include-pattern ^/A/
add-source-group
required true
add-priority
label src1
end-priority
end-source-group
add-source-group
required true
add-priority
label src2
end-priority
end-source-group
add-source-mapping /A/* 1,2 3,4 should be converted to:
include-pattern ^/A/
add-source-group
required true
add-priority
label src1
label src2
end-priority
end-source-group
add-source-group
required true
add-priority
label src3
label src4
end-priority
end-source-group
Data services can be configured to have a hierarchy of minimum requirements met and take action depending on whether these requirements are met. This works by using the following configuration items
| Configuration | Description |
|---|---|
| min-peers | If at least this number of peers are not up in a priority then the priority is regarded as down. |
| min-priorities | If at least this number of priorities are not up in a source group then the source group is regarded as down. |
| min-source-groups | If at least this number of source groups are not up in a Data service then the Data service is regarded as down. |
This configuration interacts with the required flag of the source group.
Type: Floating Point Value
Default: 10.000000
Global request timeout for all services
Type: Floating Point Value
Default: -1.000000
Global request timeout for all sources
Type: Boolean
Default: FALSE
Enable service rebalancing
Type: Floating Point Value
Default: 5.000000
Time to wait after a peer comes up before rebalancing
Type: Boolean
Default: FALSE
Balance subscriptions based on the peer with the smallest number of subscriptons across all services
add-data-serviceStarts the definition of data service.
Format:
add-data-service
service-name [value]
request-timeout [value]
exclude-pattern [value]
include-pattern [value]
add-source-group
required [boolean]
retry-time [value]
add-priority
label [value]
end-priority
end-source-group
end-data-service
Example:
add-data-service
service-name FX
exclude-pattern ^/I/X
include-pattern ^/I/
include-pattern ^/B/
request-timeout 20
add-source-group
required true
retry-time 45
add-priority
label sourceA
end-priority
add-priority
label sourceB
label sourceC
end-priority
end-source-group
add-source-group
required false
add-priority
label source1
label source2
end-priority
end-source-group
end-data-service
In this example, the service FX is defined which matches the patterns ^/I/ and ^/B/ but not the symbols ^/B/. There are two service groups, one providing required data and another providing optional data.
For the required group, there are two priorities defined, requests will be preferentially sent to sourceA, however should it not be available then sourceB and sourceC will be asked in a round-robin fashion.
For the non-required group, two sources are defined, requests will be sent to source1 and source2 in a round-robin fashion.
Type: Integer
Default: 1
Minimum: 0
Minimum number of up source groups required
Type: String
Default: None
Name of the service group
Type: Function
Default: None
Affinity configuration
Type: Integer
Default: 3
Acceptable Values:
| Name | Value | Desc |
|---|---|---|
| active | 1 | |
| broadcast | 2 | |
| rest | 4 |
Type of the service group
Type: Floating Point Value
Default: -1.000000
This option configures the timeout for all requests for this service. Should no response be received from peers within this time, the object will be assumed to be not available.
The default value of -1 means that requests will never timeout.
Type: Floating Point Value
Default: -1.000000
Timeout for all discards for this service
Type: Function
Default: None
Patterns to exclude
Type: Function
Default: None
Patterns to include
Type: Integer
Default: 0
Acceptable Values:
| Name | Value | Desc |
|---|---|---|
| down | 0 | |
| optional | 0 | |
| limited | 1 | |
| ok | 2 | |
| up | 2 |
Required state for this service
Type: Boolean
Default: FALSE
This option configures the service so that all nodatas received are propagated to the user code and any outstanding requests for the nodata subject are discarded.
The default value is for the legacy behaviour.
Type: Boolean
Default: FALSE
Disable the automatic generation of object status
Type: Boolean
Default: FALSE
Enable service rebalancing
Type: Floating Point Value
Default: -1.000000
Time to wait after a peer comes up before rebalancing
add-source-groupAdd a source group
Type: Integer
Default: 1
Minimum: 0
Minimum number of up priorities
Type: Function
Default: None
Affinity configuration
Type: Boolean
Default: FALSE
Whether this source group is required
Type: Boolean
Default: FALSE
Whether this source-group is a source of contributions
Type: Floating Point Value
Default: 30.000000
Minimum: 1.000000
Time to retry when we've looped through all priorities
add-priorityStart a priority group
Type: String Array
Default: None
Remote peer label
Type: String Array
Deprecated: Yes
Default: None
Remote peer label
Type: Integer
Default: 1
Minimum: 0
Minimum number of up peers required
Type: String
Default: None
Regex to capture remote peer labels
Type: Integer
Default: 16384
Default number of entries in the active object hashtable. This size can be overridden by putting a value in the obj-hash-size option of the add-peer entry.
Type: Integer
Default: -1
Time in minutes past midnight to clean up broadcast objects
Type: Floating Point Value
Default: -1.000000
Default delay in seconds before cleaning STALE broadcast objects from the cache
Type: Integer
Default: 1440
How often to cleanup broadcast objects
Type: Integer
Default: -1
How old broadcast objects need to be for cleanup