Obtain container data from several sources
Here we explain how to define and populate a container using more than one Integration Adapter.
For some background information about how containers work, see here.
Overview
The contents of a container can be defined by more than one DataSource application (typically an Integration Adapter or a Transformer), and the data that the elements refer to can be supplied by other DataSource applications. This decouples the source of the data from the source of the container and makes it easier to reconfigure a deployed Caplin Platform to add additional sources of container elements and additional sources of container data.
Here’s an example that shows the flow of data and processing within Caplin Platform components when a client subscribes to a container and there are several Integration Adapters supplying the data.
You might like to compare the following example with the simple one in How containers work, which only uses one Integration Adapter to supply and populate the container. |
In this example, data about Fixed Income instruments is sourced from both New York and London, but must be made available to all end-users through a container called /CTR/FI/ALL
. So the FI container and the elements (FI instruments) that it refers to are obtained from four Integration Adapters.
-
Adapter NYLists, located in New York, supplies the container elements for US instrumentss
-
Adapter NYData, located in New York, supplies the data for the US instruments.
-
Adapter LONLists, located in London, supplies the container elements for European instruments.
-
Adapter LONData, located in London, supplies the data for the European instruments.
A. Subscribing to the container:
This diagram shows how the Platform components interact to subscribe to the container.
![Container usage in Caplin Platform - multiple adapters supplying container](../../../images/Container_ArchitectureMultiple_AdaptersContainerSubscription_2.png)
Following the numbered steps in the diagram:
1. The client application subscribes, through StreamLink, to the container /CTR/FI/ALL
, and this request is sent to the Liberator.
The Liberator doesn’t currently have a subscription to this container in its cache, so it looks in its configuration for a data service that can supply the container elements. The data service is configured so that the Liberator sends the subscription request for /CTR/FI/ALL
to both the NYLists and LONLists Integration Adapters.
2a. The Integration Adapter NYLists receives the request for /CTR/FI/ALL
. Since this particular container isn’t in its cache (not previously requested since the Adapter was started up), NYLists obtains the list of elements (subjects) that this particular container refers to, and populates the container with just the subject names that it knows about – the US ones, /FI/US/A
, /FI/US/B
, /FI/US/C
, and so on.
2b. The Integration Adapter LONLists also receives the request for /CTR/FI/ALL
. As for NYLists, this particular container isn’t in its cache, so LONLists obtains the list of elements (subjects) that this particular container refers to, and populates the container with just the subject names that it knows about – the European ones, /FI/EU/A
, /FI/EU/B
, /FI/EU/C
, and so on.
In reality, the requests for /CTR/FI/ALL
are sent to both Integration Adapters at the same time, so the natural order of the elements in the returned container depends on which Adapter responds first.
3a. The NYLists Adapter returns the container elements for the US instruments to Liberator, where they’re cached.
3b. The LONLists Adapter returns the container elements for the European instruments to Liberator, where they’re cached too.
B. Obtaining the container data:
Now we show how the Platform components interact to obtain the data for the FI instruments listed in the container.
![Container usage in Caplin Platform - multiple adapters populating container with data](../../../images/Container_ArchitectureMultiple_AdaptersContainerData_2.png)
4a and 4b. Liberator examines the container contents and subscribes to the subject of each container element that isn’t already in its cache. For each unsubscribed subject it looks in its configuration for data services that can supply the data. In contrast to the example of one DataSource adapter in How containers work, there are two data services defined for obtaining the instrument data, which result in the Liberator obtaining the data for subjects of the form /FI/US/
from the New York Integration Adapter NYData, and data for subjects of the form /FI/EU/
from the London Integration Adapter LONData.
5a. The NYData Adapter receives the requests for the US instruments (subjects /FI/US/A
, /FI/US/B
, /FI/US/C
, and so on). It returns the data images for the corresponding instruments, either from its cache (cached when the instrument has previously been subscribed to since the Adapter was started up), or by requesting the data from the relevant Bank system or external data feed if it isn’t already in the cache. Liberator receives the data images for the subscribed container elements and caches them.
5b. The LONData Adapter receives the requests for the European instruments (subjects /FI/EU/A
,
/FI/EU/B
, /FI/EU/C
, and so on). It returns the data images for the corresponding instruments, in the same way as the NYData Adapter, and Liberator receives the data images and caches them.
6. Liberator sends the client the container image, and the data images of all the instruments referred to by the container elements.
7. (This step isn’t shown in the diagram.) The Liberator sends the client all subsequent updates to the subscriptions. These can be:
7a. Changes to the container structure; for example, new elements appearing in the container.
7b. Updates to the subjects that the container elements refer to; for example, a change to the Yield
field of instrument /FI/C
.
Liberator configuration
Here’s how you’d configure Liberator to route requests for /CTR/FI/ALL to the four Integration Adapters.
First, here’s the configuration for accessing the NYLists and LONLists Adapters that supply the containers:
add-peer remote-name NYLists remote-label NYLists remote-type active end-peer add-peer remote-name LONLists remote-label LONLists remote-type active end-peer add-data-service service-name ContainersFI include-pattern ^/CTR/FI/ add-source-group required TRUE add-priority remote-label NYLists end-priority end-source-group add-source-group required TRUE add-priority remote-label LONLists end-priority end-source-group end-data-service
The two add-peer
items define the Liberator’s connections to the NYLists and LonLists Integration Adapters.
Subscriptions to the container are managed through the data service called ContainersFI
, defined by the add-data-service
item. The container subject /CTR/FI/ALL
matches the include-pattern ^/CTR/FI/
, so the required data service is ContainersFI
.
The data service definition has two add-source-group
items, the first of which has a single add-priority
item with the remote-label NYLists
, which directs the Liberator to the NYLists Integration Adapter. Similarly the second add-source-group
directs Liberator to the LONLists Adapter. So the Liberator sends a subscription request for the container /CTR/FI/ALL
to both the NYLists and LONLists Integration Adapters.
Now here’s the configuration for accessing the NYData and LONData Adapters that supply the data to populate the containers:
add-peer remote-name NYData remote-label NYData remote-type active end-peer add-peer remote-name LONData remote-label LONData remote-type active end-peer add-data-service service-name InstrumentsFIUS include-pattern ^/FI/US/ add-source-group required TRUE add-priority remote-label NYData end-priority end-source-group end-data-service add-data-service service-name InstrumentsFIEU include-pattern ^/FI/EU/ add-source-group required TRUE add-priority remote-label LONData end-priority end-source-group end-data-service
Two more add-peer
items define the Liberator’s connections to the NYData and LonData Integration Adapters.
Subscriptions to the data that populates the container are managed through two data services called InstrumentsFIUS
and InstrumentsFIEU
, defined by add-data-service
items.
The subject /FI/US
matches the include-pattern ^/FI/US/
, so the required data service for the US based-instruments is InstrumentsFIUS
. The data service configuration for InstrumentsFIUS
has a single add-priority
item with the remote-label NYData
; this directs the Liberator to the NYData Integration Adapter as the source of the container’s data for US FI instruments.
Similarly, the subject /FI/EU
matches the include-pattern ^/FI/EU/
, so the required data service for the European based-instruments is InstrumentsFIEU
. The data service configuration for InstrumentsFIEU
has a single add-priority
item with the remote-label LONData
; this directs the Liberator to the LONData Integration Adapter as the source of the container’s data for European FI instruments.
For simplicity, these example configurations don’t include failover, which you would most likely need in a production deployment. For example, see How can I… Set up server failover capability in the Deployment Framework pages. |
See also:
-
Filtering and sorting containers using Transformer’s Refiner Service blade
-
How can I… Configure container usage in Liberator
-
How can I… Create containers in an Integration Adapter
-
How can I… Obtain a container snapshot in a CSV or XLSX file
-
Reference: DataSource peers configuration
-
Reference: Data services configuration
-
Reference: Object configuration