Obtain container data from several sources

Here we explain how to define and populate a container using more than one Integration Adapter.

For some background information about how containers work, see here.

Overview

The contents of a container can be defined by more than one DataSource application (typically an Integration Adapter or a Transformer), and the data that the elements refer to can be supplied by other DataSource applications. This decouples the source of the data from the source of the container and makes it easier to reconfigure a deployed Caplin Platform to add additional sources of container elements and additional sources of container data.

Here’s an example that shows the flow of data and processing within Caplin Platform components when a client subscribes to a container and there are several Integration Adapters supplying the data.

You might like to compare the following example with the simple one in How containers work, which only uses one Integration Adapter to supply and populate the container.

In this example, data about Fixed Income instruments is sourced from both New York and London, but must be made available to all end-users through a container called /CTR/FI/ALL. So the FI container and the elements (FI instruments) that it refers to are obtained from four Integration Adapters.

  • Adapter NYLists, located in New York, supplies the container elements for US instrumentss

  • Adapter NYData, located in New York, supplies the data for the US instruments.

  • Adapter LONLists, located in London, supplies the container elements for European instruments.

  • Adapter LONData, located in London, supplies the data for the European instruments.

A. Subscribing to the container:

This diagram shows how the Platform components interact to subscribe to the container.

Container usage in Caplin Platform - multiple adapters supplying container

Following the numbered steps in the diagram:

1. The client application subscribes, through StreamLink, to the container /CTR/FI/ALL, and this request is sent to the Liberator.

The Liberator doesn’t currently have a subscription to this container in its cache, so it looks in its configuration for a data service that can supply the container elements. The data service is configured so that the Liberator sends the subscription request for /CTR/FI/ALL to both the NYLists and LONLists Integration Adapters.

2a. The Integration Adapter NYLists receives the request for /CTR/FI/ALL. Since this particular container isn’t in its cache (not previously requested since the Adapter was started up), NYLists obtains the list of elements (subjects) that this particular container refers to, and populates the container with just the subject names that it knows about – the US ones, /FI/US/A, /FI/US/B, /FI/US/C, and so on.

2b. The Integration Adapter LONLists also receives the request for /CTR/FI/ALL. As for NYLists, this particular container isn’t in its cache, so LONLists obtains the list of elements (subjects) that this particular container refers to, and populates the container with just the subject names that it knows about – the European ones, /FI/EU/A, /FI/EU/B, /FI/EU/C, and so on.

In reality, the requests for /CTR/FI/ALL are sent to both Integration Adapters at the same time, so the natural order of the elements in the returned container depends on which Adapter responds first.

3a. The NYLists Adapter returns the container elements for the US instruments to Liberator, where they’re cached.

3b. The LONLists Adapter returns the container elements for the European instruments to Liberator, where they’re cached too.

B. Obtaining the container data:

Now we show how the Platform components interact to obtain the data for the FI instruments listed in the container.

Container usage in Caplin Platform - multiple adapters populating container with data

4a and 4b. Liberator examines the container contents and subscribes to the subject of each container element that isn’t already in its cache. For each unsubscribed subject it looks in its configuration for data services that can supply the data. In contrast to the example of one DataSource adapter in How containers work, there are two data services defined for obtaining the instrument data, which result in the Liberator obtaining the data for subjects of the form /FI/US/ from the New York Integration Adapter NYData, and data for subjects of the form /FI/EU/ from the London Integration Adapter LONData.

5a. The NYData Adapter receives the requests for the US instruments (subjects /FI/US/A, /FI/US/B, /FI/US/C, and so on). It returns the data images for the corresponding instruments, either from its cache (cached when the instrument has previously been subscribed to since the Adapter was started up), or by requesting the data from the relevant Bank system or external data feed if it isn’t already in the cache. Liberator receives the data images for the subscribed container elements and caches them.

5b. The LONData Adapter receives the requests for the European instruments (subjects /FI/EU/A,

/FI/EU/B, /FI/EU/C, and so on). It returns the data images for the corresponding instruments, in the same way as the NYData Adapter, and Liberator receives the data images and caches them.

6. Liberator sends the client the container image, and the data images of all the instruments referred to by the container elements.

7. (This step isn’t shown in the diagram.) The Liberator sends the client all subsequent updates to the subscriptions. These can be:

7a. Changes to the container structure; for example, new elements appearing in the container.

7b. Updates to the subjects that the container elements refer to; for example, a change to the Yield field of instrument /FI/C.

Liberator configuration

Here’s how you’d configure Liberator to route requests for /CTR/FI/ALL to the four Integration Adapters.

First, here’s the configuration for accessing the NYLists and LONLists Adapters that supply the containers:

add-peer
   remote-name          NYLists
   remote-label         NYLists
   remote-type          active
end-peer

add-peer
   remote-name          LONLists
   remote-label         LONLists
   remote-type          active
end-peer

add-data-service
   service-name         ContainersFI
   include-pattern      ^/CTR/FI/
   add-source-group
      required          TRUE
      add-priority
         remote-label   NYLists
      end-priority
   end-source-group
   add-source-group
      required          TRUE
      add-priority
         remote-label   LONLists
      end-priority
   end-source-group
end-data-service

The two add-peer items define the Liberator’s connections to the NYLists and LonLists Integration Adapters.

Subscriptions to the container are managed through the data service called ContainersFI, defined by the add-data-service item. The container subject /CTR/FI/ALL matches the include-pattern ^/CTR/FI/, so the required data service is ContainersFI.

The data service definition has two add-source-group items, the first of which has a single add-priority item with the remote-label NYLists, which directs the Liberator to the NYLists Integration Adapter. Similarly the second add-source-group directs Liberator to the LONLists Adapter. So the Liberator sends a subscription request for the container /CTR/FI/ALL to both the NYLists and LONLists Integration Adapters.

Now here’s the configuration for accessing the NYData and LONData Adapters that supply the data to populate the containers:

add-peer
   remote-name          NYData
   remote-label         NYData
   remote-type          active
end-peer

add-peer
   remote-name          LONData
   remote-label         LONData
   remote-type          active
end-peer

add-data-service
   service-name         InstrumentsFIUS
   include-pattern      ^/FI/US/
   add-source-group
      required          TRUE
      add-priority
         remote-label   NYData
      end-priority
   end-source-group
end-data-service

add-data-service
   service-name         InstrumentsFIEU
   include-pattern      ^/FI/EU/
   add-source-group
      required          TRUE
      add-priority
         remote-label   LONData
      end-priority
   end-source-group
end-data-service

Two more add-peer items define the Liberator’s connections to the NYData and LonData Integration Adapters.

Subscriptions to the data that populates the container are managed through two data services called InstrumentsFIUS and InstrumentsFIEU, defined by add-data-service items.

The subject /FI/US matches the include-pattern ^/FI/US/, so the required data service for the US based-instruments is InstrumentsFIUS. The data service configuration for InstrumentsFIUS has a single add-priority item with the remote-label NYData; this directs the Liberator to the NYData Integration Adapter as the source of the container’s data for US FI instruments.

Similarly, the subject /FI/EU matches the include-pattern ^/FI/EU/, so the required data service for the European based-instruments is InstrumentsFIEU. The data service configuration for InstrumentsFIEU has a single add-priority item with the remote-label LONData; this directs the Liberator to the LONData Integration Adapter as the source of the container’s data for European FI instruments.

For simplicity, these example configurations don’t include failover, which you would most likely need in a production deployment. For example, see How can I…​ Set up server failover capability in the Deployment Framework pages.

See also: