Sorting and filtering with Transformer Refiner

Transformer is a real-time data transformation engine. The data transformation services are implemented in Transformer Modules. Caplin provides a Transformer module called Refiner that allows large lists of data (containers) to be efficiently filtered and sorted in real time on behalf of client applications.

Objectives

In the previous tutorial we created a Pricing Integration Adapter which we connected to Transformer. However, Transformer contains no modules and is not altering the data being sent/received by the adapter in any way. We will now deploy the Refiner module and use it to sort and filter container responses.

Refiner

The Refiner Service Blade is a pre-built Transformer module which, when deployed enables the clients to request filtered or sorted containers. The module is responsible for filtering or sorting the adapter’s response according to the client’s criteria before returning the messages.

Follow the steps below:

  1. Download Refiner. To deploy Refiner, copy CPB_RefinerService-<version>.zip to your kits directory.

  2. Run ./dfw deploy to deploy the Refiner Service blade. Restart your Deployment Framework.

  3. Use Liberator Explorer to request "/CONTAINER/FX/MAJOR". This returns the un-filtered and un-sorted container which we’ve seen before.

    tutorial trnsf refiner original

  4. Request "/FILTER/CONTAINER/FX/MAJOR?sort=InstrumentName:text:ascending". The result will be sorted by instrument name in alphabetical order:

    tutorial trnsf refiner sorted

  5. Request "/FILTER/CONTAINER/FX/MAJOR?filter=(BidPrice>1.7)". The result will contain only records whose BidPrice field is greater than 1.7:

    tutorial trnsf refiner filtered

As you can see, the Refiner module expects to receive requests with a certain pattern:

/FILTER/<Container Subject>?<Filter and Sort Configuration>

The Refiner module will perform the following steps when it receives the requests:

  1. Extract the base container subject (everything between the "/FILTER" and the question mark).

  2. Request the base container from whichever integration adapter provides it.

  3. When the base container and all the records arrive at the Refiner, it will apply the filter and sort criteria as specified after the question mark. For example, if the filter is "BidPrice>1.7" it will exclude records containing values in the BidPrice field that don’t match this filter.

  4. Having applied the sort and filter criteria, Refiner will return a container on the original subject ("/FILTER/<Container Subject>?<Filter and Sort Configuration>") containing only the records that match the filter criteria, and organised into an order based on the sort criteria.

Refiner will also intercept all updates to the records in the base container and revalidate them against the sort and filter criteria, so the sorted and filtered container will update in real time. For example, if the BidPrice field of a record changes from 1.8 to 1.6 it no longer meets the filter criteria, so Refiner will update the sorted and filtered criteria to remove this record, and it will disappear on the front end.

Also note that in this example you are requesting the /FILTER subjects by manually typing them into the Liberator Explorer, which requires you to know the syntax Refiner expects. In a real front end application the user would not be expected to do this - you would provide GUI controls for sorting and filtering and build the /FILTER subject yourself in code.

Review

Having completed this tutorial you should be familiar with how to deploy Refiner and request filtered and sorted data via Transformer.

More information about Transformer can be found on the website.