Writing a Transformer Pipeline Module

Transformer is a real-time data transformation engine. The data transformation services are implemented in Transformer Modules. The Transformer API is used to create custom Transformer modules (in C, Java, Lua or JavaScript) which act on data messages being issued to client applications. Transformer houses the modules and manages the data coming in and out of them.

The following sections take you through how to write a Transformer pipeline module in the Lua scripting language. Following that, there’s an optional exercise in which you can write a pipeline module in JavaScript.

Download the presentations for this course.

Writing a Lua Pipeline Module

The Pricing Integration Adapter is connected to Transformer. However, Transformer contains no modules apart from the Refiner module, and is not altering the data being sent/received by the adapter in any way. In this tutorial you will implement a Transformer module using the pipeline module (Lua API). This module will calculate the spread between the Bid and Ask prices sent by the Pricing Adapter adding a "Spread" field to the record before sending this on to Liberator and the client applications.

Create a new blade

The first step is to create a new blade, directly in the Deployment Framework. Create a directory with the name of the blade (e.g. "SpreadLuaBlade") and the following subdirectories and files within it. (We will fill these configuration and script files as we go along.) This directory should be created inside the kits directory of the Deployment Framework.

Blade Fields

This module is a new backend component which needs to have a fields configuration. It needs to know about all the fields configured in the Pricing Integration Adapter, and additionally, a new "Spread" field which it will be appending to the price update records.

Fill in the fields.conf file:

# - Latency chaining fields
add-field        LATENCY_TIMESTAMP      -10301
add-field        LTY_INIT_TS            -10302
add-field        LTY_LIST_EVENT         -10303
add-field        LTY_LIST_TS            -10304

# - Add further fields below
add-field        Time                   -10001
add-field        Last                   -10002

add-field InstrumentName          -45028 0 0
add-field BidPrice                -45029 0 0
add-field AskPrice                -45030 0 0
add-field BidPriceId              -45031 0 0
add-field AskPriceId              -45032 0 0
add-field BidGFA                  -45033 0 0
add-field AskGFA                  -45034 0 0
add-field Timestamp               -45035 0 0
add-field Spread                  -45036 0 0

Liberator and Transformer Configuration

The new blade must include configuration for the core components that it interacts with. This configuration should typically consist of:

Data service definitions for the Liberator that allow the Liberator to request the data provided by the blade via the pipeline.
Transformer pipeline component configuration for the Lua module.

Fill the Liberator configuration in SpreadLuaBlade/Liberator/etc/rttpd.conf:

# Liberator configuration for data from the
# SpreadLuaBlade Lua module blade.

add-data-service
  service-name SpreadLuaBlade${THIS_LEG}
  include-pattern ^/FX
  add-source-group
    required true
    add-priority
      label transformer${THIS_LEG}
    end-priority

    if "${FAILOVER}" == "ENABLED"
      add-priority
        label transformer${OTHER_LEG}
      end-priority
    endif
  end-source-group
end-data-service

This is fairly standard configuration. The specific points to note are:

The blade name is used in service-name configuration options.
The macros THIS_LEG and OTHER_LEG are used to differentiate the configuration of primary and secondary failover legs.

Fill the Transformer configuration in SpreadLuaBlade/Transformer/etc/pipeline.conf:

# Location of the Lua module blade's pipeline components

pipeline-paths ${ccd}/pipeline
add-pipeline
  id SpreadLuaBlade${THIS_LEG}
  pipeline-file SpreadLuaBlade.lua
  listener-regex ^/FX/.*
  flags  suppressupdate
  update-func     update
end-pipeline

The specific points to note are:

The blade name is used as the pipeline id, and in the Lua module name that is defined by the pipeline-file configuration option.
The ccd macro defines the current directory path of the file in which it is referenced. So in this example, because the ccd reference is in the file SpreadLuaBlade/Transformer/etc/pipeline.conf, the macro defines the path SpreadLuaBlade/Transformer/etc.
The line pipeline-paths ${ccd}/pipeline/ instructs the Transformer to look for the blade’s pipeline files in the directory SpreadLuaBlade/Transformer/etc/pipeline/
The flags suppressupdate means all messages caught by your pipeline are not sent automatically, so you will have to do it explicitly.

Creating the Lua Module Script

Place the following code into SpreadLuaBlade/Transformer/etc/pipeline/SpreadLuaBlade.lua:

require("calc")

function update(packet)
  local bidPrice = packet:getfield("BidPrice")
  local askPrice = packet:getfield("AskPrice")
  packet:setfield("Spread", askPrice - bidPrice)
  packet:send()
end

The specific points to note are:

The require("calc") instruction imports the calculation API.
An update function is defined, which obtains the BidPrice and AskPrice fields from each incoming record whose subject begins with /FX/ (the Transformer’s pipeline module configuration for the SpreadJSBlade controls which records are selected - see the line listener-regex ^/FX/.*). Note that the name of this function must match that defined in the update-func option in the pipeline module configuration.
local bidPrice = packet:getfield("BidPrice") retrieves the value of the BidPrice field.
packet:setfield("Spread", askPrice - bidPrice) adds a "Spread" field to the record and sets its value.
packet:send() sends the new response message on to Liberator.

Try it out!

Follow the steps below:

Activate the blade: ./dfw activate SpreadLuaBlade
Restart the Deployment Framework components, and start up the mock server and your Pricing Integration Adapter.
Use Liberator Explorer to request the container "/CONTAINER/FX/MAJOR". You should see that the "Spread" field has now been added among the others and is calculated with each new price update:

Optional exercise: Writing a JavaScript Pipeline Module

Besides implementing pipeline modules in the Lua scripting language, you can also code them in JavaScript.

To do this, follow the same steps as in Writing a Lua Pipeline Module above, but with the following differences:

Creating the JS blade directory in the Deployment Framework

The blade is called SpreadJSBlade, so create a directory called SpreadJSBlade in the kits directory of the Deployment Framework. Add the subdirectories and files shown in Writing a Lua Pipeline Module, but with the name of the pipeline script (code) file set to SpreadJSBlade.js

Liberator and Transformer Configuration

Fill the Liberator configuration in SpreadJSBlade/Liberator/etc/rttpd.conf:

# Liberator configuration for data from the
# SpreadJSBlade JavaScript module blade.

add-data-service
  service-name SpreadJSBlade${THIS_LEG}
  include-pattern ^/FX
  add-source-group
    required true
    add-priority
      label transformer${THIS_LEG}
    end-priority

    if "${FAILOVER}" == "ENABLED"
      add-priority
        label transformer${OTHER_LEG}
      end-priority
    endif
  end-source-group
end-data-service

Fill the Transformer configuration in SpreadJSBlade/Transformer/etc/pipeline.conf:

# Location of the JavaScript module blade's pipeline components

pipeline-paths ${ccd}/pipeline
add-pipeline
  id SpreadJSBlade${THIS_LEG}
  pipeline-file SpreadJSBlade.js
  listener-regex ^/FX/.*
  flags  suppressupdate
  update-func     update
end-pipeline

Creating the JavaScript Module Script

Place the following code into SpreadJSBlade/Transformer/etc/pipeline/SpreadJSBlade.js:

var log = require("log");
function update(packet)
{
  var subject = packet.getSubject();
  var bidPrice = packet.getField("BidPrice").value;
  var askPrice = packet.getField("AskPrice").value;
  var spread = askPrice - bidPrice;

  log.log(log.INFO, "Spread for <" + subject + "> is " + spread + "\n");
  packet.setField("Spread", spread);
  packet.send();
}

The specific points to note are:

var log = require("log") imports the JavaScript pipeline’s Logging API.
An update function is defined, which obtains the BidPrice and AskPrice fields from each incoming record whose subject begins with /FX/ (the Transformer’s pipeline module configuration for the SpreadJSBlade controls which records are selected - see the line listener-regex ^/FX/.*). Note that the name of this function must match that defined in the update-func option in the pipeline module configuration.

After obtaining the subject name of the incoming record, the function obtains the bidPrice and askPrice fields from the record and uses them to calculate the spread.

In JavaScript, the expression adkPrice - bidPrice subtracts a floating-point value from a floating-point value and may result in precision errors. In production code you would use a library such as bignumber.js for floating-point arithmetic.

The logging API (log.log()) is called to log the spread calculated for the subject.
The new Spread field is then added to the record (packet.setField()) and then packet.send() sends the new response message on to the Liberator.