Create a Lua-based Transformer Service blade

Here we explain how to create a Service blade for Transformer, where the blade is written as a Transformer pipeline modulein the Lua programming language. The core of Transformer is written in C, so to create a module in Lua, you have to do it using Transformer’s Lua Pipeline API.

If you’ve already read How Can I… Create a Java-based Transformer Service blade, you’ll notice that the Java blade described there does exactly the same thing as the blade we create here. This is entirely deliberate, and should allow you to see the differences and similarities between using Java and Lua for creating Transformer Service blades.

Contents:

Before you start
Summary
Create the blade’s directory structure
Define the new fields
Update the core component configuration
Write the Lua module script
Start the new Transformer Service blade
Package the new blade

Before you start

This example shows how to create a Lua-based Service blade called SpreadLuaBlade that calculates the spread between Bid and Ask prices. At run time, the Transformer receives the prices from a Pricing Adapter (an Integration Adapter), and passes them on to the SpreadLuaBlade pipeline module, which calculates the price spread from the Bid and Ask prices. The Transformer sends the prices, including the new spread field, on to a Liberator, which forwards them to subscribing client applications, as shown in this diagram:

Diagram showing the SpreadLuaBlade pipeline module in Transformer, together with a Pricing Adapter and a Liberator

You’ll need a Transformer and Liberator to help test your Service blade while you develop it, and a Pricing Adapter that can supply the Bid and Ask prices. In this example we’ve assumed that the corresponding record fields are called bidPrice and askPrice respectively.

If you don’t have Transformer and Liberator already deployed on a machine for this purpose, you’ll need to deploy one or both of these components to the Deployment Framework on your development machine. To do this, follow the instructions in How can I… Deploy Platform components to the Framework. You won’t need to set up HTTPS connections or KeyMaster while you’re developing the blade, so you can ignore the topics about those things.

If you need to develop a suitable Pricing Adapter first, see How can I… Create a Java-based Adapter blade.

In the following steps you’ll be using the dfw command of the Deployment Framework. Before entering any dfw command as ./dfw <command-name>, make sure your current (working) directory is set to the Deployment Framework’s topmost directory. For a list of dfw commands, click here.

Blade structure

What’s in a Lua Service blade? Assuming the blade is located in in the Deployment Framework at <Framework-root>/kits/<blade_name>/<blade_name>/, relative to this path:

the blade’s field definitions are in blade_config/fields.conf
Liberator related configuration for the blade is in Liberator/etc/rttpd.conf
Transformer related configuration for the blade is in Transformer/etc/transformer.conf
the configuration for the Lua pipeline is in Transformer/etc/pipeline.conf
the Lua source code for the pipeline is in Transformer/etc/pipeline/<BladeName>.lua

Summary

In brief, you create the blade using the following steps, which are given in more detail further down the page:

Create the blade’s directory structure.
Define the new fields.
Update the core component configuration:
1. Define additional Liberator configuration.
2. Define the Lua Pipeline configuration in Transformer.
Integrate the initial core component configuration with the Deployment Framework.
Write the Lua script for the Pipeline module.
Start the new Transformer Service blade.
Package the new blade.

Create the blade’s directory structure

In a directory that’s outside the Deployment Framework (say Temp_Lua_blade), create the following directory structure:

Directory structure of a Lua-based Transformer Service Blade

At this stage, you don’t need to create the .conf and .lua files that are also shown in the structure - you’ll do that in subsequent steps.

Define the new fields

The blade must define any new fields that it will use, which in this instance means our new Spread field.

Create a text file called fields.conf in Temp_Lua_blade/SpreadLuaBlade/blade_config/

Using a suitable text editor, add the following configuration to fields.conf

#
# Field definitions for the Transformer SpreadLuaBlade
#
add-field Spread 45036

You don’t have to use the field number we’ve shown here; you can choose your own value as long as it’s unique within your particular Caplin Platform installation.

You don’t have to list in the fields.conf file all the fields belonging to the incoming records that the blade will receive, because they’re already defined in the Pricing Adapter’s fields.conf file. For example, The BidPrice and AskPrice fields used in the code samples further down this page would have been defined there. When you deploy your new SpreadLuaBlade blade, the Deployment Framework will make the Pricing Adapter’s field definitions available to it (since the Pricing Adapter will also have been deployed as a blade).

For more about defining fields, see Field definitions in the DataSource Features and Concepts, and Field definition format.

Update the core component configuration

The new blade has to include configuration for the core Platform components that it interacts with. In this case, the core components are Liberator, which will subscribe to data supplied by the Transformer module, and Transformer, which houses the module.

Define additional Liberator configuration

Configuration details for the Liberator are stored in the rttpd.conf file

Create a text file called rttpd.conf in Temp_Lua_blade/SpreadLuaBlade/Liberator/etc/

Using a suitable text editor, add the following configuration to rttpd.conf

#
# Liberator configuration for data from
# the SpreadLuaBlade Lua module blade.
#

add-data-service
   service-name SpreadLuaBlade${THIS_LEG}
   include-pattern ^/FX/
   add-source-group
      required true
      add-priority
         remote-label transformer${THIS_LEG}
      end-priority
      if "${FAILOVER}" == "ENABLED"
         add-priority
            remote-label transformer${OTHER_LEG}
         end-priority
      endif
   end-source-group
end-data-service

This configuration defines a data service through which the Liberator requests the data supplied by the SpreadLuaBlade. It also defines, through the add-source-group … end-source-group block, how the Liberator should fail the data service over to an alternate Liberator (this only applies if failover has been enabled for the deployment - see How can I… Set up server failover capability).

This is fairly standard configuration. The important points to note are:

The service-name configuration item defines the name of the data service, which in this case is (by convention) the name of the blade, SpreadLuaBlade, that supplies the data.
The include-pattern item specifies that the data service is to supply data for all subscriptions whose subjects start with the string /FX/
The macros THIS_LEG and OTHER_LEG are used to differentiate the configuration of primary and secondary failover legs.

Define the Lua Pipeline configuration in Transformer

Configuration details for the Transformer are kept in the pipeline.conf file. They identify the new SpreadLuaBlade blade for Transformer, and define the location of its Pipeline components.

Create a text file called pipeline.conf in Temp_Lua_blade/SpreadLuaBlade/Transformer/etc/

Using a suitable text editor, add the following configuration to pipeline.conf

#
# Transformer pipeline module configuration for the
# SpreadLuaBlade Lua-based Transformer Service Blade.
#

pipeline-paths ${ccd}/pipeline
add-pipeline
   id SpreadLuaBlade${THIS_LEG}
   pipeline-file SpreadLuaBlade.lua
   listener-regex ^/FX/.*
   flags suppressupdate
   update-func update
end-pipeline

The important points to note about this configuration are:

The blade name is used both as the pipeline’s id, and in name of the Lua module defined by the pipeline-file configuration option.
The ccd macro defines the current directory path of the file in which it is referenced. So in this example, when the blade is deployed, the ccd reference will be in in the file <Framework-root>/kits/SpreadLuaBlade/SpreadLuaBlade/Transformer/etc/pipeline.conf, ccd will then define the path <Framework-root>/kits/SpreadLuaBlade/SpreadLuaBlade/Transformer/etc/.
The line pipeline-paths ${ccd}/pipeline/ instructs Transformer to look for the blade’s pipeline files in the relative directory SpreadLuaBlade/Transformer/etc/pipeline/ (as per the ccd macro explained above).
The line pipeline-file SpreadLuaBlade.lua defines the name of the file where the pipeline module’s Lua code is located
The line flags suppressupdate means that messages received by your Lua pipeline module are not sent on automatically, so you will have to do it explicitly in the pipeline module’s Lua code (see the call to packet:send() in the Lua module script below).
The line update-func update defines the name of the Lua function (in the pipeline-file) that is called every time a record is passed to the pipeline module.

That’s all the configuration changes you need for the time being.

Integrate the initial core component configuration with the Deployment Framework

Now you’ve created the initial configuration for the core components that use your Transformer Service blade, you can deploy this configuration in your development environment.

ZIP up the blade directory structure in your temporary work area into a skeleton blade kit, The .zip file must have a name of the form <blade_name>-<version_number>. .zip

The topmost directory in the .zip file must be the top level directory of the blade.

So in our example:

The .zip filename is SpreadLuaBlade-<version_number>.zip (say SpreadLuaBlade-000001.zip),

The topmost directory in the .zip file is SpreadLuaBlade.

Navigate to <Framework-root> and deploy the skeleton blade kit:

./dfw deploy ../Temp_Lua_blade/SpreadLuaBlade-000001.zip

The ./dfw deploy command should respond with:

Boot-strapping the Deployment Framework

   Unpacking SpreadLuaBlade kit SpreadLuaBlade-000001.zip
   SpreadLuaBlade-000001.zip successfully unpacked and stored in kits/archive

   Activating SpreadLuaBlade

   Blades ok

And then ./dfw versions should show the new blade:

Deployment Framework           6.0.4-267113

   Core components                Version
   -----------------------------------------------------------
   Liberator                      6.1.0-275608
   Transformer                    6.1.0-275716

   Deployed blades                Version            State
   -----------------------------------------------------------
   PricingAdapter                 2013.09.27.1138    Active
   SpreadLuaBlade                                    Active

   Built-in blades                                   State
   -----------------------------------------------------------
   BlotterExport                                     Inactive
   DemoDataSource                                    Inactive
   DirectConnection                                  Active
   HTTP                                              Active
   HTTPS                                             Inactive
   JavaOpenPermissioning                             Inactive
   LiberatorJMX                                      Active
   LiberatorWebsite                                  Active
   MinimalLiberatorWebsite                           Inactive
   OpenPermissioning                                 Active
   ServerIdentification                              Active
   TransformerJMX                                    Active

Note that deploying the skeleton SpreadLuaBlade has automatically put it in the active state.

Start the deployed core components and the SpreadLuaBlade blade:
```
./dfw start
```

At this stage the Liberator and Transformer are running with the new configuration. The new blade won’t run of course, because no executable code has been written for the Lua pipeline module.

In a Web browser, navigate to the Liberator’s status page, where you should see the SpreadLuaBlade data service:

Liberator status page showing SpreadLuaBlade data service

The URL of the Liberator is of the form http://<URL_of_liberator_server>:<liberator_port>. You can find the <URL_of_liberator_server> and <liberator_port> by entering the command ./dfw info. In a development environment, the URL would typically be http://localhost:18080. When the Liberator home page is displayed, select the View Status button.

Write the Lua module script

You write the functional part of the blade as a Lua module script in the SpreadLuaBlade.lua file. The task that we defined for this module is very simple to accomplish using Lua (which is one of the reasons why we use Lua).

Create a text file called SpreadLuaBlade.lua in <Framework-root>/kits/SpreadLuaBlade/SpreadLuaBlade/Transformer/etc/pipeline/

Using a suitable text editor, add the following Lua code to SpreadLuaBlade.lua

require("calc")
function update(packet)
   local bidPrice = packet:getfield("BidPrice")
   local askPrice = packet:getfield("AskPrice")

   packet:setfield("Spread", askPrice - bidPrice)
   packet:send()
end

Here’s an explanation of what the Lua code does:

require("calc") imports the Lua calculation API.
An update function is defined, which obtains the BidPrice and AskPrice fields from each incoming record whose subject begins with /FX/ (the Transformer’s pipeline module configuration for the SpreadLuaBlade controls which records are selected - see the line listener-regex ^/FX/.*). Note that the name of this function must match that defined in the update-func option in the pipeline module configuration.
The packet:setfield() function creates a new field called Spread, by subtracting the current value of the BidPrice field from the current value of AskPrice.
The new field is then added to the record (packet:setfield()) and the record is sent on to the Liberator (packet:send()).

Start the new Transformer Service blade

If you have exclusive use of the Pricing Adapter in your development environment, restart the whole system on your development server machine:
```
./dfw start
```
This stops the deployed core components and the Pricing Adapter blade and restarts them; in this case the Liberator, Transformer and the new Integration Adapter. (The command works out from the configured hostname settings which components it needs to start on the machine where the command is running.)
Alternatively, if you share the Pricing Adapter with other developers so you don’t want to shut it down just to get your new Integration Adapter running:
1. Make sure the Pricing Adapter is running:
  ./dfw status
2. Then start the Liberator and Transformer:
  ./dfw start Liberator
  ./dfw start Transformer
When the everything is running:
1. Look at the Liberator’s status page to check that the Liberator is connected to the Transformer and can see the data service (SpreadLuaBlade in our example) for which the new Lua-based Transformer Service blade supplies data.
2. Use the Liberator’s Liberator Explorer to request data for the relevant subjects supplied by the Transformer Service blade, and check that the data is returned (via Transformer) as expected, with a correctly calculated Spread field included.

Package the new blade

When you’ve developed and tested the new blade to your satisfaction, you’ll need to package it up so it can be deployed on your production system - see How can I… Package a custom blade.

See also:

How can I… Package a custom blade
How can I… Set up server failover capability
How can I… Deploy Platform components to the Framework
How can I… Create a Java-based Transformer Service blade
How can I… Start and stop components and blades
Reference: Transformer’s Lua Pipeline API Documentation ("Transformer Pipeline Module API Reference Documentation")
Reference: Transformer Pipeline configuration