Pipeline Concepts: Data Dictionary

The standard library supplied with the pipeline module enforces the use of a Data Dictionary to store configuration such as fields and formatting information.

Using the data dictionary enables all field configuration to be located in a single configuration file rather than being present in many script files.

For a price field, the following configuration options define a field and format options:

database-insert "/DataDictionary" "PriceField"             "Price"
database-insert "/DataDictionary" "PriceFormat"            "dp:places=5;"

These are then used by the datadict::addformattedfield function to produce an output field in the correct format:

datadict.addformattedfield(packet, "PriceKey", "PriceFormat", "1.0")

Caplin also recommend the use of a data dictionary when handling incoming data, this again makes changing the field configuration a lot easier and allows reuse of scripts between different markets as can be seen in the following example which handles two markets which use different price fields:

add-pipeline
  id             equity-london
  listener-regex ^/I/*\.L
  pipeline-file  equity.lua
  update-func    update
  add-info       "IncomingPriceField"       "GEN_VAL1"
  add-info       "IncomingPriceFieldFormat" "frac;dp:places=5;"
end-pipeline

add-pipeline
  id             equity-nasdaq
  listener-regex ^/I/*\.O
  pipeline-file  equity.lua
  update-func    update
  add-info       "IncomingPriceField"       "TRDPRC_1"
  add-info       "IncomingPriceFieldFormat" "frac;dp:places=5;"
end-pipeline

In both case, the same update function is called, this may look similar to the following snippet:

function update(incoming)
     local outgoing = createrecord(outgoingsymbolname)

     local price = datadict.getfield(incoming, "IncomingPriceField", "IncomingPriceFieldFormat")

     datadict.addformattedfield(output, "PriceField", "PriceFieldFormat", price)

     outgoing:send()
end