Pipeline configuration

These configuration items enable you to add a pipeline to Transformer and specify how Transformer is to use it.

Configuration file

Transformer’s pipeline configuration must be kept in a separate configuration file called pipeline.conf

If you’re using the Deployment Framework, put changes and additions to this configuration in <Framework-root>/global_config/overrides/servers/Transformer/etc/pipeline.conf, where <Framework-root> is the Deployment Framework’s topmost directory.

Configuration items

Pipeline configuration items:

add-pipeline

add-pipeline specifies the configuration of a Lua or JavaScript pipeline that is to be loaded into Transformer.

Syntax:

add-pipeline
   id              [string]
   pipeline-file   [string]
   add-info        [array of strings]
   discard-func    [string]
   exclude-regex   [array of strings]
   flags           [integer]
   initialise-func [string]
   listener-regex  [array of strings]
   log-flags       [integer]
   nodata-func     [string]
   no-delay-load   [boolean]
   provider-exclude-regex [array of strings]
   provider-regex  [array of strings]
   request-func    [string]
   required-fields [array of strings]
   status-func     [string]
   type            [integer]
   udp-command     [array of strings]
   udpsig-func     [string]
   update-func     [string]
end-pipeline
Option Type Default Description

add-info

array of strings

NULL (no extra information added)

Specifies extra information that’s added to the pipeline module database (see the Pipeline Database configuration). Each pair of strings in the supplied array is a key + value pair, where the key part constitutes a secondary key in the database. The secondary key + value pair is stored in the database under the primary key /PIPELINE/<pipeline_id>

For example, add-info foo bar stores the value "bar" in the database under the primary key /PIPELINE/<pipeline_id> and the secondary key "foo".

discard-func

string

[none]

The name of the Lua or JavaScript function to call when an update is received whose subject matches the provider-regex pattern, and no modules or peers wish to receive updates for this subject.

The signature of the function must be: function <discard-name>(peer, subject)

The peer argument is the index of the Transformer configuration’s add-peer entry for the peer that sent the update (the first add-peer entry has index 0, the second has index 1, and so on.)

The function must return 0 if the discard can’t be handled, or 1 if it can.

This function is always called on the main thread.

exclude-regex

array of strings

[an empty list]

A space-separated list of one or more regular expressions used to match (select) the subject names of updates that are not to be processed by this pipeline.

You can put multiple instances of this option in the add-pipeline specification.

Also see listener-regex.

The equivalent option for filtering out the subject names of subscription requests and discard requests is provider-exclude-regex.

flags

string or integer

0

(no flags specified)

Flags that specify the behaviour of this pipeline. You can specify a flag by name or numeric value:

  • none or 0: No flags specified.

  • manyprovider or 1: The pipeline can receive multiple requests and discards for the same subject.

  • initallatstart or 2: The pipeline’s initialisation function, as defined by the initialise-func option, is to be called on all the pipeline’s threads as soon as the Transformer starts up and the pipeline has been loaded. This setting only works when the no-delay-load option is TRUE.

  • clearupdate or 4: Updates coming into this pipeline will be cleared after processing, to prevent them from being processed by other pipelines or other Transformer modules.

  • suppressupdate or 8: Incoming updates won’t be distributed to connected peers.

  • accessory or 16: Subscriptions on this pipeline are accessory subscriptions. This means that the pipeline doesn’t itself subscribe to any of the subjects for which it listens for updates (see listener-regex). Instead, it assumes that some other pipeline or Transformer module will subscribe to these subjects, and it just listens for any incoming updates relating to them.

Use the | operator to specify compatible multiple flag values; for example: flags manyprovider|initallatstart

id

string

[automatically generated]

A unique identifier for the pipeline. This is used by the add-info option, and is also recorded against entries in the log file specified by the logfile configuration item.

If you don’t define an id, Transformer generates one internally when it loads the pipeline. Its name is pipeline-<index>, where index is the ordinal number of this add-pipeline configuration item within the Transformer’s configuration, starting from zero. For example, the internal id of the twentieth add-pipeline item in the configuration is pipeline-19.

initialise-func

string

[none]

The name of the Lua or JavaScript function to call when the script named in pipeline-file is first loaded.

The signature of the function must be: function <initialise-func-name>()

The initialise function is called when the script is first loaded; it’s always called on the main thread. Subsequently, it may also be called multiple times, depending on the setting of the Pipeline threads configuration item num-threads

The script is normally loaded when a matching update is received, but if you also want the initialise() function to be called when the Transformer is started up, use the no-delay-load option.

listener-regex

string array

[any subject matches, unless it’s excluded by exclude-regex]

A space-separated list of one or more regular expressions used to match (select) the subject names of updates that are to be passed to the Lua or JavaScript function specified in the update-func option. This option’s also used to match the subject names of nodata messages that invoke the Lua or JavaScript function specified in the nodata-func option.

You can put multiple instances of this option in the add-pipeline specification.

Also see exclude-regex, update-func and nodata-func.

The equivalent option for selecting the subject names of subscription requests and discard requests is provider-regex.

log-flags

integer|string

0

(no extra logging)

Flags that specify additional items that can be logged to the logfile for this pipeline. You can specify a flag by name or numeric value:

  • none or 0: No additional logging.

  • update or 1: Additionally log received updates.

  • nodata or 2: Additionally log received nodata indications.

  • filter or 4: Additionally log received filters.

  • timer or 8: Additionally log timer interactions.

  • all or -1: All of update, nodata, filter and timer.

You can only specify one flag value. This means that you can either specify just one type of additional item to log, or you can specify all to include all of the additional items in the log.

To specify additional logging for dynamically created pipelines, use pipeline-logflags.

nodata-func

string

[none]

The name of the Lua or JavaScript function to call when a nodata message is received whose subject (symbol) matches one of the patterns specified by listener-regex.

The signature of the function must be: function <nodata-func-name>(peer, symbolname, flags)

The peer argument is the index of the Transformer configuration’s add-peer entry for the peer that sent the nodata message (the first add-peer entry has index 0, the second has index 1, and so on.)

In the implementation of the function, don’t inspect the flags parameter, since it has no significance.

This function can be called from any of the pipeline’s worker threads (see num-threads in the Pipeline threads configuration).

no-delay-load

boolean

FALSE

When TRUE, this option forces the Lua script or JavaScript module named in pipeline-file to be loaded as soon as the Transformer starts up, rather than being loaded only when first required.

pipeline-file

string

[none]

The name of the file containing the code of the pipeline that’s to be loaded.

For example:

  • Lua: pipeline-file myscript.lua

  • JavaScript: pipeline-file myscript.js

provider-exclude-regex

array of strings

[an empty list]

A space-separated list of one or more regular expressions used to match (select) the subject names of of subscription requests and discard requests that are not to be processed by this pipeline.

You can put multiple instances of this option in the add-pipeline specification.

Also see provider-regex.

The equivalent option for filtering out updates is exclude-regex.

provider-regex

array of strings

[any subject matches, unless it’s excluded by provider-exclude-regex]

A space-separated list of one or more regular expressions used to match (select) the subject names of subscription requests and discard requests that are to be processed by this pipeline.

You can put multiple instances of this option in the add-pipeline specification.

Also see provider-exclude-regex, request-func and status-func.

The equivalent option for selecting the subject names of updates is listener-regex.

request-func

string

[none]

The name of a Lua or JavaScript function that’s called when a subscription request is received whose subject (symbol) matches one of the patterns specified by provider-regex. If you don’t specify a provider-regex, the request function is never called.

The signature of the function must be: function <request-func-name>(peer, symbolname)

The peer argument is the index of the Transformer configuration’s add-peer entry for the peer that sent the subscription request (the first add-peer entry has index 0, the second has index 1, and so on.)

The function must return 0 if the request can’t be handled, or 1 if it can.

This function is always called on the main thread.

required-fields

array of strings

[an empty list]

A space-separated list of one or more field names. When the pipeline receives an update (filtered according to the settings of listener-regex and exclude-regex), the update will only be processed if at least one of the fields listed in required-felds is present in the update.

Each field name in the list must correspond to a field name in an add-field configuration item in the relevant fields.conf file.

status-func

string

[none]

The name of the Lua or JavaScript function to call when an update to the status of a subject is received, where the subject matches the provider-regex pattern.

The signature of the function must be: function <status-func-name>(feed, subject, status, code, status_message)

where:

  • feed is the index of the Transformer configuration’s add-peer entry for the peer that sent the status update (the first add-peer entry has index 0, the second has index 1, and so on.)

  • status is the subject status (OK, STALE or LIMITED).

  • code is always 0 (not used).

  • status-message gives more detail about the subject status.

This function is called on the pipeline thread for the subject being processed.

type

integer

0 (none)

The type of this pipeline, as a name or value:

  • none or 0: This pipeline is not a default pipeline.

  • default or 1: This pipeline is a default pipeline that will capture and process all updates not caught by other pipelines.

Note that the default setting of this option is not default.

udp-command

[array of strings]

[none]

A list of UDP commands that this pipeline will respond to.

When Transformer receives a UDP command that’s in this list, it calls the pipeline’s Lua or JavaScript function that’s specified by the udpsig-func option.

Example: udp-command source_up

udpsig-func

string

[none]

The name of the Lua or JavaScript function to call when Transformer receives a UDP command.

The signature of the function must be: function <udpsig-func-name> (msg)

where msg is a string containing the UDP message, the first field of which is the UDP command.

This function can be called from any of the pipeline’s worker threads (see num-threads in the Pipeline threads configuration).

update-func

string

[none]

The name of the Lua or JavaScript function to call when an update is received whose subject matches one of the patterns specified by listener-regex. If you don’t specify a listener-regex, the update function is never called.

The signature of the function must be: function <update-name>(update)

where the update parameter is of type dsdata (see the DSData Update Packets module in the Lua Pipeline API documentation or the JavaScript Pipeline API documentation).

This function can be called from any of the pipeline’s worker threads (see num-threads in the Pipeline threads configuration).

logfile

logfile specifies the filename of the pipeline activity log in which pipeline messages are recorded. All log messages generated using the Logging routines package of the Pipeline Module API are also sent to this file (see the Lua Pipeline Module API Documentation or the JavaScript Pipeline API documentationhttps://www.caplin.com/developer/api/transformer_pipeline_sdk/latest/[)]. Pipeline database activity is also recorded in this log. (see database-log-options in Pipeline database configuration.)

The filename can contain the parameters %a and %h At run time, %a is replaced by the Transformer’s application name (see the DataSource configuration item application-name), and %h is replaced by the host name of the machine on which the Transformer is running.

Syntax: logfile <log-file-name>

Type: string

Default value: pipeline.log

log-level

log-level specifies the severity of errors and events that are reported to the pipeline’s log file (see logfile).

Syntax: log-level <log-level-name>

Type: string

Default value: INFO

Values accepted:

Log-level-name Description

DEBUG

Reports all errors and events.

INFO

Reports events and information regarding normal operation, and all events included in the WARN, NOTIFY, ERROR and CRITICAL debug levels.

WARN

Reports minor errors and all events included in the NOTIFY, ERROR and CRITICAL log levels.

NOTIFY

Reports errors and all events included in the ERROR and CRITICAL log levels.

ERROR

Reports serious errors and all events included in the CRITICAL log level.

CRITICAL

Reports critical errors that prevent the pipeline from running.

pipeline-logflags

pipeline-logflags specifies flags that make dynamic pipelines log additional items to the logfile. (A dynamic pipeline is a pipeline that’s created at run-time by another pipeline.)

Syntax: pipeline-logflags <flag-name1> <flag-name2> <flag-name3> …​

Type: array of strings

Default value: none (no additional logging for dynamic pipelines)

Values accepted:

Flag name Meaning

none

No additional logging

update

Additionally log received updates

nodata

Additionally log received nodata indications

filter

Additionally log received filters

timer

Additionally log timer interactions

all

All of update, nodata, filter and timer.

Example:

pipeline-logflags update nodata

This configuration causes all dynamic pipelines to log both received updates and received nodata indications in addition to the standard log items.

To specify additional logging for a non-dynamic pipelines, use the log-flags option of add-pipeline.

pipeline-paths

pipeline-paths specifies a space-separated list of the directory paths where Transformer’s pipeline scripts and libraries can be found. You can separate the scripts and libraries across directories for ease of maintenance.

The directory path can contain the parameter %r At run time, %r is replaced by the root directory (application-root) under which the Transformer runs.

Syntax: pipeline-paths <list-of-directory-paths>

Type: array of strings

Default value: %r/lib/pipeline (This is where the core Caplin-supplied packages reside)

pipeline-share-state

pipeline-share-state specifies when TRUE that multiple simultaneous instances of the same dynamically-loaded pipeline share their compilation state within a particular thread. This saves memory, because once the code in the pipeline’s Lua or JavaScript file has been loaded and compiled, subsequent calls on the same pipeline (in the same thread) will reuse the compiled code, rather than loading and compiling more copies of the file.

Syntax: pipeline-share-state <boolean>

Type: boolean

Default value:

  • Transformer 7.0: TRUE

  • Transformer 6.2: FALSE

Example add-pipeline configuration

Here is some example pipeline configuration. This example is for a pipeline written in Lua.

add-pipeline
   # This is a default pipeline
   type                    default
   pipeline-file           mydefaultpipeline.lua
   initialise-func         myinitfunc
   listener-regex          ^/I/.*\.L
   exclude-regex           ^/I/VOD.L
   update-func             myupdatehandler
   nodata-func             mynodatahandler
   provider-regex          ^/I/.*\.L
   provider-exclude-regex  ^/I/VOD.L
   request-func            myrequesthandler
   status-func             mystatushandler
end-pipeline

  • The pipeline is specified as a default pipeline, which means that it captures and processes all updates that aren’t caught by other pipelines. You have to explicitly declare that it’s a default pipeline (type default), because otherwise the default configuration setting of type none would apply, which would mean the pipeline wasn’t a default one.

  • Since there is no id option in the configuration, the pipeline’s identifier is automatically generated when the pipeline is loaded.

  • The pipeline script is loaded from the file mydefaultpipeline.lua in the (default) directory %r/lib/pipeline and it’s initialised by a call to the function myinitfunc().

  • The myupdatehandler() function of the Lua script mydefaultpipeline.lua handles all updates whose subjects have the form /I/<NAME>.L (specified by listener-regex), except the subject /I/VOD.L/ (specified by exclude-regex). The function mynodatahandler() handles all nodata messages received for the same subjects.

  • The myrequesthandler() function handles all subscription requests for the same subjects, but in this case, the subject filters must be specified by provider-regex and provider-exclude-regex.

    The mystatushandler() function handles all status change messages received for these subjects.

If your pipeline is written in JavaScript rather than Lua, the only thing that changes in the above configuration is the name of the pipeline-file, which would become mydefaultpipeline.js

See also: