Activating the Persistence Service 6.2

The Persistence Service provides a persistent key-value store to Transformer modules and StreamLink clients. The key-value store is persistent across server restarts and shared between nodes in a Transformer cluster.

The service was introduced in Transformer 6.2.3 and rewritten for Transformer 7.0. Modules written using Transformer 6.2’s Persistence API are not compatible with Transformer 7.0’s Persistence API. For guidance on upgrading to Tranformer 7.0, see Upgrading to Caplin Platform 7.

The Persistence Service is required by the following Transformer modules: the Watchlist Service, the Alerts Service, and the High-Low Service.

Requirements

The Persistence Service requires the persistence option in Transformer’s licence.

Activating the Persistence Service blades

The Persistence Service comprises two blades, which are deactivated by default:

  • PersistenceService: This blade provides persistence services to Transformer modules, including the PersistenceServiceClient module (below). See the Java Transformer Module (JTM) package com.caplin.transformer.module.persistence.

  • PersistenceServiceClient: This blade provides persistence services to StreamLink clients. It requires the PersistenceService module to be active.

To activate the Persistence Service blades, run the commands below from the root of your Deployment Framework:

$ ./dfw activate PersistenceService
$ ./dfw activate PersistenceServiceClient

Datastores

The Persistence Service supports two types of datastore:

  • File-backed persistence: Data is persisted to Transformer’s local filesystem. This configuration is not supported for production use.

  • Database-backed persistence: Data is persisted to a database server.

Configuring file-backed persistence

File-backed persistence is configured by default.

File-backed persistence is unsupported in production environments, and, from version 7 of the Persistence Service, file-backed persistence cannot be used in Transformer clusters.

To configure file-backed persistence, edit the file <Framework-root>/global_config/overrides/PersistenceService/Transformer/etc/persistence.conf so that it is consistent with the values in the table below.

Configuration item Value Comment

enable-file-database

TRUE

Set to TRUE to enable file-backed persistence.

add-database-params/init-string

${ccd}/persistence

The file-system directory that file-backed persistence should persist data to.

Configuring database-backed persistence

To configure the Persistence Service to use database-backed persistence:

  1. Create the Persistence Service’s database table

  2. Add the database’s JDBC driver to Transformer’s classpath

  3. Configure the connection to the database

  4. Disable repeated database writes in a Transformer cluster

1. Create the Persistence Service’s database table

Choose a SQL database server that has a JDBC driver and that supports authentication by username and password.

Create a table with two text columns. The first column stores persistence keys and the second column stores persistence values. Define the column that stores the persistence keys as the primary key of the table. You may choose any name for the table and its columns: the table name and column names are configurable in the Persistence Service’s configuration file (see Defining the persistence configuration items).

Here’s an example MySQL table definition that uses the default table and column names defined in the Persistence Service’s configuration file:

CREATE TABLE persistence (
    pers_key VARCHAR(100),
    pers_val TEXT,
    PRIMARY KEY (pers_key)
);

You may need to increase the width of the 'pers_key' column if a module or StreamLink client uses keys greater than 100 characters in length.

2. Add the database’s JDBC driver to Transformer’s classpath

To add the database’s JDBC driver to Transformer’s classpath:

  1. Create the following directory: <Framework-root>/global_config/overrides/PersistenceService/Transformer/lib/java

  2. Copy the JDBC driver JAR file to the directory created in step 1.

  3. Make the following two changes to the file: <Framework-root>/global_config/overrides/PersistenceService/Transformer/etc/java.conf

    1. Add the following line (where filename is the name of JDBC driver JAR file):

      jvm-global-classpath "${ccd}/../lib/java/filename"

    2. Uncomment the following line:

      jvm-location ${JVM_LOCATION}

3. Configure the connection to the database

To configure database-backed persistence, edit the file <Framework-root>/global_config/overrides/PersistenceService/Transformer/etc/persistence.conf so that it is consistent with the values in the table below.

Configuration item Value Comment

enable-file-database

FALSE

Set to FALSE to use database-backed persistence.

persistence-classid

com/caplin/transformer/persistence/JdbcPersistorImpl

The implementation of Persistor used to write data to JDBC databases. Unless you write your own Persistor, use the value in this table.

add-database-params/init-string

jdbc:mysql://localhost/rtas

The JDBC URL of the database. The example value in this table is for a local MySQL database called 'rtas'.

add-database-params/driver-name

com.mysql.cj.jdbc.Driver

The name of the JDBC driver’s main class. The example value in this table is for Oracle’s MySQL Connector/J 6

add-database-params/username

user1

The username of the database account.

add-database-params/password

password

The password of the database account.

add-database-params/table-name

persistence

The name of the table used to store persistence data.

add-database-params/columns

pers_key pers_val

The names of the 'key' and 'value' columns, specified in that order and separated by a space.

add-database-params/extra-params

DEBUG

Extra parameters accepted by the Persistor. The com.caplin.transformer.persistence.JdbcPersistorImpl accepts one parameter: a log level.

4. Disable repeated database writes in a Transformer cluster

The Persistence Service 6.2 supports synchronisation of persisted data between nodes in a Transformer cluster. A change to a persisted value on one node will replicate to other nodes in the cluster.

By default, any change made to a persisted value results in a write to the database server. This includes changes made by the Persistence Service’s cluster synchronisation. As a result, when a value is persisted on one cluster node, the value will be written to the database server as many times as there are nodes in the cluster. This is inefficient and unnecessary.

To stop each cluster node from persisting changes made during cluster synchronisation, add the following configuration option to <Framework-root>/global_config/overrides/PersistenceService/Transformer/etc/persistence.conf on each node in the cluster.

dont-persist-cluster-messages   TRUE

Configuring permissioning

To allow users to interact with the Persistence Service Client, configure Liberator’s Permissioning Service according to the specification below. For more information about permissioning, see User Authentication and Permissioning.

Rules

Define rules that trigger when users contribute (publish) to the Persistence Service Client’s CONTROL and RECORD subjects. The strings used for the permission namespace and for the action are arbitrary; you are free to customise them to your own naming scheme.

Message Subject Rule Type Permission Namespace Action Action Ref Product Ref

^/PRIVATE/PERSISTENCE/%u/CONTROL

WRITE

PERSISTENCE_SERVICE

PERSISTENCE-CONTROL

-

ALL_PRODUCTS

^/PRIVATE/PERSISTENCE/%u/RECORD/.*

WRITE

PERSISTENCE_SERVICE

PERSISTENCE-RECORD

-

ALL_PRODUCTS

Permissions

Assign users the following permissions.

Product Set Permission Namespace Action Authorisation

.*

PERSISTENCE_SERVICE

PERSISTENCE-CONTROL

ALLOW

.*

PERSISTENCE_SERVICE

PERSISTENCE-RECORD

ALLOW

^/PRIVATE/PERSISTENCE/%u/CONTROL

-

VIEW

ALLOW

^/PRIVATE/PERSISTENCE/%u/RECORD/.*

-

VIEW

ALLOW

See also: