Benchmarking Caplin Platform

Benchsrc and Benchrttp are internal Caplin tools for performance testing the Caplin Platform, made available to customers on request on an unsupported basis.

Benchsrc is used to generate Record and Container updates. Benchrttp is used to generate client sessions, exerting load on the Platform. Using either or both of these it is possible to performance test the Caplin Platform under varying message and client load.

Download the presentations for this course.

Overview

In this tutorial, you will learn how to:

  • deploy the Benchsrc blade

  • configure Benchsrc to provide record updates

  • send UDP commands to change the update rate from Benchsrc

  • set up and configure a Benchrttp client

  • send UDP commands to Benchrttp to add more clients

  • send UDP commands to record the current latency status

Requirements

This tutorial requires the latest development snapshots of the following two internal Caplin testing tools:

  • Bench RTTP

  • Bench Source

These internal tools are made available by Caplin to customers "as is" on an unsupported basis and with minimal documentation. The primary purpose of these tools is to support Caplin’s testing pipelines, and features may be added, changed, or removed without notice to support this purpose. For the full terms and conditions, see the release notes on the Caplin Download site.

If you would like access to these tools, please contact your Caplin account manager.

Benchsrc

In previous tutorials, you learnt how to set up the Deployment Framework and how to deploy Liberator and Transformer. These are necessary prerequisites. Deploying the Benchsrc blade is a similar process.

Follow the steps below

  1. Copy the Benchsrc kit to the kits directory of your Deployment Framework.

  2. Deploy the Benchrc kit:

    ./dfw deploy

  3. Configure Benchsrc to run on the localhost:

    ./dfw hosts Benchsrc localhost

  4. Start the Deployment Framework:

    ./dfw start

  5. Open Liberator Explorer and try requesting the subject /BENCHSRC/A.

    If you have difficulty authenticating, check your permissioning adapter is running.

    You should see a record updating in real-time.

Support for Liberator versions earlier than 8

Benchsrc is a Caplin DataSource 8 application. To use Benchsrc with a version of Liberator earlier than 8, add the following configuration to disable Dynamic Fields in Benchsrc:

global_config/overrides/Benchsrc/DataSource/etc/benchsrc.conf
datasrc-local-flags nofields

Configure a data set in Benchsrc

You can configure the records provided by the Benchsrc, tailoring the data provided to the Platform.

Open global_config/overrides/Benchsrc/DataSource/etc/benchsrc.conf. Here you will find the following configuration:

global_config/overrides/Benchsrc/DataSource/etc/benchsrc.conf
add-object-set
        id                        ACTIVESET
        updates-rate-per-object   1
        prefix                    /

        add-field
            fieldnum            1
            type microseconds
        end-field

        add-field
            fieldnum            2
            type seqnum
        end-field

        add-field
                fieldnum        22
                type            price
                min-value       428.10
                max-value       428.30
                format          "%.2f"
        end-field

        add-field
                fieldnum        25
                min-value       428.40
                max-value       428.60
                type            price
                format          "%.2f"
        end-field
end-object-set

Analysis:

  • id: This option is used to set the ID for the object set. This will be used when sending UDP messages to change the update rate.

  • prefix: This option is used to set the prefix of the object set. E.g. If you set the prefix to be /BENCHSRC/TUTORIAL/ that means any request subject starting with /BENCHSRC/TUTORIAL/ will apply this object set.

  • updates-rate-per-object: This option is used to set the update rate for the object set. E.g. When set to 4, this means each record from this object set will update 4 times per second. (Note: This doesn’t mean that for each second it will update 4 times. Rather, it means that after running for a long time, the average number of updates per second is 4.)

  • add-field / end-field: These options are used to configure the fields of the object. The options to configure the fields are:

    • field-num: This option is used to set the field number with which the field name is encoded (where the field name is found in fields.conf).

    • type: This option describes the kind of updates which this field will send. The possible types are:

      • milliseconds: When the record updates the field will contain the current timestamp in milliseconds

      • microseconds: When the record updates the field will contain the current timestamp in microseconds

      • seqnum: Record updates are numbered in sequence i.e. seqnum is 1 on the first update, 2 on the second update … n on the nth update.

      • price: This field’s value will be a random decimal number in the format defined by the format option

      • list: When the record updates, it will update the next item in the list defined by the values option. E.g. If we define values as A B C then on the first update the field’s value will be A, on the second B, on the third C, on the fourth A again, and so on.

      • fixed: The field will have the same value on each record update.

    • format: This option is used to define format of the decimal number for a field. E.g. %.2f, %d

    • values: This option is used to define the possible values of a field.

    • image-only: When this option is set, the field will only appear in the first update.

Example 1. Try this Yourself

Define an object set that meets the following requirements:

  • It applies to any record starting with /BENCHSRC/TUTORIAL

  • The record updates once every 2 seconds

  • The record contains following fields:

    • A millisecond timestamp

    • A list with options: Cat, Dog, Pig, Cow

    • A random price between 0 and 20

Add the fields to global_config/fields.conf

You can try it out by requesting /BENCHSRC/TUTORIAL/0 in Liberator Explorer.

Send UDP commands to change update rate

In the previous section you’ve just seen how the update rate is set in the config file. However, on every configuration change Benchsrc must be restarted for the changes to take effect. This is inefficient when running multiple test cases. Benchsrc also supports a UDP interface so that commands (such as changing the update rate) can be sent at runtime. However, note that UDP messages are not guaranteed to be received by the Benchsrc, especially under heavy load. Therefore you should always check the logs to make sure that the UDP command is received.

The UDP command tool can be found here: active_blades/Benchsrc/DataSource/bin/udpsend.exe

This is the command to change the update rate at runtime:

udpsend -s <ipaddress> -p <udp-port> updates-rate-per-object <new-update-rate> <id>

So if you want to change the update rate to one update every 10 seconds for the object set defined in the previous section:

  • ipaddress: Since the benchsrc is running on your local machine, the ip address will be 127.0.0.1

  • udp-port: You can find the udp-port defined in global_config/overrides/Benchsrc/DataSource/etc/benchsrc.conf as ${THIS_LEG}5010. You will also find in global_config/environment.conf that THIS_LEG is 1. Thus, the udp-port is 15010

  • new-update-rate: This should be the update rate that we want to change to. It should be 0.1

  • id: This should be the id if the object set e.g. ACTIVESET.

Example 2. Try this Yourself

  1. While Liberator, Transformer and Benchsrc are running use the udpsend tool to change the update rate to one update every 10 seconds. You should notice that the updates for the /BENCHSRC/TUTORIAL/0 object suddenly slow down in Liberator Explorer. You should also notice the following two lines in the benchsrc log file:

    INFO: Processing UDP Command 'updates-rate-per-object' with arguments '0.1 ACTIVESET'
INFO: UDP:updates-rate-per-object ACTIVESET 0.1

  2. Next try to change the update rate to 10 times per second.

  3. You should notice that after you change the update rate, the record begins updating once every second, not 10 times per second as expected. Can you tell why? It is because the default throttling for Liberator is once per second. To turn it off, add this line to global_config/overrides/Benchsrc/Liberator/etc/rttpd.conf

    object-throttle-off

  4. Restart the environment, refresh Liberator Explorer and request /BENCHSRC/TUTORIAL/0. Change the update rate to 10 updates per second and you should notice the very fast update rate in the Liberator Explorer.

Benchrttp

The Benchrttp client generator is not shipped as a Platform blade and cannot be deployed in the Deployment Framework.

Follow the steps below:

  1. Copy the Benchrttp package to convenient a location on your hard drive. Open up a new cygwin console window and unzip the Benchrttp package using cygwin unzip command.

  2. Open the Benchrttp configuration located in etc/benchrttp.conf.

  3. Ensure that it is configured to connect to the correct Liberator host and port:

    server           localhost
port             18080

    Notice the following default configuration in this file:

    clients          1

    You don’t need to change this at the moment, but this configuration option defines how many users you want to create.

    connect-time   0.1

    This option defines the interval between user logins. When set to 0.1, there will be a 0.1 second interval between a user’s login and the next.

  4. Next you need change the object-set that you would like to request. Replace:

    add-object-set
        id              BCASTSET1
        num             20
        max             100
        format          /A/Xd
end-object-set

    with

    add-object-set
        id             TEST1
        num            10
        max            1000
        format         /BENCHSRC/TUTORIAL/Xd
end-object-set

    For each client 10 objects will be chosen randomly from 1000 with names /BENCHSRC/TUTORIAL/X 00000, /BENCHSRC/TUTORIAL/X X00001, etc.

  5. Then remove this line (Java is not needed for Benchrttp):

    include-file      java.conf

  6. The last thing is to add this line:

    microsecond-latency

    This means the latency is calculated at the microsecond level and also the timestamp sent from the benchsrc is microsecond level.

  7. Restart the Deployment Framework and start Benchrttp by running bin/benchrttp start (You can stop Benchrttp by running bin/benchrttp stop)

  8. Open the logs: var/event-benchrttp.log. Here you should see the indication that all clients have logged in and the total amount of clients is 1:

    INFO: All Clients 1

Send UDP commands to add more clients

The client option only allows you to set the number of users before you start the Benchrttp. When you want to add more users during the test, you should use UDP commands to add more users.

The syntax of the UDP message to add more clients should be:

./udpsend -s <ipaddress> -p <benchrttp-udp-port> clients <number-of-clients>

For example, if we want to set the number of clients to 10 clients.

  • ipaddress: Should be 127.0.0.1 as Benchrttp is running on the localhost

  • benchrttp-udp-port: Should be the udp-port set in the benchrttp.conf which is 13001

  • number-of-clients: Should be 10.

Example 3. Try this yourself

Use udpsend (in the bin directory) to change the number of clients to 10.

Check the logs and wait until you see:

INFO: All Clients 10

Send UDP commands to record the current latency

Benchrttp and Benchsrc are predominately used for performance testing where an important metric to measure is the message latency. Benchrttp calculates latency automatically from an embedded timestamp but to achieve this you must make sure your Benchsrc provides a field called Time with the timestamp in it. The timestamp can be either in milliseconds or microseconds. If it is in microsecond make sure you have added microsecond-latency in your benchrttp.conf.

The syntax of the udp message to check the latency status should be:

./udpsend -s <ipaddress> -p <benchrttp-udp-port> stats <time-in-seconds>

Follow the steps below:

  1. If we want to get the average latency in 30 seconds, the command is:

    ./udpsend -s 127.0.0.1 -p 13001 stats 30

  2. When you execute this command the logs should show:

    INFO: Processing UDP Command 'stats' with arguments '30'

  3. Wait for around 30 seconds, when the logs should show something similar to:

    STATS AVERAGE    30     5.06   10 1674 - min 0 max 10000 sd 946 nummax 24 sum 1361074620 num 1519 bw 210 2101 contrib 0 max 0

    Where:

    Value Description

    30

    The amount of time that we used to check the latency

    5.06

    The average update rate per client per second during that 30 seconds. It means each client got an average of 5.06 updates every second. (Note: that is not per subscription)

    10

    The number of clients

    1674

    The average latency in 30 seconds in microseconds

    min 0

    The minimum latency in 30 seconds was 0 microseconds

    max 10000

    The maximum latency in the 30 seconds was 10000 microseconds

    sd 946

    The standard deviation

Review

Having completed this tutorial you should be familiar with how to use Benchsrc and Benchrttp to create a performance testing environment for Liberator.

Just a reminder that performance tests should be run in Linux rather than Windows due to performance considerations - Windows kits are for development purposes only.