Troubleshooting checklist

This page provides a systematic approach to troubleshooting RET Adapter issues. It outlines a series of steps to follow when debugging the RET Adapter Toolkit stack.

An example of an issue could be "rates aren’t being received", or "blotters aren’t populated", rather than "a particular trade has failed".

Preliminaries

Have you identified what feature is broken, and which adapter provides that feature?

See the Adapters Feature List for a list of available features.

Make sure logging is enabled for debugging:

  1. Enable StreamLinkJS logs in the client by adding ?debug=finer to the query

  2. Enable TrAPI logs by adding ret_log_enabled=true to the trapi-connection.properties file for the relevant adapter

Connectivity

Check the Liberator Status page - is the Adapter connected to Liberator? If you can’t access this page you will need to open a console and navigate to you Deployment Framework’s topmost directory, then enter the following command:

./dfw status

If the output from the above status command indicates that there is no connection, check the adapter logs. Start with java-adapter.log (which is the Java standard output log) to see if the Adapter has crashed or if any Java classes are missing.

Check the all-adapter.log and observe any TrAPI connection errors (these are usually trading exceptions thrown by TrAPI or AdminAPI). Search keyword: TradingException.

Ensure that only one instance of the Adapter is configured on one machine to connect to the Liberator.

Ensure there are no infrastructure issues that may affect connectivity. For example, check the SCS Relay status.

Configuration

Check the configuration relating to the broken feature. The Liberator Configuration Reference explains how to correctly configure Caplin Platform components.

Log review

Review the adapter’s all-adapter.log log file. Observe the time stamps within the logs and assess the importance of any exceptions thrown that are logged during the action in question.

With regards to log levels - an ERROR does not necessarily mean a fatal error; instead it could refer to something routine such as a bad response from RET or the client. While investigating adapter issues, it is advised that you investigate things that are relevant. Do not waste effort observing error lines if they do not appear significant or relevant to the issue at hand.

Some exceptions may be routine, so it is important to identify the relevant exceptions. This can also highlight any connectivity problems - you will need to investigate any connection errors such as checking the status of the RET SCS Relay.

Full stack logs

Often, there is no failure that would indicate a problem with the adapters. In this case it is necessary to trace the failing action through the entire stack: Client → Liberator → Transformer → Adapter → TrAPI.

For example, when a blotter subscription is sent by a client, we would expect to see it pass through Liberator, Transformer, and then one of the Blotter Adapters.

First identify the subscription by its Subject and the time it was requested (for Trading and Orders issues, follow the order id):

  • Does the subscription appear in the Liberator event log? If not, was it rejected in the Auth log? See Permissioning Issues further down.

  • Does the subscription appear in the Transformer event log? If not, was any connectivity established?

  • Does the subscription appear in the Adapter log? If not, is the Liberator or Transformer service correctly configured? Are there any connectivity issues present?

  • If the failing action appears in the adapter’s all-adapter.log, is it possible to follow how it was processed?

  • Each step of processing the action is logged, and therefore any missing steps would be suspicious. Was the action explicitly logged as failed?

  • If a response is sent to the client, does it arrive as expected? Trace back the steps to confirm how it is sent.

Sending logs to Caplin support

Use the ./dfw dump command to create a configuration dump.

Use the ./dfw logs command to create a log dump.

Use the zip utility to create a single, compressed archive. Do not use RAR or P7 compression algorithms.

A screenshot of the web client is always useful.

Selective or specific log snippets are unhelpful at this point, as it is likely that logs will need to be looked at end-to-end. In general, logs provided as text files are most useful as they can be processed quickly using a grep command-line utility and other tools.

Permissioning issues

If you suspect a permissioning issue, review the Liberator auth.log. This must be set to DEBUG before any useful information is captured (configuration details are explained in the Liberator Configuration Reference).

If the failing action is rejected by the Auth module, it will log an Authentication result: Failure: Denied message.

Review the debug level messages logged during start up. Are the correct permissions being received? If not, then review the Permissioning Adapter logs and configuration.

  • Is the Permissioning Adapter UP and communicating with Liberator? Check this by Navigating to the Liberator status page http://localhost:18080/status

  • Are the correct permissions being sent? Check the Permissioning Adapter’s all-adapter.log file to confirm that permissions were loaded correctly.

  • Is the underlying source of the permissioning config correctly set up? Is it providing the right permissions?

  • Are any group or user permissions missing?