View packet logs using logcat

The logs produced by most Caplin Platform components are simple text files that can be viewed using a suitable text display utility or text editor, such as the Linux commands cat, more, and vim. However, the packet logs are in binary format and must be viewed using the Caplin logcat utility. Here’s how to do this.

Requirements

If the logcat command’s -F option is used, and the fields file referenced by the -f option contains the CONFIG_BASE variable, then you must enable the Deployment Framework’s command-line completion extension. See Setting up dfw command completion.

From version 7.1 of the Caplin Platform, packet logs use a new binary format to accommodate high-resolution timestamps. Packet logs created by DataSource 7.1 components can only be read by logcat 7.1 or greater. For more information on high-resolution timestamps in Caplin Platform 7.1, see Timestamp resolution.

Logcat locations

There’s a copy of the logcat utility in the bin directory of each installed Liberator, Transformer and Integration Adapter.

In the Caplin Deployment Framework, you’ll find logcat in the following places:

  • <Framework-root>/servers/Liberator/bin

  • <Framework-root>/servers/Transformer/bin

  • <Framework-root>/kits/<Adapter-blade-name>/<Adapter-blade-name>/DataSource/bin

where <Framework-root> is the Deployment Framework’s topmost directory.

You can generally use any instance of logcat in your Framework to view any of the packet logs produced by any component running under the Framework. However, we recommend that you normally use the latest logcat whenever possible; that is, the logcat provided in the latest kit version out of all the component kits you’ve deployed in the Framework.

Logcat syntax

You use logcat in a similar way to the standard Linux cat command.

Syntax
./logcat [options] file…
Options
-h, --help

Displays detailed information about the available logcat options.

-F, --print-field-names

Displays the names of the fields in each log record (the default is to just print the field numbers). The field names are obtained from the component’s fields file (see -f option).

If the fields file, or a file referenced by the fields file, uses the environment variable CONFIG_BASE, then you must enable the Deployment Framework’s command-line completion extension. See Setting up dfw command completion.
-f <fields-file>, --fields-file <fields-file>

The location (directory path) and name of the fields file.

The default setting, if you don’t provide a -f option, is fields.conf in the current directory.

If the fields file, or a file referenced by the fields file, uses the environment variable CONFIG_BASE, then you must enable the Deployment Framework’s command-line completion extension. See Setting up dfw command completion.
--print-info

Displays the version, type and source of the log. The timezone reported is that of the local machine that the logs were written on.

Example
Logcat: Log Type 'packet' Version 5 created by 'rttpd' in timezone 'Europe/London'
-l, --print-flag-names

Displays the flags that have been sent with each data object; for example F_NONACTIVE|F_CREATEPARENT|F_CREATEOBJECT

For an explanation of what each flag means, see the Data Object Flags module of the DataSource C (DSDK) API Documentation.

-t <type>, --log-type <type>

Tells logcat to interpret the supplied file as a particular type.

This option takes an argument. Currently the only argument available is packet which makes logcat interpret the file as a packet log. Normally logcat determines the type of the binary log file by looking at a header record within the file. However, you’ll need to use the -t option if you’ve split the file into smaller pieces, because the header will be missing from the smaller file. See Splitting packet logs into smaller files, below.

Example
logcat -t packet mypacket.log
-U <update-file>, --update-file <update-file>

Specifies the name (and optionally the path) of a file to which logcat will write update rate statistics, based on an analysis of the supplied packet log file.

The file will contain an entry for every 1 second period during which any message arrived. Each of these lines will contain:

  • A string showing the time

  • The UNIX time-stamp

  • The number of each type of message received (update, request, discard, nodata, status)

  • The number of update bytes for each one second period in the packet log.

The peak update rate and the peak update byte rate will appear at the end of the file.

You can specify both -U and -P in the same command. They are complementary: -U compiles update statistics across all peers and -P compiles update statistics for each peer.

Example 1. Compiling statistics from a Liberator packet log

To compile update statistics for a Liberator packet log, packet-rttpd.log, run the command below:

logcat -U update-file.txt packet-rttpd.log

The update information for packet-rttpd.log is written into update-file.txt.

-P, --peer-update-files

Available from: DSDK 7.1.29, Liberator 7.1.29

Compile update-rate statistics for each peer and write them to output files, one file per peer. The file names of output files are automatically generated and take the format updates-peer_name.

Each output file contains a one-line entry for every 1 second period during which any message arrived. Each entry contains:

  • A string showing the time

  • The UNIX time-stamp

  • The number of each type of message received (update, request, discard, nodata, status)

  • The number of update bytes for each one second period in the packet log.

The peak update rate and the peak update byte rate appear at the end of the file.

You can specify both -U and -P in the same command. They are complementary: -U compiles update statistics across all peers and -P compiles update statistics for each peer.

Example 2. Generating peer update statistics for a Liberator packet log

Consider a packet log, packet-rttpd.log for a Liberator connected to two peers: transformer and permissioning.

To compile update statistics from the log file, run the command below:

logcat -P packet-rttpd.log

The command above generates two output files:

  • updates-transformer

  • updates-permissioning

To compile update-rate statistics across all peers too, run the command below:

logcat -U update-file.txt -P packet-rttpd.log

The command above generates three output files:

  • update-file.txt

  • updates-transformer

  • updates-permissioning

--log-version <version_num>

Specifies the version of the log file, which is then checked against the log versions that this instance of logcat can handle.

This option only has an effect when you pipe the log file to logcat from another tool, such as tail

Example
tail -f <packet-log-file> | logcat --log-version 7

Since the log file header, which identifies the log version, isn’t usually available to logcat via tail, logcat takes the supplied <version_num> to be the log version instead and formats the file contents accordingly.

If <version_num> is greater than the maximum log file version that this instance of logcat can handle, logcat terminates with a version warning message. So in the above example, if logcat cannot handle log versions greater than 5, it displays the following message and then exits:

Logcat: The Log 'Pipe' is of a type and version (DSBLog, p, 7) that I can't handle
-x, --print-id

Displays the datasrc-id of the peer referred to in a packet.

By default, logcat displays the peer’s name as referred to by it’s datasrc-local-label, if that is available.

-z <timezone>, --timezone <timezone>

Sets all times in the log to the specified timezone. To find the required timezone look in the system folder zoneinfo, typically found at /usr/share/lib/zoneinfo or /usr/share/zoneinfo

In this folder there are sub folders with timezone information in them; for example, if there’s a subfolder called US/Mountain', set the timezone option as -z US/Mountain

Usage examples

These examples are for a Liberator packet log. The current working directory is the Liberator’s var directory, where the packet logs are usually located. The default name of Liberator’s packet log is packet-rttpd.log — see the datasrc-pkt-log configuration item in DataSource peers configuration (part 2).

Example 3. Printing information about a binary packet log

Enter the command below:

../bin/logcat --print-info packet-rttpd.log

The output from the command is shown below:

Logcat: Log Type 'packet' Version 5 created by 'rttpd' in timezone 'Europe/London'
Example 4. Converting a binary packet log file to text

Enter the command below:

../bin/logcat packet-rttpd.log

The output from the command is shown below (only the first few lines are shown):

2015/09/22-14:10:57.112 +0100: 0.0.0.0 < PEERINFO 102 transformer1 0  ACTIVE|CONTRIB Handshake data:
datasrc-version=6.2.5-301125, hostname=johne-960, platform=i686-pc-win32, technology=DSDK, transformer-version=6.2.5-301153,
pipeline-module-loaded=true, pipeline-module-version=6.2.5-301153, format-module-loaded=true,
format-module-version=6.2.5-301153, autopub-module-loaded=true, autopub-module-version=6.2.5-301153,
cluster-module-loaded=true
2015/09/22-14:10:57.112 +0100: 127.0.0.1 > PEERINFO 101 liberator1 0  BROADCAST Handshake data:
datasrc-version=6.2.5-301125, hostname=johne-960, platform=i686-pc-win32, technology=DSDK, liberator-version=6.2.6-301198,
liberator-auth-module=openauth
2015/09/22-14:10:57.113 +0100: 127.0.0.1 > HEARTBEAT transformer1 15 0
2015/09/22-14:10:57.114 +0100: 127.0.0.1 < HEARTBEAT transformer1 15 0
2015/09/22-14:11:00.297 +0100: 0.0.0.0 < PEERINFO DemoDataSource1 DemoDataSource-johne-960 0 DemoDataSource1 ACTIVE|CONTRIB
Handshake data: datasrc-version=6.2.5-301125, hostname=johne-960, platform=i686-pc-win32, technology=DSDK
2015/09/22-14:11:00.297 +0100: 127.0.0.1 > PEERINFO 101 liberator1 0  BROADCAST Handshake data:
datasrc-version=6.2.5-301125, hostname=johne-960, platform=i686-pc-win32, technology=DSDK, liberator-version=6.2.6-301198,
liberator-auth-module=openauth
2015/09/22-14:11:00.297 +0100: 127.0.0.1 > HEARTBEAT DemoDataSource1 15 0
2015/09/22-14:11:00.298 +0100: 127.0.0.1 < HEARTBEAT DemoDataSource1 15 0
2015/09/22-14:11:00.392 +0100: 127.0.0.1 < DATAUPDATE2 DemoDataSource1 1 8240 /EXAMPLES/BROADCAST/ABC 222 3
-11001=Value 0.243902 -11002=Value 0.149254 -11003=Value 0.294118
2015/09/22-14:11:00.396 +0100: 127.0.0.1 < DATAUPDATE2 DemoDataSource1 2 8240 /EXAMPLES/BROADCAST/DEF 222 3
-11001=Value 1.#INF00 -11002=Value 0.144928 -11003=Value 0.416667
2015/09/22-14:11:00.397 +0100: 127.0.0.1 < DATAUPDATE2 DemoDataSource1 3 8240 /EXAMPLES/BROADCAST/GHI 222 3
-11001=Value 0.128205 -11002=Value 0.172414 -11003=Value 0.161290
...

The log entry timestamps in the example above have a resolution of one millisecond. DataSource 7.1 applications timestamp packet log entries to a resolution of one microsecond, and logcat 7.1 outputs log entry timestamps with a longer fractional second component (6 digits). For example: 2015/09/22-14:10:57.112578 +0100

Displaying the last part of the log file in real time

You can use the Linux tail command with logcat to display the last part of the log file and update the screen when more data appears.

Example
tail -f packet-rttpd.log | ../bin/logcat -t packet

Splitting packet logs into smaller files

Packet logs can become very large. To make it easier to view a large packet log, you can split it into smaller files first, using the Linux split command.

Example
split -b 10m packet.log

This example splits the log into separate files of 10 megabytes each.

The split command can produce a lot of files if you’re not careful with the size parameter.

When you use logcat to view one of the split files (other than the first one), you must tell it that the file is a packet log, because the file’s header will now be missing:

logcat -t packet packet-xab

See also: