Logging configuration

These DataSource configuration items define what information a DataSource application can log and how the log files are automatically cycled.

For an overview of how logging works, including more about log levels and log file cycling, see Logging in DataSource applications.

"this DataSource application" means the DataSource application for which you are defining the configuration. "DataSource peer" or "peer" means a DataSource application that this DataSource application communicates with.

add-log

add-log provides configuration options that, for a particular type of log file, override one or more of the global logging configuration items. You specify the type of log file in the name option, and add other options to customise the file’s log level, log monitor level, and log file cycling characteristics.

Use in: C, Java

Syntax:

add-log
   name          [string]
   level         [string]
   maxsize       [integer]
   monitor-level [string]
   offset        [integer]
   period        [integer]
   suffix        [string]
   syslog-level  [string]
   time          [integer]
   use-parent-handlers [boolean]
end-log
Option Type Default Description

level

string

[value of log-level]

This option only applies to the Event log.

It specifies the severity of the errors and events that this DataSource application reports to the log file for this type of log message. It overrides the log-level item.

For the list of valid values, see log-level.

Also see the syslog-level option.

maxsize

integer

[value of log-maxsize]

The maximum size in bytes for this type of log file. The file size is checked every log-cycle-period (or every period option of this add-log, if defined), and the log file is cycled if it exceeds maxsize. A value of 0 means there’s no maximum size limit, but then the log file is always cycled every period / log-cycle-period regardless of its size.

This option overrides the log-maxize item.

monitor-level

string

[value of log-monitor-level]

This option only applies to the Event log.

It specifies the threshold at which log messages about this DataSource application’s events and errors are published to the monitoring subsystem. It overrides the log-monitor-level item.

For the list of valid values, see log-monitor-level.

This option is not available in Java-based DataSource applications.

name

string

[none]

The name (type) of log to which the rest of this add-log's options apply.

Note that the log cycle filename for the log is constructed from a combination of the run-time filename implied by name, the value of suffix/log-cycle-suffix, and the value of offset/log-cycle-offset.

If you’re writing your own C-based DataSource application, using the C DataSource API, you can define your own log files in addition to the standard ones. In that case, name must correspond to the name of a log object that you’ve created in the code. (In Java-based DataSource applications you can either log to the standard DataSource event log, or create your own logger when you create an instance of the DataSource class.)

Caplin-supplied DataSource applications, such as Liberator have a set of built-in log files with standard types that you can specify in name, as follows:

  • event_log: (All DataSource applications) The event log file. The run-time filename is defined by event-log.

  • http_access_log: (Liberator only) The HTTP access log file. The run-time filename is defined by Liberator’s http-access-log configuration item.

  • javaauth_log: (Liberator only) The Java auth log file.The run-time filename is auth-rttpd.log (this is the log file produced by Java-based Liberator Permissioning Service blades).

  • news_log: (Liberator only) The log file that stores news headlines. The run-time filename is defined by Liberator’s news-log configuration item.

  • object_log: (Liberator only) The object log file that keeps a record of all request and discard commands for objects. The run-time filename is defined by Liberator’s object-log configuration item.

  • packet_log: (All DataSource applications) The log file that records each packet received from the application’s DataSource peers. The run-time filename is defined by the datasrc-pkt-log configuration item.

  • request_log: (Liberator only) The log file that records RTTP requests made to Liberator. The run-time filename is defined by the Liberator’s request-log configuration item.

  • session_log: (Liberator only) The log file that records messages about client connections, disconnections, logins and logouts. The run-time filename is defined by Liberator’s session-log configuration item.

  • xmlauth_log: (Liberator only) The XMLauth log file. The run-time filename is defined by Liberator’s auth-logfile configuration item in the configuration for the XMLauth auth module.

offset

integer

[value of log-cycle-offset]

The number of minutes to take off the time before using it to create a cycled log filename suffix. The suffix is formatted according to log-cycle-suffix or the suffix option of this add-log.

A value of -1 means the offset is the same as log-cycle-period (or, if defined, the period option of this add-log).

This option overrides the log-cycle-offset item. It works in the same way as log-cycle-offset; for examples see log-cycle-offset.

period

integer

[value of log-cycle-period]

The interval in minutes for cycling this type of log file. Each day, at every interval after the first log cycle defined by log-cycle-time (or the time option of this add-log), the log is checked to see if it should be cycled. If the maxsize option of this add-log is defined, or log-maxsize is defined, the log file is cycled if it exceeds the maximum allowed size. If neither of maxsize or log-maxsize is defined, the log file is always cycled. regardless of its size.

This option overrides the log-cycle-period item.

For how to set up weekly log cycling for a particular type of log file, see the time option.

For log files that grow rapidly, a sensible configuration would be to set a suitable maximum size for the file (see the maxsize option), and set the period option to 5, so the file is checked every five minutes.

suffix

string

[value of log-cycle-suffix]

A suffix format that’s passed to the strftime() system function; the resulting string is appended to the name of the cycled log file. For information about the possible format strings, see the LINUX/UNIX manual page for strftime()

This option overrides the log-cycle-suffix item.

syslog-level

string

[value of

log-syslog-level]

Since: DataSource 6.2.4

syslog-level specifies the severity of the errors and events that this DataSource application reports to the operating system’s log for this type of log message. It overrides the log-syslog-level item.

The log-level item, and level option (if specified) take precedence over syslog-level. For example, if syslog-level is set to DEBUG but log-level is set to INFO, only log messages with severity INFO or higher go to the syslog.

For the list of valid values, see log-syslog-level.

This option is not available in Java-based DataSource applications

time

integer

[value of log-cycle-time]

The time in minutes from midnight (local time) at which this DataSource application first checks this type of log file for cycling.

This option overrides the log-cycle-time item.

The subsequent times at which the application checks the log file for cycling is determined by log-cycle-period (or the period option of this add-log). If neither the maxsize option of this add-log nor log-maxsize is defined, the log file is always cycled, regardless of its size.

If time is greater than 1440 (24 hours), the cycle time is taken from the start of the week (midnight on Sunday night). For example, a value of 1460 sets cycle time to 00:20 hrs on Monday. This allows you to cycle the log file weekly on a specific day if the period option is set to 10080 (7 days). For example, set time to 2880 (48 hours) and period to 10080, and the log file is checked for cycling at midnight every Monday.

To set weekly log cycling for all files, see log-cycle-time.

If you don’t specify this option and you don’t specify log-cycle-time, the default value of 240 minutes (4 hours) applies, so the log file is first checked for cycling at 04:00 hrs (04:00 am) each day.

use-parent-handlers

boolean

FALSE

This option applies to Java-based DataSource applications only.

When set to TRUE, the application must send log records for this type of log file to its parent Logger (see setUseParentHandlers in the documentation of the Java Logger class).

This option overrides the log-use-parent-handlers item.

event-log

event-log specifies the file name of this DataSource application’s event Log. This log contains messages about starting up, shutting down, connections to DataSource peers, and other general and debug information.

The log filename can contain the parameters %a and %h At run time, %a is replaced by the DataSource application-name, and %h is replaced by the host name of the machine on which this DataSource application is running.

By convention, log filenames have the format: <log type>-<application name>.log

Example for Liberator:

event-log event-%a.log

In this example, because Liberator’s application name is "rttpd", the configuration item names the Liberator’s event log file as event-rttpd.log.

Use log-dir to specify the directory where log files are to be created.

Use in: C, Java

Syntax: event-log <log-file-name>

Type: string

Default value: event-%a.log (in Transformer it’s just %a.log)

log-cycle-offset

log-cycle-offset specifies the number of minutes to take off the time before creating a cycled log filename suffix that’s formatted according to log-cycle-suffix.

For example, assume log-cycle-offset is 10, and cycled logs are given a filename suffix of the form .hh-mm Then when the logs are cycled at 04:00 hrs, the resulting files all have the suffix .03-50

A value of -1 means the offset is the same as log-cycle-period. For example, assume log-cycle-period is the default of 4 hours, and log-cycle-offset is -1. When the logs are cycled at 04:00 hrs on Tuesday, the value passed to strftime()for formatting is 04:00 on Monday. This makes the timestamp suffix in the cycled log filenames more useful, as the suffix indicates the date and time from which the data in each file starts, rather than the date and time when the logs were cycled.

You can override this setting for a particular type of log file by specifying the offset option of the add-log configuration item.

Use in: C, Java

Syntax: log-cycle-offset <time-offset-in-minutes>

Type: integer

Default value: -1 (= value of log-cycle-period)

log-cycle-period

log-cycle-period specifies the interval in minutes for cycling logs. Each day, at every log-cycle-period after the first log cycle defined by log-cycle-time (or by the time option of an add-log item), each log file is checked to see whether it should be cycled. If the maxsize option of the add-log for the file is defined, or log-maxsize is defined, the log file is cycled if it exceeds the maximum allowed size. If neither of maxsize or log-maxsize is defined, the log file is always cycled, regardless of its size.

You can override the log-cycle-period setting for a particular type of log file by specifying the period option of the file’s add-log configuration item.

For log files that grow rapidly, a sensible configuration would be to set a suitable maximum size for the file (see the maxsize option of add-log), and set the period option of add-log to 5, so the file is checked every five minutes.

For how to set up weekly log cycling, see log-cycle-time.

Use in: C, Java

Syntax: log-cycle-period <interval-in-minutes>

Type: integer

Default value: 1440 minutes (= 24 hours)

log-cycle-suffix

log-cycle-suffix defines a suffix format that’s passed to the strftime() system function; the resulting string is appended to cycled log files. You can override this setting for a particular type of log file by specifying the suffix option of the add-log configuration item.

For information about the possible format strings, see the LINUX/UNIX manual page for strftime()
We recommend that you don’t use a suffix format, such as %T, that includes seconds. This is because if the server is busy when the log file cycles, the create time of the log file may be out by a second, and this can prevent old (unwanted) log files from being overwritten. This can be a particular problem for packet log files, which can grow very large.

Use in: C, Java

Syntax: log-cycle-suffix <strftime-format-string>

Type: string

Default value: %u This appends the log filenames with a day number between 1 and 7, representing Monday (1) to Sunday (7).

log-cycle-time

log-cycle-time specifies the time in minutes from midnight (local time) at which log files are first checked for cycling. You can override this setting for a particular type of log file by specifying the time option of the add-log configuration item.

The subsequent times at which this DataSource application checks log files for cycling is determined by log-cycle-period (or the period option of add-log). If neither the maxsize option of the add-log configuration item for the log file nor log-maxsize is defined, the log file is always cycled, regardless of its size.

You can override the log-cycle-time setting for a particular type of log file by specifying the time option of the file’s add-log configuration item.

If log-cycle-time is greater than 1440 (24 hours), the cycle time is taken from the start of the week (midnight on Sunday night). For example, a value of 1460 sets cycle time to 00:20 hrs on Monday. This allows you to set weekly log cycling for all log files on a specific day if log-cycle-period is set to 10080 (7 days). For example, set log-cycle-time to 2880 (48 hours) and log-cycle-period to 10080, and all log files are checked for cycling at midnight every Monday.

To set weekly log cycling for an individual file, see the time option of add-log.

Use in: C, Java

Syntax: log-cycle-time <minutes-past-midnight>

Type: integer

Default value: 240 minutes (= 4 hours) So the log files are first checked for cycling at 04:00 hrs (04:00 am) every day.

log-dir

log-dir specifies the default directory (as an absolute or relative path) where this DataSource application’s log files are created.

Use in: C, Java

The directory path can contain the parameters %r and %a At run time, %r is replaced by the root directory (application-root) under which this DataSource application runs, and %a is replaced by the DataSource application-name.

Syntax: log-dir <directory-path>

Type: string

Default value: %r/var

log-level

log-level specifies the severity of the errors and events that are reported to the log files when this DataSource application is running.

You can override this setting for the Event log only by specifying the level option of the add-log configuration item.

Also see log-syslog-level and syslog-facility.

Use in: C, Java

Syntax: log-level <log-level-name>

Type: string

Default value: INFO

Values accepted:

log-level-name CORRESPONDING JAVA LOG LEVEL DESCRIPTION

DEBUG

FINE

Reports all errors and events.

INFO

INFO

Reports events and information regarding normal operation, and all errors included in the WARN, NOTIFY, ERROR and CRIT log levels.

WARN

WARNING

Reports minor errors and all errors included in the NOTIFY, ERROR and CRIT log levels.

NOTIFY

WARNING

Reports errors regarding data corruptions and all errors included in the ERROR and CRIT log levels.

ERROR

SEVERE

Reports serious errors regarding network connections and all errors included in the CRIT log level.

CRIT

SEVERE

Reports critical errors that prevent the DataSource application from running.

For Java-based DataSource applications only, the actual log level recorded against a log entry in the log file is the Java level. For example, if you specify a log-level setting of NOTIFY, the log file will contain just entries with Java log levels WARNING and SEVERE.

log-level doesn’t set the threshold at which log messages about the DataSource application’s events and errors are published to the monitoring subsystem; to do this, use log-monitor-level.

log-max-history

log-max-history specifies the maximum number of log lines that are retained for viewing via JMX monitoring tools such as the Caplin Management Console. At any one time, up to log-max-history of the most recently generated log lines can be viewed.

Use in: C

Syntax: log-max-history <max-number-of retained-log-lines>

Type: integer

Default value: 10

log-maxsize

log-maxsize specifies the maximum log file size in bytes. The size of each log file is checked every log-cycle-period (or every period of the add-log for the log file type, if defined), and a log file is cycled if it exceeds log-maxsize. A value of 0 means there’s no maximum size limit, but each log file is always cycled every period / log-cycle-period regardless of its size.

You can override this setting for a particular type of log file by specifying the maxsize option of the add-log configuration item.

Use in: C, Java

Syntax: log-maxsize <max-size-in-bytes>

Type: integer

Default value: 0 bytes (meaning that log files are cycled every time they are checked)

log-syslog-level

This configuration item is only available in DataSource version 6.2.4 or later.

log-syslog-level specifies the severity of the errors and events that are reported to the operating system’s log when this DataSource application is running.

You can override this setting by specifying the syslog-level option of the add-log configuration item.

The log-level item, and level option of add-log (if specified) take precedence over log-syslog-level. For example, if log-syslog-level is set to DEBUG but log-level is set to INFO, only log messages with severity INFO or higher go to the syslog.

Also see syslog-facility and log-level.

Use in: C

Syntax: log-syslog-level <log-level-name>

Type: string

Default value: ERROR

Values accepted:

log-level-name CORRESPONDING JAVA LOG LEVEL DESCRIPTION

DEBUG

FINE

Reports all errors and events.

INFO

INFO

Reports events and information regarding normal operation and all errors included in the WARN, NOTIFY, ERROR and CRIT debug levels.

WARN

WARNING

Reports minor errors and all errors included in the NOTIFY, ERROR and CRIT debug levels.

NOTIFY

WARNING

Reports errors regarding data corruptions and all errors included in the ERROR and CRIT debug levels.

ERROR

SEVERE

Reports serious errors regarding network connections and all errors included in the CRIT debug level.

CRIT

SEVERE

Reports critical errors that prevent the DataSource application from running.
log-syslog-level doesn’t set the threshold at which log messages about the DataSource application’s events and errors are published to the monitoring subsystem; to do this, use log-monitor-level.

log-use-parent-handlers

log-use-parent-handlers specifies, for Java-based DataSource applications only, that the application must send log records to its parent Logger (see setUseParentHandlers in the documentation of the Java Logger class). The standard DataSource logger class that’s used when this item isn’t set is com.caplin.datasource.DataSource.

You can override this setting for a particular type of log file by specifying the use-parent-handlers option of the add-log configuration item.

Use in: Java

Syntax: log-use-parent-handlers <boolean>

Type: boolean

Default value: FALSE

syslog-facility

This configuration item is only available in DataSource version 6.2.4 or later.

syslog-facility specifies the syslog facility to use when this this DataSource application’s error and event messages are recorded in the operating system’s system log (syslog).The message severities to be logged are determined by log-syslog-level and the syslog-level option of the add-log configuration item.

Refer to your operating system manual for instructions on how to set up syslog to receive these messages.

Use in: C

Syntax: syslog-facility <facility-name>

Type: string

Default value: local6

Default log cycling behaviour

If you don’t specify any log cycling configuration, then by defaultall log files are cycled at 04:00 hours each day, so that a separate log file for each type of log is created for each day. You can of course override this default setting for a particular type of log file by specifying the relevant options in an add-log …​ end-log configuration entry.

The default settings are equivalent to the following configuration:

# Cycle logs every time they are checked:
log-maxsize       0

# Cycle logs for the first time at 4 hrs after midnight:
log-cycle-time    240

# Cycle logs every 24 hrs after log-cycle-time:
log-cycle-period  1440

# Cycled log filename has suffix 1 - 7 (representing Mon - Sun):
log-cycle-suffix  .%u

# Cycled log filename has day number of the day when the logged data starts:
log-cycle-offset  -1
This default configuration can create very large log files if your system has lots of fast moving data. It’s useful to have as much log data as possible in an individual file, but you should change the logging configuration appropriately if the files grow too large.

See also: