Overview - Brief Description of Feature

System calls are calls between application and system processes and the operating system - the operating system API.  This feature traces calls between processes and the operating system in in-memory circular buffers.  One buffer is kept per-process.  The tracing logic saves pertinent input and output parameters including a limited amount of data for each system call.  All trace entries are timestamped with millisecond granularity.  

The InSight trace facility is very efficient and the additional code path added for each system call is barely measurable even on the slowest processors still in use today.

The operating system api consists of file and device "handle" functions such as open/close/read/write plus others such as a timer function and an asyncronous wait function.  In all, there are approximately 50 different system calls in 4690.

Problem - What Problem Does the Feature Solve?

This feature solves two main problems: visibility into what was going on just prior to a problem, including unexpected terminal restarts, and visibility of application and system processing paths so performance can be analyzed and improved.

Solution - Short Description of How the Feature Works

There is a single, well-defined, entry point (or interface) for system calls to the operating system.  This interface is called the 'supervisor interface' or supif for short.  InSight installs a monitor at this interface in order to trace the calls.

The circular buffers are saved to the controller on snapshot or full terminal dumps and they are found and formatted by the included InSight dump formatter.  Additionally, for development and test lab analysis, this feature can stream the system call trace data to disk files on the controller.

Some characteristics of system call tracing:

• In-memory trace buffers are 16KB per process by default.  Since the trace information is kept in raw binary format, this is typically enough to have the history need to understand problems.
• By default, 32 bytes of data buffers are traced.  For example, on a 512 byte file write, only the first 32-bytes are traced.
• When logging to the controller, the default is to log only application system calls.  System process calls are not logged although they can be.
• As a simple example, the following problem was once discovered using this tool: a java application that read a lengthy configuration file during ipl was reading that file one byte at a time.  Each byte read of the file resulted in 4 messages between the controller and the terminal as well as a physical disk read on the controller. A simple change to the java application changed this to 512 byte reads (which is the 4690 maximum).

The following is an example of a few formatted system call log lines: