Log File Management
From ARSWiki
The extensive logging capability of the AR System® platform provides support personnel and application developers with invaluable information when faced with identifying and isolating performance problems and defects in workflow. Unfortunately the amount of disk space on the production application server is limited, so logs cannot be captured indefinitely. They must be moved off to an alternate file system, processed, and then archived.
Contents |
Naming Conventions
The naming convention for AR System® Log files is based on two properties:
- The type of data captured in the log
- The date the log file was captured.
The naming convention for AR System® log files dictates that the date the log was captured by appended to the prefixes outlined in the above table. The following pattern should be used when specifying the date a log file was captured:
MMDDYY
When it becomes necessary to record/capture multiple log files in a single day (due to the fact that AR System® log files are presently limited to a size of 2 GB), a single, lower-case, alphabetic character [a-z] should be appended after the date stamp.
All log files should have a suffix of “.log”.
For example, the first AR System® SQL log file captured on May 10, 2004 should have the following name: arsql051004a.log.
Combining Multiple Log Files into a Single File
At times, it can be beneficial to combine several AR System® log files into a single physical file in order to get a better visual representation of the activity that occurs on the AR System® server. In these situations, the name of the log file should incorporate the “prefixes” of each of the types of log files being captured in the file. The prefixes should appear in an order based on the way they appear in the table above.
NOTE: The “ar” portion of the name should only appear as the first two characters in the log file name.
For example, the first log file captured on May 10, 2004 which contains escalation, filter, API, and SQL activity should have the following name: aresclfilterapisql051004a.log.
Archiving Log Files
Due to space and performance considerations, AR System® log files are only captured on the production AR System® server. All necessary parsing and compressing of log files should be performed on a separate physical server in order to avoid causing any adverse impact to the production application server.
On a UNIX/LINUX platform, the AR System log files should be stored in the following path: /var/log/arsystem.
Under the aforementioned path, a directory for each day of the month is created. The directory names use the following pattern: YYYYMMDD. For example, the log files for May 10, 2004 are stored in the following directory on the development server environment:
/var/log/arsystem/20040510
Under this directory tree, the following table lists the standard sub-directories and provides a description of the type of information stored in each directory.
server - All log files captured on the application server with the exception of user and DSO logs are stored in this sub-directory.
user - All user log files captured for a given day are stored in this directory.
so - All distributed server log files captured for a given day are stored in this directory.
integration - The log files created by any products that are integrated into the AR System server environment.
Using the May 10, 2004 day as an example, using the conventions described above, you could have the following directory structure.
/var/log/arsystem/20040510
|-- server
|-- user
|-- dso
|-- integration
Parsing Log Files that Span Multiple Days
Given that AR System® Log files are limited to a maximum size of 2 GB and the CNSMS solution is very complex, it becomes necessary to “rotate” log files periodically throughout the day. As such, a standard practice for log file management is to combine the separate log files captured throughout the day into a single log file before the file is archived.
This parsing and archiving is accomplished through the use of the grep, cat, and gzip UNIX utilities.
NOTE: Microsoft has released a free download for the Windows platform entitled “Microsoft Services for UNIX” which provides Windows executable equivalents of may UNIX commands and utilities. This download can be obtained from the following URL: http://www.microsoft.com/windows/sfu/default.asp.
Before parsing the log files for a given day, all the files should be downloaded to a machine other than the production AR System® server. Please keep in mind that during this process, you will need to have enough free space equal to twice the size of the log files to be processed. This is because you’re going to be combining each individual log file into a single log file and then removing the original, individual log files.
Once all log files for a given day have been copied to a working directory on a machine other than the production server, the grep command should be used to search for log file entries that are relevant to a given day.
For example, for May 10, 2004 we captured API, SQL, and User log files. We have the following log files downloaded to our temporary directory:
arapisql050904d.log arapisql051004a.log arapisql051004b.log arapisql051004c.log arapisql051004d.log aruser050904.log aruser051004.log
Since the capturing of log files is performed manually and most people don’t stay up until 12:00 AM midnight to change the log file name, log files typically cross over one day to the next. This is why when processing the logs for May 10th, we typically have log files from the previous day – May 9th.
To parse through the combined API and SQL log files, the following commands should be used:
grep “Mon May 10 2004” arapisql050904d.log > arapisql051004.log grep “Mon May 10 2004” arapisql051004a.log >> arapisql051004.log grep “Mon May 10 2004” arapisql051004b.log >> arapisql051004.log grep “Mon May 10 2004” arapisql051004c.log >> arapisql051004.log grep “Mon May 10 2004” arapisql051004d.log >> arapisql051004.log grep “Tue May 11 2004” arapisql051004d.log > arapisql051104.log
Please note that the first line processes the arapisql050904d.log file looking for activity that occurred on May 10th. Using the suggested log file rotation schedule, this log file would have started on the 9th of May at 11:00 PM and been stopped on the 10th of May at 7:00 AM. The first command listed above will extract a list of API and SQL activity that occurred from 5/10/2004 12:00 AM until 5/10/2004 7:00 AM.
It is also important to note the difference between the “greater than” symbols used to redirect the output from the grep command to the target file. Using a single “greater than” symbol (“>”) tells the operating system to redirect output to the named file. If the file exists, it is to be overwritten. Using two “greater than” symbols (“>>”) instructs the operating system to append the output of the program to the target file if it exists. If the target file doesn’t exist then it will be created.
In the command examples provided above, the first command line creates the arapisql051004.log and the second through fifth commands append data to that file. Since we want the combined log file for the 10th of May to contain information from 5/10/2004 12:00 AM until 5/10/2004 11:59:59 PM, we’re actually parsing through the files on the 11th of May. As such, the beginning of the last log file, arapisql051004d.log, contains activity from 11:00 PM until 11:59:59 PM on 5/10.
The end of the arapisql051004d.log log file contains activity from 12:00 AM on 5/11 until 7:00 AM on 5/11. As such the sixth (and last) command line from the above example captures the information for this time and redirects it to the new file: arapisql051104.log.
The AR System® User log files is a much simpler process since an entire day’s worth of activity is generally less than 100,000 bytes. Using the 10th of May as our example, as we did with API and SQL filters, the following steps will walk you through the process of archiving user log files.
For the purposes of this example, it is assumed that User logging was started at 11:00 PM on May 9th, 2004. Information was captured to the aruser050904a.log file. On Monday, May 10th at 7:00 AM, the User log file was rotated and User logging continued with the aruser051004a.log file. On Tuesday, May 11th at 7:00 AM, the log files were rotated and the aruser051104a.log file was used to capture AR System® User logs.
This leaves us with the following files which have been copied from the production server to our “working” environment.
aruser050904a.log aruser051004a.log
The following commands will be issued to parse through the aforementioned log files to consolidate the AR System® User log entries for Monday, May 10th in to a single file.
grep “Mon May 10 2004” aruser050904a.log > aruser051004.log grep “Mon May 10 2004” aruser051004a.log >> aruser051004.log
Since the aruser051004a.log log file was turned on and capturing AR System® User log information from 7:00 AM on May 10th until 7:00 AM on May 11th, we need to extract the User log information for 12:00 AM until 7:00 AM on May 11th from the aruser051104a.log log file. This is performed by using the following command.
grep “Tue May 11 2004” aruser051004a.log >> aruser051104.log
The aruser051104.log log file should be retained and used on Wednesday, May 12th. User log information from the aruser051104a.log log file which started capturing data at 7:00 AM on May 11th should be appended to this file (aruser051104.log). Compressing Log Files Since the combined log files which record AR System® activity for a given day can be several gigabytes in size, the files are compressed before they are placed in the archive folder on the development server. The UNIX gzip utility is used to compress the log files using the following command syntax:
gzip arapisql051004.log
In the above command example, the log file arapisql051004.log will be compressed and renamed to arapisql051004.log.gz. When the compression operation is completed, the original file is removed. Using this technique, it is not uncommon for a daily, combined AR System® API and SQL log which is natively between four and five gigabytes in size to compress down to around two hundred and fifty megabytes – 1/20th the size of the uncompressed file!
Once compressed, the aforementioned log file is moved to the following directory:
/var/log/arsystem/20040510/server
