Pegasus Enhancement Proposal (PEP)
PEP #: 302
PEP Type:Functional
Title: Enhanced logfile support
Version: 1.0
Created:
Authors: Muni S Reddy
Status: Approved
Version History:
Version |
Date |
Author |
Change Description |
0.1 |
|
Dave Sudlik |
Placeholder
PEP |
0.2 |
9 May 2007 |
Muni S Reddy |
Added
Config properties, their behavior and
How Pruning and deletion of obsolete logfiles
is done. |
0.3 |
15 May 2007 |
Muni S Reddy |
Added
definition for Pruned Logfile and Obsolete
Logfile. Replaced Config variable with Config
Property. Changed "How the obsolete logfiles
deleted" section. |
0.4 |
21 May 2007 |
Muni S Reddy |
Added Config properties in the standard format.
Acquired a lock before pruning logfile.
Modified Rationale 4. |
0.5 |
31 May 2007 |
Muni S Reddy |
Removed
Configuration variables. Changed the file name to avoid ':'
symbol. |
0.6 |
|
Muni S Reddy |
Answered
Roger's comments. |
0.7 |
|
Muni S Reddy |
Modified the
algorithm to address the comments. Moved all the comments to Discussion
section. |
0.8 |
18
June 2007 |
Muni S
Reddy |
Modified the
algorithm to make logging thread safe. |
1.0 |
8 October 2007 | Muni S Reddy | Modified the algorithm as per the comments in Code review. |
Pegasus supports POSIX style syslogs
and flat file logs.
Syslogs are supported on Linux, HP-UX and z/OS
platforms.
If PEGASUS_USE_SYSLOGS is defined, log records are written to
the syslog file.
Otherwise, log records are written to the Pegasus logfiles.
Pegasus Log files are flat files and
opened in append mode. These files keep growing and there is no limit
defined on the size of these logfiles.
Today the OpenPegasus logfile can grow without bounds. There is no mechanism to limit its size and manage the "pruning" of the logfile, including retention of pruned entries. Logfiles are not thread safe as reported in Bug # 2509.
The proposal is to add support to:
This solution is for the platforms ( i.e Windows , i5) where Syslogs are not available. The proposed solution adds additional condition checks before actual logging takes place. This solution prunes all logfiles (PegasusAudit.log, PegasusDebug.log, PegasusError.log,PegasusStandard.log and PegasusTrace.log) once the size of the logfile exceeds the PEGASUS_MAX_LOGFILE_SIZE.
1. Pruned Logfile :
Logfile created once the logfile size exceeded the defined size.
This PEP introduces a constant PEGASUS_MAX_LOGFILE_SIZE , which defines the size at which the pruning of the logfile will occur. Whose value is 32 Megabytes.
The following pseudocode outlines the pruning
algorithm:
The new code for the above low level design is added in Logger::_putInternal() of Logger.cpp file.
Files Modified:
1. Logger.cpp
Modules modified:
1. pegcommon
Rationale
1. Pruned logfiles
are not deleted, as system administrator would do the cleanup activity.
Action |
Planned |
Actual |
Comment |
PEP Submitted |
|
|
|
PEP Reviewed |
|
|
|
PEP Approved |
|
|
|
Code Committed |
|
|
|
(mreddy
) Deleting the files is
concerned, we
can retain them. But
please let me know the advantages of having lot of old logfiles.
The logfile size being use in one of our
platforms is
4 MB. That is why we opted for 4MB. We can increase this to reasonably
good
number. Please suggest if you are looking for a specific logfile
size. We have already discussed about the keeping a config
property to define the size of the logfile.
But the
architecture team is not willing to introduce one more config
property.
(r_kumpf) This algorithm does not guarantee that a log file
will not
grow beyond PEGASUS_MAX_LOGFILE_SIZE. There is a race condition where
two
threads could do the precheck at the same
time.
Neither would cause the file to grow too large independently, but
writing both
messages would exceed the size limit. If it is not important to
guarantee the
size limit, the algorithm could be simplified. Instead of doing a precheck based on the size of the message it is
about to
write, it could check whether the file is _already_ past the limit.
Muni : To Avoid the race condition, after
acquiring the lock, Read the
current logfile size again and check the
condition as
in step 2. I am adding this as Step 4.
With this
change race condition is avoided as step 5, 6, and 7 are executed only
when the
latest size satisfies the condition. We would like to retain prechecking ,as
either precheck or post check would
not acquire
guarantee the accurate size of the logfile.
Size
of
the message to be written is discarded from the condition check ,
to
decide whether we need to prune the logfile
or not.
r_kumpf
) I suggest using
the word
'old' rather than
'pruned', as the meaning seems more obvious. What is the rationale for
this
choice of file name, as opposed to something like
'PegasusStandard.log.1' or 'PegasusStandard-old.log'?
Muni : As this PEP is about pruning log
file, We opted to prefix 'pruned' to the
Name of the logfile. It can also be
changed to PegasusStandard.log.1,
but not PegasusStandard-old.log as the
original name
getting modified. It would better to prefix or suffix to the old name.
[(carson_hovey)
OpenVMS can only have one '.' in a filename.
PegasusStandard.log.1 is not legal. To create a new version of a file
on
OpenVMS, just close the file and open a new one of the same name.
(Muni):
The pruned log file will be renamed as logfile-timestamp.log
as expalined in Step 5 of Algorithm.
(r_kumpf) What about the algorithm is specific to this file?
Is it
really more difficult to make the behavior consistent across the log
files?
Muni: Yes. The Algorithm is
specific to
Standard logfile as of now.
No, It is
not
difficult to extend this to other logfiles. One additional
condition has to be added in the step1. In the implementation
appropriate file name will be picked up in steps 5, 6 and
7.
How to extend this to other logfiles will
be
explained with comment along with the code check-in.
[
(r_kumpf) Is there a possibility that
another thread
could be writing to this file when we close it here?
(mreddy) Yes, There is a
possibility
of one thread writing to the logfile, when we are trying to close the
logfile.
This can be addressed by adding a static variable logWritersCount
(AtomicInt),
which counts the number of threads writing to the logfile currently.
logWritersCount is incremented before step 10 of the algorithm and
decremented
after step 10 of the algorithm. In step4, before closing the file, wait
until
logWritersCount becomes Zero.
(dave_sudlik) I talked
with Muni and
Venkat about this. There are other windows as well -- for example,
trying to
get the size of the current log file (in step 1) in one thread while
another
thread has just renamed the logfile. I'm not sure if there's a clean
way to do
this without more extensive serialization. What about using an rwlock?
Get the
ReadLock for normal logging activity, and if it is determined that
pruning is
required, get the WriteLock. I'm thinking that most logging is done on
error or
non-mainline paths where the additional serialization would not be
noticed --
does that sound right? Something like (brackets are really braces):
<
Obtain readlock
if (current logfile size <
PEGASUS_MAX_LOGFILE_SIZE)
<
write the log record
return
>
>
<
Obtain writelock
if (current logfile size >=
PEGASUS_MAX_LOGFILE_SIZE)
<
do the necessary pruning
>
write the log record
>
Copyright (c) 2006 Hewlett-Packard Development Company, L.P.; IBM Corp.;
EMC Corporation; Symantec Corporation; The Open Group.
Permission is hereby granted, free of charge, to any person obtaining a
copy
of this software and associated documentation files (the "Software"), to
deal in the Software without restriction, including without limitation
the
rights to use, copy, modify, merge, publish, distribute, sublicense,
and/or
sell copies of the Software, and to permit persons to whom the Software
is
furnished to do so, subject to the following conditions:
THE ABOVE COPYRIGHT NOTICE AND THIS PERMISSION NOTICE SHALL BE INCLUDED
IN
ALL COPIES OR SUBSTANTIAL PORTIONS OF THE SOFTWARE. THE SOFTWARE IS
PROVIDED
"AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING
BUT NOT
LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR
PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT
HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN
AN
ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN
CONNECTION
WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
Template last modified: March 26th 2006 by Martin Kirk
Template version: 1.11