PEP

Pegasus Enhancement Proposal (PEP)

PEP #: 241

Title: OpenPegasus Stress Testing

Version: 1.5

Created: 28 October 2005

Authors: John Alex, Willis White, Ed Boden

Status: Approved

Version History:

Version	Date	Author	Change Description
1.0	28 October 2005	John Alex, Willis White, Ed Boden	Initial Submission
1.1	27 November 2005	John Alex, Willis White	Updated PEP based on previous review.
1.2	28 November 2005	John Alex	Attached the stress testing overview diagram
1.3	5 December 2005	John Alex	Updated PEP based on review (12/02/05). (This version of PEP was "Approved" on 12/09/2005)
1.4	11 April 2006	John Alex	Updated PEP with changes to New Files section and changed the tolerance check interval from 5 minutes to 1 minute.
1.5	19 April 2006	John Alex	Updated changes to PEP based on Bug# 4962 (Opened for reviewing code for this PEP) - Replaced "stressTestController" with "TestStressTestController". - Incorporated new directory structure. (Final version re-approved on 04/25/06)

Abstract: A functional PEP that illustrates the OpenPegasus Stress Testing.

Definition of the Problem

As indicated in the concept PEP #224 (OpenPegasus Test Architecture), there is clearly a need to develop a comprehensive stress test suite for OpenPegasus. OpenPegasus Stress tests are designed to stress the CIM Server until it fails or runs to successful completion. These tests are expected to stress the CIM server to achieve various possible execution limits, mainly to crash or hang the server. To understand how stress testing differs, or relates to, other kinds of testing (e.g. performance, continuous hours of operation, functional) please see PEP 224.

Proposed Solution

The proposed tests are expected to exert excessive loads on the CIM server. The main intent of these tests are to see how fast, or whether, we can get the CIM server to crash or hang. These tests are designed to allow some flexibility in adjusting the desired stress level on the CIM server.

Execution model assumptions

The proposed client-side functionality concerning stress test will be tested only on Windows and Linux. Note, that the CIM server that is being tested can be running on any of the OpenPegasus supported operating system.
The CIM server that is being stress tested is expected but not required to be running on a different system than the proposed stress client tests. This is mainly to isolate the CIM server that is under test on a different system, in order to avoid the added strain put on the same system by these clients.
All of the proposed stress test clients invoked by a given execution of the TestStressTestController are expected to be stress testing the same CIM server.
Diagnosis of CIM Server hangs or crashes caused by the proposed stress tests is expected to be done using already existing debugging and tracing techniques or tools.

OpenPegasus Stress Test Overview:

OpenPegasus Stress Test will be comprised of three major components, as follows:

TestStressTestController - Command that invokes one or more of the “Stress Test Client” concurrently.
Stress Test Clients - Client Programs that may contain one or more of the CIM Operation(s).
Stress Test Configuration - A flat file containing test configuration information.

The "TestStressTestController" will read the configuration file or the "Stress Test Configuration" and then, invoke the "Stress Test Client" instances specified in the "Stress Test Configuration". Each of the client instances will be a separate process and independent of each other. These Clients will in turn connect to the same CIM Server and perform their respective operations concurrently. The following diagram illustrates, a quick overview of the OpenPegasus Stress tests.

OpenPegasus Stress Testing overview diagram

TestStressTestController

The “TestStressTestController” command will be written in C++ and compiled into an executable.

The user may specify the name of the configuration file that is to be used for the tests.

When a configuration file is not specified at command line, then the "TestStressTestController" command will use the default configuration file if it exists, otherwise take the default values hard coded in the TestStressTestcontroller.
This command will also verify the validity of the configuration values specified in the configuration file.

This command will invoke one or more of the Stress Test Clients specified below.

This command will end for any of the following reasons:

When "Duration" as specified in the Stress Test configuration file has elapsed.

If the specified configuration file cannot be found in the configuration directory.

If the configuration file has any invalid parameter or value.

A failure in one (or more) of the Stress Test Clients. The configuration can be setup to allow some margin of client failures that might be attributed to connection timeouts, etc, if the goal is to look for only server problems that show up as crashes, etc. By default, any client failure will be reported and result in termination of the stress test (this corresponds to a ToleranceLevel of 0%).

When the command ends, it will as part of its cleanup also stop or end all the stress clients that it had originally started.

The command can be run using the following options:

TestStressTestController [-s] [-h hostname] [-p portnumber] [-u username ] [-w password] [<configfile_name>]

                --help
                    --verbose

Options:
--help.                   Writes this command's usage statement to std out.
-h hostname            Connect to CIM Server on specified hostname.
-p portnumber          Connect to CIM Server on specified portnumber. Default(5988)
-s           Use SSL protocol between Stress Client and the CIM Server.
--verbose                Writes the command´s verbose messages to standard output.
-u username           Connect to CIM Server using the specified username.
-w password            Connect to CIM Server using the specified password.

Operands:
<configfile_name>
                            Specifies the name of the configuration file that is to be used for the tests.

All the options specified here will have precedence over similar parameters specified in the configuration file.

Stress Test Clients

The following table lists the Stress Test Clients and the associated CIM Operations that each of these clients will perform on the CIM Server. Each of the test clients listed in the following table will be programmed in C++ and compiled into a separate executable. These clients once started will run repeatedly until stopped by the TestStressTestController command. The expected stress factor imposed on the CIM server is directly proportional to the number of these client instances that are running concurrently against the same CIM Server.

STRESS TEST CLIENTS	DESCRIPTION
TestModelWalkStressClient	For all the detectable namespaces this client will perform an enumeration of all Classes, Instances, References, Associators & Qualifiers. This will also do a getClass, getInstance & getQualifier on the found enumerations of Classes, Instances and Qualifiers respectively.
TestWrapperStressClient	This is a wrapper client that will allow existing clients or programs to be executed. The intended client program that is expected to be executed must be specified in the configuration file when using this client. For example, this client can be used to run pegasus CLI's to stress test the CIM server.

Note:

The above list of stress test Clients can grow as needed.

Possible Future "Stress Test Clients":

TestInvalidOperationStressClient
TestIndicationStressClient
TestProviderStressClient
TestExecQueryStressClient
TestInvokeMethodStressClient
TestAssociationStressClient
TestExportClientStressClient
TestDenialOfServiceStressClient

Every "Stress Test Client" will,

receive as arguments the appropriate parameters as specified in the configuration file from the "TestStressTestController".

connect to the CIM Server running on the specified hostname. However, if the hostname is not specified then the client will attempt to connect with the CIM server on the same system the client is invoked on.
on completion or failure of its operation as specified in the above table, continue to repeat the operations until stopped by the "TestStressTestController".
indicate in a file when it does or does not receive a valid response (exception/CIM Error/connection failure) for every one minute interval. This will help the "TestStressTestController" determine the status of each of the the "Stress Test Client".

The “Stress Test” Configuration File

The default configuration filename for the stress test will be named as "default_stresstest.conf".
The user may specify the name of the configuration file that is to be used for the tests.
All the configuration files must exist in the configuration directory as indicated under the section "OpenPegasus Stress Test file location" below.
The Configuration file can contain the following common values that would in turn provide the factors required for the stress tests:

Duration of Tests. (Duration in minutes all the “Stress Test Client” are to be run).
HostName. (Host name or IP address of the system where the CIM server is expected to be running.)
Port. (Specifies Port number. The default port is 5988)
Number of Clients. (Number of concurrent clients) per Stress Test Client.
Name Space. ( Name space that will be utilized by the Stress Test Clients)
Class Name. ( Name of the existing class that will be utilized by the Stress Test Clients.)
SSL. (Use SSL protocol between Stress Client and the CIM Server)
UserName. (User name that the Stress Test Clients will use to connect to the server.)
Password. ( Password that is associated to the user name.
ToleranceLevel. (This represents the percentage of the started "Stress Test Clients" that can be allowed to fail for a one minute interval without resulting in the termination of the stress tests. Note that any reported client failure in two consecutive one minute interval corresponds to a "failure status" for that client. The default is 0%, which means no failures are allowed, or any failures will result in termination of the stress test.)
ClientDuration. (Specifies duration in minutes for the corresponding stress client before it should be stopped.)
ClientWait. (Specifies waiting period in minutes before the corresponding stress client is re-started again.)
ClientName. (The name of the Client program that will be used by a TestWrapperStressClient instance)
Options. (The options that are desired for the corresponding Client program that will be used by a TestWrapperStressClient)

Instances of the Clients may also have unique configurable values. Unique values can be specified for each instance in the configuration file as shown in the Sample Configuration File below. If any of the above parameters are repeated for a client instance, then that particular instance(s) will utilize the new value.
Parameters "ClientName" and "Options" are only used for "TestWrapperStressClient". ClientName is a required parameter for TestWrapperStressClient.
The parameter "Hostname" will have to be the same for all the Clients.

Note:

Options or values specific for each client or instance must be placed after the corresponding client or instance. Otherwise, the order of the configuration variables in the configuration file is not important.
"ClientDuration" when used without "ClientWait" will not be re-run.
"ClientWait" will only apply when used with "ClientDuration".
When using a Client program like a "CLI" or any other "clients" for the "TestWrapperStressClient", ensure that options associated to the Client program will connect to the same CIM Server.
Running without a failure implies that the client is communicating with the CIMServer and has recieved responses other than an exception or a CIMError.

The following table lists the valid names as input for the corresponding Stress Test Clients in the Configuration file:

Stress Client Tests	Valid Names
TestModelWalkStressClient	ModelWalkStressClient
TestWrapperStressClient	WrapperStressClient

Default Values:

Default Hostname will be the local system where the stress tests are run.
Default port will be 5988.
Log files will be named as "stressTest_<DDMMYYhhmmss>.log "

      where,
   "DD" denotes the day.
           "MM" denotes the month.
           "YY" denotes the year.
           "hh" denotes the hour.
           "mm" denotes the minutes.
   "ss" denotes the seconds.

Default "Duration" will be set as 180 minutes.
Default number of instances for each of the known clients, when the configuration file does not exist is "5".
Default NameSpace when needed will be "root/cimv2".
Default UserName and Password will be blank.
Default will not use SSL.
Default percentage for "ToleranceLevel" will be set as 0%.
Default ClientWait & ClientDuration will be blank.
Default Clients for the stress tests will be set as follows:

ModelWalkStressClient=5
WrapperStressClient=5
    [ClientName=CLI, Options=niall]

Note:
     The above default sets up the stress tests for the following test clients to be run concurrently:
          Five ModelWalkStressClient instances using default values.
          Five WrapperStressClient instances running the command "CLI niall".

A Sample Configuration File:

#Machine Host of Server to be tested - The machine the
#CIM server is running on. If left blank then default is the local host.

Hostname=narten

# Total Duration of the Stress Client Test set as 300 minutes.

Duration=300

        # Tolerance level will be set as 0%
        ToleranceLevel=0

        # Number of Client instances
        # The format for running a client program:

# <client program name>=<number of instances of client program to run concurrently>
# [option1=<data1>, option2=<data2>, option3=<data3>, ..... ]
# specify all the options for a Client instance or instances within a square bracket "[ .... ]".
# TestWrapperStressClient can only have the parameters "ClientName" and "Options".
# [ClientName=<ClientProgram>, Options=<actual options for Client program> ]

ModelWalkStressClient=1
[NameSpace=/root/cimv2,ClassName=CIM_door,ClientDuration=60,ClientWait=10]
ModelWalkStressClient=5

WrapperStressClient=2
[ClientName=CLI,Options="-l narten niall"]
WrapperStressClient=10
[ClientName=CLI,Options="-l narten ei pg_computersystem"]
WrapperStressClient=5
[ClientName=osinfo,Options="-h narten",ClientDuration=120]

# This sets up the stress tests for the following stress test clients to be run concurrently:
#       - One ModelWalkStressClient which will use the Name space "/root/cimv2" and ClassName "CIM_door"
#          overriding any default value. This client will be run for 60 minutes then stopped for 10 minutes before it is
#          re-started again. This process will be repeated for the duration of the stress tests.
#       - Five ModelWalkStressClient's using common values.
#       - Two WrapperStressClients will run the commands "CLI -l narten niall" concurrently.
#       - Ten WrapperStressCleints will concurrently run the command "CLI -l narten ei pg_computersystem".
#       - Five WrapperStressCleints will run the command "osinfo -h narten" concurrently for 120 minutes and then stop.

# Note that the 17 wrapper clients instances are also connecting to the same hostname as the other clients.

Running a Stress Test using default values:

Start the CIM Server on a specific hostname.
Run the following command : (Ensure that the hostname specified below matches where the above CIM Server resides.)

TestStressTestController -h <hostname>

The "TestStressTestController" will search for the default file in the configuration directory.
Assuming that we are using default values but connecting to the specified host, the "stressController" will then invoke the Five ModelWalkStressClient instances using default values & the Five WrapperStressClient instances running the command "CLI niall -l <hostname>" . (See section Default Values above for reference)
The "TestStressTestController" will log status and other information in its log file.
Each instance of the invoked stressClients(ModelWalkStressClient & WrapperStressClient) will connect to the specified hostname via the default port and run its operations.
Note that the stress clients will repeat these operations until it unexpectedly ends or is stopped by the "TestStressTestController".
TestStressTestController will monitor the overall status of these stress clients for every one minute interval.

If the overall stress client status is less than or equal to the tolerance level then all is well and all the client instances are allowed to chug along.
if the overall stress client status is greater than the tolerance level, then the TestStressTestController will terminate all the client instances and subsequently end the stress tests while reporting & loging this failure.

If all is well then the tests will continue till the end of the duration of the stress tests, which for default will be 180 minutes.

OpenPegasus Stress Test file Location

The OpenPegasus Stress Test command (TestStressTestController) and the source code for all the Stress Test Clients will be placed in the OpenPegasus CVS source tree at $PEGASUS_ROOT/test/StressTestController/ and $PEGASUS_ROOT/pegasus/test/StressTestClients/ respectively.

The following table shows the location of some of the other files associated with these tests:

	Configuration directory	Log directory
LOCATION	$PEGASUS_ROOT/test/StressTestController/	$PEGASUS_HOME/test/StressTestController/log/

OpenPegasus Stress Test Output

When starting the tests, the "TestStressTestController" command will:

state that the stress test was initiated on this machine, on this day and time.

display the name of the configuration file used for the tests.
display all the configuration values it used for the tests.

During the course of the tests the command will,

display simple status information at every one minute interval .

When the stress tests ends it will,

state that the tests were ended on this day and at this time.

state the actual duration of the tests.

state if it failed or successfully completed all the tests.

If the percentage of the stress test clients that fails exceeds the "ToleranceLevel" percentage, then the command will report just that.
If the verbose flag is opted, then for every 100 iteration of the operation for each client instance the tests will output a percentage of failure. For example it could say "WrapperStressClient [PID#] ran 400 times with a 10% failure".

Rationale

This is a follow-on PEP for the concept PEP#224(OpenPegasus Test Architecture).

New files

Following files/directories will be added to the OpenPegaus cvs tree as a result of this PEP:

$PEGASUS_ROOT/test/StressTestController/Makefile
$PEGASUS_ROOT/test/StressTestController/stressTestController.cpp
$PEGASUS_ROOT/test/StressTestController/stressTestControllerMain.cpp
$PEGASUS_ROOT/test/StressTestController/stressTestController.h
$PEGASUS_ROOT/test/StressTestController/stressTestControllerException.cpp
$PEGASUS_ROOT/test/StressTestController/stressTestControllerException.h
$PEGASUS_ROOT/test/StressTestController/default_stresstest.conf
$PEGASUS_ROOT/test/StressTestController/tests/Makefile
$PEGASUS_ROOT/test/StressTestController/tests/invalid_noclients.conf
$PEGASUS_ROOT/test/StressTestController/tests/invalid_configuration.conf
$PEGASUS_ROOT/test/StressTestController/tests/results.master
$PEGASUS_ROOT/test/StressTestClients/Makefile
$PEGASUS_ROOT/test/StressTestCients/TestStressTestClient.h
$PEGASUS_ROOT/test/StressTestClients/TestStressTestClient.cpp
$PEGASUS_ROOT/test/StressTestClients/ModelWalkStressClient/Makefile
$PEGASUS_ROOT/test/StressTestClients/ModelWalkStressClient/TestModelWalkStressClient.cpp
$PEGASUS_ROOT/test/StressTestClients/WrapperStressClient/Makefile
$PEGASUS_ROOT/test/StressTestClients/WrapperStressClient/TestWrapperStressClient.cpp

Following files will be modified as a result of this PEP:
$PEGASUS_ROOT/Makefile
$PEGASUS_ROOT/TestMakefile
$PEGASUS_ROOT/test/Makefile

Testing

Unit test cases and Manual functional tests will be run.

Other Considerations

The "Stress Test Clients" can be running on multiple physical windows or Linux operating systems. However, to invoke these clients the "TestStressTestController" command must be executed on each of those systems manually.
The scope of the "TeststressTestController", "Stress Test Clients" and the "Stress Test configuration file" will reside within the Operating system image that it is running on.
A new target called "stresstests" will be introduced in the Makefile, that will build and run the stress tests.
A usage tests will be incorporated into "make tests" for the TestStressTestController in the $PEGASUS_ROOT/test/StressTestController/tests/Makefile.
Compilation of the "Stress Test Clients" and the associated files will be included in the nightly build. However, testing of the "Stress Test Clients" and the associated files will not be included in the nightly tests.
To avoid further chaos in the community on the matter of "unsupported scripting language" we are choosing C++ instead as the language of choice for the "TestStressTestController".

Schedule

Target release is Pegasus 2.6.

Action	Planned Date	Actual Date	Comments
PEP submitted	10/31/05	10/31/05
PEP reviewed	11/04/05	12/09/05
PEP approved	12/16/05	12/09/05 & reapproved for minor changes on 04/25/06
Code submitted	01/30/06	04/11/06
Code Approved		04/25/06

Discussion

One may utilize these tests to validate the stablity of the CIM Server. For instance, when someone makes a major change to Pegasus, these tests may be used as a benchmark to revalidate its stablility.
One may schedule these tests to be run on a regular basis to re-confirm the stablity of the CIM Server periodically.
Platform depended configuration. (Probably a future add on as required.)

Copyright (c) 2004 EMC Corporation; Hewlett-Packard Development Company, L.P.; IBM Corp.; The Open Group; VERITAS Software Corporation

Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions:

THE ABOVE COPYRIGHT NOTICE AND THIS PERMISSION NOTICE SHALL BE INCLUDED IN ALL COPIES OR SUBSTANTIAL PORTIONS OF THE SOFTWARE. THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.

Template last modified: March 9th 2004 by Martin Kirk
Template version 1.8