Pegasus Enhancement Proposal (PEP)

PEP #: 322

Title: Track Generated Indications Data

Version: 1.2

Created: 27 March 2008

Authors: Yi Zhou

Status: Approved

Version History:

Abstract: Tracking the number of CIM indications generated by providers and matched with subscriptions to facilitate problem diagnosis.

Definition of Term

Orphan indication: An indication generated by a provider that does not match any active subscriptions.

Definition of the Problem

Pegasus currently has no mechanism to identify which subscriptions cause a large number of indications to be generated or whether an indication provider generates significant numbers of orphan indications. Such data could be used to identify a subscription that should use a more selective condition or an indication provider that requires service.

Proposed Solution

This PEP proposes tracking generated indications counts by provider and by subscription. The proposed solution involves creating a new schema in order to retrieve the generated indications data, and updating indication service to record the generated indications counts.

Track Generated Indications Counts

Indication service is updated to track these generated indications counts:

• The number of generated indications by an active provider
• The number of generated indications by an active provider for each subscription it serves
• The number of orphan indications generated by an active provider

If an indication provider becomes inactive for any reason (deregistered, disabled, fails, and unload), its counts of generated indications are reset. The counts are also reset on cimserver start-up.

Hash Function

    struct ProviderIndCountTableEntry
    {
        String providerModuleName;
        String providerName;
        Uint32 indCount;
        Uint32 orphanIndCount;
    };

Following hash function will be used here to generate the hash code for this hash table since the one available from HashTable today is not efficient.

    Uint32 hash(const String& key)
    {
        Uint32 hashCode = 0;

        const Uint16* p = (const Uint16*)key.getChar16Data();

        Uint32 keySize = key.size();

        if (keySize > 1)
        {
            hashCode = p[0] + p[keySize/2] + 3*p[keySize - 1]; 
        } 

        return hashCode;
    }

We have used providers currently registered in CVS to evaluate the efficiency and hash code distribution for both the hash function available today and the new hash function. 76 providers are used for this experiment. The results indicate that the performance of hashing for the new hash function is improved by about 25% - 35% and the distribution is slightly worse (standard deviation of the chain lengths is 1.324 with current hash function available today and the standard deviation of the chain lengths is 1.359 for the new hash function).

Update Indication SubscriptionTable

A "matchedIndCountPerSubscription" field is used to store the number of generated indications by an active indication provider for a given subscription. This field is added into struct providerClassList which is one of the contents in active subscription table entry.

Function SubscriptionTable::getMatchingClassNamespaceSubscriptions will be modified to return an array of pair of subscription with associated activeSubscriptionsKey so that the key does not need to be regenerated when a matched indication count is updated.

Following interfaces are added into SubscriptionTable class:

    /**
        Updates entries in the Active Subscriptions table to increase
        matched indication counts for a provider which serves the
        subscriptions.

        @param   activeSubscriptionsKeys   the keys of matched
                                           Subscriptions which are
                                           served by the provider
        @param   providerInstance          A PG_Provider instance
                                           representing the provider that 
                                           serves the subscriptions
     */
    void updateMatchedIndicationCounts(
        const CIMInstance& providerInstance,
        const Array& subscriptionsKeys);

    /**
        Enumerates PG_SubscriptionIndicationData instances using the data
        stored in the Active Subscriptions table.

        @return All the PG_SubscriptionIndicationData instances.
    */
    Array<CIMInstance> enumerateSubscriptionIndicationDataInstances();

    /**
        Enumerates PG_SubscriptionIndicationData instance names using the data
        stored in the Active Subscriptions table.

        @return All the PG_SubscriptionIndicationData instanceName.
    */
    Array<CIMObjectPath> enumerateSubscriptionIndicationDataInstanceNames();
 
    /**
        Gets the PG_SubscriptionIndicationData instance for the specified CIM
        object path.

        @param instanceName CIMObjectpath specifies a CIM instance to be
            returned
        @return The specified PG_SubscriptionIndicationData instance.
                If the specified instance does not exist, throw a
                CIMObjectNotFoundException
    */
    CIMInstance getSubscriptionIndicationDataInstance(
        const CIMObjectPath& instanceName);

Class ProviderIndicationCountTable

Update Generated Indications Count

Function IndicationService::_handleProcessIndicationRequest will be modified to update matched indication count by calling new function updateMatchedIndicationCounts() if a generated indication by a provider matches subscriptions. This function is invoked after all matched subscriptions are processed and indication is sent to handlers. So update hash table will not affect the timing of the indication delivery.

The generated indication count by an active provider and orphan indication count will be updated by invoking new function incrementEntry().

Generated Indication Data Schema Additions

This PEP proposes adding the PG_ProviderIndicationData class and the PG_SubscriptionIndicationData class to the root/PG_Internal namespace. The PG_ProviderIndicationData class provides definitions for PG_ProviderIndicationData which represents generated indications count by an active provider. The PG_SubscriptionIndicationData class provides definitions for PG_SubscriptionIndicationData which represents the number of generated indications by an active provider for each subscription it serves.

Note: The FilterName and HandlerName formats in PG_SubscriptionIndicationData class are designed to match cimsub.

    class PG_ProviderIndicationData
    {
            [Key, Propagated ("PG_ProviderModule.Name"),
             Description ("Module name of an active provider.")]
        string ProviderModuleName;

            [Key, Propagated ("PG_Provider.Name"),
             Description ("Name of an active provider.")]
        string ProviderName;

            [Description ("Number of indications generated by this "
                 "provider since it became active.")]
        uint32 IndicationCount;

            [Description ("The number of indications generated by this"
                 "provider that did not match a subscription.")]
        uint32 OrphanIndicationCount;
    };

    class PG_SubscriptionIndicationData
    {
            [Key,
             Description ("Name of the filter that defines the criteria and "
                 "data of the indications for this subscription. The format is "
                 "namespace:filtername where the namespace is the namespace "
                 "of filter instance created, and the filtername is the value "
                 "of property Name in the filter instance.")]
        string FilterName;

            [Key,
             Description ("Name of the handler that addresses delivery of the "
                 "indications for this subscription. The format is "
                 "namespace:classname.handlername where namespace is the "
                 "namespace of handler instance created, the classname is "
                 "the class of the handler instance, and handlername is the "
                 "value of property Name in the handler instance.")]
        string HandlerName;

            [Key,
             Description ("Source namespace of this subscription.")]
        string SourceNamespace;

            [Key, Propagated ("PG_ProviderModule.Name"),
             Description ("Module name of an active indication provider "
                 "which serves this subscription.")]
        string ProviderModuleName;

            [Key, Propagated ("PG_Provider.Name"),
             Description ("Name of an active indication provider "
                 "which serves this subscription.")]
        string ProviderName;

            [Description ("The number of indications generated by this "
                 "provider that match this subscription.")]
        uint32 MatchedIndicationCount;
    };

Retrieve Generated Indications Data

The statistic data of generated indications can be retrieved by instance operations. To retrieve indication data for providers, enumerate instances of the PG_ProviderIndicationData class in the root/PG_Internal namespace. To retrieve indication data for subscriptions that are served by the active providers, enumerate instances of the PG_SubscriptionIndicationData class in the root/PG_Internal namespace.

Since separate operations are used for retrieving statistic data for active providers and for subscriptions the combined data snapshot can not be guaranteed to be consistent. If a use case needs to use both data sets (e.g. retrieves statistic data for providers first, then retrieves the statistic data for subscriptions that are served by the active providers) and the difference between two data sets is large, in order to find out what causes the large difference (e.g. did many indications be generated in the time interval between two operations?), it is recommended to collect statistic data for providers again, so there is a baseline data to compare with.

Supported Instance Operations

IndicationService is updated to support EnumerateInstances, GetInstance, and EnumerateInstanceNames operations for both class PG_ProviderIndicationData and class PG_SubscriptionIndicationData.

Test Plan

The generated indications count information is tested in the automated test suite. A test client is added to pegasus/src/Pegasus/IndicationService/tests/IndicationsCount directory. This test client creates various subscriptions, triggers the IndicationTestProvider to generate indications by creating an invoke method request, retrieve indication statistic data, and verifies the result.

The IndicationTestProvider is updated responding to the invoke method request from the test client to generate various indications.

Build Option

This documentation for the PEGASUS_ENABLE_INDICATION_COUNT build option is proposed for inclusion in PEP 308:

PEGASUS_ENABLE_INDICATION_COUNT

Description: If true, a version of OpenPegasus is build that CIM Server tracks the number of CIM indications generated by providers and matched with subscriptions. If false or not set, indication statistic support will not be included.

Default Value: true

Recommended Value (Development Build): true

Recommended Value (Release Build): true

Required: No

Considerations:  The number of CIM indications generated by providers and matched with subscriptions is stored in hash tables which causes memory usage to increase by 4 bytes for each active subscription and by about 400 bytes for each active indication provider.

Discussion

• Marek Szermutzky: As of today we already have a mechanism in place in OpenPegasus for statistics. Why not extend on what is already in place, but create a second approach with a new schema ? The Indication Service is already large enough as it stands today. Instead of extending it even more we could make use of the 'Internal Provider' approach. That would exhibit a number of advantages. To name just two: code readability and footprint(can be enabled/disabled easily).

  Karl Schopmeyer: We expect for this to change in a new set of classes from the DMTF because the old statistics proved inadequate.

  Marek Szermutzky: Why does that have to reside in the PG_Internal namespace ? Is there nothing in the CIM Schema for statistical data about indications ? And if there is nothing, was it considered getting those classes added to the CIM Schema ?

  Today, DMTF does not have a schema to address statistical data about indications. The DMTF is considering standardized CIM Schema for indication statistical data, But it will not be available in the Pegasus 2.8 time frame.

  Currently, some indication data is known only by the Indication Service (e.g. which subscription matches an indication and which indication is an orphan indication). It is not clear how this information can be relayed efficiently from indication service to an "Internal provider".

• Marek Szermutzky: In abundance of a section for general comments I will put this here. Seeing as the implementation of this PEP will increase memory footprint and instruction path length there should be a way to enable/disable the gathering of this statistical data. When disabled there should be a very small impact to performance(memory,path length etc.). If not planned to be ever enabled, there should be no impact at all.

  Karl Schopmeyer: I think that given the possibility of embedded systems we should consider how to completely remove this type of extension.

  A new build option PEGASUS_ENABLE_INDICATION_COUNT is introduced to give a way to enable/disable this functionality. Please see section "Build Option" for details.

  The number of CIM indications generated by providers and matched with subscriptions is stored in hash tables which causes memory usage to increase by 4 bytes for each active subscription and by about 400 bytes for each active indication provider.

• Karl Schopmeyer: Is there any value to including indication size in this in addtion to the count. We include the sizes in the existing statistics for operations but remember that the operation requests and response are expected to have significantly more data size variation than the indications. I am asking here, I do not have a clear answer.

  There is no use case today to indicate that the information of indication size can add value to help support engineer to isolate the problem.

• Marek Szermutzky: How is the non-thread safety of HashTable handled ? Is there going to be even more locking in the IndicationService ?

  HashTables are locked briefly to increment indication counts and matched subscriptions counts. The ProviderIndCount hash table is used only for tracking indication counts, so the locking should have minimal effect on other indication processing.

• Marek Szermutzky: I am convinced there is a way for a more 'creative' algorithm for the key. At least the hash code should not just be generated using the functions today available from HashTable. They are inefficent, as they read every character from a String.

  A new hash function will be used here to generate the hash code. Please see Hash Function section for details.

• Karl Schopmeyer: Should these be subclasses of something common. There are a lot of reasons for this not the least of which is just having a lot of classes at the root level.

  Currently, it is not clear where these classes would fit in the existing model. Karl will provide input if a suitable location is determined.

Copyright (c) 2006 Hewlett-Packard Development Company, L.P.; IBM Corp.; EMC Corporation; Symantec Corporation; The Open Group.

Permission is hereby granted, free of charge, to any person obtaining a copy  of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions:

THE ABOVE COPYRIGHT NOTICE AND THIS PERMISSION NOTICE SHALL BE INCLUDED IN ALL COPIES OR SUBSTANTIAL PORTIONS OF THE SOFTWARE. THE SOFTWARE IS PROVIDED  "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.

Template last modified: March 26th 2006 by Martin Kirk
Template version: 1.11