PEP #: 12
TITLE: Finish Association Provider Interface and Testing
Version : 1.7
Authors: Karl Schopmeyer
State: Accepted, Version 1.7
Approvals Required: Architecture Team
Type: Design
Created: 30 January 2003
Version History:
| Version | Date | Author | Change Description |
|---|---|---|---|
| 1.0 | 30 January 2003 | K. Schopmeyer | Initial Submission |
| 1.1 | 1 February 2003 | K. Schopmeyer | Revise to clarify tasks |
| 1.2 | 1 February 2003 | K. Schopmeyer | Correct Branch Name of the CVS work Branch |
| 1.3 | 13 March 2003 | K. Schopmeyer |
|
| 1.4 | 13 Mach 2003 | K. Schopmeyer | Add issue concerning the provider response (see issues section) |
| 1.5 | 25 March 2003 | K.Schopmeyer | Incorporate Comments from Review |
| 1.6 | 4 April 2003 | K. Schopmeyer | Make corrections from discussions |
| 1.7 | 10 April 2003 | K. schopmeyer | Add section "Definition of the solution" based on HP request |
Abstract: Pegasus has several components of the Association provider interface not tested and therefore not necessarily running.
This PEP provides a list of these tasks, the proposed fixes and the time to complete.
The priority of this work is high to a group of users who need the association provider interface for SNIA interoperability testing within a week. This is one of the MUST tasks for Pegasus 2.2 (See PEP 15)
The CIM Operations for Associators, Associator Names, References, and Reference Names do not work completely and correctly in the Pegasus implementation for a number of reasons. At one time the code did work. The code for the repository does work and returns both class and instance responses from the repository.
Provider Interface - The Association provider interface has been written but not completely tested. The provider interface header exists, the base of a sample provider exists in Sample provider (structure but no response generating code) but at some time it was removed from the build so it has not been built as the interfaces were modified and tested. Associations are used in Pegasus but we do not have a working example of an association provider. There is at least one specific known hole (CIMOM Handle) and there may be some others because it was not integrated into the test suite or test and display tools.
Operation Dispatcher - This code today calls only one provider and provides no aggregation of responses. The idea that an associators type call would call only one provider was simply wrong.
Provider Manager - There is some old code that was hacked in to make this function work for one platform and the general structure of the code to make all 4 operations work is in place. However, the code makes the provider manager assume some of the CIM operations functions and further, there are a number of areas that are simply broken because many functional changes have been made to Pegasus since this code was build and without associatiors tests, the code was not retested.
The associations request distribution algorithm - We also believe that the basic algorithm for distributing requests for associations is not as complete as it has to be and that the complete algorithm must be defined and the code (both distribution to providers and what is acquired from the repository on class requests) must be defined more carefull and the functions probably expanded.
End-End Tests - There is very little Pegasus test support for associations. The associations operations are very complex with multiple filtering capabilities and the algorithms for what is to be acquired are alos complex and an extensive test is required to determine if the associations are working and the algorithms operating properly.
Unit Tests - The Unit tests (e.x. the common\tests directory short-circuited the associations components of the unit tests in things like building classes, reference properties, instances, etc. and these tests need to be extended to assure that the basic objects work as expected with association type objects.
The solution is to correct the behavior of a number of modules (see list of tasks) so that Pegasus matches the behavior defined in this section for the associations operations (referenceNames, references, AssociatorNames, and Associators Operations).
The CIMOM (CIMOperationsDispatcher) must separate class level operations from instance level operations and route the operations as follows:
Class Level Associatior Type Operations - Route to the appropriate function in CIM Repository so that the appropriate classes or class names are returned. The CIM Repository performs a search for objects that match the parameters of the request and returns the appropriate object. Class level associator operations do not need to call the providers. The repository contains all required information to respond with the correct list of objects or object names. The algorithms below define the characteristics of the search for objects for the response. The CIM repository has available all of the parameters of the associator type call and therefore generates a single response that aggregates all of the objects required of the response.
Instance Level Associator Type Operations - These operations must be routed to the appropriate providers defined for the association operations and the responses from these providers aggregated into a single operation response. NOTE: the CIM Instance repository is a possible provider. Because of the nature of the associator type requests, we can expect multiple providers to be involved in generating response. The CIMOM must be capable of selecting the appropriate providers, issuing requests to these providers and aggregating the responses. Each provider generates a set of response objects defined by the parameters of the call and the CIMOperationsDispatcher aggregates the multiple responses into a single coordinated response.
Note that despite the fact that there are 4 separate operations (associators, associatornames, references, referencenames), they are all basically the same algorithm and processing. In all cases, the initial objective is to find association objects that point back to the target CIMObjectPath (the one input in the request). Thus if the input object is CIM_Disk I want all associations that have a reference back to CIM Disk (see below for some of the complexities of deciding which associations point back to CIM_Disk). Beyond that, associators request that we go further and return the object at the the other end of the association whereas references simply request the association object itself. The difference between the names function and the non-names function is whether we return the object or just the name.
Generally Associator operations are processed the same as other operations with the following exceptions:
1. The repository processing is significantly more complex. We are not simply returning or modifying the target object, we are searching for pointers to that object.
2. They are inherently multi-provider, much as the enumerate instance operations. Each operation may go to more than one provider or to the repository and providers.
3. They take either a class or instance on input and the processing is significantly different depending on whether the input is class or instance. They are in effect two different operations (a class operation and an instance operation) wrapped into a single operation. Note that other operations could have taken this same approach (ex. getObject whereas we have getClass and getInstance as seperate CIM Operations).
The algorithm is defined as follows:
1. Receive and Decode CIMOperation input - Same as other operations. The resulting operation request (ex. referenceNames is routed to the CIMOperationDispatcher
2. CIMOperation Dispatcher request dispatching
if (objectpath == class)
Send operation to repository for processing for each class.
Repository function searches for assoc classes (see assoctable.cpp) and returns complete response
else // response is an instance
Get list of classes on which we could have responses (possible Association List)
(This is the list of classes generated by a class level referenceNames call)
lookup associationproviders for the classes that can have associations to this cimobjectpath
for each class in possible association list
{
if (provider exists)
send to request to provider (this is an async operation sent to the correct provider manager)
else
send request to the instance repository.
}
3. CIM Repository Processing
CIMRepository associator function (ex. referencenames) searches repository for valid class responses for the input target class using the parameters provided int he call. The algorithm defined below defines the search mechanism. NOTE: In the current pegasus implementation, the CIM repository ncludes a module called associatioTBL that creates/updates a table of associations when the classes are modified. This table contains all of the parameters possible in the associator requests. An associator class request is implemented as a search through this table for entries that match the input parameters.
4. Provider Processing
The provider is accessed via the Pegasus Provider API defined below. Each provider that has a possible association with the target CIMObject receives a request. The provider is required to generate a response of the objects/object names that match the calling parameters.
NOTE: The provider is required also to process the boolean parameters for includeQualifiers and classOrigin if they exist.
5. CIMOperation Response processing
Gather provider responses until all responses received.
If only a single response exists
{
if the response is an object
if path component empty
add patch compnent
forward it to the encoder functions
}
else
{
If any responses are good
{
filter out all error responses and discard them.
aggregate remaining response objects into a single response.
if the response type is an object
for each response object
if the path component is empty
add the pathcomjponent
forward the aggregated response to encoder functions.
{
else
send one response (normally first in list) to encoder and delete remainder of responses
}
Association requests are significantly different than other CIM operation requests. Whereas other requests respond with information directly about the object defined or at most about the object and objects further down in the hierarchy (ex. classes and subclasses or instances of the target class and of its subclasses) the association requests are demanding to know associations that could exist (the class request) or do exist(the instnace based request) that point back to the target CIMObjectPath input. There is no direct pointer out to the associations, you have to determine which associations point back to the input target. Thus, for a class request, which association classes could contain references to the target class input with the request. This is more complex than simply those associations that directly reference the target class in a reference since it must account for the hierarchy nature of the CIM model. In effect, it is an organized search through possible associations for associations with reference properties that point back to this class. However, this is further complicated by the fact that:
1. Associations defined at higher levels in the class hierarchy could define associations for the target Object Name. Thus, an association defined for the superclass of the input CIMObjectPath could be used as an association for the current class. NOTE: The current code in the associationTable module does not account for this characteristics. This is being extended now.
2. Associations themselves have a hierarchy and subclasses of a defined association that points back to the target class could also associate back to this class. The class association code in the CIM repository implementation does account for this characteristic today.
Thus, the algorithm for a) returning classes to class requests and b) returning instances to instance requests must take the superclasses of the target class and subclasses of possible association requests into account.
It is not clear today that the different implementations do this the same way or that we all agree with the same set of rules or allgorithm.
We are working today with several other people to define a set of use cases and example that demonstrates the correct behavior and defining this behavior completely for Pegasus and the DMTF.
NOTE: At the minute, the use cases, algorithm, etc. are defined in a separate document since they contain diagrams.
There is a specified API for association requests passed to providers. This is defined in src\Provider\CIMAssociationProvider.h It provides a interfaces for each of the 4 associator type calls (references, etc.). This PEP does not propose to change that API. the parameter definitions for these calls remain the same. However, we propose to change the conception of the call so that that providers may be constructed around the associator classes rather than the association referenced classes. This does not change the API.
The provider associated with specified operations on a CIM Class is defined through provider registration. Provider registrations specifically provides for the registration of association providers that will provide responses to association operation requests.
NOTE: To complete Pegasus 2.2 we will not change the registration functions. However, we propose that since these operations are so expensive, we can make a significant gain in the future by limiting the number of providers called and that we can further limit the number of providers called by using some of the supplementary information in the operation requests (ex. result class, etc.). Thus, in the future we propose to extend the provider registration to allow this information to be optionally included with registration.
The tasks required to provide complete association functionality are as follows:
Except for the CIMOM Handle and some possible changes to the lookupAssociationProvider interface which is being done by Mike Day as part of the merge of IBM and HP local branches, the work is in process today and everything except the CIMOM handle should be finished by 20 March 2003. It can be integrated from the local CVS ranch to the main trunk after that. The completion of the sample providers and test tools should allow us to surface any problems rapidly.
Until approval has been received, the work on this extension will be put into a CVS branch LOCAL_ASSOCPROV-BRANCH (taken from the TAG LOCAL_ASSOCPROV-BRANCH tag). and can be moved into the main trunk when testing is complete.
The status of this work today (verison 1.6 of this PEP) is as follows:
1. ProviderManager clean up. This has been accomplished and the provider manager, provider registration manager, etc. all respond properly to requests issued from the CIMOperationDispatcher. The changes involved a) changes to the a)lookupassociationproivder functions and b) the provider request and response processing.
2. AssociationProvider completed for testing and and additional provider (FamilyProvider) written that provides a simple set of instnaces, associations and relationships that can be used for testing.
3. CIMOperationsRequestDispatcher functions rewritten and tested. The changes to the dispatcher involved changing the dispatcher to issue multiple requests and aggregate responses and to correct problems in the whole lookupassociationprovider functionality.
4. Algorithm for deciding what to return (which providers to call and exactly which classes to request from the reposiory) largely defined and review proceeding of this definition to assure that we are correctly processing associator type functions. Today the algorithm works and we are trying to optimize as assure that we cover all cases that the specificaiton requires. NOTE: We are also considering that we should propose changes to the cim/xml specification to clarify the issues around associators type processing.
5. MOF written and installed for a complete association test provider. The test provider has been initially written and is being extended now.
6. Single executable test utility is being written now as probably the simplest means of executing the extensive options agains the test provider, the repository, and the test mof. This tool will be in a test directory under the association test provider.
7. CLI utility extended so that the associations functions can be executed through this tool including all of the options. This is an alternative to the automated test tool for all types of testing.
The test version of this code has been committed to the CVS association test branch and is being used in testing today as part of the SNIA interoperability tests.
Association operations do work today. The open issues are the algorithm, cleaning up the code, lack of testing of some of the options and filters, and efficiency of operation.
Karl Schopmeyer has committed to do this work except for the CIMOM handle.
1. Algorithm - Assure ourselves that the algorithm for responding to requests is correct and complete. The algorithm or selecting classes/instances for response to these requests is complex, more complex than simply direct relationships to the defined input CIMObjectPath. We must verify that the algorithm is correct in that we are responding with the correct set of classes or instances (in particular the classes since that responsibility is between the CIMOperationDispatcher and the CIMRepository). See the discussion of response algorithm below.
CONCLUSION: There are problems with the algorithm used today to get association classes for references, referencenames, etc. It does not take into account the superclasses of the target class which may have associations that apply to the target class. This algorithm will be extended in the CIMRepository code to take into account superclasses of the target class.
2. Class vs. Instance Choice - Clarification of issues of determining whether we are receiving a class or an instance on input and definition of a standard test mechanism for this decision. The code today uses the existence of keybindings to determine if the received request is a class or an instance.
The association requests are the only requests where the input object can be either a class or an instance (type CIMObjectPath). The CIMOM must make a routing and processing decision based on whether it is a class input or an instance. This is typically a statement much as the following:
Boolean isClassRequest = (request->objectName.getKeyBindings ().size () == 0)? true: false;
This test has a problem in that there is the possibility of a degenerate class type, the singleton class where there need not be any keys. The degenerate form would not have any keys and would be allowed only one instance. This is apparently defined in the CIM specification. In that case, this test would interpret the instance as a class since there are no keys.
CONCLUSION: For Pegasus 2.2 we do not propose to try to fix this problem because 1) there appear to be no singleton classes defined in any of the CIM schema so the problem should not occur, 2) it is not clear that it is fixable since for a singleton it is not clear that there is any way to know whether the input is class or instance even if you know that the class is singleton.
3. The Role Property - There may be one more issue or change proposal that has not been looked at yet, the type for the role and result role parameter. In the past all of the additional parameters of the associations were defined as strings internally to Pegasus. With the advant of the CIMName type, the names of those parameters that represented class information were modified to be of the type CIMName as shown below:
Array<CIMObjectPath> referenceNames(
const CIMNamespaceName& nameSpace,
const CIMObjectPath& objectName,
const CIMName& resultClass = CIMName(),
const String& role = String::EMPTY
);
This has an advantage for these parameters in that it clearly allows the NULL option for the parameter internally wheras there was always confusion between NULL and empty string ("") before. It also forces an additional test for legality of the name itself.
However, the role parameter (which is expected to be a valid property name) is still processed through the interface as a String type. The question arises whether we should consider changing this type to a more formalized specific type like CIMName.
At this point we have not seriously considered the possibility of a change but we should review this before the association work is committed. The negative aspect of this is that there are already implementations and this could seriously affect those implementations.
CONCLUSION: Concluded in discussion that this will NOT be changed.
4. Remove the Pegasus Option for Assocation traversal - There is a CIM config option today that allows association operations to be completely disabled. This is supported as runtime option through cimconfig. The issue is whether there is a use case for keeping this option after associations operate correctly or removing ti completely.
CONCLUSION: This will be removed as an option once the association functions are completed, installed in the trunk and tested.
5. Provider response with CIMObjects (The Path Component) - The association specification requires that the response to an associator or reference request return the type ObjectWithPath, not simply the object itself. The references operation below shows this:
<objectWithPath>*References (
[IN] <objectName> ObjectName,
[IN,OPTIONAL,NULL] <className> ResultClass = NULL,
[IN,OPTIONAL,NULL] string Role = NULL,
[IN,OPTIONAL] boolean IncludeQualifiers = false,
[IN,OPTIONAL] boolean IncludeClassOrigin = false,
[IN,OPTIONAL,NULL] string PropertyList [] = NULL
)
However, the Pegasus code (and all other implementations) define the return as cimply the CIMObject and that "withPath" is a part of the cimobject itself, an optional part. This leads to a problem in the providers where they do not put the path component on and create an illegal response. Typically in Pegasus, this causes a CIMException in the client xml processing since the path component is required at that point.
The issues is whether we should do something to protect against this behavior where the object is returned without a path component. The alternatives appear to be:
a. Do nothing
b. Figure out some way that the object is considered illegal at the level of the provider without the path component.
c. Add a test to the CIMOM response mechanism to test for the path component and add it if necessary. this is a very low code cost (about 8 instruction total per response type but add processing cost to each response.
Despite the fact that it looks like a simplistic error, the problems in interoperability testing have been extensive because a) it does not get found until it gets to the client, b) the implementations are inconsistent on the client side and some implementations have been modified to allow this error XML to be processed by the client because the error was so extensive in the servers. This was one of the big errors in the SNIA interoperability testing.
CONCLUSION: The architecture team concluded that the overriding philosophy should be that anything that causes failures in the response should be tested and corrected in the CIMOM. Things that cause only erronous responses should probably be left to the provider. This came up in a disucsion of the issue defined here and also the issue of the use of includeQualifier in the enumerateinstances functions. While the specification is not clear, the general usage by other implementaitons is that includeQualifier returns all class level qualifiers. Implementations use this to determine keys from the information as opposed to getting the key information from the classes. The Pegasus CIMOM does NOT test for the existence of qualifers and add them if the provider does not add them. Actually most other implementations probably do the same but provide a function "createInstancefromclass" that builds the instances with qualifiers dynamically so that it is not really a major issue with these implementations. In the case of Pegasus, it requires code be developed for each instance provider so that today most providers do not correctly respond to the includeQualifiers options on instance requests. We concluded that in this case, it is NOT a CIMOM responsibility because the result does not "break" the response, it simply represents an incorrect response. Hosever, since the missing path components effectively break the response (The Pegasus client generates an exception for this response), we must test and, if necessary complete this response. The responses will be tested in the dispatcher response functions for missing path component and if it is missing it will be added.
March 18 2003 - Group Review. The items discussed were:
1. Error Handling in Aggregated Responses - The Pseudo Code representing the aggregation process is to simplistic and does not reflect the needs of aggregating good and error responses back into an aggregated response. It was noted that this is more simple than the actual response and the pseudo code will be modified to reflect the code. However, there is a potential problem in that any errors in a partially good/parttly bad response are simply discarded so that there is no record of strange things that come from providers in this case. We aggregate the good components and discard the bad ones. Further, if all the responses are in error, we simply pick the first error and do not try to analyze error types or severity whidch could be misleading to the client.
CONCLUSION: Pseudo Code to be further defined. This was corrected in Verison 1.5
2. The Target Provider for Association Operations - It was commented that while we today passs the client API to the provider (which means that the target object is really the end point of an association, not the association class itself), it would seem logical to be able to define providers in terms of the association classes themselves. Thus given a Class A and Class B connected by an association X, all of the provider calls are for association information based on either A or B, not for associations from X. This means that we must define association providers for A and B rather than being able to define a single association provider based on X itself. It would seem to be simpler to make the provider request to X, the association provider than to make it to the end point providers. The advantages of this seem to be:
a) You could define a single provider for an association, not one for each end point.
b) It would be conceptually easier to define the association provider as a separate entity, independent of the other providers so that associations could be modified, added, etc. independent of the instance providers.
This is a significant conceptual change from the existing definition although it may not be any API change. Thus, today we make a referenceNames request to the provider as follows
/** Enumerate the association objects that refer to a particular target CIM Object (Instance).
@param context contains security and locale information relevant for the lifetime
of this operation.
@param objectName - The target CIM Object whose referring object names are to be returned.
* Note that only instances will be forwarded to the provider. All class level
* requests are handled by the CIM Server internally.
*
@param resultClass - NULL, MUST be a valid CIM Class name. It acts as a filter
* on the returned set of Object Names by mandating that each returned Object Name
* MUST identify an Instance of this Class (or one of its subclasses), or this Class
* (or one of its subclasses)
*
@param role The Role input parameter, if not NULL, MUST be a valid Property name.
* It acts as a filter on the returned set of Objects by mandating that each returned
* Object MUST be associated to the source Object via an Association in which the
* source Object plays the specified role (i.e. the name of the Property
* in the Association Class that refers to the source Object MUST match the value
* of this parameter.
@param handler asynchronously processes the results of this operation.
@exception NotSupported
@exception InvalidParameter
*/
Note that the other association type calls are simply expansions of the referencenames calls that add additional filters to the call including and ask for further processing of the response list (i.e. get the reference objects themselves or get the objects/names at the other end of the association):
associationClass - Added by the Associator and AssociatorNames calls and acts as a filter on the returned set of names by mandating that each returned name identifies an Object that MUST be associated to the source Object via an Instance of this Class or one of its subclasses. Or put it simply, defines the association class used as the basis for this response changing the request from one of associations through all possible association classes to one through a single class.
resultRole - A valid Property name. It acts as a filter on the returned set of names by mandating that each returned name identifies an Object that MUST be associated to the source Object via an Association in which the named returned Object plays the specified role (i.e. the name of the Property in the Association Class that refers to the returned Object MUST match the value of this parameter).
Today we find all possible associations for the defined end point (actually the same as the class level request) but we ask for the end point classes for these associations and this becomes the set of classes that defines the set of providers. We call every provider that has an end point association provider defined.
Thus, if we have Class A and class B with association X, and the client request is for referenceNames for an instance if Class A, we call the association provider for Class A with the request. If the same request was made but with Class B as the target CIM Object, the Pegasus CIM Server would call the Class B association provider. We do not try to call any providers for Class X.
Conceptually, the provider must:
If this is a reference names request, the provider simply return the resulting referenceNames. If it is a reference, return the resulting association objects. If it is an associationName request (after additional filtering for assocClass, etc). return the other objectName of the other references in the association objects themselves. Finally, if it is an associator call, return the objects identified by the other reference in the association class
If, on the other hand, we call providers for the association objects themselves, the provider has a simpler role in that he already (in theory) knows the association instances and simply has to filter out the ones associated with the defined endpoint. If the client requested references for Class A, we call the provider for the association class X. If the client requested references for Class B, we again call the provider for association class X.
There are some issues and questions with this change today including:
Note that it appears that we do not have to change anything in the interface itself. We simply have to
Note: This does mean that the conceptual use of these entities is different for the CIM instance Repository than it would be for the provders.
CONCLUSION: So far this seems very reasonable. It actually makes the code simpler because the list of classes use to generate the list of possible providers is in reality the response from a referenceNames request with the class name of the target CIM Object Name input as the operation target. The documentation will be modified to provide this definition.