Specification [1] requires that transfer encoding with value="chunked" be supported by any client claiming to be HTTP 1.1 compliant (specification [1] section 3.6.1 at end). Currently, the pegasus client does not do this. However, the pegasus server is currently compliant because HTTP does not require servers to send chunked even if the client requests it to do so.

Large responses can utilize huge amounts of memory. This feature allows the message to be sent over the network in "chunks" as they are sent down by the server application. Each chunk is at a minimum a complete CIM object. After each XML encoded object(s) is sent to the client, the memory is released. This means that a message that would otherwise amount to 100mb (of XML encoding and application data) can be broken up and sent in pieces so that all 100mb are not in memory at once. This realizes a big memory savings since both XML encoded objects and application objects will be released after they are sent to the client as they become available.  See response thresholding below on how this will happen.

Today, a server application cannot control whether the delivered responses go immediately to the client. All the data of the request must be collected first then allowed to be sent to the client. (i.e. ResponseHandler.complete() must be called first before anything reaches the client). There are many cases in which data could be sent as it becomes available to (or processed by) the server application. The parts could in fact be interpreted independently if sent separately (as opposed to sending the entire message back). This proposal would address this problem.

Example:

Client requests EnumerateInstances on all volumes (LUNs) of a Storage Device. The server application asks the embedded device for this data. There exists 10 volumes, but 1 is extremely busy and cannot service the request for many seconds. 9 out of the 10 are communicated to the server application immediately, but the server application needs to wait for the 10th before responding to the client. The 9 are independent (in this case) and could have been sent seconds earlier as they became available. This could also solve potential client timeout problems when in fact some data is ready earlier than the set timeout (but see next note).

 

Important note:

Although the proposed changes to the server allow sending pieces of the application data as they arrive, the pegasus client will NOT realize this faster response because this proposal states that the client will still wait for the entire message to arrive before handing the data to the client application. This section simply states that this change has the potential benefit for the client IF the client side is changed as well.

Today, the Pegasus client is incompatible with non-pegasus CIM/HTTP servers that support (and send) transfer encodings with value="chunked" (even after checking for HTTP 1.1 compatibility)

The following HTTP features will NOT be supported. Their syntax will be parsed, but the values will be dropped and an error given if warranted. Note that the terminology "features not supported" simply means that HTTP level chunked encoding does not apply to the feature in question. That feature will still of course work normally. Many of these below bring up a variety of issues. Refer to discussion section as well. These are features not supported in phase 1

Only header values of "chunked" and "identity" (i.e. "do nothing") values are supported on a client receive. Any others are not supported. The server will not support parsing of any kind of transfer encodings to the body of an incoming request. Specification [1]  section 3.5 lists all values . This also applies to the statement in Specification [1]  section 14.41 (which allows other values from other specifications):

"Additional information about the encoding parameters MAY be provided by other entity-header fields not defined by this specification."

This statement is NOT supported.

Specification [1] section 3.9 describes qvalues (or Quality Values). These parameters apply transfer encoding values in order stated. They reside in the TE header field as extension syntax. Since only a single value of "chunked" or "identity" will be supported, the qvalues for transfer encodings are not needed nor supported.

Chunk extensions are meta data intended for the end application and have no meaning nor behavior rules at the protocol level. They reside on the chunk length line. Specification [1]  section 3.6.1

A trailer is a header that may optionally occur after the last chunk and is considered part of the message body. It is detected if the first byte after the last chunk is not the start of the message terminator. In general, the contents of the trailer will be ignored EXCEPT for anything relating to errors. This exception is due to the fact of Specification [2] section 3.3.11 which deals with CIMError embedded into the trailer. The server will ALWAYS send back the "trailer" field in the header as well as the trailer IF and only if there was any server error AND the client requested trailers or chunking. See behavior section if this occurs

The TE header field will be ignored by the server EXCEPT for values of "chunked" or "trailers".  This exception is due to the fact of Specification [2] section 4.2.19.  The client WILL make use of this field for requests (see behavior)

6 Proxy/Gateway behavior

CIM features not supported

The following describe CIM features that are not supported. Note that the terminology "features not supported" simply means that immediate asynchronous behavior does not apply to the feature in question. That feature will still of course work normally.

1 Indications

Indications will never be sent with any chunked transfer encodings

2 Operations

CIM operations "InvokeMethod" and "ExecQuery" (and the above mentioned Indications) will not be sent asynchronous. Only because to make these asynchronous would require more work which is not planned at this time. The HTTP header will still have transfer-encoding keyword, but the message will only be sent when these operations complete() method is called by the server application.

Internal features not supported

Note that the terminology "features not supported" simply means that asynchronous behavior from the deliver() calls do not apply to the feature in question. That feature will still of course work normally

1 Interface types

Pegasus users can configure different "interface-types" such as java (JMPI) or C (CMPI) or remote CMPI. The default interface type (C++) is currently the only interface to be officially supported (for phase2) in the interest of time. This of course can be changed in the future by those who need these other interfaces to support asynchronous behavior.

A side note:
Since the code to be changed is common between all interface types, it is likely that the rest will probably work anyway.  The reason that these others are "unsupported" for phase 2 is that the necessary testing time to confirm that they support chunking is not planned at this time..

2. Out-of-process-providers

Out of process providers (see PEP-72) will not be supported in the interest of time. Again, this can be changed in the future for those who need the support.

The Pegasus client will be modified to detect the transfer-encoding with value="chunked" in the header and the subsequent chunks in the body. All chunks will be read until the final chunk arrives (except for errors - see below) and any optional trailers. The chunk lengths, chunk extensions, chunk terminators and trailer will be stripped out and the message content length will be set according to the data that arrived (minus chunk meta data). The resulting message now appears the same as if no transfer encodings were applied. If a trailer was received with an error, any data received will be discarded and the client application will receive the CIM error. If there was any parsing error during the receiving of the message, the client application will receive the appropriate error as if the server had sent it directly. No client code execution path needs to be changed after this to preserve the behavior today. The client will support the
receiving of transfer encodings regardless of whether the client requested the response in this form. This has to be done to accommodate non-pegasus servers that send chunked transfer encodings according to the looser rules in Specification [1]  (Specification [2] requires the transfer encodings to be requested in order for them to be sent back in that form).

The client will not send transfer encodings in the body but will ALWAYS send both "chunked" and "trailers" with its request in the "TE" header field. Old servers that do not understand these values should be ignoring them.

The server will look for the "TE" header field. If the values of "chunked" or "trailers" occur, then chunked transfer encoding WILL be sent back. Other values in the TE header field are ignored.

The server will be modified to allow transfer encodings of type "chunked" to be sent asynchronously to the client by the server application before the entire response is ready. However, this will only occur if the client request has values of "chunked" or "trailers" in the "TE" header field. (specification [2] requirement). Even though the HTTP protocol allows any number of bytes representing any state of any kind of data, the minimum chunk will contain (at least) one complete CIM object. This will eliminate any unnecessary complication on the client side for future asynchronous client receiving. 

Any detected non-supported transfer encoding values received in the client (see above) or duplicated supported values,  will throw out the entire HTTP message even if chunked is one in the list. This comes from Specification [1] section 3.6. (see error conditions). The server will not support any transfer encoding in the request body.

The server (or client) may run into conditions which would generate the following HTTP standard errors while processing a response or request.
 

•     "400 Bad Request"
•     "413 Request Entity Too Large"
•     "500 Internal Server Error"
•     "501 Not Implemented"

If the client detected these errors, then these are returned to the client application as if the server itself sent this error.  Any data that was received will be discarded. Also, if there was a CIMError or CIMStatusCode in the trailer, the same thing will happen as just described but using the numbers and data in these fields. Except for  "413 Request Entity Too Large" (when there was too much data in the response or request), all other errors and behavior are new to this PEP.
 
Conditions causing above errors:

•     Bad request results in any condition where the client or server interprets the data and meta data as not conforming to the specifications.
•     Entity too large results whenever any cumulative component of the protocol becomes too large to process for a (currently) 32 bit unsigned number.
•     Not implemented results when unsupported or duplicated transfer encodings are detected.
•     Anything else is an internal error, which of course is a bug that must be fixed.

The server will send back CIMError, CIMStatusCode in the trailer IF a CIM error occurred AND a trailer or chunk was requested by the client.  The server will also place these keywords in the trailer header line whether there is an error or not (if client requested so).  This behavior is required by specification [2]. A CIM error does not necessarily mean an HTTP level error (listed above). A CIM error is typically generated by the server application.  If the operation requested generated many threads to handle each implied class in the request, then each thread has the potential for error. This error may occur in mid stream of any given thread. The current solution is to stop the flow of data of that one thread but allow the rest to continue.

Notes:

- The HTTP level CIMError is not to be confused with the DMTF CIM_Error class that is transmitted in the XML payload.

- There are known open issues with error conditions. See discussion below.

- Users of the client side pegasus code should see no behavior change (also see error handling in discussion section).  Cuurently, there are no NEW requirements for the client to have to reconnect after a HTTP level failure (which now includes chunked transfer-encoding parsing)

2 Internal Server behavior

This section describes the new proposed behavior in the server that occurs above the HTTP level. This behavior is not dictated by any standard. These changes are necessary to take advantage of the chunking changes being added to the HTTP level. In summary, the modifications being proposed below change the deliver() method call in the ResponseHandler class hierarchy (that is called from within the server application code) from being synchronous to asynchronous.  More API/class detail can be found at http://cvs.opengroup.org/pegasus-doc

Summary of functional areas:

• HTTPConnection                           (phase 1)
  
• CIMOperationRequestDecoder     (no changes)
  
• CIMOperationRequestDispatcher (phase 2)
  
• ProviderManagerService               (phase 2)
  
• ProviderManager                          (phase 2)
  
• CIMOperationResponseEncoder    (phase 1, phase 2)
  

It is important to understand the data flow today before the new modifications can be understood. Today the flow of an incoming request (operation) and subsequent response is as follows on the server side:

• HTTPConnection reads in XML encoded request (via TCP) and sends it to CIMOperationRequestDecoder
• CIMOperationRequestDecoder decodes it into a CIMRequest and sends it to CIMOperationRequestDispatcher
• CIMOperationRequestDispatcher calls the appropriate operation method. This may involve breaking the request into separate threads to feed it to each registered provider dealing with a specific class.  Unregistered or static data requests go to the repository. Some requests that are internal go to the control providers that are internal to pegasus. The remaining  request(s) are sent to ProviderManagerService.
• The ProviderManagerService looks up the registered interface (i.e JMPI, CMPI, default) and calls into the appropriate ProviderManager instance to service the request.
  
• ProviderManager takes each request, creates a ResponseHandler instance that points to the original CIMRequest and an allocated CIMResponse object correlating to the requested operation. It then calls the registered provider that deals with the class and operation.
• The provider calls Responsehandler.deliver() to collect the response data locally in the ProviderMananger
  
• The provider calls Responsehandler.complete() to transfer the response data from the handler to the newly allocated response message.
  
• After complete(), the Provider returns from the call stack through ProviderManager to ProviderManagerService. This in turn sends the CIMResponse to CIMOperationRequestDispatcher (using callback function pointers indicated within the async sending instance (i.e AsyncOpNode))
• CIMOperationRequestDispatcher waits for all launched threads to come back from ProviderManagerService and/or repository service and keeps each response locally in the OperationAggregate instance. This instance (one per request) keeps track of every response per request issued.  It then places all responses of requested operation on one list (if there is more than one data response on different threads).  It then sends the new cumulative complete response message to CIMOperationResponseEncoder
• CIMOperationResponseEncoder takes the CIMResponse and XML encodes it according to class and operation, then sends it to HTTPConnection
• HTTPConnection sends the complete message to client (via TCP)

1 HTTPConnection

class:          HTTPConnection
dir:              Common
file(s):         HTTPConnection.cpp HTTPConnection.h

proposal:  (phase 1 changes - see HTTP behavior above)

2 CIMOperationRequestDispatcher

class:            CIMOperationRequestDispatcher, OperationAggregate
dir:                Server
file(s):           CIMOperationRequestDispatcher.cpp CIMOperationRequestDispatcher.h

proposal:

OperationAggregate:

The response functionality that comes back to this class via a callback from ProviderManagerService and/or repository  (see CIMOperationRequestDispatcher::_forwardForAggregationCallback()) will funnel through a modified OperationAggregate class. The function of the OperationAggregate class now becomes a sequencer rather than an aggregator. This is because each message coming from ProviderManagerService/repository are object(s) that need to be passed immediately down to CIMOperationResponseEncoder and then to the client. It will not wait for the completed message from all launched threads for that request. This has ramifications for error conditions. (see below).  Each CIMResponseMessage will contain information about the chunk (the index/count and a complete flag). This means the resequence function will restamp the chunks according to its own chunk count which is a summation of all chunks (objects) that came in from any thread (including synchronous repository calls) at this point for this request.  It will also restamp the complete flag when the last complete of the last thread and/or repository response is received. OperationAggregate will have an additional member tracking completed responses. For every thread that sends a chunk with the complete flag on, this is counted as that thread having finished. At this time, repository responses will always be complete as they are synchronous in nature to start with. It is very important that all other threads (for this request) are locked out while resequence executes. Once the response is resequenced, it is now ready to be sent down to the encoder and back out to the client. It is also important to note that we still need to be locked for sequential execution of the resquenced response because the client MUST get the first chunk first because it contains the opening XML tags and the client MUST get the last chunk last as it contains the closing XML tags. What happens in between could in theory be sent in parallel, but for the sake of simpler code each chunk will be sent to the client sequentially after resquencing.

Note:
To reiterate for clarification, the thresholding works on a per thread within a request basis. Right now, there is no concept of a "global" per request threshold.  If this did exist, it would enable the entire request to threshold the number of objects before sending to the client. This of course can be proposed for later phases.

The algorithm will look something like:

void OperationAggregate::resequenceResponse(CIMResponseMessage& response)
{
    // lock out all other threads so we can restamp the current chunk of the current thread.
    _mutex.lock(pegasus_thread_self());

    response.setIndex(_totalReceived++);
    if (response.isComplete() == true)
    {
        _totalReceivedComplete++;

        // only one of many threaded messages are complete, set to incomplete
        // until ALL completed messages have come in
       
        if (_totalReceivedComplete < _totalIssued)
            response.setComplete(false);
      
    }
    _mutex.unlock();
}


enqueuing:

There needs to be a method to synchronously send the response in this service. The code may look like:


/*
 * send the given response synchronously using the given aggregation object.
 * return whether the sent message was complete or not. The parameters are
 * pointer references because they can be come invalid from external deletes
 * if the message is complete after queuing. They can be zeroed in this
 * function preventing the caller from referencing deleted pointers.
 */

Boolean
CIMOperationRequestDispatcher::_enqueueResponse(OperationAggregate *&poA,
                                                                                                CIMResponseMessage *&response)
{
    lock();

    Boolean isComplete = false;

    try
    {
        poA->resequenceResponse(*response);
        isComplete = response->isComplete();

        // reset the response destination id to what was set in the request
        response->dest = poA->_dest;

        Uint32 type = poA->getRequestType();
        //request->getType()
       
        switch(type)
        {
            case CIM_ASSOCIATORS_REQUEST_MESSAGE :
            case CIM_REFERENCES_REQUEST_MESSAGE :
            case CIM_REFERENCE_NAMES_REQUEST_MESSAGE :
            case CIM_ASSOCIATOR_NAMES_REQUEST_MESSAGE :
                // execute today's code that exists and sets host and namespace if missing in handle*ResponseAggregation where * is the operation

            default :
                PEGASUS_ASSERT(0);
                break;
            
        } // switch
       
         // this is a synchronous call today that passes response to encoder
        lookup(response->dest)->enqueue(response);
        } // try

        catch(...){}

        if (isComplete == true)
        {
            // also deletes the copied request attached to it
            delete poA;
            poA = 0;
        }

        // after sending, the response has been deleted externally
        response = 0;

        unlock();
        return isComplete;
}



callbacks:

Any callback that is assigned on the request side will be enhanced to call the resequencer (above) and then send the response sequentially to the encoder. The callbacks to be modified are _forwardForAggregationCallback and  _forwardRequestCallback. The aggregation callback is used when there are multiple threads servicing the one request and hence has to "aggregate" (in the original context) all the parallel responses coming back. The other callback is used when only a single (but still asynchronous) response is anticipated. Even still, the returned response must be resequenced with potential repository responses. The aggregation callback may look like:

void CIMOperationRequestDispatcher::_forwardForAggregationCallback(
    AsyncOpNode *op,
    MessageQueue *q,
    void *userParameter)
{
      // today's code ... here

        //new code
        Boolean isComplete = response->isComplete();
        if (isComplete == false)
        {
            // put back the async request because there are more chunks to come.
            op->put_request(asyncRequest);
        }
        else
        {
            // these are per thread instantiations
            delete asyncRequest;
            delete asyncReply;
            op->release();
            service->return_op(op);
        }

        // After resequencing, this flag represents the completion status of
        // the ENTIRE response to the request.

       
        isComplete = service->_enqueueResponse(poA, response);
}


The other callback will have similar behavior

3 SimpleResponseHandler

class(es):       SimpleResponseHandler SimpleInstanceResponseHandler SimpleObjectPathResponseHandler SimpleMethodResultResponseHandler SimpleIndicationResponseHandler    SimpleObjectResponseHandler SimpleInstance2ObjectResponseHandler SimpleValueResponseHandler SimpleClassResponseHandler
dir:                 ProviderManager2
file(s):            SimpleResponseHandler.cpp SimpleResponseHandler.h


proposal:

NOTE: this proposal does NOT change any interfaces!

The base class SimpleResponseHandler, is one of the classes in the multiple inheritance hierarchy of the derived operation response classes (see below).  the remaining classes (listed above) all inherit the base class. The following protected virtual functions will be added to  this base class and all derived classes:

Also,  a new function will be defined in the base class that all deliver() calls will call underneath to determine whether to send the data asynchronously (i.e immediately) or not. This function will be called for both the deliver() and complete() virtual function calls and will be protected. The deliver() call will always call send() with isComplete = false and complete() will always call send() with IsComplete = true.


The complete call will now just call the send method (see OperationResponseHandler below for operation behavior)



The body of the code will live in SimpleResponseHandler.cpp and be as follows:


The comments note that the base class and the derived simple classes may not be a derived operation class. If this is the case, then there is no data to send asynchronously and the deliver() and complete() calls default to today's standard synchronous mode. However, if the instantiated class is an OperationResponseHandler class (see below) or a derived  OperationResponseHandler class, then this function will call the operation send function.  The operation class header file will only be included in the this source file and not the header.

Notes:

- The ResponseHandler class hierarchy is very involved with 2 levels of multiple inheritance. It is important to have an understanding of the hierarchy to understand this proposal.

- The new functions cannot be made abstract in the base class because this class has already been defined to be concrete. The new functions in this base class will effectively be a "do nothing" function whereas the derived classes will then redefine these functions

- SimpleMethodResultResponseHandler size() function represents the parameter out list, not the return value. The clear() function clears the out parameters and the return value.

- Some of the derived simple classes (listed above) are used in operation derived classes that were stated as not supporting asynchronous delivers (such as indications), however the new functions are still needed for correctness and consistency

4 OperationResponseHandler

class(es):       OperationResponseHandler GetInstanceResponseHandler  EnumerateInstancesResponseHandler EnumerateInstanceNamesResponseHandler  CreateInstanceResponseHandler GetPropertyResponseHandler SetPropertyResponseHandler ExecQueryResponseHandler AssociatorsResponseHandler AssociatorNamesResponseHandler ReferencesResponseHandler  ReferenceNamesResponseHandler InvokeMethodResponseHandler
dir:                 ProviderManager2
file(s):            OperationResponseHandler.cpp OperationResponseHandler.h


proposal:

NOTE: this proposal does NOT change any interfaces!

The OperationResponseHandler class is one of the inherited classes of the other classes listed above. These classes are the operation based response handler classes.  Any handler instantiated as one of the above derived classes has the potential to send data back asynchronously from a deliver() call. (although some are not going to be supported - see not supported section). Keep in mind, although OperationResponseHandler is a base class to the others listed above, it is also a concrete class. This means this base class may be instantiated as well, but then there is no data expected back on the operation response because the SimpleResponseHandler hierarchy actually contains any data returned.

The following protected virtual functions will be added to this base class and all derived classes:

        // transfer any objects from handler to response. this does not clear() the handler
        virtual void transfer() {}

        // validate whatever is necessary before the transfer
        virtual void validate() {}

These functions are needed for a single send() call to do virtual tasks. The transfer() takes the data stored in the SimpleResponseHandler side of the hierarchy (if this is a derived operation instantiation) and transfers it to the response on the OperationResponseHandler side of the hierarchy. The validate() function is used to validate the data before a transfer is executed. Typically this is used to validate the minimum number of responses being sent. The base class is a "do nothing" because it is a concrete class as well as there is no data to transfer or validate. Because these new methods exist, the derived complete() call can now be eliminated. It will just exist as a single virtual in SimpleResponseHandler

The following private members will be added:


        Uint32 _responseObjectTotal;
        Uint32 _responseMessageTotal;
        Uint32 _responseObjectThreshold;
   

The _responseObjectTotal represents how many total objects (up to any point in time) have been sent asynchronously.

The _responseMessageTotal is how many responses (up to any point in time) have been sent asynchronously. Note that each response may have 0 to N objects! The message total is roughly analogous to HTTP level chunks EXCEPT for responses that ONLY represent opening or closing XML but NOT any data.

The _responseObjectThreshold represents the threshold used for this handler to represent the maximum object count needed before a response is sent asynchronously (except if a response is marked as complete, in which case it is sent regardless). For operations that do not have any expected data (e.g.  CreateSubscription), responses are not sent until marked complete. After testing different thresholds for performance, a default value of 100 has been given if not set. This is not an ordained number, but one that was seen through many different testing scenarios as being somewhere between time/performance friendly and memory friendly for many applications. Of course any particular environment may be different.  If a particular application has need to change this number, then another container can be added to the OperationContext class in which the application can call SetContext on it. This container does not exist as of yet, but can be added in future phases if deemed necessary.  Note that thresholding is on a per instance handler. One request may generate many handlers and this does not apply to control providers nor the repository service.

Also, the threshold is a compile time define. If server applications cannot deal with the proposed default threshold, then the OperationResponseHandler.cpp file can be recompiled with a different number passed in. Note that 0 will be treated as ~0 which is the same as having no threshold (message will be sent when complete() is called).

The code will be as follows:

#ifndef PEGASUS_RESPONSE_OBJECT_COUNT_THRESHOLD
#define PEGASUS_RESPONSE_OBJECT_COUNT_THRESHOLD 100
#elif PEGASUS_RESPONSE_OBJECT_COUNT_THRESHOLD  == 0
#undef PEGASUS_RESPONSE_OBJECT_COUNT_THRESHOLD
#define PEGASUS_RESPONSE_OBJECT_COUNT_THRESHOLD  ~0
#endif


This send method will be called from SimpleResponseHandler.send() . The send algorithm will look similar to:

void OperationResponseHandler::send(Boolean isComplete)
{
    SimpleResponseHandler *simpleP = dynamic_cast<SimpleResponseHandler*>(this);

    // It is possible to instantiate this class directly (not derived)
    // The caller would do this only if the operation does not have any data to
    // be returned

    if (! simpleP)
    {
        // if there is no data to be returned, then the message should NEVER be
        // incomplete (even on an error)
        if (isComplete == false)
            PEGASUS_ASSERT(false);
        return;
    }

    SimpleResponseHandler &simple = *simpleP;
    PEGASUS_ASSERT(_response);
    Uint32 objectCount = simple.size();

    // have not reached threshold yet
    if (isComplete == false && objectCount < _responseObjectThreshold)
        return;

    CIMResponseMessage *response = _response;

    // for complete responses, just use the one handed down from caller
    // otherwise, create our own that the caller never sees but is
    // utilized for async responses underneath

    if (isComplete == false)
        _response = _request->buildResponse();

    _response->setComplete(isComplete);

       _responseObjectTotal += objectCount;
    Uint32 responseMessageCount = response->getIndex();

    // since we are reusing response for every chunk,keep track of original count
    _response->setIndex(_responseMessageTotal++);

    validate();
    transfer();
    simple.clear();

    _response->operationContext.
        set(ContentLanguageListContainer(simple.getLanguages()));

    // call thru ProviderManager to get externally declared entry point

    if (isComplete == false)
    {
       ProviderManagerService::handleCimResponse(*this);
    }

    // put caller's allocated response back in place
    _response = response;
}

5 ProviderManagerService

class:             ProviderManagerService
dir:                 ProviderManager2
file(s):            ProviderManagerService.cpp ProviderManagerService.h


proposal:

The provider manager service will add a method to handle the asyncronous response. This is a logical home for this new function since the service handles messaging to begin with.

The method will look similar to:

/*static */ void
ProviderManagerService::handleCimResponse(CIMRequestMessage &request,  CIMResponseMessage &response)
{
    CIMStatusCode code = CIM_ERR_SUCCESS;
    String message;

    try
    {
        // only incomplete messages are processed because the caller ends up
        // sending the complete() stage
        PEGASUS_ASSERT(response.isComplete() == false);
       
        AsyncLegacyOperationStart *requestAsync =
            dynamic_cast<AsyncLegacyOperationStart *>(request._async);
        PEGASUS_ASSERT(requestAsync);
        AsyncOpNode *op = requestAsync->op;
        PEGASUS_ASSERT(op);
        PEGASUS_ASSERT(! response._async);
        response._async = new AsyncLegacyOperationResult
            (requestAsync->getKey(), requestAsync->getRouting(), op, &response);
       
        // set the destination
        op->_op_dest = op->_callback_response_q;
       
        MessageQueueService *service =    
            dynamic_cast<MessageQueueService *>(op->_callback_response_q);

        PEGASUS_ASSERT(service);

        // the last chunk MUST be sent last, so use execute the callback
        // not all chunks are going through the dispatcher's chunk
        // resequencer, so this must be a synchronous call here
        // After the call is done, response and asyncResponse are now invalid
        // as they have been sent and deleted externally
       
        op->_async_callback(op, service, op->_callback_ptr);
       
    }

    catch(CIMException &e)
    {
        code = e.getCode();
        message = e.getMessage();
    }
    catch(Exception &e)
    {
        code = CIM_ERR_FAILED;
        message = e.getMessage();
    }
    catch(...)
    {
        code = CIM_ERR_FAILED;
        message = cimStatusCodeToString(code);
    }

    if (code !=  CIM_ERR_SUCCESS)
        response.cimException = PEGASUS_CIM_EXCEPTION(code, message);
}


Note that responses marked as complete do not come through this call. The handler.send() call simply stamps the response and returns out of the provider stack (see OperationResponseHandler.send()) if marked complete. This allows the original response allocated in ProviderManagerService to be used as the final completed response to be sent as it is today and NONE of the ProviderManagerService calling code needs to be modified!

This algorithm is fairly similar to what happens today after the provider has queued up the data and completed its function call. The difference here is that this will call the callback to CIMOperationRequestDispatcher and wait for the complete cycle of the response from the point of deliver() all the way to when HTTPConnection does a tcp.send() to the client. This waiting is important for these reasons:

• If we did not wait, the many chunks per thread would easily overwhelm the global message queue (Imagine an EnumerateInstances on a class with 50+ derived classes each with 10000 instances or chunks). By waiting, we do not aggravate the multiple thread launching paradigm that occurs today nor do we have to test for "queue full" which can have many deadlock conditions.
• Keeps the memory usage low as the chunked response messages would not "pile up" in a busy queue (per thread)
• If we did not wait, there would be the possibility of a "complete" flagged response arriving at CIMOperationRequestDispatcher before previous response within the same thread in a multi-processor machine.
  

The overhead to send each response through the mechanism includes a new CIMResponseMessage and 2 async helper instances (AsyncOpNode, AsyncLegacyOperationResult).  It is simply dealing with locks and pointers during the deliver() process. Keep in mind that the amount of data being sent is the same. As the object response threshold is set lower, the overhead becomes higher, but of course the trade-off is not using exorbitant amounts of memory at once.

6 CIMOperationResponseEncoder

class:            CIMOperationResponseEncoder
dir:                Server
file(s):           CIMOperationResponseEncoder.cpp CIMOperationResponseEncoder.h

proposal:

The encoder change simply has to convey to the sendResponse method the message index and if the message is complete or not. This is a phase 1 change.


class:            XmlWriter
dir:                Common
file(s):           XmlWriter.cpp


proposal:

The formatters will be modified to understand first, middle, last. Any combination of all 3 states may occur. "middle" implies actual object data (i.e. "body" argument).  A given response may have all of or just one of these states. This is a phase 1 change.


Array<Sint8> XmlWriter::formatSimpleIMethodRspMessage(
    const CIMName& iMethodName,
    const String& messageId,
    HttpMethod httpMethod,
    const ContentLanguages & httpContentLanguages,   
    const Array<Sint8>& body,
    Uint64 serverResponseTime,
        Boolean isFirst,
        Boolean isLast)
{
    Array<Sint8> out;

        if (isFirst == true)
        {
            // NOTE: temporarily put zero for content length. the http code
            // will later decide to fill in the length or remove it altogether
            appendMethodResponseHeader(out, httpMethod, httpContentLanguages, 0, serverResponseTime);
            _appendMessageElementBegin(out, messageId);
            _appendSimpleRspElementBegin(out);
            _appendIMethodResponseElementBegin(out, iMethodName);
            // output the start of the return tag. Test if there is response data by:
            // 1. there is data on the first chunk OR
            // 2. there is no data on the first chunk but isLast is false implying
            //    there is more non-empty data to come. If all subsequent chunks
            //    are empty, then this generates and empty response.
            if (body.size() != 0 || isLast == false)
                _appendIReturnValueElementBegin(out);
        }

    if (body.size() != 0)
    {
            out << body;
    }

        if (isLast == true)
        {
            if (body.size() != 0 || isFirst == false)
                _appendIReturnValueElementEnd(out);
            _appendIMethodResponseElementEnd(out);
            _appendSimpleRspElementEnd(out);
            _appendMessageElementEnd(out);
        }

    return out;
}

7 Error Conditions

Today, if any of the responding threads contains an error, the data for that thread is discarded (except if all threads have errors, it chooses the first one). To reiterate from above, the thread data will simply be terminated at the point that the error is encountered since some data may have already been sent to the client. (see  discussion on errors below)

See "Definition of the Problem" section

The targeted code base is 2.4 for phase 1 and 2.5 for phase 2. The date is whenever that is scheduled to be released.

NOTE: as of last discussion, the change will ONLY go into monitor1. It is assumed PEP-183 will reconcile monitor1 vs monitor2

Comment here on the proposed design/methodology for asynchronous deliver() calls. There are many implementation issues and I want to make sure they are covered. Some issues that have been brought up:
 

• There was ordering of chunks concern since now that many providers can be delivering chunks asynchronously.  The concern here would be that automated tests may fail if a certain order of results is expected.
  
• The desired result is to have synchronous behavior at the point of sending each chunk through the system. Calling callbacks directly achieves this behavior. Is this safe ? (see handleCimResponse()).
• There is concern that the deliver() calls may take an undetermined amount of time (when the threshold is hit). Will this affect timing issues within any given provider ?
  
• Other issues ?