PEP

Pegasus Enhancement Proposal (PEP)

PEP#: 349

PEP Type: Functional

Title: Improve the availability of the CIMOM by better isolation from faulty providers.

Status: Approved

Version History:

Version	Date	Author	Change Description
0.1	6 Oct 2009	Sahana Sampige Prabhakar (sahana@hp.com), Chakravarthy Racharla(chakru@hp.com)	Initial Submission.
0.2	3 Nov 2009	Sahana Sampige Prabhakar (sahana@hp.com)	Incorporated review comments.
0.3	6 Nov 2009	Sahana Sampige Prabhakar (sahana@hp.com)	Updated the discussion section with responses to review comments.
0.4	9 April 2010	Sahana Sampige Prabhakar (sahana@hp.com)	Updated with changes after implementing the feature.
0.5	29April 2010	Sahana Sampige Prabhakar (sahana@hp.com)	Incorporated review comments.

Abstract: Since the CIMOM is the core component of the manageability stack, it should be operational at all times. A faulty provider (a provider with a bug due to which it does not send responses) should not bring the CIMOM operations to a stand still. Currently in OpenPegasus the Provider Manager Service creates one thread per request which is not released until the response is sent back from the provider. In a situation where a provider is faulty and periodic requests are sent to this provider, many threads will be busy waiting for the response. Thus over a period of time the cimserver process reaches its process thread limit and cannot process any further requests. This inhibits the cimserver from processing requests for well behaved providers as well and the entire manageability stack becomes dysfunctional. It is quite possible that the client connection has timed while waiting for the response from this faulty provider. In such a situation, there is no point in the CIMOM holding on to the resources used up for this request. The proposed PEP tries to isolate the CIMOM from these faulty out of process providers. At the same time care is taken to ensure minimal impact of these changes on well behaved providers. The goal is to limit the effect of the faulty behavior to the cimprovagt hosting that provider. This PEP does not address in process providers. This approach does not help free up resources if clients use a very large timeout and do not disconnect.

Definition of the Problem

Currently, one faulty provider can potentially bring the cimserver operations to a stand still. Consider a scenario where there are requests being sent periodically to a faulty provider. A faulty provider is one which is not sending responses to requests sent to it. Below are the side effects in the current implementation in OpenPegasus:

1. The ProviderManagerService creates one thread for every request which needs to be sent to a provider. This thread is blocked while waiting for a response from the provider. If there are many requests to an unresponsive provider, over a period of time the cimserver thread limit is reached and cimserver cannot process any further requests due to insufficient thread resources. Thus, one faulty provider prevents all other providers from getting used.

2. The file descriptors used for the socket connection with the client are not closed until the entire response arrives. It is quite possible for the Operating System to run out of file descriptors when responses are not received for concurrent requests to a faulty provider. If the client has disconnected due to a timeout, this socket can be closed without causing any loss of data.

3. The OperationAggregator in the CIMOperationRequestDispatcher is not deleted until responses are received from every provider to which the request was sent. So, an unresponsive provider will hold up the OperationAggregator object from deletion. If the client has disconnected then there is no point sending the response or retaining the OperationAggregator. On the other hand, if the client connection is active then we have to retain the OperationAggregator.

4. The Provider Agent has a thread to read requests from a pipe which it uses for communication with the cimserver. Every new request is allocated a new thread. In case of a faulty provider these threads will not be released and the reader thread will yield until threads become available. Hence the Provider Agent does not read new requests from the cimserver and the write buffer at the cimserver’s end gets full. When the pipe buffer is full, the ProviderAgentContainer::_processMessage() function is blocked on the write, while holding the _agentMutexLock which blocks all further request to the Provider.

Proposed Solution

The proposed solution is to respond asynchronously to all requests sent to the OOP Router. When the provider responds, the response is sent back asynchronously using a callback function. Periodically the outstanding requests for a provider module are checked to see if the client is still waiting for the response. If not, the resources used for this request are freed up. The changes address only out of process providers.

The details of the changes proposed by this PEP are as below:

1. The ProviderAgentContainer::_processMessage function will send a request to the ProviderAgent and return. The response will be sent asynchronously using a callback function. This change applies only to out-of-process providers.

2. A new virtual function called isActive () will be added to the MessageQueue class. This function will indicate whether the MessageQueue is active or not. By default this function will return true and do nothing else. The HTTPConnection class will have a specialized implementation. In this implementation a check will be performed on the non-blocking socket to see if it is active by reading 1 byte. Since the current thread is processing the request, its safe to try to read 1 byte from the socket as there should be no data on the socket, If read returns a message of size zero, it is an indication that the client has closed the connection and the socket at the server end is closed using the HTTPConnection::_closeConnection function (The HTTPConnection::needsReconnect() function proposed in Bug 8605 will be used ).

3. A new member, “Boolean isAsyncResponsePending” will be added to OutstandingReuqestEntry class to indicate whether a response is pending or not. This is set to false by default. It will be set to true when the response is expected to arrive asynchronously. Entries which have this flag set are periodically checked for client disconnects and associated resources are cleaned up (more below).

4. All responses will be handled asynchronously are periodically checked to see if the client is still waiting for the response. This is done by fetching the bottom most queueid from the QueueIdStack of the request message. The isActive() (as implemented by HTTPConnection) function will indicate whether the connection is active or not. A response which will indicate that the connection can be closed is sent back to the ProviderManagerService. This will close the socket file descriptor used for communication between the cimserver and the client. This will have no impact on requests sent using sent SendWait since the bottom most queueid will not be HTTPConnection.

5. A new call back function will be added to ProviderManagerService to handle asynchronous responses from the OOPProviderManagerRouter. This function, asycResponsePendingCallback() will be similar to responseChunkCallback(). This function is called to asynchronously send the response to the destination mentioned in the request message. Its prototype will be as below

typedef void (*PEGASUS_ASYNC_RESPONSE_CALLBACK_T)(

CIMRequestMessage* request, CIMResponseMessage* response);

6. To handle requests sent to all providers for example the request to stop all providers, the _forwardRequestToAll agents() function will construct a ResponseAgrregation count object like below :

class RespAggCounter
{
public:
RespAggCounter(Uint32 count)
{
_expectedresponseCount = count;
}
Boolean isComplete(CIMexception &e)
{
AutoMutex mtx(_mutex);
if (e.getCode() != CIM_RC_ERR_OK)
{
_exception = e;
}
_receivedResponseCount++;
return receivedResponseCount == expectedresponseCount ;
}
CIMException getException()
{
return _exception;
}
private:
Mutex _mutex;
Uint32 _expectedresponseCount, _receivedResponseCount ;
CIMException _exception;
};

RespAggCounter is passed to ProviderAgentContainer::processMessage which is stored in OutStandingRequestEntry. WaitSemaphore is removed altogether. When the responseProcessor thread receives the response from the provider, it calls asyncResponseCallback only when the response is complete. For other normal requests this object is null. This makes sure that the approach to handle responses is similar for requests sent asynchronously or synchronously. It does not improve the availability of the cimserver from faulty providers which don’t respond to requests sent using MessageQueueService::SendWait(). But it does enable the CIM_STOP_ALL_PROVIDERS_REQUEST_MESSAGE and CIM_SUBSCRIPTION_INIT_COMPLETE_REQUEST_MESSAGE messages to be sent in parallel to all provider modules hence improving the cimserver startup and shutdown time.

7. The OutstandingRequestTable needs to be checked at regular intervals if the entries with pending responses need to be cleaned up if the client has closed the connection. A new function called cleandDisconnectedClientRequests() will be added to ProviderAgentContainer which will be invoked similar to the way idle provider unload is initiated. This function will check if the client is inactive and send a response using the asyncResponseCallback() if the client is inactive. This will free up the file descriptors and OperationAggregator associated with the disconnected clients. If the client is inactive then the OutstandingRequestEntry is updated to indicate that the client is no longer waiting for the response. If not, nothing is done. The trigger to initiate this cleanup will be from the CIMServer::runForever() thread.

8. The ProviderAgentContainer::_uninitialize() function can be called as a consequence of a graceful shutdown of a provider or due to a sudden closure of the pipe connection with the cimserver. If the provider went down gracefully and if there are pending asynchronous requests then they will be retried on a new thread. This was previously handled using the _REQUEST_NOT_PROCESSED response. If the provider went down unexpectedly and there are pending asynchronous requests a response with an exception is sent using the new asyncResponseCallback.

9. The cimprovagt will be optimized to continuously read from the socket irrespective of the provider behavior. Currently, when there are insufficient thread resources, the ProviderAgent run thread yields waiting for resources to be available. This holds up the run thread from reading from the socket. If the provider is not releasing the other threads due to a bug in the provider, the read from the pipe does not happen. At the same time if further requests are piled up for this provider, the write buffer of the pipe from the server end gets full. This will block the write to the pipe and subsequent requests to this provider will be blocked on the ProviderAgentContainer::_pipeToAgent mutex. This can be avoided if the yield is removed and response with CIMException set to CIM_ERR_FAILED and an appropriate message is sent to the client. This will result in a change in behavior for the client. The client will now receive a CIM_ERR_FAILED exception instead of a CONNECTION_TIMEOUT_EXCEPTION. This change will expose the faulty or slow behavior of the provider to the client rather than stop the cimprovagt from reading any further requests. This change will not impact the following requests :

· CIM_INITIALIZE_PROVIDER_AGENT_REQUEST_MESSAGE

· CIM_NOTIFY_CONFIG_CHANGE_REQUEST_MESSAGE

· CIM_DISABLE_MODULE_REQUEST_MESSAGE

· CIM_STOP_ALL_PROVIDERS_REQUEST_MESSAGE

· CIM_SUBSCRIPTION_INIT_COMPLETE_REQUEST_MESSAGE

· CIM_INDICATION_SERVICE_DISABLED_REQUEST_MESSAGE

· Should CIM_EXPORT_INDICATION_REQUEST_MESSAGE also be included here?

10. When the cimprovagt cannot create new threads a message will be logged in the Syslog.

Rationale

The implementation of this PEP will improve isolation of effects of some faulty OOP providers on the CIMOM. The faulty provider should not cause CIMOM to consume excessive resources and prevent servicing requests to other providers. The proposed changes will not affect the in-process providers (BasicProviderManagerRouter).

Schedule

OpenPegasus 2.11 release.

Discussion

1. (venkat_puvvada) Is this logic applied to all the requests? For example internal client do not timeout and some internal control providers sends requests using SendWait().
(Sahana_prabhakar) No. It will not be applied to CIM_EXPORT_INDICATION_REQUEST_MESSAGE, CIM_STOP_ALL_PROVIDERS_REQUEST_MESSAGE, CIM_SUBSCRIPTION_INIT_COMPLETE_REQUEST_MESSAGE, CIM_INDICATION_SERVICE_DISABLED_REQUEST_MESSAGE, CIM_NOTIFY_CONFIG_CHANGE_REQUEST_MESSAGE, CIM_ENABLE_MODULE_REQUEST_MESSAGE, CIM_DISABLE_MODULE_REQUEST_MESSAGE. These requests do not inundate the cimprovagt and so these will be handled as before. Also this change applies only to Out of process providers.

2. (venkat_puvvada) What is this new Active() function used for ? I think MessageQueueService can be running or not running at present, also HTTPConnection is a MessageQueue not MessageQueueService. For Server, reading 1 byte from the socket may not be good idea to determine whether client has closed the connection or not.
(Sahana_prabhakar) Since the current thread is processing the request, I would think it’s safe to try to read 1 byte from the socket, how else can one verify if the client connection is alive? Yes, you are right; this function will go into MessageQueue class.

3. (venkat_puvvada) Where this topmost queueId for the request is appended? HTTPConnection queueId will not be present for all requests. I think for requests which have the HTTPConnection queue-id topmost queueid would be CIMResponseEncoder. How the Monitor entries are are cleaned up in this case?
(Sahana_prabhakar) That should be the queueid at the bottom of the queue. That is the queueid at location zero of the QueueIdStack. The _closeConnection function in HTTPConnection class will be used to cleanup the monitor entries.

4. (venkat_puvvada) I think there are some more services which send requests to providers. For example CIMExportRequestDispatcher, any more services missing here?
(Sahana_prabhakar) CIMExportRequestDispatcher was the only other one that I could find. But the CIM_EXPORT_INDICATION_REQUEST_MESSAGE that it sends will be processed as before. I could not find any other.

5. (venkat_puvvada) I think OperationAggregator object pointer is passed as a parameter to the callback function which is stored in the AsyncOpNode. Each request have the same OperationAggregator pointer, how this is set to NULL or deleted?
(Sahana_prabhakar) Good catch. Will correct that. It will not be set to null. Instead, this response will be processed like any other response.

6. (venkat_puvvada) Note that current responseChunkCallback() method assumes that request has been sent using callback method(SendAsync), new method asycResponsePendingCallback() must be able to handle other scenarios like request has been sent using the SendWait() or SendForget().
(Sahana_prabhakar) In the current design asycResponsePendingCallback will not be called for requests sent using SendWait & SendForget, but for completeness of this function, an assert to check whether the AsyncOpNode flags is ASYNC_OPFLAGS_CALLBACK will be added.

7. (venkat_puvvada) What is the timeinterval used to invoke this cleanup of outstanding requests entries? Note that if there many instances to be returned response processor thread might be sending chunked responses , in that case wait semaphore is not signalled until the last response is arrived. This scenario can be used for optimization without sending RESPONSE_PENDING message to appropriate service.
(Sahana_prabhakar) To add this optimization, the processMessage thread has to be intimated when the response chunk arrives. For this I will either have to introduce another entry in the outstanding request table or use another semaphore. When this is compared with sending a response pending message, the later seems better from a performance perspective.

8. (venkat_puvvada) This scenario must be handled carefully, since many providers may be running in the same cimprovagt, some requests like CIM_SUBSCRIPTION_INIT_COMPLETE_REQUEST_MESSAGE need to be processed since there are no error checks are performed against responses for these requests currently.
(Sahana_prabhakar) Will process these requests synscronously - CIM_STOP_ALL_PROVIDERS_REQUEST_MESSAGE, CIM_SUBSCRIPTION_INIT_COMPLETE_REQUEST_MESSAGE, CIM_INDICATION_SERVICE_DISABLED_REQUEST_MESSAGE, CIM_NOTIFY_CONFIG_CHANGE_REQUEST_MESSAGE, CIM_ENABLE_MODULE_REQUEST_MESSAGE, CIM_DISABLE_MODULE_REQUEST_MESSAGE

9. (Roger_kumpf) Is there value in having the ProviderAgentContainer respond synchronously in some cases and asynchronously in others? I wonder if it wouldn't make sense to have all responses come asynchronously. It would simplify the implementation and improve testing of the asynchronous case. Maybe there is a performance penalty, I don’t know. The clean-up could still be done via regular traversals of the OutstandingRequestTable, with status maintained in the table as you've proposed. This approach should also avoid the need to introduce a CIM_PENDING_RESPONSE_MESSAGE message type, which feels a bit clumsy.The CIM_DISCARD_RESPONSE_MESSAGE message type also does not seem to fit in well. Perhaps this case could/should also be handled via some sort of callback? (If the approach suggested in the paragraph above is taken, there wouldn't be a way to return this message at the appropriate time, anyway.) I suspect the code you are adding will be tricky to test, since it often will only be exercised in an exceptional condition. So I'd advise changing the mainline implementation where feasible rather than introducing separate logic to handle the exception case. Stress testing will also be valuable, since race conditions are a significant risk in the area in which you are working.

(Sahana Prabhakar) Good point. All requests sent using SendAsync will be handled asynchronously.

10. (venkat_puvvada) How the _REQUEST_NOT_PROCESSED responses are handled?

(Sahana Prabhakar) The _uninitialize function can be called as a consequence of a graceful shutdown of a provider or due to a sudden closure of the pipe connection with the cimserver. If the provider went down gracefully and if there are pending asynchronous requests then they will be retried on a new thread. If the provider went down unexpectedly and there are pending asynchronous requests a response with an exception is sent using the new asyncResponseaCallback.

11. (venkat_puvvada) Why all the responses mentioned below cannot be sent asynchronously?

(Sahana Prabhakar) The intent of this change is to avoid misbehaved providers from hogging system resources. In this case its threads. The intent here is to not yield when there are no threads available but send an exception response to the client. But this logic will not be applied to requests which have to be processed immediately like stop all providers requests, subscription init complete etc. These requests do not inundate the cimserver hence they cannot cause the problems which this PEP is trying to address.

12. (venkat_puvvada) A response aggregation object similar to dispatcher can be used to handle requests sent using SendWait.

(Sahana Prabhakar) Ok.

Permission is hereby granted, free of charge, to any person obtaining a copy of

this software and associated documentation files (the "Software"), to deal in

the Software without restriction, including without limitation the rights to

use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of

the Software, and to permit persons to whom the Software is furnished to do so,

subject to the following conditions:

The above copyright notice and this permission notice shall be included in all

copies or substantial portions of the Software.

THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR

IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS

FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR

IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN

CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.

Template last modified: February 17th 2009 by Martin Kirk
Template version: 1.15