ZMap implements 'servers' which provide access to databases either via an interactive connection (eg ACEDB) or a passive one (eg pipeServers) and a few layers of software have been implemented to allow this. These layers have thoer own private data and information is passed bewteen the layers via callback functions which are driven by a number of requests and responses defined in zmapServerProtocolHandler.c. At each layer requests and responses are copied from one data structure to another and request codes and responses code are translated to something else.
A further section below deals with passing request coordinates to a server.
Hopefully this little document may help someone else find their way through the code.
Here we document what has to be done to add new data from an external script and pass this through to Otterlace, which will take us from the script output itself through the layers of server code and via the view and control code back to Otterlace. Adding parameters from Otterlace or ZMap config or user action requires a similar process but in reverse.
The present example is due to a request from Anacode to have the entire output of a script's STDERR reported instead of a nominal single line error message. As the existing single line error message has to remain (eg to report other errors) we cannot reuse this and have to implement an extra data item. We also require the exit code of the script to ensure that data is valid (and complete).
Note that returning STDERR is no more complex than returning an exit code - we simple read all of it and assume we have enough memory
In case of a failure a pipe server script will produce diagnostic information on STDERR, and this may occur before or during normal data output. If ZMap reads the whole stream without an error it will report success, but if not it will read the pipe script's STDERR. Previously (if there was an error) the last non blank line of this was reported as the error message and now the requirement is to output all of this on ZMap's STDERR and also send it to Otterlace via XRemote.
In this case we will report an error message such as "script failed".
There is a function pipe_server_get_stderr() that takes a PipeServer struct as argument which is called in case of any error. This can be left as is to read STDERR whenever an explicit error occurs and also called at normal EOF on the pipe script's STDOUT.
So step 1 is to add some code to read STDERR and set an error condition if this is non-null.
There's a new requirement to return STDERR always and return the exit code of pipe Server scripts. In case of a non zero exit code the data should be dropped as it may be incomplete, and this could influence annotation.
ZMap interfaces to servers via a series of requests the fetch styles, DNA, and features, and to check the script's exit code and return STDERR it is neatest to add a new step to this protocol.
In zmapServerProtocol.h add an enum for the request type and a ZMapServerReqStatus data structure. Add this to ZMapServerReqUnion (in zmapServerProtocol.h) and zMapServerRequestCreate() and zMapServerCreateRequestDestroy() (sic) and zMapServerRequestHandler() in zmapServerProtocolHandler.c. and then add a function like getStatus() to call the server function zMapServerGetStatus() (that you have to add) which is itself a wrapper for the server->funcs->get_sequence() call.
Why are there all these layers? AFAICS it's because each one has it's own data which is out of scope in all the others. Note that the zmapServer layer holds the request and response data in the server request data structure that differs for each request type and the actual remote server module that processes the request does not have access to this data structure and handles each data item explicitly.
Now add a server function to the ZMapServerFuncs structure in zmapServerPrototype.h and add a matching function prototype. Note that to follow the existing design it is necessary to have all the data outputs specified as arguments. Revist the code mentioned above and adjust the calls to match.
Add code for this function to zMapServerGlobalInit() in zmapServer.c. Note that this implies that all server modules must implement this function.
In pipeServer.c add the get_status function to pipeGetServerFuncs() and write the matching getStatus() function based on the prototype defined above. For pipe servers take into account that they also implement file servers and need to take into account that there is no exit code or STDERR in this case.
Do the same in acedbServer.c and dasServer.c
This is done by calling zmapViewStepListAddServerReq() in createConnection() when we are running a script that is expected to terminate, which for the moment is all pipe scripts. As we can only get an exit code from a terminated script this seems fairly safe. Note that the terminate call itself is for the ZMap thread that runs the script and responds to the various requests above.
Add the request type to dispatchContextRequests() and processDataRequests().
Adjust the display_after flags to be this new request where appropriate (in createConnection())
Features are displayed via the getFeatures() in processdataRequests() and we have to do the same stuff for ZMAP_SERVERREQ_FEATURES and ZMAP_SERVERREQ_GETSTATUS depending on whether or not we get the exit code (we don't for ACEDB, it doesn't exit). This is independant of reporting 'features_loaded' to Otterlace, which is something that happens via checkStateConnections(). This function tests the server thread status and picks up on error conditions at any point in the series of requests, and we report status to Otterlace when the thread has completed its step list of requests.
This is done by a callback set by zmapControl.c calling zmapViewInit() and the function we end up in via view_cbs_G->load_data() is dataLoadCB()
The problem to solve now is to get the exit code and STDERR output which is available from the getStatus() function to the dataLoadCB function and this requires the use of two more data structures:
There is a slight problem reporting errors from any particular step in the series of requests in that the exit code and STDERR output are recovered by an explicit request and the existing code aborts making these requests in case of an error. Existing code did not test the script's return code and only detected failure in the Open call by failing to read a valid GFF header. Obviously a script could fail at any point and we need to detect failure at each possible step int he request protocol.
The current interface consists of different data structures for each request and repsonse and to provide an error response for each request with ancillary data would require changing the code for each request in all the server modules and at each layer of the server interface and the calling code in zmapView.
There is another complication in that the code in zmapSlave.c aborts the thread if the external server dies and as it stands we cannot return the exit code or STDERR output by the existing interface.
To fix this we can stop the slave thread from exiting and continue to process other requests. Note that zmapServer.c only makes requests if the previous repsonse was not ZMAP_SERVERRESPONSE_SERVERDIED. This implies that the pipeServer should set ZMAP_SERVERRESPONSE_SERVERDIED rather than ZMAP_SERVERRESPONSE_REQFAIL.
What would be better would be the have the pipe server check the status of the external script. This could be done by adding a callback to report the child process exiting and report ZMAP_SERVERRESPONSE_SERVERDIED if that happened. However, this would happen asyncronously and the likely result would then be a race condition, with some of the failures operating as at present.
To avoid a) rewriting the server code and b) making a large number of global changes the following will be done:
In zmapView.c when replies are received to requests there is a flag that controls whether or not to display the data recieved. This previously could be set for sequence or features requests, both of which were handled by the same response code in zmapView.c. They use different data structures but both are accessed as ZMapServerReqGetFeatures. (this is a bug).
The flag 'display_after' is misnamed as it controls storing the returned data (merging into the View's context) and then displaying it. If we flag the getStatus request as the one to display after then by then we have lost the data that was returned by getFeatures.
We could query the exit status fo the script before requesting features but this could lead to a deadlock where the script is de-scheduled due to waiting to write to a pipe that is not being read, and we have to store the features data in the ConnectionData struct for use later.
Due to a previous implementation where the View's step list data structures were used to ensure that all active connections did each request in step and all finished at the same time attempts to add a getStatus call by using the function zmapViewStepListAddServerReq() failed as this requires zmapViewStepListAddStep(step_list, ZMAP_SERVERREQ_GETSTATUS, REQUEST_ONFAIL_CANCEL_THREAD) to have been called first. This is hidden away in zmapViewConnectionStepListCreate().
This implies that the current implementation prevents re-ordering of requests for different types of server. Note: some of the requests are required to be in the given order as some require data retrieved by a previous request. Some of this may be passed in arguments or perhaps held secretly by the server module.
If a request succeeds then the code in zmapView.c calls zmapViewStepListStepProcessRequest() which has the side effect of marking the current step as finished. Not doing this causes the step list to stop where it is as it is marked as waiting for a response. This implies that although the action REQUEST_ONFAIL_CONTINUE had been programmed it has never been run.
zmapServer.c tries to preserve SERVERDIED status by remembering if that response was received in the past from that server. Unfortunately some of the functions do not implement this (eg getStyles() stylesHaveModes() and loose this information: if a server dies in the Open call and we then carry on through the step list then calls such as getSequence() and getFeatures() would not return ZMAPTHREAD_RETURNCODE_SERVERDIED status despite code suggesting otherwise. (in fact they would hopefully return ZMAPTHREAD_RETURNCODE_REQFAIL status instead as programmed and this would be translated into ZMAPTRHEAD_REPLY_REQFAIL by zmapServerprotocolHandler.c)
acedbServer.c does return a spontaneous ZMAPTHREAD_RETURNCODE_SERVERDIED code for both getSequence and getFeatures(), but note that dasServer.c does not, and besides zmapServerProtocolHandler translates this into ZMAPTRHEAD_REPLY_REQFAIL regardless.
The return code ZMAPTHREAD_RETURNCODE_REQFAIL gets turned into ZMAPTHREAD_REPLY_REQERROR by zmapSlave.c. At the level below that zmapServerProtocolHandler.c turns the return code into ZMAPTHREAD_RETURNCODE_REQFAIL if it's not ZMAPTHREAD_RETURNCODE_OK or ZMAPTHREAD_RETURNCODE_UNSUPPORTED, although this varies between functions.
The solution is to make the server default to the last response, except for the getStatus call. Note that exisitng code that handles ZMAPTRHEAD_REPLY_REQERROR causes the step list to hang as mentioned above and needs fixing.
zmapServerProtocolHandler.c appears to have been coded to make up its own return codes if it does not get an OK status. This prevents any attempt so pass return codes from the actual server thread upsteam futile, and it is necessary to convert these codes explicitly.
The root cause of this is attempting to represent more than different kind of information with one value: script status, thread status, request response status all combined into one value used in zmapView.c and during transit through these layers they get mangled.
A further example of how this gets tied in impossible knots is that if we have an error in the GFF the server reports ZMAP_SERVERRESPONSE_REQFAIL wheihc gets mapped onto ZMAPTHREAD_RETURNCODE_REQFAIL which gets mapped onto ZMAPTHREAD_REPLY_REQERROR. This error does not get reported because the thread has run successfully.
Some of the requests are handled directly by zmapServerProtcolHandler.c eg GET_STYLES if a styles file is given. This does not go through the server code and a SERVERDIED status cannot be returned from this.
In zmapView.c checkStateConnections() polls the server thread step lists and on a status change (detected via the ZMAPTHREAD_REPLY_codes) will call zmapViewStepListStepProcessRequest() for ZMAPTHREAD_REPLAY_GOTDATA status, which is set if zmapServerProtcolhandler returns ZMAPTHREAD_RETURNCODE_OK. This only sets the step list state to finished if the process function returns success and if not this would hang. I didn't fix this one.
Returning response codes that correspond to actual failure states exposes an obscure bug in zMapServerProtocolHandler.c which contains the following code which has not enough brackets:
case ZMAP_SERVERREQ_NEWCONTEXT: { ZMapServerReqNewContext context = (ZMapServerReqNewContext)request_in ; /* Create a sequence context from the sequence and start/end data. */ if (thread_rc == ZMAPTHREAD_RETURNCODE_OK && (request->response = zMapServerSetContext(server, context->context) != ZMAP_SERVERRESPONSE_OK)) {
Working on this code I cannot escape from the feeling that people are laughing at me. It's also immensely frustrating that such a simple task can be so complex.
We ask a server what its's done with:
/* Call the registered slave handler function. */ slave_response = (*(thread->handler_func))(&(thread_cb->slave_data), request, &reply, &slave_error) ;
/* Signal that we failed. */ zmapVarSetValueWithErrorAndData(&(thread->reply), ZMAPTHREAD_REPLY_REQERROR, error_msg, request) ;
/* Sometimes for Errors we need to return some of the data as it included request * details and so on. */ void zmapVarSetValueWithErrorAndData(ZMapReply thread_state, ZMapThreadReply new_state, char *err_msg, void *data) { thread_state->state = new_state ; thread_state->reply = data ; }
gboolean zmapVarGetValueWithData(ZMapReply thread_state, ZMapThreadReply *state_out, void **data_out, char **err_msg_out) { *state_out = thread_state->state ; if (thread_state->reply) { *data_out = thread_state->reply ; } }
So to summarise:
I think that comes to 14:
include/ZMap/zmapServerProtocol.h zmapServer/zmapServer.c zmapServer/zmapServer.h zmapServer/zmapServerProtocolHandler.c zmapServer/zmapServerPrototype.h zmapServer/acedb/acedbServer.c zmapServer/das/dasServer.c zmapServer/pipe/pipeServer.c zmapServer/pipe/pipeServer_P.h zmapView/zmapView.c zmapControl/zmapControl.c include/Zmap/zmapView.h zmapView/zmapViewUtils.c zmapView/zmapView_P.h
Given a request from otterlace to load data we visit the following modules:
Due to the chosen implementation of a number of layers that communicate though one level only there are a number of different status codes that get translated as the progress from failure detection to reporting. In the case of the Open Connection request these are:
In the features_loaded XRemote message that current status code refers to the thread status not the script exit code. This is set to 1 for 'thread complete OK' and 0 otherwise. There is a message that gives helpful info on a good day These will remain as is.
Two extra will be added: <exit_code value="code" />> <stderr value="war and peace" /> "code" consists of the least significant 8 bits of the exit code from the script, or 0x100 if the script had an abnormal exit. If it is zero then we assume that the script completed OK.
<zmap> <request action="features_loaded"><client xwid=0x12345678\" /> <featureset names="EST_Gibbon" /> <start value="123456" /> <end value="234567" /> <status value="1" message="OK" /> <exit_code value="0" /> <stderr value=""> </request></zmap>
<zmap> <request action="features_loaded"><client xwid=0x12345678\" /> <featureset names="EST_Gibbon" /> <start value="123456" /> <end value="234567" /> <status value="0" message="GFF line 5 - Mandatory fields missing in: /> <exit_code value="0" /> <stderr value="here is my story"> </request></zmap>
<zmap> <request action="features_loaded"><client xwid=0x12345678\" /> <featureset names="EST_Gibbon" /> <start value="123456" /> <end value="234567" /> <status value="0" message="pipe script reports failure" /> <exit_code value="3" /> <stderr value="here is my story"> </request></zmap>
This should be trivial but experience shows that it involves a lot of code not immediately obvious. Other code modifications could require following the same trail, and looking at the command structure rigourously can highlight areas of code that work for present use only. Compilation did in fact show several paths through the code that were missed.
Why is the trail not obvious? it's partly due to the naming conventions not being followed and in fact are sometimes blatantly confusing.
Here we just write up the process of adding 'dataset' to request logic, this is a parameter that should be specified globally and passed to pipe server scripts, and is also needed for Blixem. We change the original interface that involves a lot of individual arguments passed to functions and use a data structure which will allow similar changes with much less work.
There are a few interfaces to feature request code (eg via XRemote, from the command line, from the ZMap config file, from the columns dialog) and to catch then all we need to start on the common path and work outwards. Starting at random in zmapAppconnect.c with the function
void zmapAppCreateZMap(ZMapAppContext app_context, char *sequence, int start, int end)
typedef struct { char *dataset ; /* eg human */ char *sequence ; /* eg chr6-18 */ int start, end ; /* chromosome coordinates */ } ZMapFeatureSequenceMapStruct, *ZMapFeatureSequenceMap ;
This gives us a paths through the code like these:
Note here that zmapViewLoadFeatures() takes separate start and end coordinates and no ZMapFeatureSequenceMap but does have the view which contains view->view_sequence which has the sequence name and dataset defined. This is necessary as we can request a subset of the data eg via request from mark.
createConnection() sets up a step list of requests for the view and these get executed eventually via checkStateConnections() - this is a break in the obvious trail through the code. Parameters are set up for each individual request by dispatchContextRequest() rather than on creating the request data structures in step list creation. We guess that this is a historical issue relating to ACEDB which returns data (eg styles) that we then pass back to allow the acedbServer module to function. This could be resolved by combining the different request structures into one, setting this up as a common parameter and updating this on each request. This would mean not having to call dispatchContextRequests().
Of relevance here is the dispatchContextRequests/ZMAP_SERVERREQ_OPEN which sets up start and end coordinates for the server, and this is where we have to add sequence name and dataset from the view->view_sequence struct. Then the request is process via the zmapThreads interface and there is a sequence of data structures that have data copied between them in a similar process to the one described above for returning status codes.
The ZMapFeatureSequenceMap contains the sequence name and dataset and can be extende to conatin further information if needed, so the best plan is to pass that struct through to the pipe servers. Note that it contains start and end coordinates of the sequence in question but we can request data for a part of the sequence and have to pass start and end separately (as implemented previously).
The first step is to add a ZMapFeatureSequenceMap to the ConnectionData struct, which is private to zmapView.c. This is used by zmapView.c to hold information set up by createConnection() and various requests in the step list so that it can be passed to requests by dispatchContextRequest(). Note that each request has a private data structure containing the information for that request only and that we could achieve the same reult more simply by making the ConnectionData struct accessable to the server code.
Step 2 is to add a ZMapFeatureSequenceMap to the ZMapServerReqOpen struct, which is defined in zmapServerProtocol.h and to copy from the ConnectionData struct in dispatchContextRequest().
Step 3 is in zmapServerProtocolHandler.c/zMapServerRequestHandler() where we take the data in the ZMapServerReqOpen struct and pass the field individually to zMapServerOpenConnection(). Bizzarely, zmapServer.c includes zmapServer_P.h which does not include zmapServerProtocol.h. This means that to add this extra parameter we have to change zmapServer.c and each of the server modules. To avoid having to do this again we can include zmapServerProtocol.h in the server module and use the data structures defined there. It's a public header. Ideally we could remove the request specific data structures and adapt the view's private ConnectionData structure (and then be able to add or change parameters easily) but that would involve changing a lot of code.
zMapViewCreate() is given information about a single sequence (the reference?) and stores this in a list of ZMapFeatureSequenceMap structs which looks like it can have only one member. zmapCreateView() takes the first one of these and assigns it to view->view_sequence. view->sequence_mapping gets to hold the list of the one sequence and there's a further assignment that looks out of date and possibly a lurking problem:
zmap_view->session_data->sequence = zmap_view->view_sequence->sequence ;
zmapCreateView() also calls addWindow() which calls zMapWindowCreate()with zmap_view->view_sequence. myWindowCreate() stores the master sequence in the window.
This of course needs to be converted to use ZMapFeatureSequenceMap. The sequence data is derived from the mark of from the block start and end coordinate, and sequence and dataset can be derived in a similar way if not defined (as at present) in XRemote.
This is called indirectly via window_cbs_G->command(). Note that zmapWindowFetchData() also appears in the code in zmapWindowRemoteReceive.c in iffed out code that has been moved to zmapViewRemoteReceive.c.
This corresponds to the user main window 'create' button ???? This is likely to change when XRemote is updated. dataset is defined from the app_context data now included in the ZMapStruct.
The following code does not work in a generalised way and uses defaulted values that don't operate in in as flexible a way as other parameters.
When started from otterlace ZMap does not read its configuration in a way that results in dataset or sequence being accessable. What was actually happening was that teh zmap view was created vvia the new_view XRemote command that didi not have access to the configuration data. This is a faairly obscure area to understand, refer to Application Callbacks for a summary.
ZMap should create a view window if the "win-id" arg is not given. The function that checks this did not work, instead reported that an argument was set if it was coded as a possible arg rather than given on the command line.