ZMap Server layering

Orientation

ZMap implements 'servers' which provide access to databases either via an interactive connection (eg ACEDB) or a passive one (eg pipeServers) and a few layers of software have been implemented to allow this. These layers have thoer own private data and information is passed bewteen the layers via callback functions which are driven by a number of requests and responses defined in zmapServerProtocolHandler.c. At each layer requests and responses are copied from one data structure to another and request codes and responses code are translated to something else.

A further section below deals with passing request coordinates to a server.

Hopefully this little document may help someone else find their way through the code.

An example: Passing an exit code and STDERR from a pipe to Otterlace

Here we document what has to be done to add new data from an external script and pass this through to Otterlace, which will take us from the script output itself through the layers of server code and via the view and control code back to Otterlace. Adding parameters from Otterlace or ZMap config or user action requires a similar process but in reverse.

The present example is due to a request from Anacode to have the entire output of a script's STDERR reported instead of a nominal single line error message. As the existing single line error message has to remain (eg to report other errors) we cannot reuse this and have to implement an extra data item. We also require the exit code of the script to ensure that data is valid (and complete).

Note that returning STDERR is no more complex than returning an exit code - we simple read all of it and assume we have enough memory

A script reports an error

In case of a failure a pipe server script will produce diagnostic information on STDERR, and this may occur before or during normal data output. If ZMap reads the whole stream without an error it will report success, but if not it will read the pipe script's STDERR. Previously (if there was an error) the last non blank line of this was reported as the error message and now the requirement is to output all of this on ZMap's STDERR and also send it to Otterlace via XRemote.

In this case we will report an error message such as "script failed".

There is a function pipe_server_get_stderr() that takes a PipeServer struct as argument which is called in case of any error. This can be left as is to read STDERR whenever an explicit error occurs and also called at normal EOF on the pipe script's STDOUT.

So step 1 is to add some code to read STDERR and set an error condition if this is non-null.

Change of spec! ... Interfacing to the ZMap server protocol

There's a new requirement to return STDERR always and return the exit code of pipe Server scripts. In case of a non zero exit code the data should be dropped as it may be incomplete, and this could influence annotation.

ZMap interfaces to servers via a series of requests the fetch styles, DNA, and features, and to check the script's exit code and return STDERR it is neatest to add a new step to this protocol.

Add a new request type and data structure

In zmapServerProtocol.h add an enum for the request type and a ZMapServerReqStatus data structure. Add this to ZMapServerReqUnion (in zmapServerProtocol.h) and zMapServerRequestCreate() and zMapServerCreateRequestDestroy() (sic) and zMapServerRequestHandler() in zmapServerProtocolHandler.c. and then add a function like getStatus() to call the server function zMapServerGetStatus() (that you have to add) which is itself a wrapper for the server->funcs->get_sequence() call.

Why are there all these layers? AFAICS it's because each one has it's own data which is out of scope in all the others. Note that the zmapServer layer holds the request and response data in the server request data structure that differs for each request type and the actual remote server module that processes the request does not have access to this data structure and handles each data item explicitly.

Add more functions to allow servers to implement the interface

Now add a server function to the ZMapServerFuncs structure in zmapServerPrototype.h and add a matching function prototype. Note that to follow the existing design it is necessary to have all the data outputs specified as arguments. Revist the code mentioned above and adjust the calls to match.

Add code for this function to zMapServerGlobalInit() in zmapServer.c. Note that this implies that all server modules must implement this function.

Add get_status() functions to each of the server modules

In pipeServer.c add the get_status function to pipeGetServerFuncs() and write the matching getStatus() function based on the prototype defined above. For pipe servers take into account that they also implement file servers and need to take into account that there is no exit code or STDERR in this case.

Do the same in acedbServer.c and dasServer.c

Add this call to the View's step list when appropriate

This is done by calling zmapViewStepListAddServerReq() in createConnection() when we are running a script that is expected to terminate, which for the moment is all pipe scripts. As we can only get an exit code from a terminated script this seems fairly safe. Note that the terminate call itself is for the ZMap thread that runs the script and responds to the various requests above.

Add the request type to dispatchContextRequests() and processDataRequests().

Adjust the display_after flags to be this new request where appropriate (in createConnection())

Get the exit code and STDERR and report to Otterlace

Features are displayed via the getFeatures() in processdataRequests() and we have to do the same stuff for ZMAP_SERVERREQ_FEATURES and ZMAP_SERVERREQ_GETSTATUS depending on whether or not we get the exit code (we don't for ACEDB, it doesn't exit). This is independant of reporting 'features_loaded' to Otterlace, which is something that happens via checkStateConnections(). This function tests the server thread status and picks up on error conditions at any point in the series of requests, and we report status to Otterlace when the thread has completed its step list of requests.

This is done by a callback set by zmapControl.c calling zmapViewInit() and the function we end up in via view_cbs_G->load_data() is dataLoadCB()

The problem to solve now is to get the exit code and STDERR output which is available from the getStatus() function to the dataLoadCB function and this requires the use of two more data structures:

The ConnectionData struct is held for a Server connectiion and holds various items derived from individual requests. LoadFeaturesStruct appears to have been created specifically to pass info to our callback function. We have to copy the information we want from one structure (ZMapServerReqGetStatus) to another ConnectionData to get round scope issues and the from that one to another LoadfeaturesData to get round some more scope issues.

A problem

There is a slight problem reporting errors from any particular step in the series of requests in that the exit code and STDERR output are recovered by an explicit request and the existing code aborts making these requests in case of an error. Existing code did not test the script's return code and only detected failure in the Open call by failing to read a valid GFF header. Obviously a script could fail at any point and we need to detect failure at each possible step int he request protocol.

The current interface consists of different data structures for each request and repsonse and to provide an error response for each request with ancillary data would require changing the code for each request in all the server modules and at each layer of the server interface and the calling code in zmapView.

There is another complication in that the code in zmapSlave.c aborts the thread if the external server dies and as it stands we cannot return the exit code or STDERR output by the existing interface.

To fix this we can stop the slave thread from exiting and continue to process other requests. Note that zmapServer.c only makes requests if the previous repsonse was not ZMAP_SERVERRESPONSE_SERVERDIED. This implies that the pipeServer should set ZMAP_SERVERRESPONSE_SERVERDIED rather than ZMAP_SERVERRESPONSE_REQFAIL.

What would be better would be the have the pipe server check the status of the external script. This could be done by adding a callback to report the child process exiting and report ZMAP_SERVERRESPONSE_SERVERDIED if that happened. However, this would happen asyncronously and the likely result would then be a race condition, with some of the failures operating as at present.

A practical solution

To avoid a) rewriting the server code and b) making a large number of global changes the following will be done:

Another problem

In zmapView.c when replies are received to requests there is a flag that controls whether or not to display the data recieved. This previously could be set for sequence or features requests, both of which were handled by the same response code in zmapView.c. They use different data structures but both are accessed as ZMapServerReqGetFeatures. (this is a bug).

The flag 'display_after' is misnamed as it controls storing the returned data (merging into the View's context) and then displaying it. If we flag the getStatus request as the one to display after then by then we have lost the data that was returned by getFeatures.

We could query the exit status fo the script before requesting features but this could lead to a deadlock where the script is de-scheduled due to waiting to write to a pipe that is not being read, and we have to store the features data in the ConnectionData struct for use later.

Another problem

Due to a previous implementation where the View's step list data structures were used to ensure that all active connections did each request in step and all finished at the same time attempts to add a getStatus call by using the function zmapViewStepListAddServerReq() failed as this requires zmapViewStepListAddStep(step_list, ZMAP_SERVERREQ_GETSTATUS, REQUEST_ONFAIL_CANCEL_THREAD) to have been called first. This is hidden away in zmapViewConnectionStepListCreate().

This implies that the current implementation prevents re-ordering of requests for different types of server. Note: some of the requests are required to be in the given order as some require data retrieved by a previous request. Some of this may be passed in arguments or perhaps held secretly by the server module.

Another problem

If a request succeeds then the code in zmapView.c calls zmapViewStepListStepProcessRequest() which has the side effect of marking the current step as finished. Not doing this causes the step list to stop where it is as it is marked as waiting for a response. This implies that although the action REQUEST_ONFAIL_CONTINUE had been programmed it has never been run.

More confusion

zmapServer.c tries to preserve SERVERDIED status by remembering if that response was received in the past from that server. Unfortunately some of the functions do not implement this (eg getStyles() stylesHaveModes() and loose this information: if a server dies in the Open call and we then carry on through the step list then calls such as getSequence() and getFeatures() would not return ZMAPTHREAD_RETURNCODE_SERVERDIED status despite code suggesting otherwise. (in fact they would hopefully return ZMAPTHREAD_RETURNCODE_REQFAIL status instead as programmed and this would be translated into ZMAPTRHEAD_REPLY_REQFAIL by zmapServerprotocolHandler.c)

acedbServer.c does return a spontaneous ZMAPTHREAD_RETURNCODE_SERVERDIED code for both getSequence and getFeatures(), but note that dasServer.c does not, and besides zmapServerProtocolHandler translates this into ZMAPTRHEAD_REPLY_REQFAIL regardless.

It's even more confusing than that

The return code ZMAPTHREAD_RETURNCODE_REQFAIL gets turned into ZMAPTHREAD_REPLY_REQERROR by zmapSlave.c. At the level below that zmapServerProtocolHandler.c turns the return code into ZMAPTHREAD_RETURNCODE_REQFAIL if it's not ZMAPTHREAD_RETURNCODE_OK or ZMAPTHREAD_RETURNCODE_UNSUPPORTED, although this varies between functions.

The solution is to make the server default to the last response, except for the getStatus call. Note that exisitng code that handles ZMAPTRHEAD_REPLY_REQERROR causes the step list to hang as mentioned above and needs fixing.

The main problem is that return codes are ignored and replaced by something else

zmapServerProtocolHandler.c appears to have been coded to make up its own return codes if it does not get an OK status. This prevents any attempt so pass return codes from the actual server thread upsteam futile, and it is necessary to convert these codes explicitly.

The root cause of this is attempting to represent more than different kind of information with one value: script status, thread status, request response status all combined into one value used in zmapView.c and during transit through these layers they get mangled.

A further example of how this gets tied in impossible knots is that if we have an error in the GFF the server reports ZMAP_SERVERRESPONSE_REQFAIL wheihc gets mapped onto ZMAPTHREAD_RETURNCODE_REQFAIL which gets mapped onto ZMAPTHREAD_REPLY_REQERROR. This error does not get reported because the thread has run successfully.

Another obscurity

Some of the requests are handled directly by zmapServerProtcolHandler.c eg GET_STYLES if a styles file is given. This does not go through the server code and a SERVERDIED status cannot be returned from this.

Another obscure gotcha

In zmapView.c checkStateConnections() polls the server thread step lists and on a status change (detected via the ZMAPTHREAD_REPLY_codes) will call zmapViewStepListStepProcessRequest() for ZMAPTHREAD_REPLAY_GOTDATA status, which is set if zmapServerProtcolhandler returns ZMAPTHREAD_RETURNCODE_OK. This only sets the step list state to finished if the process function returns success and if not this would hang. I didn't fix this one.

An obscure old bug

Returning response codes that correspond to actual failure states exposes an obscure bug in zMapServerProtocolHandler.c which contains the following code which has not enough brackets:

    case ZMAP_SERVERREQ_NEWCONTEXT:
      {
        ZMapServerReqNewContext context = (ZMapServerReqNewContext)request_in ;

      /* Create a sequence context from the sequence and start/end data. */
      if (thread_rc == ZMAPTHREAD_RETURNCODE_OK
          && (request->response = zMapServerSetContext(server, context->context)
            != ZMAP_SERVERRESPONSE_OK))
        {
which sets the response code to 1 which is ZMAPSERVER_RESPONSE_BARDREQ, which then causes the server thread to abort. BADREQ is never set in the source code so it was diffcult to find.

A personal view

Working on this code I cannot escape from the feeling that people are laughing at me. It's also immensely frustrating that such a simple task can be so complex.

with code like this is there any hope?

We ask a server what its's done with:

	  /* Call the registered slave handler function. */
	  slave_response = (*(thread->handler_func))(&(thread_cb->slave_data), request, &reply, &slave_error) ;
and never look at reply.
It tell us what it's done with:
	/* Signal that we failed. */
	zmapVarSetValueWithErrorAndData(&(thread->reply), ZMAPTHREAD_REPLY_REQERROR, error_msg, request) ;
that gets handled with:
/* Sometimes for Errors we need to return some of the data as it included request
 * details and so on. */
void zmapVarSetValueWithErrorAndData(ZMapReply thread_state, ZMapThreadReply new_state,
				     char *err_msg, void *data)
{
  thread_state->state = new_state ;

  thread_state->reply = data ;
}
that is accessed with:
gboolean zmapVarGetValueWithData(ZMapReply thread_state, ZMapThreadReply *state_out,
				 void **data_out, char **err_msg_out)
{
      *state_out = thread_state->state ;

      if (thread_state->reply)
	{
	  *data_out = thread_state->reply ;
	}
}
and in the calling code there is a local variable called thread_state which refers to the thread state as opposed to the ZMapReply 'thread_state' that contains the original request.

So to summarise:

Passing sequence coordinates to a server

Why document this?

This should be trivial but experience shows that it involves a lot of code not immediately obvious. Other code modifications could require following the same trail, and looking at the command structure rigourously can highlight areas of code that work for present use only. Compilation did in fact show several paths through the code that were missed.

Why is the trail not obvious? it's partly due to the naming conventions not being followed and in fact are sometimes blatantly confusing.

Here we just write up the process of adding 'dataset' to request logic, this is a parameter that should be specified globally and passed to pipe server scripts, and is also needed for Blixem. We change the original interface that involves a lot of individual arguments passed to functions and use a data structure which will allow similar changes with much less work.

Where to start?

There are a few interfaces to feature request code (eg via XRemote, from the command line, from the ZMap config file, from the columns dialog) and to catch then all we need to start on the common path and work outwards. Starting at random in zmapAppconnect.c with the function

void zmapAppCreateZMap(ZMapAppContext app_context, char *sequence, int start, int end)
we need to change this to use:
typedef struct
{
  char *dataset ;       /* eg human */
  char *sequence ;      /* eg chr6-18 */
  int start, end ;      /* chromosome coordinates */
} ZMapFeatureSequenceMapStruct, *ZMapFeatureSequenceMap ;

and then follow through with the functions that call this and get called. Note that start and end may be diffferent from startup configuration.

This gives us a paths through the code like these:

ZMap starts up

Otterlace starts up ZMap
zmapAppRemote.c application_execute_command()
zmapAppremote.c createZMap() See note 3
zmapAppconnect.c zmapAppCreateZMap()
Otterlace opens a new ZMap view
zmapControlRemoteReceive.c control_execute_command()
zmapControlRemoteReceive.c insertView() See note 5
zmapControl.c zMapAddView()
ZMap starts from the command line
zmapAppMain.c main()
zmapAppwindow.c zmapMainMakeAppWindow()
zmapAppconnect.c zmapAppCreateZMap()
The user clicks create in then main window
zmapAppConnect.c createThreadCB()
zmapAppconnect.c zmapAppCreateZMap()

ZMap creates a view window and stores sequence info for later use

zmapAppconnect.c zmapAppCreateZMap() creates the main (manager) window
zmapManager.c zMapManagerAdd()
We drop through here or land from insertView() above
zmapControl.c zMapAddView()
zmapControl.c zmapControlAddView()
zmapControlViews.c zmapControlNewWindow()
zmapView.c zMapViewCreate()
zmapView.c createZMapView() See note 1
See note 2
zmapView.c addWindow() creates the ZMap window that the users see
zmapWindow.c zMapWindowCreate()
zmapWindow.c myWindowCreate() See Note 2

Different interfaces for requesting data

Requesting data from the stored sequence info - automatically requests initial data
zmapManager.c zMapManagerAdd()
zmapControl.c zMapConnectView()
zmapView.c zMapViewConnect()
zmapView.c zmapViewLoadFeatures()
Otterlace requests ZMap to load new features
zmapViewRemoteReceive.c view_execute_command()
zmapViewRemoteReceive.c loadFeatures() See note 3
zmapView.c zmapViewLoadFeatures()
The user requests data via the columns dialog
zmapWindow.c zmapWindowFetchData()
zmapView.c commandCB() see Note 4
zmapView.c zmapViewLoadFeatures()

Actually requesting the data

zmapView.c zmapViewLoadFeatures()
zmapView.c createConnection()
zmapView.c dispatchContextRequest() Called indirectly by an event

Note here that zmapViewLoadFeatures() takes separate start and end coordinates and no ZMapFeatureSequenceMap but does have the view which contains view->view_sequence which has the sequence name and dataset defined. This is necessary as we can request a subset of the data eg via request from mark.

createConnection() sets up a step list of requests for the view and these get executed eventually via checkStateConnections() - this is a break in the obvious trail through the code. Parameters are set up for each individual request by dispatchContextRequest() rather than on creating the request data structures in step list creation. We guess that this is a historical issue relating to ACEDB which returns data (eg styles) that we then pass back to allow the acedbServer module to function. This could be resolved by combining the different request structures into one, setting this up as a common parameter and updating this on each request. This would mean not having to call dispatchContextRequests().

Of relevance here is the dispatchContextRequests/ZMAP_SERVERREQ_OPEN which sets up start and end coordinates for the server, and this is where we have to add sequence name and dataset from the view->view_sequence struct. Then the request is process via the zmapThreads interface and there is a sequence of data structures that have data copied between them in a similar process to the one described above for returning status codes.

Getting the sequence info to the pipe servers

The ZMapFeatureSequenceMap contains the sequence name and dataset and can be extende to conatin further information if needed, so the best plan is to pass that struct through to the pipe servers. Note that it contains start and end coordinates of the sequence in question but we can request data for a part of the sequence and have to pass start and end separately (as implemented previously).

The first step is to add a ZMapFeatureSequenceMap to the ConnectionData struct, which is private to zmapView.c. This is used by zmapView.c to hold information set up by createConnection() and various requests in the step list so that it can be passed to requests by dispatchContextRequest(). Note that each request has a private data structure containing the information for that request only and that we could achieve the same reult more simply by making the ConnectionData struct accessable to the server code.

Step 2 is to add a ZMapFeatureSequenceMap to the ZMapServerReqOpen struct, which is defined in zmapServerProtocol.h and to copy from the ConnectionData struct in dispatchContextRequest().

Step 3 is in zmapServerProtocolHandler.c/zMapServerRequestHandler() where we take the data in the ZMapServerReqOpen struct and pass the field individually to zMapServerOpenConnection(). Bizzarely, zmapServer.c includes zmapServer_P.h which does not include zmapServerProtocol.h. This means that to add this extra parameter we have to change zmapServer.c and each of the server modules. To avoid having to do this again we can include zmapServerProtocol.h in the server module and use the data structures defined there. It's a public header. Ideally we could remove the request specific data structures and adapt the view's private ConnectionData structure (and then be able to add or change parameters easily) but that would involve changing a lot of code.

Note 1 - lists of sequences held by the view

zMapViewCreate() is given information about a single sequence (the reference?) and stores this in a list of ZMapFeatureSequenceMap structs which looks like it can have only one member. zmapCreateView() takes the first one of these and assigns it to view->view_sequence. view->sequence_mapping gets to hold the list of the one sequence and there's a further assignment that looks out of date and possibly a lurking problem:

  zmap_view->session_data->sequence = zmap_view->view_sequence->sequence ;

Note 2 - passing the sequence to the window

zmapCreateView() also calls addWindow() which calls zMapWindowCreate()with zmap_view->view_sequence. myWindowCreate() stores the master sequence in the window.

Note 3 - getting sequence data from XRemote

This of course needs to be converted to use ZMapFeatureSequenceMap. The sequence data is derived from the mark of from the block start and end coordinate, and sequence and dataset can be derived in a similar way if not defined (as at present) in XRemote.

Note 4 -

This is called indirectly via window_cbs_G->command(). Note that zmapWindowFetchData() also appears in the code in zmapWindowRemoteReceive.c in iffed out code that has been moved to zmapViewRemoteReceive.c.

Note 5 - XRemote interface via control (main) window

This corresponds to the user main window 'create' button ???? This is likely to change when XRemote is updated. dataset is defined from the app_context data now included in the ZMapStruct.

Loose ends of code

The following code does not work in a generalised way and uses defaulted values that don't operate in in as flexible a way as other parameters.

zmapView.c zmapViewCreate() zmap_view->session_data->sequence = zmap_view->view_sequence->sequence ;
Needs dataset as well
zmapAppconnect.c createThreadCB() dataset needs to be defined via a widget
zmapAppremote.c createZMap() dataset needs to be defined via XML
zmapAppremote.c insertView() See note 5
zmapControlWindowMenubar.c newCB() As for note 5: incomplete configuration

A problem

When started from otterlace ZMap does not read its configuration in a way that results in dataset or sequence being accessable. What was actually happening was that teh zmap view was created vvia the new_view XRemote command that didi not have access to the configuration data. This is a faairly obscure area to understand, refer to Application Callbacks for a summary.

Another problem

ZMap should create a view window if the "win-id" arg is not given. The function that checks this did not work, instead reported that an argument was set if it was coded as a possible arg rather than given on the command line.