ACEDB Server/Client system

(Acedb 4.7 RPC version)


Introduction

Acedb consists of a collection of programs that either access the database files directly (e.g. tace, xace) or get access to the database by issuing commands across a network to a server program (e.g. aceserver/aceclient). This latter approach has several advantages:

Basis of client/server communication

Textual queries:

It's useful to understand just what information passes between the client and server. There is an implicit layering in all of the acedb programs in that they all provide an interface whereby queries in plain text form, e.g.

		"find model"
		"list"
		etc.
can be passed to "interpreter" code which will then call the low level database routines required to fulfill those commands. It is at this level that the split between client and server is made. The client reads commands typed in by the user (or perhaps read from a file containing a collection of commands that the user has supplied), and then passes these commands across the network to the server. The server contains the "interpreter" code which scans the commands and makes calls to the low level database routines to access the actual database. The server then constructs a reply as plain text which it sends back to the client, the client displays this reply to the user. In this way there is no difference really between tace and aceclient, both are command line based, it's just that aceclient accesses the database via a server across the network.

Graphical displays - an aside

Graphical display is more difficult because the graphical code, for performance reasons, must itself call low level routines to access information in the database. There are however currently two methods of graphical display:

These interfaces will not be demonstrated here, the former because it is experimental, the latter because it is quite complex to set up and beyond the scope of this course.

Things you need to understand to use the server/clients

Server Instances, Port numbers and databases

If you are familiar with Unix/computers then skip this paragraph, otherwise..... We tend to think of a program as the single file on the disk that contains that program. It is more correct however to think of "instances" of a running program: many copies of the same program can be running at the same time. This is important when thinking about the server because each time a server is started up it can only access one database. Hence to allow access to many databases on a single machine, there may be many instances of the server program running at the same time, each accessing a different database.

Now, which instance of the server do you contact to access a particular database ? To understand this we need to know a bit about port numbers.

Computers make sure that the many different types of communications they receive over the network go to the correct programs by associating each program with a unique "port" number, e.g. if you look in the file /etc/services you can see a list of port numbers/programs, some of which are essential operating system services. For now all you need to know is two things:

  1. port numbers below 1024 are reserved for operating system use and you should not try to use them (the operating system will probably prevent you from using one), otherwise you can use any number that is not mentioned in /etc/services.
  2. When we start up the server we tell it what database we want to access AND which port number it is to use.

You can probably already see that the port number is like a "key" to the database. When the client contacts a server on a particular port number, the server on the other end of that port will be accessing a particular database. To access a different database we could start up another instance of a server running on a different port and using a different database.

So when you start up the client you do not say which database you want to access, instead you specify the machine you want to contact and which port on the machine you want to connect to.

(NOTE that the client and server can be on the same or different machines, the commands are the same.)

An example:

(In this and other examples "YOURDB" stands for a whatever directory you may have set up your database in, e.g. /home/johnsmith/mydatabase, so that if you see "YOURDB/wspec" it means /home/johnsmith/mydatabase/wspec)

Let's say we are working on the machine testbed and start up the server like this:


> aceserver YOURDB 2000

then the client can be started up like this:


> aceclient  testbed -port  2000

The client will now be able to access YOURDB via the server on testbed on port 2000. Note how the client needs to be told the machine and port number and then the database that will be accessed is implicit from this because of how the server was started up.

Trying out the server/client code

Simple set up of the server and client

There are several things you need to do before you can try out the client/server code:

  1. A database must be set up (you may need to use tace to do this).
  2. Files/directories must be set up for user authentification
  3. The server must be started up and ready to receive requests.
  4. The client must be started up so that it connects to the server.

For this initial exercise we will set up the client/server as simply as possible, glossing over some of the details (see Appendix A for a full description).

Step 1)

We'll assume that step 1) has been done and that the database has been set up by you in some directory of your choice, e.g. YOURDB

Step 2)

At this stage we will use the simplest option and make the database readable by anybody so that we can get on and try out the server/client. To do this you need to edit the file YOURDB/wspec/server.wrm and edit the line that begins READ_ACCESS_DIRECTORY so that it looks like this:


READ_ACCESS_DIRECTORY PUBLIC

This tells the server to let anyone have read access to the database.

Step 3)

The simplest way to start the server is simply by starting it at a terminal, this is what we will do. Make sure you have at least two xterms running on your X Windows screen. Decide what port number you want to use (must be greater than 1023 and not in /etc/services) and then start the server by typing:


> aceserver  /your/database/directory  your_port_number

If the server has started correctly then the xterm prompt will not be redisplayed, the xterm will "hang" until the server program exits.

Step 4)

You now need to start the client in a separate xterm window. Since you have started the server on your own machine you now need to know what the name of your machine is. You can find this out by typing the command "hostname":


> hostname

this should return the name by which your machine is known on the network.

Now start the client by typing:


> aceclient your_machine_name -port your_port_number

If you are successful then you should see output like this:


acedb@your_machine_name>

you can now type commands in at the "acedb" prompt to explore the database.

Trying out the server/client

tace and aceclient share the same command interface which enables you to to interact with the database, do the following to get an idea of the scope of the interface:


acedb@your_machine_name> ?
- will give you a list of all possible commands

acedb@your_machine_name> help
- will give you a brief overview of the sets of commands and what they do

acedb@your_machine_name> help topic
- will give you more detailed help on a particular topic/command

Try out a few commands:


acedb@your_machine_name> classes
- this gives a list of all the visible class names and how many objects they contain

To see the model or schema for a class try the "model" command:


acedb@your_machine_name> model oligo
- will show the model for the oligo class, you should see something like:

?Oligo  Sequence UNIQUE Text    // verbatim sequence - useful
        In_sequence ?Sequence XREF Oligo
        STS STS1 UNIQUE ?STS XREF Oligo1
            STS2 UNIQUE ?STS XREF Oligo2

When you want to stop the client simply type:


acedb@your_machine_name> quit

You could try getting access from another machine by using an xterm on someone elses machine and once again issuing the command


> aceclient your_machine_name -port your_port_number

Note how the same command gets you access to the server from a completely different machine. All that is needed are the machine name and port number. Provided your network security system allowed it, you could even use your server from a machine in a different country although you would need to use the long form of the machine name, e.g. your_machine_name.hgmp.ac.uk or whatever.

To stop the server, go to the xterm running the server and press and hold down the 'Ctrl' key and press the 'c' key. This will send a special signal to the server program that will cause it to exit (later you will learn of a better way to stop the server).

Summary

To recap, you have learnt the basic set up of the database and server/client programs, how to start the programs, basic use of the client and how to stop the programs. The following sections will discuss in more detail how to control access to a database, how to shut it down properly, how to monitor the server, and how to tell the operating system to start up the server automatically every time a client connects to the port for your database.

Getting write access

So far you have only given read access to clients signing on. Now we will go through the basic steps to get write access (once again glossing over some of the details in the interests of making things simple to start off with).

The server controls access in the following way:

When a client contacts the server, the server writes a file containing a special number into a directory that is on the shared file system common to both the servers and the clients machines. (Since NFS comes as standard with most Unix systems this should not be a problem.) If the userid of the client has the correct Unix file permissions it will be able to read this file and get the special number contained in the file. The client can then return this number to the server which acts like a key and will tell the server that it is OK to give the client access to the database. An example will show how this works:

  1. Make a directory and, to make life simple, set the permissions of the directory so that anyone can do anything to the directory:
    
           > mkdir your_directory_name
           > chmod ugo+rwx your_directory_name
           
  2. Now edit the file YOURDB/wspec/server.wrm and edit the line that begins WRITE_ACCESS_DIRECTORY so that it looks like this:
    
         WRITE_ACCESS_DIRECTORY your_directory_name
         
    (Note that you must give the full path name of the directory. )

When the server is contacted by a client it will now write the file containing the magic number in this directory. Your client will then be able to read it and send the number back to the server to get write access. Now we need to try this out.

Start the server as before:


> aceserver  /your/database/directory  your_port_number

Now start the client:


> aceclient your_machine_name -port your_port_number

To find out if you have write access, type in the "save" command:


acedb@your_machine_name> save

If you have write access you will see a message like this:


// Response: 24 bytes.
// 0 Active Objects

If not, you will see a message explaining that you do not have write access:


// Reponse: 64 bytes.
// Sorry, you do not have Write Access.
// 0 Active Objects

If you did not get write access then check through each step carefully and try again.

Although we have used this mechanism to provide write access we can also use the same mechanism to control read access by creating a new directory and using it with the READ_ACCESS_DIRECTORY keyword (in wspec/server.wrm) instead of the special "PUBLIC" keyword we used in the previous example. The server will check first for write access and then for read access.

You should also note that access can be very strictly controlled (in contrast to what we have done) so that only certain groups of users can have read or write access. Appendix A to this document explains how to do this using Unix group and file permissions. It's enough for now to note that the following combinations are possible:


					READ ACCESS
			no one		group		anyone

	no one		not allowed	OK		OK

WRITE	group		OK		OK		OK
ACCESS
	anyone		OK		OK		not allowed

The only two combinations disallowed are that no one has read/write access or that everyone has read/write access, the former renders the database inaccessible, the latter is just not a good idea !!

Shutting down the server

In a previous example we used Cntl-C to send a signal to the server which forced it to exit. This should only be used as a last resort if you are running the server on an important database. It is much better to use the "shutdown" command which will cause the server to refuse any new client connections and close down the database in an orderly way once the last client has quit. Only users with write access permission are allowed to do this, you could try the command out now that you know how to get write access.

Monitoring the server

There are two files used by the server to log errors/information:


YOURDB/database/log.wrm
YOURDB/server.log

log.wrm is used by all acedb programs that access the database to record information and errors. If your program crashes then the cause of the crash will be recorded in this file unless the crash was so bad that the program was forced to exit immediately. xace, tace and the aceserver all log errors in this file.

server.log is used by the server to record useful information that may help with debugging or show how many clients have logged on and so on.

You should look in these files if you have problems such as the server not starting, not responding normally or crashing.

It is also useful to use the Unix ps command to look at what the server process is doing, assuming the server program is still called aceserver:


> ps -l

will display something like this:


   F S  UID   PID  PPID    C PRI  NI  ADDR   SZ WCHAN  TTY      TIME CMD

8001 S 9767 17388 17337  1.2  44   0     0 7.3M event  ttyp8 0:00.35 aceserver
                  < lots more process entries >

The most useful fields are the SZ and WCHAN fields:

SZ will tell you how much memory the server is using, if you suspect that
   the server is running away and eating up more and more memory it will be
   obvious here (the aceserver is 7.3MB in the above example).

WCHAN gives you an idea of what the server process is doing, in the above
   example it is waiting for an "event", i.e. the server is effectively
   asleep and waiting for a client to connect.

You could try out this command before, during and after a client connection.

"top" is another useful command if your system supports it. It will show you a dynamic display of the top users of CPU on your system. If you suspect the server is looping out of control you can use this command to check this. In the below example you can see that the aceserver is not currently active.


> top

will display something like this:


-----------------------------------------------------------------------
load averages:  0.19,  0.08,  0.07    
109 processes: 1 running, 35 sleeping, 73 idle
CPU states:  7.5% user,  0.0% nice,  8.2% system, 84.2% idle
Memory: Real: 298M/494M act/tot  Virtual: 38M/852M use/tot  Free: 33M

  PID USERNAME PRI NICE  SIZE   RES STATE   TIME    CPU COMMAND
16721 lmc       44    0   33M   19M sleep   0:52  3.10% real-netscape
  640 root      42   -2   16M 9928K sleep   2:47  0.40% Xdec
17389 edgrif    44    0 4576K  745K sleep   0:00  0.20% aceserver
                    etc. etc.

Documentation

As well as the help files in YOURDB/whelp, you will find additional information about various aspects of programming and installing ACEDB in YOURDB/wdoc. (See the document RPC_aceserver.html for server details) You can also find more documentation from the acedb web site (http://www.acedb.org) which is hosted at the Sanger site and can be accessed from there as well (http://www.sanger.ac.uk).

Inetd and the server

So far we have run the server by starting it from an xterm, this is fine for testing but not satisfactory for providing a database service because we would like users to have access to the database whether we are logged on to the machine or not. For this we would like to have server startup automated and fortunately on Unix systems this can be done via an operating system service called inetd (which is short for "Internet Daemon"). inetd is a program that will start up server programs such as the aceserver when a client request arrives at the servers machine. Briefly it does this by listening on all the ports that these servers would normally use and then when a request comes in on a port it starts up the server that should service that port and lets it deal with the request. This section shows what you need to do to have inetd start up the aceserver.

**** TAKE CARE ****

Because you will either need root access on your machine or you will need to get your system support people to do this, we will not actually edit the required files. If you want to try this out you should make sure that you do this with the help of someone who knows how to safely edit operating system files, it is possible to leave your system in a state where it may not reboot if you do not alter these files correctly.

**** TAKE CARE ****

For this example we will assume the following:

our machine name is:                       redbox
our server program is the file:            /home/me/aceserver
the database we want to access is:         /home/mydatabase
the port we are going to use is:           2001
the nickname for this server/database is:  allgenes
the owner of the server program is:        joebloggs
(you will see these bits of information highlighted in the examples below)

We will need to alter several files in the /etc directory which is where the configuration files for most Unix operating systems are kept.

The first step is to tell the operating system which port you want to use, what protocol to use and what symbolic name we want to give to our service, to do this we need to add the following line to the bottom of /etc/services:


allgenes    2001/tcp

Now we need to tell rpc which port we are using and what our symbolic name is by adding this line to the bottom of the file /etc/rpc:


allgenes    2001

Finally we need to tell inetd which program to start up when a request comes in on the port we have named


allgenes stream rpc/tcp wait joebloggs /home/me/aceserver (contd..)
                              aceserver /home/mydatabase 2001

Any bits that are not highlighted will be the same for any aceserver entry in these files, remember that you will need a new set of entries for each new database you want to make available via a server.

What port numbers should I choose:

    0 to  1023: are reserved for operating system use. DO NOT USE THESE.

 1024 to 49151: are "registered" ports. If you setting up the server to
                be run by inetd you should use one of these (you will need
                to check in /etc/services that your number does not clash
                with one already there)

49152 to 65535: are for casual/ephemeral use. You should use one of these
                if you are just trying out the server from an xterm

Now you should be able to access the server in the same way as before by typing in:

> aceclient  redbox -port  2001

If you have trouble then check you all the steps above and when you are sure they are correct then refer to the trouble shooting guide in YOURDB/wdoc/RPC_aceserver.html

In the Future

In acedb 4_8 a sockets based version of the server will be released, although the interface is essentially the same there are a number of differences: