New Repository

Current repository is disk based

experiment and trace files in project directories

New repository is based around an Oracle database

experiement files replaced by TRACEINFO, sequence and quality tables

traces stored in a trace table

Still have project directories containing

phrap output

CAF files

gap4 databases

fofn etc.

Access to TRACEINFO, sequence, quality and traces is through a database call

Based on existing TraceArchive structure

560 million traces

16 Tbytes of disk

15-30 million traces per month

up to 1.5 Tbyes per month

Access is via a Web page for single traces

Pfetch for multiple (<500) traces

fetchWGS

Glovar

Human chromosome SNP project

Gap4

via the RAWDATA environmental variable

Database

TRACEINFO

SEQUENCE

QUALITY

TRACE

TRACEINFO – many indexes

All queries run on this table – ID's

species

project

library

All other tables accesed using ID

ID1 ... ... ...

ID2 ... ... ...

ID3 ... ... ...

ID4 ... ... ...

ID4 ... ... ...

ID5 ... ... ...

ID6 ... ... ...

ID7 ... ... ...

ID1 ... ... ...

ID2 ... ... ...

ID3 ... ... ...

ID4 ... ... ...

ID5 ... ... ...

ID6 ... ... ...

ID7 ... ... ...

ID8 ... ... ...

ID9 ... ... ...

ID10 ... ... ...

ID1 ... ... ...

ID2 ... ... ...

ID3 ... ... ...

ID4 ... ... ...

ID5 ... ... ...

ID6 ... ... ...

ID7 ... ... ...

ID8 ... ... ...

ID9 ... ... ...

ID10 ... ... ...

ID1 ... ... ...

ID2 ... ... ...

ID3 ... ... ...

ID4 ... ... ...

ID5 ... ... ...

ID6 ... ... ...

ID7 ... ... ...

ID8 ... ... ...

ID9 ... ... ...

ID10 ... ... ...

Database

TRACEINFO

SEQUENCE

QUALITY

TRACE

TRACEINFO - many indexes

All queries run on this table – ID's

species

project

library

All other tables accesed using ID

1K

1K

1K

1K

ID ... ... ...

ID ... ... ...

ID ... ... ...

ID ... ... ...

ID ... ... ...

ID ... ... ...

ID ... ... ...

ID ... ... ...

ID ... ... ...

ID ... ... ...

ID ... ... ...

ID ... ... ...

ID ... ... ...

ID ... ... ...

ID ... ... ...

ID ... ... ...

ID ... ... ...

ID ... ... ...

ID ... ... ...

ID ... ... ...

ID ... ... ...

ID ... ... ...

ID ... ... ...

ID ... ... ...

ID ... ... ...

ID ... ... ...

ID ... ... ...

ID ... ... ...

ID ... ... ...

ID ... ... ...

ID ... ... ...

ID ... ... ...

ID ... ... ...

ID ... ... ...

ID ... ... ...

ID ... ... ...

ID ... ... ...

ID ... ... ...

3K per trace 5-10%

Database

TRACEINFO

SEQUENCE

QUALITY

TRACE

TRACEINFO - many indexes

All queries run on this table – ID's

species

project

library

All other tables accesed using ID

25-100K

90-95%

ID ... ... ...

ID ... ... ...

ID ... ... ...

ID ... ... ...

ID ... ... ...

ID ... ... ...

ID ... ... ...

ID ... ... ...

ID ... ... ...

ID ... ... ...

ID ... ... ...

ID ... ... ...

ID ... ... ...

ID ... ... ...

ID ... ... ...

ID ... ... ...

ID ... ... ...

ID ... ... ...

ID ... ... ...

ID ... ... ...

ID ... ... ...

ID ... ... ...

ID ... ... ...

ID ... ... ...

ID ... ... ...

ID ... ... ...

ID ... ... ...

ID ... ... ...

ID ... ... ...

ID ... ... ...

ID ... ... ...

ID ... ... ...

ID ... ... ...

ID ... ... ...

ID ... ... ...

ID ... ... ...

ID ... ... ...

ID ... ... ...

Pros

Number of files in a project reduced by 95%

simplifies backups

Oracle is 'highly' available

Doesn't rely on NFS mounted disks

Scalable both #users and data volume

Well established recovery procedure

To look at a trace you don't need to know it's location

ASP could be modified to put traces in the database rather than the project directories

Still drive programs with a fofn

Staden iolib can already access the TraceArchive

Cons

Potentially many programs need to be changed

Database needs to have the information to produce a list of the traces in each project

Need to create many indexes to speed up all potential queries

Will need to expand the current TraceArchive schema to hold all the information in the experiment files

Things to consider

Switching to Oracle

Single point of failure

Speed of access

Backups

Data for a project is no longer in a single location

Milestones

TraceArchive changes to provide 'proof of idea'

Currently traces are stored in tarfiles which are indexed in the	database

Move traces into database

Need to understand what acesses project directories

Perl, C, C++ ?

What files are required ?

What work is required to access Oracle

Steven Leonard

2004-11-03T12:33:28