Linux Intro and NGS File Formats

NGS WTAC 2017

Tony Cox and Steven Leonard

WTSI

Shell, Running Commands

Interactive command line sessions - the shell
Open a terminal window to create a shell
Running commands
- ```
who
```
  gives a list of who is logged in
- ```
w
```
  give a list of who is logged in - and what they are doing
Commands and command line switches
- ```
df
```
  or
```
df -h
```
  to see disk space use
- switches modify the way that standard commands work
Help for commands
- ```
man df
```
  or just web search....
Different shells:
```
sh
```
,
```
csh
```
,
```
tcsh
```
,
```
bash
```
Leaving a shell: type
```
exit
```
or
```
ctrl-d
```

Shells ensure convenient and almost universal remote access to unix machine

Navigation and Manipulation of Directories

```
ls
```
list files in the current directory. Nb note also "." and ".."
```
ls -l
```
to see files in current directory - how is it different?
```
ls -lart
```
to see files in current directory - how is it different?
```
ls /usr/bin
```
list files in another directory
```
ls /usr/bin/x*
```
or files matching a wildcard
```
pwd
```
to see current directory location
```
pwd; ls
```
multiple commands on the same line
The linux directory system...
Navigating the linux directory structure - moving around the hierarchy

```
cd /usr/bin
```
```
pwd
```
```
cd
```
```
pwd
```
```
cd -
```
```
pwd
```
```
cd 
```

Navigation and Manipulation of Directories

Relative and absolute directory paths

Manipulating directories and files

mkdir -p seq/data/results; ls -l seq

create a directory hierarchy

What did the

-p

switch do?

mv seq/data seq/olddata; ls -l seq

renaming a directory (or file)

rm -r seq; ls -l seq

removing an entire directory hierarchy (be careful!)

cp file1 file2; ls -l seq

copying a file

cp -r seq/data seq/data2; ls -l seq

copying a directory recursively

cp

has many important and useful switches!

Using a semi-colon allows you to sequentially perform commands without having to wait for each one individually

External Drives - working with USB disks

Plug in your USB drive
```
df -Th
```
can help you find where the disk has been "mounted"
What did this command do? - how did using the switches
```
-Th
```
help?
Change you current directory to that directory
Directory paths with spaces - special shell charcters and how to "escape" them
Create a new test directory and move into that directory
When finished, remember to move out of the directory (close shell or change directory) and then tell the OS to "safely remove" unmount the usb drive
Now move back to your home directory, then create and move into a directory for this tutorial

Security and Permissions

```
touch myfile
```
create a new empty file
```
ls -l myfile
```
display myfile permissions

-rw-r--r-- 1 avc team117   17729 Mar 12 13:59 myfile

[Read|Write|Execute] for [User/Group/Other]
```
chmod u-w myfile
```
remove User write permission
```
chmod g+x myfile
```
add Group execute permission
```
chmod go-r myfile
```
remove Group and Other read permission
Directory permissions work in exactly the same way
Beware - you need execute permissions to be able to do anything with directories!

Useful Tips and Tricks

Speeding things up

```
touch myfile
```
to create and empty file
Use the TAB key to complete commands or arguments to save typing and time
```
history
```
shows you your shell command history
```
! #cmd
```
re-executes that command
Up/down arrows scroll you through the command history
```
Ctrl-a
```
and
```
Ctrl-e
```
move to start and end of lines
Special characters and the command line escape
```
cd
```
takes you home
```
.
```
is your current directory and
```
..
```
is your parent directory

Fasta Sequence Data Files

Text File Manipulation

what is a FASTA file?

wget 'ftp://ftp.sanger.ac.uk/pub/teams/117/WTAC/data/adapters.fasta'

another fasta file with adapter sequence

```
cat adapters.fasta
```
displays the entire file to the terminal
```
less adapters.fasta
```
simple viewer for the terminal

Use
```
/[string]
```
and PageUp/PageDown keys to search within the display
Use
```
q
```
to quit from
```
less
```

```
wc phix-illumina.fa
```
to get the number of lines, words, and characters

Fasta Sequence Data Files

Text File Manipulation Continued

```
wget ftp://ftp.sanger.ac.uk/pub/teams/117/WTAC/data/phix-illumina.fa
```
let's download a fasta sequence file for phiX (from the command line or just use your browser)
```
head phix-illumina.fa
```
to inspect the top of the file
```
tail phix-illumina.fa
```
to inspect the tail of a file
```
head -n 5 phix-illumina.fa
```
to inspect the first 5 lines
```
tail -n 5 phix-illumina.fa
```
to inspect the last 5 lines
```
grep -c -E '^>' adapters.fasta
```
count the number of sequence records
```
grep -A 2 RNA adapters.fasta
```
show (first line of) RNA related records

Fastq Sequence Data Files

and Using the Shell

What is a FASTQ file?
```
touch foo{1,3,5}{A,B} okay
```
shell creates files named using all possible expanded strings

wget ftp://ftp.sanger.ac.uk/pub/teams/117/WTAC/data/s6823_1_{1,2,t}_phix.fastq

download 3 fastq sequence files with phiX reads

```
head -n 6 s6823_1_{1,2,t}_phix.fastq
```
to inspect the first few lines of each file
```
wc s6823_1_*_phix.fastq
```
to get the number of lines, words, and characters
How many sequence records?
Phred qualities: -10*log_10(P_error) beware of different encodings

Data Redirection

```
md5sum s6823_1_1_phix.fastq
```
show the md5sum of a file

md5sum s6823_1_1_phix.fastq > s6823_1_1_phix.fastq.md5

save the output of md5sum in a file

```
cat s6823_1_1_phix.fastq.md5
```
```
history > cmd.history
```
save my shell command history
Redirection is powerful but dangerous. Be careful
Use
```
>>
```
to append to a file rather than overwrite it

Pipes

```
env
```
```
env | sort
```
```
env | grep SHELL
```
```
cat adapters.fasta | wc
```
output of a process can also be redirected to another process

cat adapters.fasta | grep -E '^>' | sort -u | wc -1

eh?

Loops

```
for f in *.fastq; do md5sum $f > $f.md5; done
```
loop through all the fastq files creating corresponding md5 files

Compression

```
gzip adapters.fasta
```
compresses file with gzip and adds a .gz suffix
```
gzip -d adapters.fasta.gz 
```
decompresses a .gz file

tar cf - s6823_1_{1,2,t}_phix.fastq > archive.tar

create an archive of the fastq files (reduced IO)

```
gzip archive.tar
```
compresses archive with gzip and adds a .gz suffix
```
tar cf - s6823_1_{1,2,t}_phix.fastq | gzip -9 > s6823_1_phix.tgz
```
uses a pipe and a redirect to create a compressed archive of the fastq files (reduced IO)
```
gunzip -c s6823_1_phix.tgz | tar tf -
```
lists the contents of the archive
```
tar xvfz s6823_1_phix.tgz 
```
extracts the contents of the archive

Process Control

```
xeyes
```
then ctrl-C to terminate it
```
xeyes
```
then
- ctrl-Z to suspend the process
- ```
bg
```
  to allow it to continue running in the background
```
xeyes &
```
to start a process running in the background
```
jobs
```
to see background processes, jobs, of this shell
```
ps
```
shows processes of this session
```
kill %1
```
to terminate the first xeyes
```
kill
```
2nd xeyes PID for the second
```
top
```
for a full terminal updating display of processes running on the system

SAM (& BAM) Sequence & Alignment Files

http://www.htslib.org/ for the specification and links
Samtools and Picard are the leading tools
Biobambam also used also at WTSI
Text (e.g. .sam files) and binary (e.g. .bam files) formats

wget ftp://ftp.sanger.ac.uk/pub/teams/117/WTAC/data/s6823_1_phix.sam

```
head s6823_1_phix.sam
```
header records start with @, tab delimited fields identified by two character tags followed by a :
```
grep -v -E '^@' s6823_1_phix.sam | head
```
separate sequence/alignment data records for paired reads (subreads), tab delimited fields (11 mandatory folowed by optional tag identified fields)

SAM format

```
grep -v -E '^@' s6823_1_phix.sam | head
```
HS1_6823:1:2208:16638:14313#168 81 phix-illumina.fa 1 37 75M = 5080 5004 GAGTTTTATCGCTTCCATGACGCAGAAGTTAACACTTTCGGATATTTCTGATGAGTCGAAAAATTATCTTGATAA G;BGIKJMGHEKJ>AOMJHJOLLHIKIQLLFHLHM?LLNJKFKOLKJMJLKHLIKNKLMIJLKJILMJ>GJGHBD X0:i:1 X1:i:0 MD:Z:75 RG:Z:1 XG:i:0 AM:i:37 NM:i:0 SM:i:37 XM:i:0 XO:i:0 QT:Z:+1144=AD RT:Z:ACAACGCA XT:A:U

QNAME HS1_6823:1:2208:16638:14313#168
FLAG 81
RNAME phix-illumina.fa
....6 other mandatory fields

SEQ

GAGTTTTATCGCTTCCATGACGCAGAAGTTAACAC.....

QUAL

G;BGIKJMGHEKJ>AOMJHJOLLHIKIQLLFHLHM.....

Samtools

obtaining and building

wget https://github.com/samtools/samtools/releases/download/1.4/samtools-1.4.tar.bz2 \
 && tar xjf samtools-1.4.tar.bz2

download and if download is successful uncompress some samtools source

```
cd samtools-1.4; ls;  make -j 2
```
```
ls; cd .. 
```
```
samtools-1.4/samtools
```
```
samtools-1.4/samtools help view
```
```
man samtools-1.4/samtools.1 
```

Samtools

using

samtools-1.4/samtools view -O bam s6823_1_phix.sam  > s6823_1_phix.bam

convert SAM to BAM

```
file s6823_1_phix.bam s6823_1_phix.sam
```
not normally interesting to look directly in binary files....
```
ls -lh s6823_1* 
```
compare the sizes of the files....
```
samtools-1.4/samtools flagstat s6823_1_phix.bam
```
for some useful stats (dependant on how it's been created)
```
samtools-1.4/samtools index s6823_1_phix.bam 
```
creating an index allows fast reference location access

samtools-1.4/samtools tview s6823_1_phix.bam phix-illumina.fa

reference based text viewer -- boring for phiX

Samtools & CRAM

New format to try to help with growing data volumes
Lossy and lossless modes

samtools-1.4/samtools view -O cram -T phix-illumina.fa s6823_1_phix.sam  > s6823_1_phix.cram

convert SAM to CRAM

```
ls -lh s6823_1* 
```
compare the sizes of the files....

samtools-1.4/samtools view -h s6823_1_phix.cram | less

to convert to SAM and pipe to a "pager" (q to quit)

CRAM functionality in samtools approaching that of BAM....

Illumina Runfolder Anatomy

Structure

```
InterOp
```
```
Data/Intensities
```
```
Data/Intensities/BaseCalls
```

Data file types: (c)locs, filter, control, bcl, cif

So...

Sequence file formats
- fasta, fastq, sam/bam : you're most likely to work with today
- cram : smaller than bam, v3 now, next likely common format
- sff, srf, sra, ztr : there are plenty of other formats which typically contain more "raw" data
- Illumina: (c)locs, filter, control, bcl → fastq
Using the shell
- is a "universal" language
- gives you a
```
history
```
- can get the computer to do the (boring) repetative stuff completely consistently
- allows you wrap an established procedure in to a script....

Questions?

samtools & pileup

Converts a read-orientated (sam/bam) file in alignment order to a reference oriented (text/binary) file

samtools-1.4/samtools faidx phix-illumina.fa

create an index for reference fasta file

```
cat phix-illumina.fa.fai 
```
it's quite boring for this reference

samtools-1.4/samtools mpileup -f phix-illumina.fa s6823_1_phix.bam | less -S

samtools/bcftools & vcf/bcf

Download and uncompress/install bcftools

wget https://github.com/samtools/bcftools/releases/download/1.4/bcftools-1.4.tar.bz2

```
tar xjf bcftools-1.4.tar.bz2
```
```
cd bcftools-1.4; make; cd ..
```

Call variants generate a (text/binary) file

samtools-1.4/samtools mpileup -t DP,SP -I -uf phix-illumina.fa -Q 25 s6823_1_phix.bam \
   | bcftools-1.4/bcftools call -vc -

Samtools & flagstat, stats/bamcheck

samtools-1.4/samtools view -u \
ftp://ngs.sanger.ac.uk/production/WTAC/data/12585_1#21.bam \
| samtools-1.4/samtools stats - | tee 12585_1#21.bam.bamstats

pull your data from ftp site, push it uncompressed through bamcheck, and both write bamcheck's output to the terminal and to a file

samtools-1.4/misc/plot-bamstats -p 12585_1#21/ 12585_1#21.bam.bamstats

create some plots using the bamcheck data

this doesn't work as gnuplot isn't installed on these machines so

wget ftp://ftp.sanger.ac.uk/pub/teams/117/WTAC/data/12585_1%2321.tar.gz

```
tar xvf 12585_1#21.tar.gz
```

```
file:///tmp/12585_1%2321/index.html
```

FastQC

From Babraham Institute - neighbours up the road...
Download and uncompress/install FastQC

wget http://www.bioinformatics.babraham.ac.uk/projects/fastqc/fastqc_v0.11.5.zip

```
unzip fastqc_v0.11.5.zip
```
```
chmod 755 FastQC/fastqc
```

Try inspecting BAM or fastq files

```
FastQC/fastqc
```

IGV

From Broad Institute
Download and uncompress/install igv

wget http://data.broadinstitute.org/igv/projects/downloads/IGV_2.3.92.zip

```
unzip IGV_2.3.92.zip
```

Inspect your BAM file with IGV
```
IGV_2.3.92/igv.sh
```

Picard

From Broad Institute
Download and uncompress/install Picard

wget https://github.com/broadinstitute/picard/releases/download/2.9.0/picard.jar

N.B. this version requires java 8
Try converting from BAM to fastq
```
java -jar picard.jar SamToFastq -h
```

Biobambam

Download and uncompress/install Biobambam

wget https://github.com/gt1/biobambam2/releases/download/2.0.72-release-20170316102450/biobambam2-2.0.72-release-20170316102450-x86_64-etch-linux-gnu.tar.gz

tar xvf biobambam2-2.0.72-release-20170316102450-x86_64-etch-linux-gnu.tar.gz

ln -s biobambam2-2.0.72-release-20170316102450-x86_64-etch-linux-gnu/bin biobambam

Try marking duplicates using Biobambam and compare with Picard

```
./bammarkduplicates -h
```

WTSI -- 23/04/2017

NGS WTAC 2017 -- Linux Intro and NGS File Formats

Linux Intro and NGS File Formats

NGS WTAC 2017

Tony Cox and Steven Leonard

WTSI

Shell, Running Commands

Navigation and Manipulation of Directories

Navigation and Manipulation of Directories

External Drives - working with USB disks

Security and Permissions

Useful Tips and Tricks

Speeding things up

Fasta Sequence Data Files

Text File Manipulation

Fasta Sequence Data Files

Text File Manipulation Continued

Fastq Sequence Data Files

and Using the Shell

Data Redirection

Pipes

Loops

Compression

Process Control

SAM (& BAM) Sequence & Alignment Files

SAM format

Samtools

obtaining and building

Samtools

using

Samtools & CRAM

Illumina Runfolder Anatomy

So...

samtools & pileup

samtools/bcftools & vcf/bcf

Samtools & flagstat, stats/bamcheck

FastQC

IGV

Picard

Biobambam