The Biopython team is proud to announce Biopython 1.54, a new stable release of the Biopython library. Biopython 1.54 comes five months after our last release and brings new features, tweaks to some established functions and the usual collection of bug fixes.

This is the first stable release to feature the new Bio.Phylo module which can be used to read, write and take data from phylogenetic trees in Newick, Nexus and PhyloXML formats. The module is the result of Eric Talevich’s Google Summer of Code project which was supported by The National Evolutionary Synthesis Center (NESCent).

Biopython now supports the reading, writing and indexing of Standard Flowgram Format (SFF) files produced in 454 sequencing. Jose Blanca (the brains behind the widely used sff_extract tool) provided code to handle SFF files and Peter Cock has integrated that code with Bio.SeqIO. Adding SFF support to SeqIO makes it possible to convert these files to the FASTQ, FASTA and QUAL formats (as trimmed or untrimmed reads).

As well as adding features the new release tweaks and extends some of the core modules:

• Both Bio.SeqIO and Bio.AlignIO will accept filenames as well as handles, as detailed here.
• The multiple sequence alignment object that underlies Bio.AlignIO has been improved.
• Bio.SeqIO can read and write EMBL nucleotide files.
• The dictionary-like objects returned by Bio.SeqIO.index() have a new method “get_raw” that gets unparsed data from a file as a string, as detailed here.
• Bio.Entrez includes some more DTD files, in particular eLink_090910.dtd, used by our NCBI Entrez Utilities XML parser.

Binaries and source files for Biopython 1.54 are available from the downloads page. The documentation has been updated to include the changes made since our last release.

A big thanks to every one who tested our beta release or submitted bugs since Biopython 1.53. And an especially big thanks to everyone who contributed to this release, including five first time contributors:

• Anne Pajon (first contribution)
• Brad Chapman
• Christian Zmasek
• Diana Jaunzeikare (first contribution)
• Eric Talevich
• Jose Blanca (first contribution)
• Kevin Jacobs (first contribution)
• Leighton Pritchard
• Michiel de Hoon
• Peter Cock
• Thomas Holder (first contribution)

BioPerl has migrated to git and GitHub!  We have also set up a mirror set of several key repositories at the great public git hosting site repo.or.cz.

If you are a current BioPerl developer (had a previous account for direct access to our prior Subversion repository), please sign up for a GitHub account and let us know your user ID.  Also, add the extra email (where ‘DEVNAME’ is your original Subversion account ID).  This should map any previous commits from the older Subversion and CVS repository to your new GitHub account.

The following are ways everyone can download the latest code.

Using git

Note you can replace ‘bioperl-live.git’ with any of the repository names (bioperl-db, bioperl-run, etc).  For BioPerl developers (GitHub collaborators) you have a choice of SSH or HTTP:

  git clone git@github.com:bioperl/bioperl-live.git

  git clone https://bioperl@github.com/bioperl/bioperl-live.git

The open read-only link (for everyone):

  git clone git://github.com/bioperl/bioperl-live.git

or using the mirror site:

  git clone git://repo.or.cz/bioperl-live.git

Using SVN (read-only)

We will also support read-only access to GitHub with Subversion.  We may allow write support at some later point.  To use svn:

  svn checkout http://svn.github.com/bioperl/bioperl-live.git

Direct downloads

Tagged releases can be found here:

http://github.com/bioperl/bioperl-live/downloads

The latest source code here:

http://github.com/bioperl/bioperl-live/archives/master

Forking BioPerl and Pull Requests

We intend on using git and GitHub to their fullest.  With that in mind, we encourage users to fork BioPerl code, make changes, commit them to your forked repository, and submit pull requests.

Documentation

We’re also in the process of updating our local developer documents for help with those who haven’t used git before.  In particular, we have a Using Git page, and have added RSS feeds for our repository commits.

Enjoy!

chris

Update: SVN version fixed, thanks to DaveMessina++ for pointing it out.

In another quirk to the FASTQ story, recent Illumina FASTQ files don’t actually use the full range of PHRED scores – and a score of 2 has a special meaning, The Read Segment Quality Control Indicator (RSQCI, encoded as ‘B’).

Hats off to Dr Torsten Seemann for raising awareness of this issue in his post on the seqanswers.com forum, referring to a presentation by Tobias Mann of Illumina which says:

The Read Segment Quality Control Indicator:

• At the ends of some reads, quality scores are unreliable. Illumina has an algorithm for identifying these unreliable runs of quality scores, and we use a special indicator to flag these portions of reads
• A quality score of 2, encoded as a “B”, is used as a special indicator. A quality score of 2 does not imply a specific error rate, but rather implies that the marked region of the read should not be used for downstream analysis.
• Some reads will end with a run of B (or Q2) basecalls, but there will never  be an isolated Q2 basecall.

So, armed with this knowledge, you might want to apply a simple trimming criteria to any Illumina FASTQ files – remove anything after and including a PHRED quality score of 2 (encoded as ASCII ‘B’).

We could do this with the rich object orientated SeqRecord based API in Biopython, but when dealing with massive FASTQ files this overhead matters. Instead we’ll stick with plain Python strings:

from Bio.SeqIO.QualityIO import FastqGeneralIterator
handle = open("B_trimmed.fastq", "w")
min_length = 10
for title, seq, qual in FastqGeneralIterator(open("untrimmed.fastq")) :
    #Find the location of the first "B" (PHRED quality 2)
    trim = qual.find("B")
    if trim == -1:
        #No need to trim
        handle.write("@%s\n%s\n+\n%s\n" % (title, seq, qual))
    elif trim >= min_length:
        #Take everything up to the first B
        handle.write("@%s\n%s\n+\n%s\n" % (title, seq[:trim], qual[:trim]))
handle.close()

The above will work fine on any recent Illumina FASTQ files using the RSQCI scheme, but on older Illumina FASTQ files the “B” character is just a low quality score – and can occur even in the middle of a read. Here trimming at the first “B” might be unwise. Instead, we can trim any trailing “B” characters – which will do the same thing on RSQCI based FASTQ files where the “B” should only appear at the end:

from Bio.SeqIO.QualityIO import FastqGeneralIterator
handle = open("B_trimmed.fastq", "w")
min_length = 10
for title, seq, qual in FastqGeneralIterator(open("untrimmed.fastq")) :
    qual = qual.rstrip("B") #Remove any trailing B characters
    length = len(qual)
    if length >= min_length:
        seq = seq[:length] #trim to match
        handle.write("@%s\n%s\n+\n%s\n" % (title, seq, qual))
handle.close()

You could easily modify this example to read from stdin and write to stdout (see this cookbook example), or take filenames as command line arguments to turn this into a general purpose “FASTQ B-trimming script”.

This is another blog post to highlight one of the neat tricks you’ll be able to do with Biopython 1.54 (which you can help test with the Biopython 1.54 beta release).

It is often useful to be able to extract a few records from a larger sequence file – for example, some sequences of interest from a full UniProt or GenBank dump. One obvious way to try to do this is to parse the file into an object representation (i.e. SeqRecord objects using Bio.SeqIO.parse(...)), filter to pick out the entries you want, and then write them back to disk (using Bio.SeqIO.write(...)). However, for complex file formats like GenBank this can be lossy (Bio.SeqIO does not support a 100% identical round trip), and Biopython don’t currently support writing out the SwissProt plain text format used by UniProt. So, that approach won’t work.

Biopython 1.52 introduced a new indexing function, Bio.SeqIO.index(...), which allows a large multi-sequence file to be treated like a Python dictionary – parsing requested records on request. This has been enhanced for Biopython 1.54 with a method get_raw(...) to extract the raw for a record as a string.

How is this useful? Well, take your large (UniProt) file, index it, then extract the records you want and write them to your output file unmodified:

from Bio import SeqIO
uniprot = SeqIO.index("uniprot_sprot.dat", "swiss")
handle = open("selected.dat", "w")
for acc in ["P33487", "P19801", "P13689", "Q8JZQ5", "Q9TRC7"]:
    handle.write(uniprot.get_raw(acc))
handle.close()

Another neat use of this functionality would be to sort entries in a sequential file format, and there is an example of that in the Biopython Tutorial (PDF).

One of the small changes coming in Biopython 1.54 (which you can try out already using the Biopython 1.54 beta) is to Bio.SeqIO and Bio.AlignIO. Previously the input and output functions had required file handles, but they will now also accept filenames.

This is a case of practicality beats purity (to quote the Zen of Python), and is particularly handy when doing very short scripts or working at the Python prompt.

For example, filtering a FASTA file to take only entries with a minimum length of 100 can be done like this (with handles):

from Bio import SeqIO
in_handle = open("example.fasta", "rU")
out_handle = open("long.fasta", "w")
records = (rec for rec in SeqIO.parse(in_handle, "fasta") if len(rec)>100)
SeqIO.write(records, out_handle, "fasta")
in_handle.close()
out_handle.close()

Using filenames it becomes much more concise – just three lines:

from Bio import SeqIO
records = (rec for rec in SeqIO.parse("example.fasta", "fasta") if len(rec)>100)
SeqIO.write(records, "long.fasta", "fasta")

This also means Python and Biopython beginners can postpone learning about file handles a little longer, although that may not be an entirely good thing

A beta release for Biopython 1.54 is now available for download and testing.

Since Biopython 1.53 was released at the end of last year, several new features and more documentation have been added, plus the usual bug fixes. For full details see the NEWS file.

All the new features have been tested by the dev team but it’s possible there are cases that we haven’t been able to foresee and test, especially for the updated multiple sequence alignment object (which is what you’ll now get when parsing alignments with Bio.AlignIO), the new Bio.Phylo module, and the Bio.SeqIO support for Standard Flowgram Format (SFF) files.

Source distributions and Windows installers are available from the downloads page on the Biopython website (biopython.org).

We are interested in getting feedback on the beta release as a whole, but especially on the new features and the Biopython Tutorial and Cookbook (PDF).

(At least) 10 people contributed to this release (so far), which includes 4 new people:

• Anne Pajon (first contribution)
• Brad Chapman
• Christian Zmasek
• Eric Talevich
• Jose Blanca (first contribution)
• Kevin Jacobs (first contribution)
• Leighton Pritchard
• Michiel de Hoon
• Peter Cock
• Thomas Holder (first contribution)

So, gather your courage, download the release, try it out and please let us know what works and what doesn’t through the mailing lists (or bugzilla).

BioPerl developers and users attended the BioPerl satellite meeting on January 13th, just prior to the GMOD Meeting.  Several items were covered on the agenda:

• In order to start addressing whole genome data with more lightweight objects, we are planning on setting up a lightweight Bio::SeqI object that has a flexible DB backend (i.e. Bio::DB::SeqFeature::Store or similar).  We are also contemplating adding lazy parsing for some parsers, possibly using the Bio::PullParserI methods (or similar) that Sendu Bala created.
• After a final  1.6 branch point release, we may ‘freeze’ BioPerl in a maintenance mode, primarily so that we can reorganize core into several more easily installed subdistributions on a branch.  New modules will essentially be additional separate repos that will depend on BioPerl core.  This reorganization has been discussed for a few years now, and as we edge closer to starting this (probably this spring) we’ll announce more details.
• Some initial thoughts on how to handle circular genomes more efficiently.  We essentially do this already, but it isn’t full-proof.
• Need some significant time dedicated towards GFF3-based coding (reimplement FeatureIO but allow some flexibility).  Rob Buels had started the initial run at splitting out FeatureIO, so next step is a true reimplementation.
• We don’t plan on including Moose support for the immediate future, feeling that it would be better to reimplement some of the classes from scratch using Moose and similar as a BioPerl 2.0, or possibly await the impending Rakudo Perl 6 alpha and start afresh using that instead of Moose.

Anything we missed?  Anything you would like to address?  Please add comments and we’ll discuss them on list.

We are pleased to announce the release of BioRuby 1.4.0. This new release contains many new features, in addition to bug fixes and improvements.

PhyloXML support: Support for reading and writing PhyloXML file format is added, developed by Diana Jaunzeikare, mentored by Christian M Zmasek and co-mentors, supported by Google Summer of Code 2009 in
collaboration with the National Evolutionary Synthesis Center (NESCent).

FASTQ file format support: Support for reading and writing FASTQ file format is added. All of the three FASTQ format variants are supported. The code is written by Naohisa Goto, with the help of discussions in the
open-bio-l mailing list. The prototype of Bio::Fastq class was first developed during the BioHackathon 2009 held in Okinawa.

DNA chromatogram support: Support for reading DNA chromatogram files are added. SCF and ABIF file formats are supported. The code is developed by Anthony Underwood.

MEME (motif-based sequence analysis tools) support: Support for running MAST (Motif Alignment & Search Tool, part of the MEME Suite, motif-based sequence analysis tools) and parsing its results are added,  developed by Adam Kraut.

Improvement of KEGG parser classes: Some new methods are added to parse new fields added to some KEGG file formats. Unit tests for KEGG parsers are also added and improved. These are contributed by Kozo Nishida.

Many sample scripts are added: Many sample scripts showing demonstrations of usages of classes are added. They were originally primitive test codes written in the “if __FILE__ == $0″ convention.

Unit tests can test installed BioRuby: Mechanism to load library and to find test data in the unit tests are changed, and target library path and test data path can be changed with environment variables.

Incompatible change: Bio::NCBI::REST needs email address: NCBI announced that all Entrez Utilities (E-utilities)  requests must contain email and tool parameters, and requests without them will return error after June 2010. In BioRuby, to set default email address and tool name, following methods are added.

• Bio::NCBI.default_email=(email)
• Bio::NCBI.default_tool=(tool_name)

Note that no default email address is preset in BioRuby. Library users must set their own email address or implement to get user’s email address in some way (from input form, configuration file, etc).

In addition, many changes have been made, including incompatible changes. For more information, see RELEASE_NOTES.rdoc and ChangeLog.

The archive is available at: http://bioruby.org/archive/bioruby-1.4.0.tar.gz

Gem file is also available at:  http://bioruby.org/archive/gems/bio-1.4.0.gem

We also put RubyGems pacakge at RubyForge and Gemcutter. You can easily install by using RubyGems.
First, check the version number by using search command:
% gem search –remote bio
and find “bio (1.4.0)” in the list. Then,
% sudo gem install bio

Hope you enjoy.

We are pleased to announce the availability of Biopython 1.53, a new stable release of the Biopython library, three months after the release of Biopython 1.52. This is our first release since migrating from CVS to git for source code control.

There have been some additions to our core objects – the Seq (and related UnknownSeq) objects gained upper and lower methods (like the string methods of the same name but alphabet aware) plus a new ungap method. The SeqFeature object now has an extract method to get the region of sequence it describes (useful for getting CDS nucleotide sequences from GenBank files). Also SeqRecord objects now support addition, giving a new SeqRecord with the combined sequence, all the SeqFeatures, and any common annotation.

SQLite support (built into Python 2.5+) was added to our BioSQL interface (Brad Chapman). This is still a little experimental as we are using a draft BioSQL SQLite schema, but this should be merged into the next BioSQL release.

Biopython now includes wrappers for the new NCBI BLAST C++ tools, which will be replacing the old NCBI “legacy” BLAST tools written in C. The plain text BLAST parser has been updated to cope as well. Nevertheless, we (and the NCBI) still recommend using the XML output for parsing.

Bio.Entrez includes the new (Jan 2010) DTD files from the NCBI for parsing MedLine/PubMed data.

The NCBI codon tables have been updated from version 3.4 to 3.9, which adds a few extra start codons, and a few new tables (Tables 16, 21, 22 and 23).

The restriction enzyme list in Bio.Restriction has been updated to the Nov 2009 release of REBASE.

The Bio.PDB parser and output code has been updated to understand the element column in ATOM and HETATM lines (based on patches contributed by Hongbo Zhu and Frederik Gwinner), and Bio.PDB.PDBList has been updated for recent changes to the PDB FTP site (Paul T. Bathen).

Finally, support for running Biopython under Jython (using the Java Virtual Machine) has been much improved (with input from Kyle Ellrott). Note that Jython does not support C code, and currently Jython does not parse DTD files (Jython Issue 1447; needed for the Bio.Entrez XML parser). However, most of the Biopython modules seem fine from testing with Jython 2.5.0 and 2.5.1.

Sources and Windows Installers are available from our downloads page.

Thanks to the Biopython development team and to everyone who has reported bugs or contributed patches since our last release.

(At least) 12 people contributed to this release, including 3 first timers:

• Bartek Wilczynski
• Brad Chapman
• Chris Lasher
• Cymon Cox
• Frank Kauff
• Frederik Gwinner (first contribution)
• Hongbo Zhu (first contribution)
• Kyle Ellrott
• Leighton Pritchard
• Michiel de Hoon
• Paul Bathen (first contribution)
• Peter Cock

This post is about paired end data (FASTA or FASTQ) and manipulating it with Biopython’s Bio.SeqIO module (see also FASTQ conversions & speeding up FASTQ).

There are two main ways of presenting paired end data in FASTA or FASTQ files:

• Paired files, with matching entries for the forward and reverse reads (probably the norm with Illumina data)
• Single files, with alternating entries for the forward and reverse reads (used by Velvet)

Converting between these two is a relatively common operation, and is normally pretty easy. There was a short example of how you might do this in Biopython on a recent (September 2009) Velvet users/Biopython mailing list discussion. That script didn’t check the record IDs matched up (but neither does the Perl script shuffleSequences_fastq.pl included with Velvet for this task).

It would be safer to check the record IDs do match. However, there are several different naming schemes for reads, most typically suffixes of /1 and /2, but also things like .f and .r get used. In the case of FASTQ files from the NCBI SRA, the reads have no suffixes, so to feed those into Velvet you may want to check they are equal and then add a suffix as shown below.

#This Python script requires Biopython 1.51 or later
from Bio import SeqIO
import itertools

#Setup variables (could parse command line args instead)
file_f = "SRR001666_1.fastq"
file_r = "SRR001666_2.fastq"
file_out = "SRR001666_interleaved.fastq"
format = "fastq" #or "fastq-illumina", or "fasta", or ...

def interleave(iter1, iter2) :
    for (forward, reverse) in itertools.izip(iter1,iter2):
        assert forward.id == reverse.id
        forward.id += "/1"
        reverse.id += "/2"
        yield forward
        yield reverse

records_f = SeqIO.parse(open(file_f,"rU"), format)
records_r = SeqIO.parse(open(file_r,"rU"), format)

handle = open(file_out, "w")
count = SeqIO.write(interleave(records_f, records_r), handle, format)
handle.close()
print "%i records written to %s" % (count, file_out)

This example uses the SRR001666 files from the NCBI SRA FTP site.

Now that works fine, and just by changing the filenames and the format name this could be used on FASTA data (or another supported file format). The bad news is it took 14 minutes to produce a 2GB FASTQ. However, going a little more low-level as discussed before can really pay off. This FASTQ-only version takes just 2 minutes:

#This Python script requires Biopython 1.51 or later
from Bio.SeqIO.QualityIO import FastqGeneralIterator
import itertools

#Setup variables (could parse command line args instead)
file_f = "SRR001666_1.fastq"
file_r = "SRR001666_2.fastq"
file_out = "SRR001666_interleaved.fastq"

handle = open(file_out, "w")
count = 0

f_iter = FastqGeneralIterator(open(file_f,"rU"))
r_iter = FastqGeneralIterator(open(file_r,"rU"))
for (f_id, f_seq, f_q), (r_id, r_seq, r_q) \
in itertools.izip(f_iter,r_iter):
    assert f_id == r_id
    count += 2
    #Write out both reads with "/1" and "/2" suffix on ID
    handle.write("@%s/1\n%s\n+\n%s\n@%s/2\n%s\n+\n%s\n" \
                 % (f_id, f_seq, f_q, r_id, r_seq, r_q))
handle.close()
print "%i records written to %s" % (count, file_out)

You can make this a little faster still by missing out most of the validation done by the Biopython FASTQ parser – but personally I wouldn’t take that risk. I’d much rather know about any errors in the data.

Peter

P.S.

Things get more interesting if you want to do quality filtering or trimming. If only one of a pair passes the quality assurance step, then you may want to keep it and treat it as an unpaired read. To give such cleaned up data to Velvet, you would need one file of alternating paired end reads, and a separate file of the orphaned effectively unpaired reads. That deserves another post going into more detail…