The Biopython team is proud to announce Biopython 1.55, a new stable release, about three months after our last stable release (Biopyton 1.54) and the beta release earlier in August.

A lot of work has been towards Python 3 support (via the 2to3 script), but unless we broke something you shouldn’t notice any changes

In terms of new features, the most noticeable highlight is that the command line tool application wrapper classes are now executable, which should make it much easier to call external tools. This is described in the updated documentation.

Additionally GenBank and EMBL parsing has been sped up, the BioSQL classes act more like Python dictionaries, and Bio.PDB should handle model numbers and a missing element column better.

Note we are phasing out support for Python 2.4. We will continue to support it for at least one further release (i.e. Biopython 1.56). This could be delayed given feedback from our users (e.g. if this proves to be a problem in combination with other libraries or a popular Linux distribution).

(At least) 12 people have contributed to this release, including 6 new people – thank you all:

• Andres Colubri (first contribution)
• Carlos Rios Vera (first contribution)
• Claude Paroz (first contribution)
• Cymon Cox
• Eric Talevich
• Frank Kauff
• Joao Rodrigues (first contribution)
• Konstantin Okonechnikov (first contribution)
• Michiel de Hoon
• Nathan Edwards (first contribution)
• Peter Cock
• Tiago Antao

Source distributions and Windows installers are available from the downloads page on the Biopython website (biopython.org).

As usual, feedback is most welcome on the mailing lists (or bugzilla).

After 10 years of development, the BioRuby paper is finally published in the Bioinformatics journal.  The article is open access, so please take a look.

BioRuby: Bioinformatics software for the Ruby programming language
Naohisa Goto, Pjotr Prins, Mitsuteru Nakao, Raoul Bonnal, Jan Aerts and Toshiaki Katayama
Bioinformatics 2010; doi: 10.1093/bioinformatics/btq475

We’ve just released a beta of Biopython 1.55 for user testing.

Since Biopython 1.54 was released three months ago, we’ve made a good start on work for Python 3 support (via the 2to3 script), but as a side effect of this we’ve had to update quite a lot of the older parts of the library. Although the unit tests are all fine, there is a small but real chance that we’ve accidentally broken things – which is why we’re doing this beta release.

In terms of new features, the most noticeable highlight is that the command line tool application wrapper classes are now executable, which should make it much easier to call external tools. This is described in the updated documentation.

Note we are phasing out support for Python 2.4. We will continue to support it for at least one further release (i.e. Biopython 1.56). This could be delayed given feedback from our users (e.g. if this proves to be a problem in combination with other libraries or a popular Linux distribution).

(At least) 10 people have contributed to this release (so far), including 5 new people:

• Andres Colubri (first contribution)
• Carlos Rios Vera (first contribution)
• Claude Paroz (first contribution)
• Eric Talevich
• Frank Kauff
• Joao Rodrigues (first contribution)
• Konstantin Okonechnikov (first contribution)
• Michiel de Hoon
• Peter Cock
• Tiago Antao

Source distributions and Windows installers are available from the downloads page on the Biopython website (biopython.org).

So, please let us know what works and what doesn’t through the mailing lists (or bugzilla).

The Biopython team is proud to announce Biopython 1.54, a new stable release of the Biopython library. Biopython 1.54 comes five months after our last release and brings new features, tweaks to some established functions and the usual collection of bug fixes.

This is the first stable release to feature the new Bio.Phylo module which can be used to read, write and take data from phylogenetic trees in Newick, Nexus and PhyloXML formats. The module is the result of Eric Talevich’s Google Summer of Code project which was supported by The National Evolutionary Synthesis Center (NESCent).

Biopython now supports the reading, writing and indexing of Standard Flowgram Format (SFF) files produced in 454 sequencing. Jose Blanca (the brains behind the widely used sff_extract tool) provided code to handle SFF files and Peter Cock has integrated that code with Bio.SeqIO. Adding SFF support to SeqIO makes it possible to convert these files to the FASTQ, FASTA and QUAL formats (as trimmed or untrimmed reads).

As well as adding features the new release tweaks and extends some of the core modules:

• Both Bio.SeqIO and Bio.AlignIO will accept filenames as well as handles, as detailed here.
• The multiple sequence alignment object that underlies Bio.AlignIO has been improved.
• Bio.SeqIO can read and write EMBL nucleotide files.
• The dictionary-like objects returned by Bio.SeqIO.index() have a new method “get_raw” that gets unparsed data from a file as a string, as detailed here.
• Bio.Entrez includes some more DTD files, in particular eLink_090910.dtd, used by our NCBI Entrez Utilities XML parser.

Binaries and source files for Biopython 1.54 are available from the downloads page. The documentation has been updated to include the changes made since our last release.

A big thanks to every one who tested our beta release or submitted bugs since Biopython 1.53. And an especially big thanks to everyone who contributed to this release, including five first time contributors:

• Anne Pajon (first contribution)
• Brad Chapman
• Christian Zmasek
• Diana Jaunzeikare (first contribution)
• Eric Talevich
• Jose Blanca (first contribution)
• Kevin Jacobs (first contribution)
• Leighton Pritchard
• Michiel de Hoon
• Peter Cock
• Thomas Holder (first contribution)

BioPerl has migrated to git and GitHub!  We have also set up a mirror set of several key repositories at the great public git hosting site repo.or.cz.

If you are a current BioPerl developer (had a previous account for direct access to our prior Subversion repository), please sign up for a GitHub account and let us know your user ID.  Also, add the extra email (where ‘DEVNAME’ is your original Subversion account ID).  This should map any previous commits from the older Subversion and CVS repository to your new GitHub account.

The following are ways everyone can download the latest code.

Using git

Note you can replace ‘bioperl-live.git’ with any of the repository names (bioperl-db, bioperl-run, etc).  For BioPerl developers (GitHub collaborators) you have a choice of SSH or HTTP:

  git clone git@github.com:bioperl/bioperl-live.git

  git clone https://bioperl@github.com/bioperl/bioperl-live.git

The open read-only link (for everyone):

  git clone git://github.com/bioperl/bioperl-live.git

or using the mirror site:

  git clone git://repo.or.cz/bioperl-live.git

Using SVN (read-only)

We will also support read-only access to GitHub with Subversion.  We may allow write support at some later point.  To use svn:

  svn checkout http://svn.github.com/bioperl/bioperl-live.git

Direct downloads

Tagged releases can be found here:

http://github.com/bioperl/bioperl-live/downloads

The latest source code here:

http://github.com/bioperl/bioperl-live/archives/master

Forking BioPerl and Pull Requests

We intend on using git and GitHub to their fullest.  With that in mind, we encourage users to fork BioPerl code, make changes, commit them to your forked repository, and submit pull requests.

Documentation

We’re also in the process of updating our local developer documents for help with those who haven’t used git before.  In particular, we have a Using Git page, and have added RSS feeds for our repository commits.

Enjoy!

chris

Update: SVN version fixed, thanks to DaveMessina++ for pointing it out.

I’m pleased to announce the acceptance of OBF’s 2010 Google Summer of Code students, listed in alphabetical order with their project titles and primary mentors:

Mark Chapman (PM Andreas Prlic) – Improvements to BioJava including Implementation of Multiple Sequence Alignment Algorithms

Jianjiong Gao (PM Peter Rose) – BioJava Packages for Identification, Classification, and Visualization of Posttranslational Modification of Proteins

Kazuhiro Hayashi (PM Naohisa Goto) – Ruby 1.9.2 support of BioRuby

Sara Rayburn (PM Christian Zmasek) – Implementing Speciation & Duplication Inference Algorithm for Binary and Non-binary Species Tree

Joao Pedro Garcia Lopes Maia Rodrigues (PM Eric Talevich) – Extending Bio.PDB: broadening the usefulness of BioPython’s Structural Biology module

Jun Yin (PM Chris Fields) – BioPerl Alignment Subsystem Refactoring

Congratulations to our accepted students!

All told, we had 52 applications submitted for the 6 slots (5 originally assigned, plus 1 extra) allotted to us by Google.  Proposals were extremely competitive: 6 out of 52 translates to an 11.5% acceptance rate.  We received a lot of really excellent proposals, the decisions were not easy.

Thanks very much to all the students who applied, we very much appreciate your hard work.

Here’s to a great 2010 Summer of Code, I’m sure these students will do wonderful work.

Rob Buels
O|B|F GSoC 2010 Administrator

In another quirk to the FASTQ story, recent Illumina FASTQ files don’t actually use the full range of PHRED scores – and a score of 2 has a special meaning, The Read Segment Quality Control Indicator (RSQCI, encoded as ‘B’).

Hats off to Dr Torsten Seemann for raising awareness of this issue in his post on the seqanswers.com forum, referring to a presentation by Tobias Mann of Illumina which says:

The Read Segment Quality Control Indicator:

• At the ends of some reads, quality scores are unreliable. Illumina has an algorithm for identifying these unreliable runs of quality scores, and we use a special indicator to flag these portions of reads
• A quality score of 2, encoded as a “B”, is used as a special indicator. A quality score of 2 does not imply a specific error rate, but rather implies that the marked region of the read should not be used for downstream analysis.
• Some reads will end with a run of B (or Q2) basecalls, but there will never  be an isolated Q2 basecall.

So, armed with this knowledge, you might want to apply a simple trimming criteria to any Illumina FASTQ files – remove anything after and including a PHRED quality score of 2 (encoded as ASCII ‘B’).

We could do this with the rich object orientated SeqRecord based API in Biopython, but when dealing with massive FASTQ files this overhead matters. Instead we’ll stick with plain Python strings:

from Bio.SeqIO.QualityIO import FastqGeneralIterator
handle = open("B_trimmed.fastq", "w")
min_length = 10
for title, seq, qual in FastqGeneralIterator(open("untrimmed.fastq")) :
    #Find the location of the first "B" (PHRED quality 2)
    trim = qual.find("B")
    if trim == -1:
        #No need to trim
        handle.write("@%s\n%s\n+\n%s\n" % (title, seq, qual))
    elif trim >= min_length:
        #Take everything up to the first B
        handle.write("@%s\n%s\n+\n%s\n" % (title, seq[:trim], qual[:trim]))
handle.close()

The above will work fine on any recent Illumina FASTQ files using the RSQCI scheme, but on older Illumina FASTQ files the “B” character is just a low quality score – and can occur even in the middle of a read. Here trimming at the first “B” might be unwise. Instead, we can trim any trailing “B” characters – which will do the same thing on RSQCI based FASTQ files where the “B” should only appear at the end:

from Bio.SeqIO.QualityIO import FastqGeneralIterator
handle = open("B_trimmed.fastq", "w")
min_length = 10
for title, seq, qual in FastqGeneralIterator(open("untrimmed.fastq")) :
    qual = qual.rstrip("B") #Remove any trailing B characters
    length = len(qual)
    if length >= min_length:
        seq = seq[:length] #trim to match
        handle.write("@%s\n%s\n+\n%s\n" % (title, seq, qual))
handle.close()

You could easily modify this example to read from stdin and write to stdout (see this cookbook example), or take filenames as command line arguments to turn this into a general purpose “FASTQ B-trimming script”.

This is another blog post to highlight one of the neat tricks you’ll be able to do with Biopython 1.54 (which you can help test with the Biopython 1.54 beta release).

It is often useful to be able to extract a few records from a larger sequence file – for example, some sequences of interest from a full UniProt or GenBank dump. One obvious way to try to do this is to parse the file into an object representation (i.e. SeqRecord objects using Bio.SeqIO.parse(...)), filter to pick out the entries you want, and then write them back to disk (using Bio.SeqIO.write(...)). However, for complex file formats like GenBank this can be lossy (Bio.SeqIO does not support a 100% identical round trip), and Biopython don’t currently support writing out the SwissProt plain text format used by UniProt. So, that approach won’t work.

Biopython 1.52 introduced a new indexing function, Bio.SeqIO.index(...), which allows a large multi-sequence file to be treated like a Python dictionary – parsing requested records on request. This has been enhanced for Biopython 1.54 with a method get_raw(...) to extract the raw for a record as a string.

How is this useful? Well, take your large (UniProt) file, index it, then extract the records you want and write them to your output file unmodified:

from Bio import SeqIO
uniprot = SeqIO.index("uniprot_sprot.dat", "swiss")
handle = open("selected.dat", "w")
for acc in ["P33487", "P19801", "P13689", "Q8JZQ5", "Q9TRC7"]:
    handle.write(uniprot.get_raw(acc))
handle.close()

Another neat use of this functionality would be to sort entries in a sequential file format, and there is an example of that in the Biopython Tutorial (PDF).

Just a friendly reminder that abstracts for BOSC 2010 are due next Thursday, April 15.  See the BOSC web site at http://www.open-bio.org/wiki/BOSC_2010 for details.  Submissions will only be accepted electronically at http://events.open-bio.org/BOSC2010/openconf.php.

Graduate students, don’t forget we are offering $250 student travel awards this year. Be sure to check the box indicating that you are a graduate student to be considered for the award.

We are also pleased to announce that Guy Coates, Group leader of the Informatics Systems Group at the Wellcome Trust Sanger Institute, and Ross Gardler, Vice President of the Apache Software Foundation, will be giving keynote presentations at BOSC.

On behalf of the BOSC 2010 organizing committee, I hope to see you there!

One of the small changes coming in Biopython 1.54 (which you can try out already using the Biopython 1.54 beta) is to Bio.SeqIO and Bio.AlignIO. Previously the input and output functions had required file handles, but they will now also accept filenames.

This is a case of practicality beats purity (to quote the Zen of Python), and is particularly handy when doing very short scripts or working at the Python prompt.

For example, filtering a FASTA file to take only entries with a minimum length of 100 can be done like this (with handles):

from Bio import SeqIO
in_handle = open("example.fasta", "rU")
out_handle = open("long.fasta", "w")
records = (rec for rec in SeqIO.parse(in_handle, "fasta") if len(rec)>100)
SeqIO.write(records, out_handle, "fasta")
in_handle.close()
out_handle.close()

Using filenames it becomes much more concise – just three lines:

from Bio import SeqIO
records = (rec for rec in SeqIO.parse("example.fasta", "fasta") if len(rec)>100)
SeqIO.write(records, "long.fasta", "fasta")

This also means Python and Biopython beginners can postpone learning about file handles a little longer, although that may not be an entirely good thing