Abstract submissions for the 11th Annual Bioinformatics Open Source Conference (BOSC 2010) are now open.

At-a-glance
BOSC is an ISMB 2010 Special Interest Group (SIG)
Date: July 9-10, 2010
Location: Boston, Massachusetts, USA
BOSC 2010 web site: http://www.open-bio.org/wiki/BOSC_2010
Abstract submission via Open Conference System site:  http://events.open-bio.org/BOSC2010/openconf.php
E-mail: bosc@open-bio.org
Bosc-announce list:  http://lists.open-bio.org/mailman/listinfo/bosc-announce

Important Dates
April 15: Abstract deadline
May 5:  Notification of accepted abstracts
May 28: Early Registration Discount Cut-off date
July 8-9:  Codefest 2010
July 9-10: BOSC 2010
August 15:  Manuscript deadline for BOSC 2010 Proceedings published in BMC Bioinformatics

The Bioinformatics Open Source Conference (BOSC) is sponsored by the Open Bioinformatics Foundation (O|B|F), a non-profit group dedicated to promoting the practice and philosophy of Open Source software development within the biological research community. To be considered for acceptance, software systems representing the central topic in a presentation submitted to BOSC must be licensed with a recognized Open Source License, and be freely available for download in source code form.

We have some exciting things planned this year, including:

• Codefest 2010 programming session for the two days preceeding BOSC:  See http://www.open-bio.org/wiki/Codefest_2010 for details.
• OpenBio Solution Challenge:  See session description below and http://www.open-bio.org/wiki/SolutionChallenge for details.
• Student Travel Fellowships:  Through generous sponsorship from Eagle Genomics and an anonymous donor, we are pleased to announce the competition for three Student Travel Awards for BOSC 2010. Each winner will be awarded $250 to defray the costs of travel to BOSC 2010.  See http://www.open-bio.org/wiki/BOSC_2010#Student_Travel_Awards for details.
• First-ever BOSC Proceedings will be published in the Open Access journal, BMC Bioinformatics.  Manuscripts will be due after BOSC on August 15.  See http://www.open-bio.org/wiki/BOSC_2010#First-ever_Published_BOSC_Proceedings for details.
• Sessions on approaches to analyzing high-throughput ‘omics data, cloud-based approaches to improving software and data accessibility, the semantic web in open source bioinformatics, see below:

We invite abstracts for talks at the following sessions:

OpenBio SolutionChallenge — Bioinformatics library providers: please join us in a friendly competition to solve a shared biological problem, demonstrating the utility of your toolkit alongside other developers. Instead of the traditional Bio* updates that we’ve had at previous conferences, this year, we’re planning to organize these talks around a central theme: the OpenBio Solution Challenge. We start with a biological question of general interest, and the project talks will focus around how you would solve that problem using your toolkit and programming language. This is meant to provide a challenge for OpenBio contributors, a nice tutorial style overview of various projects and approaches for other programmers, and a fun opportunity to compete and learn from other projects. Conference attendees will vote on their favorite solution, with the winner receiving fame and fortune (warning: fortune not guaranteed). Specific challenges are being discussed on the SolutionChallenge page and through the various Bio* mailing lists. Alternately, each project could highlight a challenge that they particularly do well, focusing tutorial-style on how to solve a particular problem.

BioPerl developers and users attended the BioPerl satellite meeting on January 13th, just prior to the GMOD Meeting.  Several items were covered on the agenda:

• In order to start addressing whole genome data with more lightweight objects, we are planning on setting up a lightweight Bio::SeqI object that has a flexible DB backend (i.e. Bio::DB::SeqFeature::Store or similar).  We are also contemplating adding lazy parsing for some parsers, possibly using the Bio::PullParserI methods (or similar) that Sendu Bala created.
• After a final  1.6 branch point release, we may ‘freeze’ BioPerl in a maintenance mode, primarily so that we can reorganize core into several more easily installed subdistributions on a branch.  New modules will essentially be additional separate repos that will depend on BioPerl core.  This reorganization has been discussed for a few years now, and as we edge closer to starting this (probably this spring) we’ll announce more details.
• Some initial thoughts on how to handle circular genomes more efficiently.  We essentially do this already, but it isn’t full-proof.
• Need some significant time dedicated towards GFF3-based coding (reimplement FeatureIO but allow some flexibility).  Rob Buels had started the initial run at splitting out FeatureIO, so next step is a true reimplementation.
• We don’t plan on including Moose support for the immediate future, feeling that it would be better to reimplement some of the classes from scratch using Moose and similar as a BioPerl 2.0, or possibly await the impending Rakudo Perl 6 alpha and start afresh using that instead of Moose.

Anything we missed?  Anything you would like to address?  Please add comments and we’ll discuss them on list.

BOSC 2010 is currently in the planning stages. It will be held for 2 days in conjunction with the 18th Annual International Conference on Intelligent Systems for Molecular Biology (ISMB 2010) in Boston, Massachusetts, USA. The dates of BOSC 2010 are July 9-10; the main ISMB Conference runs July 11-13, 2010.  The BOSC 2010 web site can be accessed here:  http://www.open-bio.org/wiki/BOSC_2010.

The BOSC organizing committee is soliciting input on the planning of BOSC 2010 so that we can make it a successful and productive conference for the O|B|F community.  You may send your suggestions to the bosc@open-bio.org e-mail address  or add suggestions to the BOSC 2010 talk/discussion wiki page at: http://www.open-bio.org/wiki/Talk:BOSC_2010. Please respond to any or all of the questions below:

1.  For the last several years BOSC has consisted mainly of one or two keynote presentations, other talks chosen from among the submitted abstracts organized into sessions by topic, updates from the Bio* projects, Lightning Talks, and informal Birds of a Feather sessions.  Would you rather see BOSC continue in this fashion, or would you support changing the format to one or all of the following:

• Tutorials where there were in depth demonstrations and code tutorials. This could be lead off by the OBF projects instead of the traditional update talks, but could feature any open source projects interested. These would be hands on sessions with real code examples, with a focus on teaching people how to leverage various code bases to make real life work easier.  Would you be willing to organize/lead such a session for your project?
• Discussion following the hands on tutorials, these would be interactive sessions focused around dealing with unsolved issues. The “speaker” would be responsible for setting up a set of discussion topics around an issue of interest, and then facilitating ideas and opinions from the attendees. The goals would be to talk through problems and gather a consensus about options for solving them.  Would you be willing to organize/lead such a session for your project?
• Mini-hackathon either before, during, or after the 2-day BOSC.  The subject of the hackathon would need to be organized by the individual project leaders/teams.  Some suggestions would be adding/extending support for next-gen sequencing; organizing bugs/tasks so that new beginners can start contributing to the project easily and working on some of those bugs/tasks; organizing some type of contest like the Genome Annotation Assessment Project (GASP) where solutions from different projects compete on arriving at some type of goal.  Would you be willing to organize/lead this type of session?
• Organizing/creating a LiveCD or Debian download of Bio* projects with documentation to support outreach to the larger bioinformatics community.  Would you be willing to organize/lead this type of session?
• What session topics would you like to see represented for traditional talks?
• Who would you like to hear as a keynote speaker?

2.  The BOSC 2010 organizing committee is in discussion with an open access journal to publish a formal Proceedings for BOSC.  If you are planning on submitting an abstract for BOSC 2010, are you interested in submitting a more formal paper to the BOSC proceedings, given that as the author you would need to pay the page charges that could run between US$500-1000?  We are likely to move ahead with plans to have a proceedings, but it would be helpful to know how many submissions to expect.

3.  Call for volunteers.  Organizing tutorial/hackathons and such will only be possible if individuals step forward to lead these sessions.  Please let us know if you would be willing to serve in any capacity.  We also need volunteers to review abstracts for the more “traditional” sessions, please let us know if you are willing to do this as well.

Timeline: We are planning on putting out the Call for Abstracts in mid-January.  To be on track, we would like to receive your input by Friday, January 8.  If you are willing to step forward to organize a tutorial/discussion/hackathon, you would need to commit by that time, although there would still be some more time to put the actual program together in the new year.

Thanks and Happy Holidays!

Kam Dahlquist
Chair, BOSC 2010 on behalf of the BOSC 2010 Organizing committee:
Brad Chapman, Michael Heur, Darin London, Anton Nekrutenko, Steffen Moeller, Jim Procter
And the O|B|F Board:
Chris Dagdigian, Nomi Harris, Hilmar Lapp, Jason Stajich

I’m delighted to announce an open access publication in Nucleic Acids Research describing the FASTQ file format based on the conventions agreed by the OBF projects:

The Sanger FASTQ file format for sequences with quality scores, and the Solexa/Illumina FASTQ variants
Peter J. A. Cock (Biopython), Christopher J. Fields (BioPerl), Naohisa Goto (BioRuby), Michael L. Heuer (BioJava) and Peter M. Rice (EMBOSS).
Nucleic Acids Research, doi:10.1093/nar/gkp1137

This will hopefully serve as a reference describing the original standard Sanger FASTQ, and the two variants from Solexa/Illumina, and how to inter-convert between them.

This post is about paired end data (FASTA or FASTQ) and manipulating it with Biopython’s Bio.SeqIO module (see also FASTQ conversions & speeding up FASTQ).

There are two main ways of presenting paired end data in FASTA or FASTQ files:

• Paired files, with matching entries for the forward and reverse reads (probably the norm with Illumina data)
• Single files, with alternating entries for the forward and reverse reads (used by Velvet)

Converting between these two is a relatively common operation, and is normally pretty easy. There was a short example of how you might do this in Biopython on a recent (September 2009) Velvet users/Biopython mailing list discussion. That script didn’t check the record IDs matched up (but neither does the Perl script shuffleSequences_fastq.pl included with Velvet for this task).

It would be safer to check the record IDs do match. However, there are several different naming schemes for reads, most typically suffixes of /1 and /2, but also things like .f and .r get used. In the case of FASTQ files from the NCBI SRA, the reads have no suffixes, so to feed those into Velvet you may want to check they are equal and then add a suffix as shown below.

#This Python script requires Biopython 1.51 or later
from Bio import SeqIO
import itertools

#Setup variables (could parse command line args instead)
file_f = "SRR001666_1.fastq"
file_r = "SRR001666_2.fastq"
file_out = "SRR001666_interleaved.fastq"
format = "fastq" #or "fastq-illumina", or "fasta", or ...

def interleave(iter1, iter2) :
    for (forward, reverse) in itertools.izip(iter1,iter2):
        assert forward.id == reverse.id
        forward.id += "/1"
        reverse.id += "/2"
        yield forward
        yield reverse

records_f = SeqIO.parse(open(file_f,"rU"), format)
records_r = SeqIO.parse(open(file_r,"rU"), format)

handle = open(file_out, "w")
count = SeqIO.write(interleave(records_f, records_r), handle, format)
handle.close()
print "%i records written to %s" % (count, file_out)

This example uses the SRR001666 files from the NCBI SRA FTP site.

Now that works fine, and just by changing the filenames and the format name this could be used on FASTA data (or another supported file format). The bad news is it took 14 minutes to produce a 2GB FASTQ. However, going a little more low-level as discussed before can really pay off. This FASTQ-only version takes just 2 minutes:

#This Python script requires Biopython 1.51 or later
from Bio.SeqIO.QualityIO import FastqGeneralIterator
import itertools

#Setup variables (could parse command line args instead)
file_f = "SRR001666_1.fastq"
file_r = "SRR001666_2.fastq"
file_out = "SRR001666_interleaved.fastq"

handle = open(file_out, "w")
count = 0

f_iter = FastqGeneralIterator(open(file_f,"rU"))
r_iter = FastqGeneralIterator(open(file_r,"rU"))
for (f_id, f_seq, f_q), (r_id, r_seq, r_q) \
in itertools.izip(f_iter,r_iter):
    assert f_id == r_id
    count += 2
    #Write out both reads with "/1" and "/2" suffix on ID
    handle.write("@%s/1\n%s\n+\n%s\n@%s/2\n%s\n+\n%s\n" \
                 % (f_id, f_seq, f_q, r_id, r_seq, r_q))
handle.close()
print "%i records written to %s" % (count, file_out)

You can make this a little faster still by missing out most of the validation done by the Biopython FASTQ parser – but personally I wouldn’t take that risk. I’d much rather know about any errors in the data.

Peter

P.S.

Things get more interesting if you want to do quality filtering or trimming. If only one of a pair passes the quality assurance step, then you may want to keep it and treat it as an unpaired read. To give such cleaned up data to Velvet, you would need one file of alternating paired end reads, and a separate file of the orphaned effectively unpaired reads. That deserves another post going into more detail…

Two of the core BioPerl developers, Jason Stajich and Chris Fields, were interviewed by FLOSS Weekly.  The interview is now available as an MP3 on the FLOSS Weekly website; several streaming versions (including podcast) are also available.

We are pleased to announce the immediate availability of BioPerl 1.6.1, the latest release of BioPerl’s core code. You can grab it here:

Via CPAN:

http://search.cpan.org/~cjfields/BioPerl-1.6.1/

Via the BioPerl website:

http://bioperl.org/DIST/BioPerl-1.6.1.tar.bz2
http://bioperl.org/DIST/BioPerl-1.6.1.tar.gz
http://bioperl.org/DIST/BioPerl-1.6.1.zip

The PPM for Windows should also finally be available this week, ActivePerl problems permitting (we will post more information when it becomes available).

Tons of bug fixes and changes have been incorporated into this release. For a more complete change list please see the ‘Changes’ file included with the distribution.

A few highlights:

• FASTQ parsing and interconversion of the three FASTQ variants (Sanger, Illumina, Solexa) now works (a concerted OBF effort!)
• Significant refactoring of Bio::Restriction methods
• Complete refactoring of Bio::Search-related tiling code, including HOWTO documentation
• GBrowse-related fixes:
• - berkeleydb database now autoindexes wig files and locks correctly
• - add Pg, SQLite, and faster BerkeleyDB implementations
• Infernal 1.0 output is now parsed
• New SearchIO-based parser for gmap -f9 output
• BLAST XML parsing essentially complete
• Installation via CPANPLUS should now work
• For those using Strawberry Perl on Windows, the latest build is expected to pass all tests.
• ‘raw’ sequence format now parsed by line or optionally as a single sequence
• SCF parsing/writing now round-trips
• Demo code for using RPS-BLAST and Bio::Tools::Run::RemoteBlast
• Bio::Tools::SeqPattern now has a backtranslate() method
• Bio::Tree::Statistics now has methods to calculate Fitch-based score, internal trait values, statratio(), sum of leaf distances
• Scripts
• - update to bp_seqfeature_load for SQLite
• - hivq.pl – commmand-line interface to Bio::DB::HIV
• - fastam9_to_table – fix for MPI output
• - gccalc – total stats
• - einfo – simple script to find up-to-date NCBI databases, list field and link values for a specific database

We will shortly release updates for BioPerl-db, BioPerl-run, and BioPerl-network. Enjoy!

chris

Biopython 1.51 onward includes support for Sanger, Solexa and Illumina 1.3+ FASTQ files in Bio.SeqIO, which allows a lot of neat tricks very concisely. For example, the tutorial (PDF) has examples finding and removing primer or adaptor sequences.

However, because the Bio.SeqIO interface revolves around SeqRecord objects there is often a speed penalty. For example for FASTQ files, the quality string gets turned into a list of integers on parsing, and then re-encoded back to ASCII on writing.

The new Bio.SeqIO.convert(…) function in Biopython 1.52 onwards makes converting from FASTQ to FASTA, or between the FASTQ variants about five times faster. It can do this because it doesn’t bother with creating any objects – it just uses Python strings.

You can use the same approach in your own scripts. For example, suppose you have a Solexa FASTQ file where you want to trim all the reads, taking just the first 21 bases (say). Why might you want to do this? Well, in Solexa/Illumina there is a general decline in read quality along the sequence, so it can make sense to trim, and some algorithms like to have all the input reads the same length. Here is how I would write this using the standard Bio.SeqIO functions:

from Bio import SeqIO
records = (rec[:21] for rec in SeqIO.parse(open("untrimmed.fastq"), "fastq-solexa"))
handle = open("trimmed21.fastq", "w")
count = SeqIO.write(records, handle, "fastq-solexa")
handle.close()
print "Trimmed %i FASTQ records" % count

This works, and is very simple and general. The same template can be used on any file formats supported by Bio.SeqIO. However, it might be a bit slow for large next generation sequence files.

Instead, we can get a little more low level – and work directly with strings. This requires you to know more about the details of the FASTQ file format. Parsing FASTQ files is surprising complicated (with nasty things like line wrapping technically allowed), so we’ll still get Biopython to do that bit – but not bother with constructing SeqRecord objects and decoding the FASTQ quality strings. On the other hand, doing the FASTQ output explicitly isn’t actually too bad once you know how things work:

from Bio.SeqIO.QualityIO import FastqGeneralIterator
trim = 21
handle = open("trimmed21.fastq", "w")
for title, seq, qual in FastqGeneralIterator(open("untrimmed.fastq")) :
    handle.write("@%s\n%s\n+\n%s\n" % (title, seq[:trim], qual[:trim]))
handle.close()

Again, the solution is a very short script – but this time it is much less flexible, and not nearly as clear what is going on. On the bright side, it is many times faster. Deciding on this trade-off is down to you, but I hope this blog post has highlighted the potential usefulness of the FastqGeneralIterator function in Bio.SeqIO.QualityIO, which you might otherwise have overlooked. To find out more, please read the built in documentation (also available online):

>>> from Bio.SeqIO.QualityIO import FastqGeneralIterator
>>> help(FastqGeneralIterator)
...


Please sign up to the Biopython mailing list if you want to discuss this topic further.

Peter

The release of Biopython 1.52 earlier this week marked the end of an era, it was our last release using CVS for source code control.

As of now, Biopython is using a git repository, hosted on github.com who kindly provide git hosting for open source projects free of charge. The BioRuby project have been using github for some time, so we are in good company.

Our existing OBF hosted CVS repository will be maintained in the short to medium term as a backup, but will not be updated.

Although many people have been involved in this move, we’d like to thank Bartek Wilczynski in particular for handling the CVS to git conversion, and the mirroring our CVS updates to git during the transition period. In the next few weeks hopefully we’ll get our git usage wiki pages perfected, as we start using git for real.

We are pleased to announce the release of BioRuby 1.3.1. This new release fixes many bugs existed in 1.3.0.

Here is a brief summary of changes.

• Refactoring of BioSQL support.
• Bio::PubMed bug fixes.
• Bio::NCBI::REST bug fixes.
• Bio::GCG::Msf bug fixes.
• Bio::Fasta::Report bug fixes and added support for multiple query sequences.
• Bio::Sim4::Report bug fixes.
• Added unit tests for Bio::GCG::Msf and Bio::Sim4::Report.
• License of BioRuby is clarified.

In addition, many changes have been made, mainly bug fixes. For more information, you can see ChangeLog.

The archive is available at: http://bioruby.org/archive/bioruby-1.3.1.tar.gz

We also put RubyGems pacakge at RubyForge as always. You can easily install by using RubyGems.
% sudo gem install bio

You can also obtain bioruby gem file from bioruby.org.
http://bioruby.org/archive/gems/bio-1.3.1.gem

Hope you enjoy.