Really impressed with Tagxedo – shaped word clouds. Here is a word cloud of my PhD thesis in the shape of a Drosophila:

Simon — Thu, 02 Jun 2011 08:08:53 +0000

I worked on developmental genetics of Drosophila melanogaster for my PhD. You can find Tagxedo at http://www.tagxedo.com/. A lot of the tools (such as adding your own images for word cloud shapes) are free in the beta, but I guess won’t be forever.

Posted via email from Simon’s posterous

Graphing protein databases

Simon — Thu, 11 Nov 2010 15:24:57 +0000

I’m giving a lecture next week to the Bioinformatics Masters students here about protein structure prediction. As part of the introduction to this topic, I have a traditional ‘data explosion’ slide, to illustrate the gap between the quantity of protein sequence data available versus the number of solved protein structures in the PDB (hence the need for bioinformatics to help fill the gap, by good prediction algorithms). When I last gave this talk (scarily, 4 years ago), this slide was just text, a description of the present size of UniProt & the PDB.

Since 2006 my lecturing style has progressed somewhat, I don’t like to have slides with just words on anymore, so I wanted to replace this slide, rather than just updating the numbers. Graphs of the growing sizes of the databases are easy to find online, but to my mind the real story here is of the gap in the sizes of the 2 databases (UniProt & PDB), and whether it is growing (or are protein structural determination methods catching up). This graph doesn’t (to my knowledge) exist, so, inspired by this question on BioStar I set out to draw them.

The first task is to retrieve numbers from each of the databases of their size at particular dates. For the PDB this is simple, because they distribute a CSV file of this information. You can get it too, it’s linked to here. For UniProt, it was non-obvious where to find this information. Every time there’s a new release, the webpage documenting that release gives the size of UniProt at the point of release (and it’s components, SwissProt and TrEMBL), but it is hard to find these pages for any release that is not current. So my approach was to download the history of UniProt from their FTP server, and use BioPython to calculate the size of each release:

import os
import sys
from Bio import SwissProt

def main():
    dirs = os.listdir("data")
    results = map(numbers, dirs)

def numbers(dir):
    directory = "data/"+dir
    h = open(directory+"/reldate.txt")
    lines = h.readlines()
    h.close()
    date = lines[1].rstrip() #more processing required to return just date
    sh = open(directory+"/uniprot_sprot.dat")
    descriptions = [record.accessions for record in SwissProt.parse(sh)]
    sprot_size = len(descriptions)
    sh.close()
    th = open(directory+"/uniprot_trembl.dat") #and the same for trembl
    descriptions = [record.accessions for record in SwissProt.parse(th)]
    trembl_size = len(descriptions)
    th.close()
    return (date,sprot_size,trembl_size)

It was only once I was coming to the end of this process (slow, because we’re dealing with 16 releases of UniProt: 150GB of data) that I found this page, which was fairly hidden away, but gives me the sizes of SwissProt from the last 25 years. Curses! So much effort seemingly gone to waste. However, there doesn’t appear to be a corresponding page for TrEMBL, which is much larger (being a conceptual translation of EMBL), and I wanted these numbers too, to illustrate the full scope of the problem. So my effort was not in vein.

Now that we have all the numbers in an appropriate format (DATE,DATABASE,SIZE), we can draw some graphs. For this I use the ggplot2 library and R, which seems to be de rigueur for pretty visualisations these days. Here’s some code:

library(ggplot2)
pdb <- read.table("/path/to/data/pdb.txt", sep=",")
colnames(pdb) = c("Year", "Database", "value")
pdb$Year <- as.Date(pdb$Year)
png("/path/to/graphs/uniprot_graphs/pdb.png", bg="transparent", width=800, height=600)
qplot(Year, value, data=pdb, geom="line", color=I("red")) + scale_x_date(format="%Y") + scale_y_continuous("Entries", formatter="comma")
dev.off()

spdb <- read.table("/path/to/data/sp_pdb.txt", sep=",")
colnames(spdb) = c("Year", "Database", "value")
spdb$Year <- as.Date(spdb$Year)
png("/path/to/graphs/sp_pdb.png", bg="transparent", width=800, height=600)
qplot(Year, value, data=spdb, geom="line", group=Database, color=Database) + scale_x_date(format="%Y") + scale_y_continuous("Entries", formatter="comma")
dev.off()

all <- read.table("/path/to/data/all.txt", sep=",")
colnames(all) = c("Year", "Database", "value")
all$Year <- as.Date(all$Year)
png("/path/to/graphs/all.png", bg="transparent", width=800, height=600)
qplot(Year, value, data=all, geom="line", group=Database, color=Database) + scale_x_date(format="%Y") + scale_y_log10("Entries", breaks=c(10^4,10^5,10^6,10^7))
dev.off()

This very simple R produces 3 plots, all of which are informative in different ways.

Plot 1 is a simple restatment of the PDB graph, which I produced just so all my graphs would look the same, it’s a pretty standard exponential curve (though admittedly the numbers are slightly smaller than the numbers you may be used to seeing on such plots).

Plot 2 compares the size of SwissProt with the size of the PDB. I’m extremely happy with this one, as it shows precisely what I wanted it to, SwissProt being much larger than the PDB, and marching away at an increasing rate. For the record, the most recent size of the PDB and SwissProt in the graph are 68,998 and 522,019 respectively (compared with when I last gave the protein structure lecture: 40,132 & 241,365).

The final plot is just to scare people. It includes TrEMBL, and had to be plotted on a log10 scale, because TrEMBL is another order of magnitude larger than SwissProt (12,347,303 sequences).

Addendum – further to all this, the problem of the gap between sequence and structure is actually more stark than presented here. Although the PDB today (11/11/10) contains 69,162 structures, they are highly redundant, and there are only 39,724 unique sequences of known structure.

5 Best Data Visualization Projects of the Year – 2009 | FlowingData

Simon — Wed, 16 Dec 2009 09:24:06 +0000

via flowingdata.com

The top visualization project of the year, according to FlowingData, is a project by Ben Fry, which shows changes to the theory of evolution over time. The project takes advantage of the publication of the full works of Darwin online to trace changes in the text of On the Origin of Species over time.
A deserved winner, and very apt, considering the batch of anniversaries that have gone by this year.

Posted via web from Simon’s posterous

Wordle

Simon — Fri, 12 Dec 2008 09:54:53 +0000

I’m a bit behind the zeitgeist here, I know, but Wordle is a pretty cool little tool. I think the cloud of my CiteULike tags fairly reflects my academic interests, and the fact that even after 5 years away from the lab, ‘wet’ techniques dominate, I really should make more of the bioinformatics involved in the papers I bookmark.

Wordle Tag Cloud of my CiteULike Library

Find it on Wordle here.

A few notes on technique. I extracted the tags from the RIS export of my CiteULike library using this Python script, I had to limit the represntation of words to 25, otherwise ‘proteomics’ would have dominated to such an extent that virtually none of the other tags would be visible (except maybe ‘protein-protein-interactions’). But then I am responsible for supporting proteomics researchers, and predicting and validating protein-protein interactions is my major personal research interest. So I guess this is fair enough.

Fuzzier Logic » Visualisation

Really impressed with Tagxedo – shaped word clouds. Here is a word cloud of my PhD thesis in the shape of a Drosophila:

Graphing protein databases

5 Best Data Visualization Projects of the Year – 2009 | FlowingData

Wordle