29 October 2010

Revisions=f(time): and the winner is Charles Darwin

I've been playing with the mediawiki API to draw a timeline of popular scientists through the last centuries using the number of revisions in wikipedia as the Y-axis. The java code generating this timeline is available here:WikipediaBioEdits.java. This code:

  • loops over the articles having a Template:infobox scientist
  • gets the categories of an article and retrieves the birth/death dates threw the categories 'Births_by_year' and 'Deaths_by_year'
  • gets one picture in an article an resizes it to 64px
  • counts the number of revisions for an article


At the end, the generated timeline looks like this: (best viewed with firefox):
1192196227323502427250425812658273528122152515791633168717411795184919031957

That's it

Pierre

27 October 2010

Coloring a black and white drawing with the GIMP: my notebook

I've been kindly asked to draw a proposal for the cover of http://www.cell.com/immunity (It's just a project and I'm not sure it will be accepted). So, here is my notebook for using GIMP (GNU Image Manipulation Program) to colorize a black and white drawing. Foreword:: just like everybody: I do hate gimp.


The Original Drawing


Scan the original drawing at 600dpi. Open it as a ".xcf" color image (the native file format of the Gimp).
(Image stored on OpenWetWare )

Wash Tint


Create a new layer with a white background under the original source (mode = multiply) and fill the shapes . The visible top layer is displayed over the layer to see the strokes.


Gradients


Add a new transparent layer (mode = normal ) and create a radial gradient (transparent-black) from the nucleus to the side of the sheet. Use the eraser to highlight the pyramid-shaped proteins.


Airbrush


Create a new transparent layer (mode = normal ) and use the airbrushes to draw some shadows.




Merge the layers


Et voila !


That's it,

Pierre

24 October 2010

Where are the alternative reading frames in the Human Genome ?

The following post was inspired by a question asked recently on Biostar :"Do exons ever have different reading frames in spliced variants?".

To find those alternative reading frames I've used the table KnownGene available at UCSC from : http://hgdownload.cse.ucsc.edu/goldenPath/hg18/database/knownGene.txt.gz. This file contains the positions of the exons for each transcript in the human genome:

mysql -h genome-mysql.cse.ucsc.edu -A -u genome -D hg18 -e 'select * from knownGene limit 10\G'

(...)
*************************** 7. row ***************************
name: uc009vis.1
chrom: chr1
strand: -
txStart: 4268
txEnd: 6628
cdsStart: 4268
cdsEnd: 4268
exonCount: 4
exonStarts: 4268,4832,5658,6469,
exonEnds: 4692,4901,5805,6628,
proteinID:
alignID: uc009vis.1
*************************** 8. row ***************************
name: uc009vit.1
chrom: chr1
strand: -
txStart: 4268
txEnd: 9622
cdsStart: 4268
cdsEnd: 4268
exonCount: 9
exonStarts: 4268,4832,5658,6469,6720,7095,7777,8130,8775,
exonEnds: 4692,4901,5810,6628,6918,7605,7924,8229,9622,
proteinID:
alignID: uc009vit.1


The following java program creates an array of bytes having a length greater than the length of the human chromosome chr1. This array is initialized with the constant 'NIL'. Then for each chromosome and each transcript, we loop over each exon and we record what was the reading frame (0, 1 or 2) at a given position. If this position was already flagged with another frame, a warning is printed to stdout.

Compilation & Execution


javac BioStar3034.java
java BioStar3034

Result


(...)
chr1:53286155-53286156 (+)
chr1:53286156-53286157 (+)
chr1:53286157-53286158 (+)
chr1:53286158-53286159 (+)
chr1:53286159-53286160 (+)
chr1:53286160-53286161 (+)
chr1:53286161-53286162 (+)
chr1:53286162-53286163 (+)
(...)
java BioStar3034 | sort | uniq | wc -l
300696



(Image from UCSC/OpenWetWare)


That's it
Pierre

11 October 2010

A custom JSP tag for mediawiki

I'm pretty happy with the following custom JSP tag for mediawiki I wrote last week (http://code.google.com/p/code915/source/browse/trunk/charpak/src/WEB-INF/src/fr/inserm/umr915/charpak/j2ee/tags/MediaWikiTag.java). For a given title, it calls the mediawiki api to test if an article exists in a mediawiki installation. In my case, this system will be useful to let my users write some custom annotations to some records from our database.

<u915:mediawiki title="${geneName}" preload="Template:Gene"/>
If the article does exist on the mediawiki installation a simple hyperlink is created. If the article does not exist, an hyperlink for creating/editing the new article is displayed with a predload option: Preloading wikitext presents the user with a partially created page rather than a blank page, possibly with inline instructions for content organization..For example, in the following web application, the blue and red icons point to some existing articles or some articles to be created:


Internals

This custom tag calls the mediawiki api (e.g api.php?action=query&format=xml&titles=Charles+Darwin )and get the XML response as a DOM document. An XPath expression is then evaluated to test if an attribute '@missing' was declared for the given article.
boolean pageFound=false;
String termUTF8 = URLEncoder.encode(term,"UTF-8");
URL url=new URL(baseurl+"/api.php?action=query&format=xml&titles="+termUTF8);
URLConnection con=url.openConnection();
in=con.getInputStream();
Document dom=this.domBuilder.parse(in);
Element e=(Element)this.xpathPage.evaluate(dom,XPathConstants.NODE);
if(e!=null && e.getAttributeNode("missing")==null)
{
pageFound=true;
}
in.close();


That's it,
Pierre

Playing with the Wordle algorithm: a tag cloud of Mesh Terms

The paper describing Wordle has been recently published (http://www.research.ibm.com/visual/papers/wordle_final2.pdf ). The algorithm was briefly described: “The most distinctive geometric aspect of a Wordle is the layout algorithm, which packs words to make efficient use of space. While many space-filling visualizations exist, they typically work by recursiveLayout proceeds according to this pseudocode:

sort words by weight, decreasing
for each word w:
w.position := makeInitialPosition(w);
while w intersects other words:
updatePosition(w);

The two key procedures here are "makeInitialPosition" and "updatePosition". The makeInitialPosition routine picks a point at random according to a distribution that takes into account the desired overall shape, and, if desired, alphabetical order. The updatePosition routine moves the word on a spiral of increasing radius, radiating from the word's starting position. The updatePosition routine is aware of constraints on the overall shape of the Wordle. Constraining the layout to a rectangular shape causes updatePosition to prefer positions inside of the strict boundaries of the playing field; a blobby overall shape accepts boundary violations. The rectangular constraint is relaxed when the spiral radius exceeds either playing field dimension. ”
Your browser does not support the <CANVAS> element !


Just for fun , I've implemented my own version of the Wordle Alogrithm. The Java code is available on github at http://github.com/lindenb/jsandbox/blob/master/src/sandbox/MyWordle.java. I won't describe the program here, but I'll just say that the code invokes a java.awt.font.TextLayout class to get the shape of the text:
Graphics2D g=(...)
FontRenderContext frc = g.getFontRenderContext();
Font font=new Font("Dialog",Font.BOLD,fontSize);
TextLayout textLayout=new TextLayout(w.getText(), font, frc);
Shape shape=textLayout.getOutline(null);
and a java.awt.geom.Area to test if two shapes intersects.

Ok, let's test this code. FIrst I'm going to dump a pubmed query as XML with another simple tool named PubmedDump. This XML file is then parsed with a javascript program called from the SAX parser saxstream.jar (previously described here):
mesh.js
importPackage(Packages.sandbox);
importPackage(Packages.java.io);
importPackage(Packages.java.awt);
var content=null;
var mesh2count={};

function startElement(uri,localName,name,atts)
{
if(name=="DescriptorName")
{
content="";
}
}
function characters(s)
{
if(content!=null) content+=s;
}
function endElement(uri,localName,name)
{
if(content!=null)
{
var count=mesh2count[content];
if(count===undefined) count=0;
mesh2count[content]=count+1;
}
content=null;
}
function endDocument()
{
var w= new MyWordle();
for(var s in mesh2count)
{
var word= new MyWordle.Word(s,mesh2count[s]);
w.add(word);
}
w.setUseArea(true);/* use shape area instead of bounding boxes */
w.setAllowRotate(true);
w.setSortType(1);/* sort by weight */
w.doLayout();

var f=new File("result.svg");
w.saveAsSVG(f);
}
This javascript code counts the occurence of each MESH term (<DescriptorName>), and when the document has been parsed, a new instance of MyWordle class is created, filled and the result is saved to a SVG file.

Invocation


Here is an example for the query "Rotavirus NSP3 NSP1":
java -jar pubmeddump.jar "Rotavirus NSP3 NSP1" |\
java -cp mywordle.jar:saxscript.jar org.lindenb.tinytools.SAXScript -n -f mesh.js


Result





That's it,
Pierre

10 October 2010

The Geek is dead. Long live the Geek :-)

congratulations to Neil! ;-)



24 September 2010

Connecting to a MongoDB database from R using the C API for MongoDB

Today, Neil posted an article titled" Connecting to a MongoDB database from R using Java". In the current post, I'll show how to use the C API for MongoDB to fetch some MongoDB data from R. The code will be somehow similar to my previous post "A stateful C function for R: parsing Fasta sequences".

OK, First, let's add a few values in mongo:

for(i=1;i> 20;++i)
db.dbsnps.save({_id:"rs"+i,name:"rs"+i});


The C code contains 3 functions.

The first function mongoRconnect connects to the MongoDB server and put the pointer into a R variable.
SEXP mongoRconnect()
{
mongo_connection* conn; /* ptr */
mongo_connection_options opts[1];
mongo_conn_return status;

conn=(mongo_connection*)malloc(sizeof(mongo_connection));
strcpy( opts->host , "127.0.0.1" );
opts->port = 27017;
status = mongo_connect( conn, opts );

return R_MakeExternalPtr(conn, R_NilValue, R_NilValue);
}
The second method mongoRdiconnect closes the connection:
SEXP mongoRdiconnect(SEXP r_handle)
{
mongo_connection* conn;
conn = (mongo_connection*)R_ExternalPtrAddr(r_handle);;
if(conn==NULL) MONGO_ERROR("conn==NULL");
mongo_destroy( conn );
free(conn);
R_ClearExternalPtr(r_handle);
return ScalarInteger(0);
}
The last method mongoRquery scans the database test.dbsnps and inserts the name of the snps into an R array:
SEXP mongoRquery(SEXP r_handle)
{
SEXP values=NULL;
mongo_cursor *cursor;
bson empty[1];
bson_empty( empty );
int i;
mongo_connection* conn = R_ExternalPtrAddr(r_handle);
SEXP* array=NULL;
int array_size=0;
//the R value contains two objects


if(conn==NULL) MONGO_ERROR("handle==NULL");
cursor = mongo_find( conn,
"test.dbsnps",/* ns */
empty,/* fields */
empty,/* return */
0,/* return */
0,/* skip */
0 /* options */
);

while( mongo_cursor_next( cursor ) )
{
bson_iterator it[1];
if ( bson_find( it, &(cursor->current), "name" ))
{
array=(SEXP*)realloc(array,(array_size+1)*sizeof(SEXP));
if(array==NULL) error("out of memory");
array[array_size]=mkChar( bson_iterator_string( it ));
array_size++;
}
}
mongo_cursor_destroy( cursor );
PROTECT(values = allocVector(STRSXP, array_size));
for(i=0;i< array_size;++i)
{
SET_STRING_ELT(values, i, array[i]);
}
free(array);
UNPROTECT(1);
return values;
}
This C code is then be called from R:
mongo <- mongo.open()
mongo.snps(mongo)
mongo.close(mongo)

Result:
[1] "rs1" "rs2" "rs3" "rs4" "rs5" "rs6" "rs7" "rs8" "rs9" "rs10"
[11] "rs11" "rs12" "rs13" "rs14" "rs15" "rs16" "rs17" "rs18" "rs19"



Source code


Makefile

R_HOME=R-2.11.0
MONGO_HOME=mongo-c-driver
run:
gcc -fPIC -I -g -c -Wall -DMONGO_HAVE_STDINT -I ${R_HOME}/include -I ${MONGO_HOME}/src mongoR.c ${MONGO_HOME}/src/*.c
gcc -shared -Wl,-soname,rmongo.so.1 -o librmongo.so *.o
${R_HOME}/bin/R --no-save < mongo.R
clean:
rm *.o


mongoR.c

(again, I'm not sure about those PROTECT/UNPROTECT ...)
#include <ctype.h>
#include <errno.h>
#include <R.h>
#include <Rinternals.h>
#include <bson.h>
#include <mongo.h>

#define MONGO_ERROR(a) { error(a); fputs(a,stdout);exit(EXIT_FAILURE);}

/**
* connect to MONGO
*/
SEXP mongoRconnect()
{
mongo_connection* conn; /* ptr */
mongo_connection_options opts[1];
mongo_conn_return status;

conn=(mongo_connection*)malloc(sizeof(mongo_connection));
if(conn==NULL)
{
MONGO_ERROR("out of memory");
}

strcpy( opts->host , "127.0.0.1" );
opts->port = 27017;

status = mongo_connect( conn, opts );
if(status!= mongo_conn_success)
{
MONGO_ERROR("connection failed");
}

/** the handle is bound a R variable */
return R_MakeExternalPtr(conn, R_NilValue, R_NilValue);
}

/**
* close the mongo connection
*/
SEXP mongoRdiconnect(SEXP r_handle)
{
mongo_connection* conn;
conn = (mongo_connection*)R_ExternalPtrAddr(r_handle);;
if(conn==NULL) MONGO_ERROR("conn==NULL");
mongo_destroy( conn );
free(conn);
R_ClearExternalPtr(r_handle);
return ScalarInteger(0);
}

/**
* get all SNPS
*/
SEXP mongoRquery(SEXP r_handle)
{
SEXP values=NULL;
mongo_cursor *cursor;
bson empty[1];
bson_empty( empty );
int i;
mongo_connection* conn = R_ExternalPtrAddr(r_handle);
SEXP* array=NULL;
int array_size=0;
//the R value contains two objects


if(conn==NULL) MONGO_ERROR("handle==NULL");
cursor = mongo_find( conn,
"test.dbsnps",/* ns */
empty,/* fields */
empty,/* return */
0,/* return */
0,/* skip */
0 /* options */
);

while( mongo_cursor_next( cursor ) )
{
bson_iterator it[1];
if ( bson_find( it, &(cursor->current), "name" ))
{
array=(SEXP*)realloc(array,(array_size+1)*sizeof(SEXP));
if(array==NULL) error("out of memory");
array[array_size]=mkChar( bson_iterator_string( it ));
array_size++;
}
}
mongo_cursor_destroy( cursor );
PROTECT(values = allocVector(STRSXP, array_size));
for(i=0;i< array_size;++i)
{
SET_STRING_ELT(values, i, array[i]);
}
free(array);
UNPROTECT(1);
return values;
}


mongo.R

dyn.load(paste("librmongo", .Platform$dynlib.ext, sep=""))

mongo.open <- function()
{
.Call("mongoRconnect")
}

mongo.close <- function(handler)
{
.Call("mongoRdiconnect",handler)
}

mongo.snps <- function(handler)
{
.Call("mongoRquery",handler)
}

mongo <- mongo.open()
mongo.snps(mongo)
mongo.close(mongo)


That's it,
Pierre