Mapping the UCSC/Web-Sequences to a world map.

People at the UCSC have recently released a new track for the GenomeBrowser

We BLATted the Internet! The DNA sequences from 40 billion webpages mapped to hg19 and other species: http://t.co/5XAsFCguE2

/ UCSC Genome Browser (@GenomeBrowser) January 23, 2014

"We're pleased to announce the release of the Web Sequences track on the UCSC Genome Browser. This track, produced in collaboration with Microsoft Research, contains the results of a 30-day scan for DNA sequences from over 40 billion different webpages. The sequences were then mapped with Blat to the human genome (...) The data were extracted from a variety of sources including patents, online textbooks, help forums, and any other webpages that contain DNA sequence. In essence, this track displays the Blat alignments of nearly every DNA sequence on the internet!"

I've mapped each genomic location from this track to a country and generated the following (unreadable) picture:

How this picture was generated

• I've downloaded the data from the UCSC using the Table browser. The data look like this:

  #bin    chrom   chromStart      chromEnd        name    score   strand  thickStart      thickEnd        reserved        blockCount      blockSizes      chromStarts     tSeqTypes     seqIds  seqRanges       publisher       pmid    doi     issn    journal title   firstAuthor     year    impact  classes locus
  585     chr1    14789   15004   3500336380      75              14789   15004   8421504 2       40,35   0,180   g       350033638000000000      0-75                          Tophat, Cufflinks and replicates - Page 2 - SEQanswers  seqanswers.com  0       0               WASH2P,WASH7P
  585     chr1    15017   15590   3500327042      381             15017   15590   8421504 2       326,55  0,518   g       350032704200000008      0-747                        Research Technologies at Indiana University      biomedapp.iu.edu        0       0               WASH7P
  585     chr1    68858   68895   3500020489      37              68858   68895   8421504 1       37      0       g       350002048900000000,350002048900000001   0-36,0-36    Genome mapability - Musings from a PhD candidate davetang.org    0       0               OR4F5
  585     chr1    69170   69479   3500359797      142             69170   69479   8421504 2       76,66   0,243   c       350035979700000000,350035979700000002   0-76,10-76    CRAM compression and TLEN SAM's field - SEQanswers      seqanswers.com  0       0               OR4F5
  585     chr1    70013   70230   3500427570      150             70013   70230   8421504 2       75,75   0,142   g       350042757000000000,350042757000000001   0-75,0-75     Inconsistency with SAM flag output? - SEQanswers        seqanswers.com  0       0               OR4F5
  585     chr1    98860   98888   3500207083      26              98860   98888   8421504 3       5,7,14  0,6,14  g       350020708300000108,350020708300000060,350020708300000239      0-24,0-21,0-21                                          Method For The Simultaneous Determination Of Blood Group And Platelet Antigen Genotypes         .freshpatents.com     0       0               OR4F5
  586     chr1    137603  138008  3500170315      405             137603  138008  8421504 1       405     0       p       350017031500015076,350017031500015074   0-135,0-270  Balding D. (2007) Handbook of Statistical Genetics       www.scribd.com  0       0               OR4F5
  586     chr1    139485  143008  3500419332      1794            139485  143008  8421504 2       65,1729 0,1794  g       350041933200000004,350041933200000000,350041933200000001,350041933200000002,350041933200000003        0-1263,0-1859,0-1852,0-1860,0-576                                               PPT  Evolution by Genome Duplication PowerPoint presentation | free to view   www.powershow.com       0       0               OR4F5
  586     chr1    141535  143008  3500270480      1372            141535  143008  8421504 24      57,60,58,59,61,59,60,59,59,62,61,58,60,58,16,59,59,59,59,57,57,59,58,58 0,61,125,187,250,314,377,441,503,566,631,695,756,819,881,919,981,1044,1107,1170,1230,1291,1353,1415   g       350027048000000003,350027048000000002   0-902,0-525                  Chen-Kung Chou 3-22-2004 www.dls.ym.edu.tw       0       0               OR4F5

• I want to generate BED file: 'chrom/start/end/country'. The 23rd column contains the URL of the web-sequence. I use the domain of the URL to try to guess the country. The following awk script was used to generate the file:

  BEGIN   {
          FS="[\t]";
          }
  
          {
          country=$23;
          for(;;)
                  {
                  slash=index(country,"/");
                  if(slash==0) break;
                  country=substr(country,1,slash-1);
                  }
          for(;;)
                  {
                  colon=index(country,":");
                  if(colon==0) break;
                  country=substr(country,1,colon-1);
                  }
          if( country ~ /\.$/ ) next;
          if( country ~ /\.com$/ ) next;
          if( country ~ /\.org$/ ) next;
          if( country ~ /\.cat$/ ) next;
          if( country ~ /\.net$/ ) next;
          if( country ~ /\.gov$/ ) next;
          if( country ~ /\.edu$/ ) next;
          if( country ~ /\.name$/ ) next;
          if( country ~ /\.info$/ ) next;
          if( country ~ /\.biz$/ ) next;
          if( country ~ /\.[0-9]+$/ ) next;
          if( index(country,".")==0) next;
          if( index(country," ")!=0) next;
          for(;;)
                  {
                  dot=index(country,".");
                  if(dot==0) break;
                  country=substr(country,dot+1);
                  }
  
                  if( country== "af") {country="afghanistan";}
                  else if( country== "ax") {country="Ålandislands";}
                  else if( country== "al") {country="albania";}
                  else if( country== "dz") {country="algeria";}
                  else if( country== "as") {country="americansamoa";}
                  else if( country== "ad") {country="andorra";}
                  else if( country== "ao") {country="angola";}
                  else if( country== "ai") {country="anguilla";}
                  else if( country== "aq") {country="antarctica";}
                  else if( country== "ag") {country="antiguaandbarbuda";}
                  else if( country== "ar") {country="argentina";}
                  else if( country== "am") {country="armenia";}
                  else if( country== "aw") {country="aruba";}
                  else if( country== "au") {country="australia";}
                  else if( country== "at") {country="austria";}
                  else if( country== "az") {country="azerbaijan";}
                  else if( country== "bs") {country="bahamas";}
                  else if( country== "bh") {country="bahrain";}
                  else if( country== "bd") {country="bangladesh";}
                  else if( country== "bb") {country="barbados";}
                  else if( country== "by") {country="belarus";}
                  else if( country== "be") {country="belgium";}
                  else if( country== "bz") {country="belize";}
                  else if( country== "bj") {country="benin";}
                  else if( country== "bm") {country="bermuda";}
                  else if( country== "bt") {country="bhutan";}
                  else if( country== "bo") {country="bolivia,plurinationalstateof";}
                  else if( country== "bq") {country="bonaire,sinteustatiusandsaba";}
                  else if( country== "ba") {country="bosniaandherzegovina";}
                  else if( country== "bw") {country="botswana";}
                  else if( country== "bv") {country="bouvetisland";}
                  else if( country== "br") {country="brazil";}
                  else if( country== "io") {country="britishindianoceanterritory";}
                  else if( country== "bn") {country="bruneidarussalam";}
                  else if( country== "bg") {country="bulgaria";}
                  else if( country== "bf") {country="burkinafaso";}
                  else if( country== "bi") {country="burundi";}
                  else if( country== "kh") {country="cambodia";}
                  else if( country== "cm") {country="cameroon";}
                  else if( country== "ca") {country="canada";}
                  else if( country== "cv") {country="capeverde";}
                  else if( country== "ky") {country="caymanislands";}
                  else if( country== "cf") {country="centralafricanrepublic";}
                  else if( country== "td") {country="chad";}
                  else if( country== "cl") {country="chile";}
                  else if( country== "cn") {country="china";}
                  else if( country== "cx") {country="christmasisland";}
                  else if( country== "cc") {country="cocos(keeling)islands";}
                  else if( country== "co") {country="colombia";}
                  else if( country== "km") {country="comoros";}
                  else if( country== "cg") {country="congo";}
                  else if( country== "cd") {country="congo,thedemocraticrepublicofthe";}
                  else if( country== "ck") {country="cookislands";}
                  else if( country== "cr") {country="costarica";}
                  else if( country== "ci") {country="cÔted'ivoire";}
                  else if( country== "hr") {country="croatia";}
                  else if( country== "cu") {country="cuba";}
                  else if( country== "cw") {country="curaÇao";}
                  else if( country== "cy") {country="cyprus";}
                  else if( country== "cz") {country="czechrepublic";}
                  else if( country== "dk") {country="denmark";}
                  else if( country== "dj") {country="djibouti";}
                  else if( country== "dm") {country="dominica";}
                  else if( country== "do") {country="dominicanrepublic";}
                  else if( country== "ec") {country="ecuador";}
                  else if( country== "eg") {country="egypt";}
                  else if( country== "sv") {country="elsalvador";}
                  else if( country== "gq") {country="equatorialguinea";}
                  else if( country== "er") {country="eritrea";}
                  else if( country== "ee") {country="estonia";}
                  else if( country== "et") {country="ethiopia";}
                  else if( country== "fk") {country="falklandislands(malvinas)";}
                  else if( country== "fo") {country="faroeislands";}
                  else if( country== "fj") {country="fiji";}
                  else if( country== "fi") {country="finland";}
                  else if( country== "fr") {country="france";}
                  else if( country== "gf") {country="frenchguiana";}
                  else if( country== "pf") {country="frenchpolynesia";}
                  else if( country== "tf") {country="frenchsouthernterritories";}
                  else if( country== "ga") {country="gabon";}
                  else if( country== "gm") {country="gambia";}
                  else if( country== "ge") {country="georgia";}
                  else if( country== "de") {country="germany";}
                  else if( country== "gh") {country="ghana";}
                  else if( country== "gi") {country="gibraltar";}
                  else if( country== "gr") {country="greece";}
                  else if( country== "gl") {country="greenland";}
                  else if( country== "gd") {country="grenada";}
                  else if( country== "gp") {country="guadeloupe";}
                  else if( country== "gu") {country="guam";}
                  else if( country== "gt") {country="guatemala";}
                  else if( country== "gg") {country="guernsey";}
                  else if( country== "gn") {country="guinea";}
                  else if( country== "gw") {country="guinea-bissau";}
                  else if( country== "gy") {country="guyana";}
                  else if( country== "ht") {country="haiti";}
                  else if( country== "hm") {country="heardislandandmcdonaldislands";}
                  else if( country== "va") {country="holysee(vaticancitystate)";}
                  else if( country== "hn") {country="honduras";}
                  else if( country== "hk") {country="china";}
                  else if( country== "hu") {country="hungary";}
                  else if( country== "is") {country="iceland";}
                  else if( country== "in") {country="india";}
                  else if( country== "id") {country="indonesia";}
                  else if( country== "ir") {country="iran";}
                  else if( country== "iq") {country="iraq";}
                  else if( country== "ie") {country="ireland";}
                  else if( country== "im") {country="isleofman";}
                  else if( country== "il") {country="israel";}
                  else if( country== "it") {country="italy";}
                  else if( country== "jm") {country="jamaica";}
                  else if( country== "jp") {country="japan";}
                  else if( country== "je") {country="jersey";}
                  else if( country== "jo") {country="jordan";}
                  else if( country== "kz") {country="kazakhstan";}
                  else if( country== "ke") {country="kenya";}
                  else if( country== "ki") {country="kiribati";}
                  else if( country== "kp") {country="northkorea";}
                  else if( country== "kr") {country="southkorea";}
                  else if( country== "kw") {country="kuwait";}
                  else if( country== "kg") {country="kyrgyzstan";}
                  else if( country== "la") {country="laopeople'sdemocraticrepublic";}
                  else if( country== "lv") {country="latvia";}
                  else if( country== "lb") {country="lebanon";}
                  else if( country== "ls") {country="lesotho";}
                  else if( country== "lr") {country="liberia";}
                  else if( country== "ly") {country="libya";}
                  else if( country== "li") {country="liechtenstein";}
                  else if( country== "lt") {country="lithuania";}
                  else if( country== "lu") {country="luxembourg";}
                  else if( country== "mo") {country="macao";}
                  else if( country== "mk") {country="macedonia,theformeryugoslavrepublicof";}
                  else if( country== "mg") {country="madagascar";}
                  else if( country== "mw") {country="malawi";}
                  else if( country== "my") {country="malaysia";}
                  else if( country== "mv") {country="maldives";}
                  else if( country== "ml") {country="mali";}
                  else if( country== "mt") {country="malta";}
                  else if( country== "mh") {country="marshallislands";}
                  else if( country== "mq") {country="martinique";}
                  else if( country== "mr") {country="mauritania";}
                  else if( country== "mu") {country="mauritius";}
                  else if( country== "yt") {country="mayotte";}
                  else if( country== "mx") {country="mexico";}
                  else if( country== "fm") {country="micronesia,federatedstatesof";}
                  else if( country== "md") {country="moldova,republicof";}
                  else if( country== "mc") {country="monaco";}
                  else if( country== "mn") {country="mongolia";}
                  else if( country== "me") {country="montenegro";}
                  else if( country== "ms") {country="montserrat";}
                  else if( country== "ma") {country="morocco";}
                  else if( country== "mz") {country="mozambique";}
                  else if( country== "mm") {country="myanmar";}
                  else if( country== "na") {country="namibia";}
                  else if( country== "nr") {country="nauru";}
                  else if( country== "np") {country="nepal";}
                  else if( country== "nl") {country="netherlands";}
                  else if( country== "nc") {country="newcaledonia";}
                  else if( country== "nz") {country="newzealand";}
                  else if( country== "ni") {country="nicaragua";}
                  else if( country== "ne") {country="niger";}
                  else if( country== "ng") {country="nigeria";}
                  else if( country== "nu") {country="niue";}
                  else if( country== "nf") {country="norfolkisland";}
                  else if( country== "mp") {country="northernmarianaislands";}
                  else if( country== "no") {country="norway";}
                  else if( country== "om") {country="oman";}
                  else if( country== "pk") {country="pakistan";}
                  else if( country== "pw") {country="palau";}
                  else if( country== "ps") {country="palestine,stateof";}
                  else if( country== "pa") {country="panama";}
                  else if( country== "pg") {country="papuanewguinea";}
                  else if( country== "py") {country="paraguay";}
                  else if( country== "pe") {country="peru";}
                  else if( country== "ph") {country="philippines";}
                  else if( country== "pn") {country="pitcairn";}
                  else if( country== "pl") {country="poland";}
                  else if( country== "pt") {country="portugal";}
                  else if( country== "pr") {country="puertorico";}
                  else if( country== "qa") {country="qatar";}
                  else if( country== "re") {country="france";}
                  else if( country== "ro") {country="romania";}
                  else if( country== "ru") {country="russia";}
                  else if( country== "rw") {country="rwanda";}
                  else if( country== "bl") {country="saintbarthÉlemy";}
                  else if( country== "sh") {country="sainthelena,ascensionandtristandacunha";}
                  else if( country== "kn") {country="saintkittsandnevis";}
                  else if( country== "lc") {country="saintlucia";}
                  else if( country== "mf") {country="saintmartin(frenchpart)";}
                  else if( country== "pm") {country="saintpierreandmiquelon";}
                  else if( country== "vc") {country="saintvincentandthegrenadines";}
                  else if( country== "ws") {country="samoa";}
                  else if( country== "sm") {country="sanmarino";}
                  else if( country== "st") {country="saotomeandprincipe";}
                  else if( country== "sa") {country="saudiarabia";}
                  else if( country== "sn") {country="senegal";}
                  else if( country== "rs") {country="serbia";}
                  else if( country== "sc") {country="seychelles";}
                  else if( country== "sl") {country="sierraleone";}
                  else if( country== "sg") {country="singapore";}
                  else if( country== "sx") {country="sintmaarten(dutchpart)";}
                  else if( country== "sk") {country="slovakia";}
                  else if( country== "si") {country="slovenia";}
                  else if( country== "sb") {country="solomonislands";}
                  else if( country== "so") {country="somalia";}
                  else if( country== "za") {country="southafrica";}
                  else if( country== "gs") {country="southgeorgiaandthesouthsandwichislands";}
                  else if( country== "ss") {country="southsudan";}
                  else if( country== "es") {country="spain";}
                  else if( country== "lk") {country="srilanka";}
                  else if( country== "sd") {country="sudan";}
                  else if( country== "sr") {country="suriname";}
                  else if( country== "sj") {country="svalbardandjanmayen";}
                  else if( country== "sz") {country="swaziland";}
                  else if( country== "se") {country="sweden";}
                  else if( country== "ch") {country="switzerland";}
                  else if( country== "sy") {country="syrianarabrepublic";}
                  else if( country== "tw") {country="taiwan";}
                  else if( country== "tj") {country="tajikistan";}
                  else if( country== "tz") {country="tanzania";}
                  else if( country== "th") {country="thailand";}
                  else if( country== "tl") {country="timor-leste";}
                  else if( country== "tg") {country="togo";}
                  else if( country== "tk") {country="tokelau";}
                  else if( country== "to") {country="tonga";}
                  else if( country== "tt") {country="trinidadandtobago";}
                  else if( country== "tn") {country="tunisia";}
                  else if( country== "tr") {country="turkey";}
                  else if( country== "tm") {country="turkmenistan";}
                  else if( country== "tc") {country="turksandcaicosislands";}
                  else if( country== "tv") {country="tuvalu";}
                  else if( country== "ug") {country="uganda";}
                  else if( country== "ua") {country="ukraine";}
                  else if( country== "ae") {country="unitedarabemirates";}
                  else if( country== "gb") {country="unitedkingdom";}
                  else if( country== "uk") {country="unitedkingdom";}
                  else if( country== "us") {country="USA";}
                  else if( country== "um") {country="unitedstatesminoroutlyingislands";}
                  else if( country== "uy") {country="uruguay";}
                  else if( country== "uz") {country="uzbekistan";}
                  else if( country== "vu") {country="vanuatu";}
                  else if( country== "ve") {country="venezuela";}
                  else if( country== "vn") {country="vietnam";}
                  else if( country== "vg") {country="virginislands,british";}
                  else if( country== "vi") {country="virginislands,u.s.";}
                  else if( country== "wf") {country="wallisandfutuna";}
                  else if( country== "eh") {country="westernsahara";}
                  else if( country== "ye") {country="yemen";}
                  else if( country== "zm") {country="zambia";}
                  else if( country== "zw") {country="zimbabwe";}
                  else { next;}
  
          printf("%s\t%s\t%s\t%s\n",$2,$3,$4,country);
          }
  

• For the world map, I've used a SVG-vectorial map from wikipedia: https://commons.wikimedia.org/wiki/File:World_V2.0.svg.
  
  
  
  The coordinates of the boundaries of each country are defined in a SVG 'path' element:

  <?xml version="1.0" encoding="UTF-8" standalone="no"?>
  <!DOCTYPE svg PUBLIC "-//W3C//DTD SVG 20010904//EN" "http://www.w3.org/TR/2001/REC-SVG-20010904/DTD/
  <svg xmlns="http://www.w3.org/2000/svg" width="8.88889in" height="4.44444in" viewBox="0 0 800 400">
    <path id="Taiwan" fill="none" stroke="black" stroke-width="1" d="M 668.85,151.22  C 668.98,150.71 ...
    <path id="Estonia" fill="none" stroke="black" stroke-width="1" d="M 460.75,68.26  C 459.95,68.11 4 ...
    <path id="Latvia" fill="none" stroke="black" stroke-width="1" d="M 461.23,72.27  C 460.75,72.20 46 ...
    <path id="Lithuania" fill="none" stroke="black" stroke-width="1" d="M 452.39,79.42  C 452.67,79.72 ...
    <path id="Byelarus" fill="none" stroke="black" stroke-width="1" d="M 453.57,81.92  C 453.87,82.37  ...
    <path id="Ukraine" fill="none" stroke="black" stroke-width="1" d="M 453.09,85.95  C 453.43,86.30 4 ...
    <path id="Moldova" fill="none" stroke="black" stroke-width="1" d="M 460.57,93.00  C 461.00,93.70 4 ...
    <path id="Syria" fill="none" stroke="black" stroke-width="1" d="M 480.33,127.61  C 481.03,127.90 4 ...
    <path id="Turkey" fill="none" stroke="black" stroke-width="1" d="M 499.47,116.91  C 499.31,116.32  ...
    <path id="Kuwait" fill="none" stroke="black" stroke-width="1" d="M 505.26,133.84  C 504.97,134.56  ...
    <path id="Saudi Arabia" fill="none" stroke="black" stroke-width="1" d="M 495.83,163.75  C 496.31,1 ...
    <path id="United Arab Emirates" fill="none" stroke="black" stroke-width="1" d="M 516.03,150.12  C  ...
    <path id="Yemen" fill="none" stroke="black" stroke-width="1" d="M 517.68,161.79  C 517.01,160.50 5 ...
    <path id="Slovenia" fill="none" stroke="black" stroke-width="1" d="M 430.55,97.07  C 429.62,97.36  ...
    <path id="Croatia" fill="none" stroke="black" stroke-width="1" d="M 439.09,103.97  C 439.01,103.46 ...
    <path id="Bosnia and Herzegovina" fill="none" stroke="black" stroke-width="1" d="M 440.44,105.02   ...
  (...)

• I've joined the data using a custom java program (available on github at: https://github.com/lindenb/jvarkit/wiki/WorldMapGenome ). The program transforms the 'path' elements to a GeneralPath
  

  $  cat map.bed |\
       java -jar dist/worldmapgenome.jar \
       -u World_V2.0.svg \
       -w 2000 -o ~/ouput.jpg \
       -R hg19.fasta

That's it,
Pierre

mapping people I'm following on twitter: (KML, java, geonames.org)

I wrote a java tool to map the people I'm following on twitter. This tool invokes the twitter API to fetch the profiles of my contacts and it uses the geonames web services to guess the geolocation of the places.

The source code is available on github at https://github.com/lindenb/jsandbox/blob/master/src/sandbox/TwitterToKML.java.
the build.xml is here.

Compilation

ant twitterkml
#get your twitter-id at "http://api.twitter.com/1/users/show.xml?screen_name=<your-twitter-username>"
java -jar dist/twitterkml.jar -g <geonames-id> -o result.kml <twitter-numeric-id>
##WAIT. I don't use the OAuth API, so my program waits until the 'rate limit' is enabled again.

'following'

View @yokofakun's followings date:2011-04-28 21:03:42.079 in a larger map

'followers'

View @yokofakun's followers date:2011-04-29 in a larger map

That's it,
Pierre

XSLT+XML+geolocation=KML "BioStar users on a world map"

Today, I'be been appointed administrator of http://biostar.stackexchange.com/. It enables me to download a XML dump of the biostar database. Regarding the users, the XML file looks like this:

<?xml version="1.0" encoding="utf-8"?>
<Users>

(...)
<row>
<Id>30</Id>
<UserTypeId>4</UserTypeId>
<Reputation>13562</Reputation>
<Views>1485</Views>
<CreationDate>2010-02-25T17:27:15.56</CreationDate>
<LastAccessDate>2011-03-03T14:20:19.17</LastAccessDate>
<HasReplies>false</HasReplies>
<HasMessage>false</HasMessage>
<LastLoginDate>2011-03-03T11:15:36.54</LastLoginDate>
<LastLoginIP>XX.XX.XX.XX</LastLoginIP>
<Email>plindenbaum@yahoo.fr</Email>
<DisplayName>Pierre Lindenbaum</DisplayName>
<DisplayNameCleaned>pierre lindenbaum</DisplayNameCleaned>
<WebsiteUrl>http://plindenbaum.blogspot.com</WebsiteUrl>
<RealName>Pierre Lindenbaum</RealName>
<Location>France</Location>
<BadgeSummary>2=9 3=27</BadgeSummary>
<AboutMe>(...)</AboutMe>
<PreferencesRaw>[1|4][2|2][3|2][10|15]</PreferencesRaw>
</row>
(...)
</Users>

Using those information, it is now easy to create a google map/KML file plotting the biostar users. Neil has already achieved this task last october using ruby, GeoIP and data-scraping.

Here I've used the following simple XSLT stylesheet to create the KML file. This stylesheet is available at stackexchange-user2kml.xsl. Each time the XSLT stylesheet catches an element

<LastLoginIP>74.125.230.82</LastLoginIP>

it invokes a geolocalization service...

<xsl:variable name="url" select="concat('http://freegeoip.appspot.com/xml/',LastLoginIP)"/>
<xsl:message terminate="no">Downloading <xsl:value-of select="$url"/> ...</xsl:message>
<xsl:apply-templates select="document($url,/Response)" mode="geo"/>
</xsl:element>

returning another XML document http://freegeoip.appspot.com/xml/74.125.230.82

<?xml version="1.0" encoding="UTF-8"?>
<Response>
<Status>true</Status>
<Ip>74.125.230.82</Ip>
<CountryCode>US</CountryCode>
<CountryName>United States</CountryName>
<RegionCode>CA</RegionCode>
<RegionName>California</RegionName>
<City>Mountain View</City>
<ZipCode>94043</ZipCode>
<Latitude>37.4192</Latitude>
<Longitude>-122.057</Longitude>
</Response>

Result

xsltproc --stringparam title BioStar --stringparam anonymous yes \
stackexchange-user2kml.xsl Users.xml > biostar.kml

The anonymized KML file was then uploaded in google maps: Et voila;!:


View Biostar-anonymized 2011-03-03 in a larger map

That's it,

Pierre

Freebase and the History of Sciences

(feed readers, this post is better displayed on the web site)
I've been looking for a way to get a structured description of the biographies of the scientists threw the History. One of my investigation led to wikistory, a webstart application based on the data extracted from Wikipedia by the project DBPedia.

However, the data collected from DBPedia are mostly based on the infoboxes and most of them are missing or are incomplete. (as an example this is ok for Darwin (happy birthday) but there is no box for Georges-Louis Leclerc, Comte de Buffon (last accessed February 14th 2008 20H46)). Moreover the informations stored in those infoboxes are missing the fields I needed: gender, a short biography, parents, children.... etc....

I eventually decided to go back to look after freebase wich I tested a few monthes ago and which was also introduced at scifoo:

A screenshot of my final result is presented below:

and you can test this interface here:

http://lindenb.integragen.org/xulhistory/history.php

This is an interactive XUL page (it will only work with firefox) with a timeline containing a few hundred of scientists.


Just for fun, I also generated a time-based KML file for google-earth with those data.

http://lindenb.integragen.org/xulhistory/history.kml

The source code used to create this XUL page is available at here.

Here is how I proceeded:
On freebase I created my own type "scientist" enclosing some fields such as "short bio", "known for", "students", etc... (I don't know how to define 'inverse properties' in freebase: if A was the teacher of B, how can I automatically say that B was the student of A ?). This type scientist was added to some of the freebase records and completed (I think that Freebase also parsed the infoboxes in wikipedia to build their database, that is why most their records are almost empty). For example see: Buffon.


All the persons associated with my type scientist can be retrieved using the following MQL query:

{"qname1":{"query":[{"guid":null,"type":"/user/lindenb/default_domain/scientist"}]}}

The result looks like this...

{
"status": "200 OK",
"code": "/api/status/ok",
"qname1": {
"code": "/api/status/ok",
"result": [
{
"guid": "#9202a8c04000641f800000000000cb7c",
"type": "/user/lindenb/default_domain/scientist"
},
{
"guid": "#9202a8c04000641f800000000000f65e",
"type": "/user/lindenb/default_domain/scientist"
},

(...)
{
"guid": "#9202a8c04000641f80000000003b7d80",
"type": "/user/lindenb/default_domain/scientist"
},
{
"guid": "#9202a8c04000641f80000000003bd1ef",
"type": "/user/lindenb/default_domain/scientist"
}
]
}

For each gui we can retrieve the types associated with the record.

{"qname1":{"query":{"guid":"#9202a8c04000641f800000000000cb7c","type":[]}}}

The types associated with the record #9202a8c04000641f800000000000cb7c" were returned as follow:

{
"status": "200 OK",
"code": "/api/status/ok",
"qname1": {
"code": "/api/status/ok",
"result": {
"guid": "#9202a8c04000641f800000000000cb7c",
"type": [
"/common/topic",
"/people/person",
"/people/deceased_person",
"/book/author",
"/user/mikelove/default_domain/influence_node",
"/user/lindenb/default_domain/scientist",
"/award/award_winner"
]
}
}
}

For each type, I fetched the fields this record.

{"qname1":{"query":{"guid":"#9202a8c04000641f800000000000cb7c","*":null,"type":"/people/person"}}}

here is the response from freebase:

{
"status": "200 OK",
"code": "/api/status/ok",
"qname1": {
"code": "/api/status/ok",
"result": {
"creator": "/user/metaweb",
"profession": [
"Naturalist",
"Biologist",
"Geologist"
],
"places_lived": [
null,
null,
null,
null
],
"education": [
null,
null
],
"children": [
"George Howard Darwin",
"Horace Darwin"
],
"guid": "#9202a8c04000641f800000000000cb7c",
"employment_history": [],
"id": "/topic/en/charles_darwin",
"religion": [
"Agnosticism",
"Christianity",
"Unitarianism",
"Church of England"
],
"date_of_birth": "1809-02-12",
"parents": [
"Robert Darwin",
"Susannah Darwin"
],
"metaweb_user_s": [],
"type": "/people/person",
"attribution": "/user/metaweb",
"permission": "/boot/all_permission",
"timestamp": "2006-10-22T08:53:38.0061Z",
"signature": [],
"weight_kg": null,
"key": [
"Charles_Darwin",
"Charles_Robert_Darwin",
"Darwin$0027s",
"Mary_Darwin",
"Darwin$002C_Charles",
"C$002E_R$002E_Darwin",
"Charles_R$002E_Darwin",
"8145410",
"Charles_Darwin$0027s",
"Charles_darwin",
"charles_darwin",
"CR_Darwin",
"Charles_R_Darwin",
"71b891f5-92bb-42be-9c45-98b8f56a3177"
],
"nationality": [
"United Kingdom"
],
"spouse_s": [
null
],
"name": "Charles Darwin",
"gender": "Male",
"sibling_s": [
null
],
"height_meters": null,
"place_of_birth": "Shrewsbury",
"quotations": []
}
}
}

And so on, using this kind of queries I was able to fetch the birth dates, the geographical coordinate of the places, the pictures, etc...

The result is available here:

http://lindenb.integragen.org/xulhistory/history.php

That's it.
Pierre

Update 2010-08-12 : source code

history.js

var XUL={
NS:"http://www.mozilla.org/keymaster/gatekeeper/there.is.only.xul"
};

var XHTML={
NS:"http://www.w3.org/1999/xhtml"
};

function StartDate(year,month,dayOfMonth)
{
this.year=year;
this.month=month;
this.dayOfMonth=dayOfMonth;
}

StartDate.prototype.days=function()
{
var d= this.year*365.25;
if(this.month!=null)
{
d+=(this.month*(365.25/12.0));
if(this.dayOfMonth!=null)
{
d+=this.dayOfMonth;
}
}
return d;
}



function EndDate(year,month,dayOfMonth)
{
this.year=year;
this.month=month;
this.dayOfMonth=dayOfMonth;
}

EndDate.prototype.days=function()
{
var v= 0;
if(this.month!=null)
{
if(this.dayOfMonth!=null)
{
v+=(1+this.dayOfMonth);
v+=this.month*(365.25/12.0);
}
else
{
v+=(this.month+1)*(365.25/12.0);
}
v+=this.year*365.25;
}
else
{
v+=(1+this.year)*365.25;
}
return v;
}


var MY={
now: null,
iconSize:64,
screenWidth:15000,
minDate:null,
maxDate:null,
debug:function(msg)
{
var message=document.getElementById("message");
if(message==null) return;
MY.removeAllChild(message);
message.appendChild(document.createTextNode(msg==null?"null":msg));
},
x1:function(person)
{
return MY.convertDate2Pixel(person.birthDate);
},
x2:function(person)
{
var d= person.deathDate;
if(d==null)
{
d= MY.now;
}
return MY.convertDate2Pixel(d);
},
convertDate2Pixel:function(date)
{
return MY.screenWidth*((date.days()-MY.minDate.days())/(MY.maxDate.days()-MY.minDate.days()));
},
removeAllChild:function(root)
{
if(root==null) return;
while(root.hasChildNodes())
{
root.removeChild(root.firstChild);
}
},
loaded:function()
{
var d= new Date();
MY.now = new EndDate(d.getFullYear(),1+d.getMonth(),d.getUTCDate());

var set= new Array()
for(var i=0;i< persons.length;++i)
{
for(var j=0;j< persons[i].profession.length;++j)
{
set[persons[i].profession[j] ]=1;
}
}
MY.fillListBox("profession",set);

//knownfor
set= new Array()
for(var i=0;i< persons.length;++i)
{
for(var j=0;j< persons[i].knownFor.length;++j)
{
set[persons[i].knownFor[j] ]=1;
}
}
MY.fillListBox("knownfor",set);

//country
set= new Array()
for(var i=0;i< persons.length;++i)
{
for(var j=0;j< persons[i].nationality.length;++j)
{
set[persons[i].nationality[j] ]=1;
}
}
MY.fillListBox("country",set);

//AWARDS
set= new Array()
for(var i=0;i< persons.length;++i)
{
for(var j=0;j< persons[i].awards.length;++j)
{
set[persons[i].awards[j] ]=1;
}
}
MY.fillListBox("awards",set);

MY.pileup();
},
fillListBox:function(id,set)
{
var root=document.getElementById(id);
if(root==null) return;
var array2= new Array(set.length);
for(var p in set)
{
array2.push(p);
}
array2.sort();
for(var j=0;j< array2.length;++j)
{
var item=document.createElementNS(XUL.NS,"listitem");
item.setAttribute("label",array2[j]);
item.setAttribute("value",array2[j]);
root.appendChild(item);
}
},
selectedItems:function(id)
{
var set= new Array();
var root=document.getElementById(id);
if(root==null) return set;
var selected= root.selectedItems;

for(var i=0;i<selected.length;++i)
{
set.push(selected[i].value);
}
return set;
},
date2text:function(date)
{
var s=""+date.year;
if(date.month!=null)
{
s+=" ";
switch(date.month)
{
case 1: s+=("Jan"); break;
case 2: s+=("Feb"); break;
case 3: s+=("Mar"); break;
case 4: s+=("Apr"); break;
case 5: s+=("May"); break;
case 6: s+=("Jun"); break;
case 7: s+=("Jul"); break;
case 8: s+=("Aug"); break;
case 9: s+=("Sep"); break;
case 10: s+=("Oct"); break;
case 11: s+=("Nov"); break;
case 12: s+=("Dec"); break;
default: s+=date.month; break;
}
if(date.dayOfMonth!=null)
{
s+=" "+date.dayOfMonth;
}
}
return s;
},
containsSet:function(set,subset)
{
if(subset.length==0) return true;
for(var i=0;i< subset.length;++i)
{
if(set.indexOf(subset[i])!=-1) return true;
}
return false;
},
update:function()
{
for(var i=0;i< persons.length;++i)
{
persons[i].selected=true;
}
var sel=MY.selectedItems("profession");

for(var i=0;i< persons.length && sel.length>0;++i)
{
if(!MY.containsSet(persons[i].profession,sel))
{
persons[i].selected=false;
}
}
sel=MY.selectedItems("knownfor");

for(var i=0;i< persons.length && sel.length>0;++i)
{
if(!persons[i].selected) continue;
if(!MY.containsSet(persons[i].knownFor,sel))
{
persons[i].selected=false;
}
}

sel=MY.selectedItems("country");

for(var i=0;i< persons.length && sel.length>0;++i)
{
if(!persons[i].selected) continue;
if(!MY.containsSet(persons[i].nationality,sel))
{
persons[i].selected=false;
}
}

sel=MY.selectedItems("awards");

for(var i=0;i< persons.length && sel.length>0;++i)
{
if(!persons[i].selected) continue;
if(!MY.containsSet(persons[i].awards,sel))
{
persons[i].selected=false;
}
}

sel=MY.selectedItems("gender");

for(var i=0;i< persons.length && sel.length>0;++i)
{
if(!persons[i].selected ) continue;
if(sel.indexOf(persons[i].gender)==-1)
{
persons[i].selected=false;
}
}

for(var i=0;i< persons.length ;++i)
{
persons[i].node.style.opacity=(persons[i].selected?1.0:0.3);
}
},
simpleTag:function(tag,text)
{
var e = document.createElementNS(XHTML.NS,tag);
e.appendChild(document.createTextNode(text));
return e;
},
bold:function(text) { return MY.simpleTag("h:b",text);},
italic:function(text) { return MY.simpleTag("h:i",text);},
underline:function(text) { return MY.simpleTag("h:u",text);},
pileup:function()
{
var remains=new Array(persons.length);
for(var i=0;i< persons.length;++i) remains[i]=persons[i];
MY.minDate=null;
MY.maxDate=null;
for(var i=0;i< remains.length;++i)
{
var o=remains[i];
if(MY.minDate==null || o.birthDate.days() < MY.minDate.days())
{
MY.minDate= o.birthDate;
}
if(o.deathDate!=null && (MY.maxDate==null || MY.maxDate.days()< o.deathDate.days()))
{
MY.maxDate= o.deathDate;
}
}
if(MY.minDate==null || MY.maxDate==null) return;
/*
var nLine=-1;
while(remains.length>0)
{
++nLine;
var first=remains[0];
remains=remains.slice(1);
first.y=nLine;

while(true)
{
var best=null;
var bestIndex=-1;
for(var i=0;i< remains.length;++i)
{
var next=remains[i];
if(MY.x1(next)< MY.x2(first)+5) continue;
if(best==null ||
(MY.x1(next)-MY.x2(first) < MY.x1(best)-MY.x2(first)))
{
best=next;
bestIndex=i;
}
}
if(best==null) break;
first=best;
first.y=nLine;
remains.splice(bestIndex,1);
}
}
*/
var timeline=document.getElementById("timeline");
MY.removeAllChild(timeline);
var MARGIN=2;
var HEIGHT=MY.iconSize+MARGIN*2;
for(var i=0;i< persons.length;++i)
{
var o= persons[i];
var stack= document.createElementNS(XUL.NS,"stack");
var style="top:"+Math.round(o.y*(HEIGHT+10))+"px;"+
"left:"+Math.round(o.x1)+"px;"+
"width:"+Math.round(o.x2-o.x1)+"px;"+
"height:"+Math.round(HEIGHT)+"px;"+
"background-color:black;"+
"color:white;"+
"border-width:2px;"+
"border-color:red;"+
"overflow:hidden;"+
"font-size:11px;"+
"opacity: 1;"
;
o.node=stack;
stack.setAttribute("style",style);


var hbox= document.createElementNS(XUL.NS,"hbox");
hbox.setAttribute("flex","1");
stack.appendChild(hbox);
if(o.img!=null)
{
var img= document.createElementNS(XUL.NS,"image");
hbox.appendChild(img);
img.setAttribute("src",o.img);
img.setAttribute("style","width:"+MY.iconSize+"px;height:"+MY.iconSize+"px;");
}

var div= document.createElementNS(XHTML.NS,"h:div");
div.setAttribute("style","width:"+Math.round(MY.x2(o)-MY.x1(o)-(o.img==null?0:MY.iconSize))+"px;");
//div= document.createElementNS(XUL.NS,"label");
hbox.appendChild(div);

var anchor= document.createElementNS(XHTML.NS,"h:a");
anchor.appendChild(document.createTextNode(o.name));
anchor.setAttribute("href","http://www.freebase.com/view/guid/"+o.guid);
anchor.setAttribute("target",o.guid);
anchor.setAttribute("title",o.name);
div.appendChild(anchor);

div.appendChild(document.createTextNode(" : "));
if(o.birthDate!=null)
{
div.appendChild(document.createTextNode(MY.date2text(o.birthDate)));
if(o.birthPlace!=null)
{
div.appendChild(document.createTextNode(" at "+o.birthPlace));
}
div.appendChild(document.createTextNode(" - "));
}
if(o.deathDate!=null)
{
div.appendChild(document.createTextNode(MY.date2text(o.deathDate)));
if(o.deathPlace!=null)
{
div.appendChild(document.createTextNode(" at "+o.deathPlace));
}
}
div.appendChild(document.createTextNode(":"+o.shortBio+" "));
if(o.knownFor.length>0)
{
div.appendChild(MY.bold("Known For:"));
for(var k in o.knownFor)
{
div.appendChild(document.createTextNode(o.knownFor[k]+" "));
}
}
timeline.appendChild(stack);
}


}

};

history.xul

<?xml version="1.0" encoding="UTF-8"?>
<?xml-stylesheet href="chrome://global/skin/" type="text/css"?>
<window
id="main-window"
title="History Of Science"
orient="horizontal"
xmlns="http://www.mozilla.org/keymaster/gatekeeper/there.is.only.xul"
xmlns:h="http://www.w3.org/1999/xhtml"
onload="MY.loaded();"
>
<!-- FIREFOX 2.0 IS REQUIRED TO SEE THIS FILE !!! -->
<script src="history.js"/>
<script src="person.js"/>
<script src="http://www.google-analytics.com/urchin.js" type="text/javascript"></script><script type="text/javascript">_uacct = "UA-307413-2";urchinTracker();</script>
<vbox flex="1">
<hbox>
<h:div><h:h1>History Of Sciences</h:h1> 2008 <h:a title="plindenbaum@yahoo.fr" href="mailto:plindenbaum@yahoo.fr">Pierre Lindenbaum PhD</h:a>.
Testing <h:a title="freebase" href="http://www.freebase.com">Freebase</h:a> to fetch the biographies of famous scientists. See also my <h:a href="http://plindenbaum.blogspot.com/2008/02/freebase-and-history-of-sciences.html">blog</h:a> and the <h:a href="http://maps.google.com/maps?f=q&amp;hl=en&amp;geocode=&amp;q=http%3A%2F%2Flindenb.integragen.org%2Fxulhistory%2Fhistory.kml&amp;ie=UTF8&amp;ll=53.956086,-13.007812&amp;spn=99.233765,284.0625&amp;t=h&amp;z=2" title="kml">KML file</h:a> for Google Earth. </h:div>
</hbox>
<scrollbox flex="6" style='overflow: auto;'>
<stack id="timeline" flex="6" style="background-color:lightgray;color:black;width:200px;border-style:solid;border-color: black;border-width:1px;">
<!-- content goes here -->
</stack>
</scrollbox>

<vbox flex="1">
<hbox><description id="message"/></hbox>
<hbox flex="1">

<listbox id="gender" seltype="multiple" rows="2" flex="1">
<listhead>
<listheader label="Gender"/>
</listhead>
<listitem label="Man" value="Man"/>
<listitem label="Woman" value="Female"/>
</listbox>

<listbox id="profession" seltype="multiple" rows="5" flex="1">
<listhead>
<listheader label="Profession"/>
</listhead>
</listbox>

<listbox id="country" seltype="multiple" rows="5" flex="1">
<listhead>
<listheader label="Country"/>
</listhead>
</listbox>

<listbox id="awards" seltype="multiple" rows="5" flex="1">
<listhead>
<listheader label="Awards"/>
</listhead>
</listbox>

<listbox id="knownfor" seltype="multiple" rows="5" flex="1">
<listhead>
<listheader label="Known for"/>
</listhead>
</listbox>
<button id="yes" label="Update" flex="1" oncommand="MY.update();"/>
</hbox>
<hbox><label value="Updated: 2008-03-02"/></hbox>
</vbox>

</vbox>
</window>

person.js

var persons=[
{
name:"Thales",
guid:"9202a8c04000641f800000000003b246",
gender:"Male",
x1:0.0,
x2:527.9741361723788,
y:0,
node:null,
selected:true,
nationality:["Ancient Greece"],
shortBio:"pre-Socratic Greek philosopher and one of the Seven Sages of Greece. Many regard him as the first philosopher in the Greek tradition, while some also consider him the \"father of science.\"",
profession:["Mathematician","Philosopher"],
birthDate:new StartDate(-634,null,null),
birthPlace:null,deathDate:new EndDate(-542,null,null),
deathPlace:null,
knownFor:["Thales\' theorem"],
img:null,
awards:[]
}
,
{
name:"Anaximander",
guid:"9202a8c04000641f8000000000004e68",
gender:"Male",
x1:147.60567247829948,
x2:505.2655711757174,
y:1,
node:null,
selected:true,
nationality:["Ancient Greece"],
shortBio:"pre-Socratic philosopher",
profession:["Mathematician"],
birthDate:new StartDate(-608,null,null),
birthPlace:null,deathDate:new EndDate(-546,null,null),
deathPlace:null,
knownFor:["Apeiron"],
img:null,
awards:[]
}
(...)
,
{
name:"Brian Greene",
guid:"9202a8c04000641f80000000001f50ef",
gender:"Male",
x1:14744.621902781384,
x2:15000.473095104098,
y:5,
node:null,
selected:true,
nationality:["United States"],
shortBio:"Theoretical physicist and one of the best-known string theorists.",
profession:["Physicist","Scientist","Science writer"],
birthDate:new StartDate(1963,2,9),
birthPlace:"New York, New York",deathDate:null,
deathPlace:null,
knownFor:["The Elegant Universe","The Fabric of the Cosmos","String theory"],
img:"9202a8c04000641f80000000049ae0ba.png",
awards:[]
}
];

Mapping NCBI/PUBMED

In my previous post I showed how I used the tag <Affiliation> from the XML/pubmed records to extract the mails and the names from the authors of a paper. I've slightly changed the source code of this program to find the country of origin of each paper. To retrieve the country I used:
1) the suffix of the mail (if any)
2) the name of the country (if any)
3) the name of the city (a few famous one such as Standord, for the US or UK)

My program takes as input a pubmed query and the ouput is the number of papers per year and per country. I put a few results on ManyEyes. As an example with the query "Rotavirus" with 1000 records, I was able to retrieve 887 countries.

Publications in "Bioinformatics", "BMC Bioinformatics", "Plos Comp. Biol."

Publications about "Rotavirus"

publications about malaria, anopheles, plasmodium etc...

Google Developer Day France 2007

Today, I attended the french session of the Google Developer Day at Paris. Most of the conference dealt with the 33 (!) google APIs , mashups and the last (and most interesting) was about the google web toolkit. The place was full of testosterone and, sorry, nothing about science (scholar...).

Patrick Chenzon made a general introduction about the google APIs. He also wrote a short demo on how to create a mashup with the map API.
http://wordpress.chanezon.com/
http://del.icio.us/chanezon

Gears was introduced (the buzz word of the day) with a demo using reader.google.com

David Aubespin talked about the google gadget. (I've already used this technology with pubmed , see google-gadget-for-bioinformatics.html)
There is now a scratchpad to edit and test your gadget. We also had a presentation about the new gadget system now used within google maps: mapplets. For example my NCBI gadget could be modified to extract the content of the <Affiliation> tag, getting the latitude and longitude of this address and display it on the map.


Jean François Wassong talked about the google map API. There is also a new tool , the Mashup Editor an AJAX development framework and a set of tools that enable developers to quickly and easily create simple web applications and mashups with Google services like Google Maps and Google Base.

For the last talk Didier Girard spoked about the google web toolkit. He created example on how to create an AJAX based interface using the GWT, he also used GWT Designer to design the graphical interface.


Pierre