Registering a tool in the @ELIXIREurope regisry using XML, XSLT, JSON and curl. My notebook.

The Elixir Registry / pmid:26538599 "A portal to bioinformatics resources world-wide. With community support, the registry can become a standard for dissemination of information about bioinformatics resources: we welcome everyone to join us in this common endeavour. The registry is freely available at https://bio.tools."

In this post, I will describe how I've used the bio.tools API to register some tools from jvarkit.

Authenticate with your credentials

using curl, the 'bio.tools' service returns a authentication token.

$ curl -s \
 -H "Content-type: application/json" \
 -X POST \
 -d '{"username":"my-login@univ-nantes.fr","password":"password1234"}' \
 https://bio.tools/api/auth/login |\
 python -m json.tool
{
    "token": "74dedea023dbad8ecda49ac57bb1074acd794f"
}

Creating a JSON describing the tool.

The tool I'm goind to use is VCFhead. A very simple tool printing the first variants of a VCF file. In jvarkit I don't write the code parsing the arguments, everything is described using a XML file that is going to be processed with a XSTL stylesheet to generate an abstract java code handling the options, etc....

xsltproc command2java VcfHead.xml > AbstractVcfHead.java

For VcfHead the XML descriptor is available here: https://github.com/lindenb/jvarkit/blob/master/src/main/java/com/github/lindenb/jvarkit/tools/misc/VcfHead.xml.

<?xml version="1.0" encoding="UTF-8" standalone="yes"?>
<app xmlns="http://github.com/lindenb/jvarkit/" xmlns:h="http://www.w3.org/1999/xhtml" app="VcfHead" package="com.github.lindenb.jvarkit.tools.misc" >
  <description>Print the first variants of a VCF.</description>
  <input type="vcf"/>
  <output type="vcf"/>
  <options>
    <option name="count" type="int" opt="n" longopt="count" min-inclusive="0" default="10">
      <description>number of variants to be printed</description>
    </option>
  </options>
  <documentation>
    <h:h3>Example</h:h3>
   (...)
  </documentation>
</app>

Using a first XSLT stylesheet https://github.com/lindenb/jvarkit/blob/master/src/main/resources/xsl/jsonxelixir.xsl, 'VcfHead.xml' is firstly converted to the 'infamous' JSONx (JSON+XML) format .

xsltproc jsonxelixir VcfHead.xml > VcfHead.jsonx

The JSONx file:

<?xml version="1.0"?>
<jsonx:object xmlns:jsonx="http://www.ibm.com/xmlns/prod/2009/jsonx" xmlns:c="http://github.com/lindenb/jvarkit/" xmlns="http://www.w3.org/1999/xhtml" xmlns:x="http://www.ibm.com/xmlns/prod/2009/jsonx">
  <jsonx:string name="accessibility">Public</jsonx:string>
  <jsonx:string name="affiliation">univ-nantes.fr</jsonx:string>
  <jsonx:string name="cost">Free</jsonx:string>
  <jsonx:array name="platform">
    <jsonx:string>Linux</jsonx:string>
    <jsonx:string>Mac</jsonx:string>
  </jsonx:array>
  <jsonx:string name="version">1.0</jsonx:string>
  <jsonx:string name="homepage">https://github.com/lindenb/jvarkit/wiki/VcfHead</jsonx:string>
  <jsonx:array name="function">
    <jsonx:object>
      <jsonx:array name="input">
        <jsonx:object>
          <jsonx:object name="dataType">
            <jsonx:string name="term">File name</jsonx:string>
            <jsonx:string name="uri">http://edamontology.org/data_1050</jsonx:string>
          </jsonx:object>
          <jsonx:array name="dataFormat">
            <jsonx:object>
              <jsonx:string name="term">VCF</jsonx:string>
              <jsonx:string name="uri">http://edamontology.org/format_3016</jsonx:string>
            </jsonx:object>
          </jsonx:array>
        </jsonx:object>
      </jsonx:array>
      <jsonx:array name="output">
        <jsonx:object>
          <jsonx:object name="dataType">
            <jsonx:string name="term">File name</jsonx:string>
            <jsonx:string name="uri">http://edamontology.org/data_1050</jsonx:string>
          </jsonx:object>
          <jsonx:string name="dataDescription">any format</jsonx:string>
          <jsonx:array name="dataFormat">
            <jsonx:object>
              <jsonx:string name="term">VCF</jsonx:string>
              <jsonx:string name="uri">http://edamontology.org/format_3016</jsonx:string>
            </jsonx:object>
          </jsonx:array>
        </jsonx:object>
      </jsonx:array>
      <jsonx:array name="functionName">
        <jsonx:object>
          <jsonx:string name="term">Formatting</jsonx:string>
          <jsonx:string name="uri">http://edamontology.org/operation_0335</jsonx:string>
        </jsonx:object>
      </jsonx:array>
      <jsonx:string name="functionDescription">Print the first variants of a VCF.</jsonx:string>
    </jsonx:object>
  </jsonx:array>
  <jsonx:string name="description">Print the first variants of a VCF.</jsonx:string>
  <jsonx:object name="docs">
    <jsonx:string name="docsTermsOfUse">https://opensource.org/licenses/MIT</jsonx:string>
    <jsonx:string name="docsGithub">https://github.com/lindenb/jvarkit/wiki/VcfHead</jsonx:string>
    <jsonx:string name="docsHome">https://github.com/lindenb/jvarkit/wiki/VcfHead</jsonx:string>
    <jsonx:string name="docsCitationInstructions">https://github.com/lindenb/jvarkit/wiki/Citing</jsonx:string>
    <jsonx:string name="docsDownloadSource">https://github.com/lindenb/jvarkit/archive/master.zip</jsonx:string>
    <jsonx:string name="docsDownload">https://github.com/lindenb/jvarkit/archive/master.zip</jsonx:string>
  </jsonx:object>
  <jsonx:array name="collection">
    <jsonx:string>jvarkit</jsonx:string>
  </jsonx:array>
  <jsonx:object name="credits">
    <jsonx:array name="creditsInstitution">
      <jsonx:string>Institut du Thorax, Nantes, France</jsonx:string>
    </jsonx:array>
    <jsonx:array name="creditsDeveloper">
      <jsonx:string>Pierre Lindenbaum</jsonx:string>
    </jsonx:array>
  </jsonx:object>
  <jsonx:array name="interface">
    <jsonx:object>
      <jsonx:string name="interfaceType">Command line</jsonx:string>
    </jsonx:object>
  </jsonx:array>
  <jsonx:string name="name">VcfHead</jsonx:string>
  <jsonx:array name="topic">
    <jsonx:object>
      <jsonx:string name="term">Omics</jsonx:string>
      <jsonx:string name="uri">http://edamontology.org/topic_3391</jsonx:string>
    </jsonx:object>
  </jsonx:array>
  <jsonx:string name="license">MIT License</jsonx:string>
  <jsonx:array name="language">
    <jsonx:string>Java</jsonx:string>
  </jsonx:array>
  <jsonx:array name="resourceType">
    <jsonx:string>Tool</jsonx:string>
  </jsonx:array>
  <jsonx:string name="maturity">Stable</jsonx:string>
  <jsonx:array name="contact">
    <jsonx:object>
      <jsonx:string name="contactURL">https://github.com/lindenb</jsonx:string>
      <jsonx:string name="contactName">Pierre Lindenbaum</jsonx:string>
      <jsonx:array name="contactRole">
        <jsonx:string>Developer</jsonx:string>
        <jsonx:string>Maintainer</jsonx:string>
        <jsonx:string>Helpdesk</jsonx:string>
      </jsonx:array>
    </jsonx:object>
  </jsonx:array>
  <jsonx:object name="publications">
    <jsonx:string name="publicationsPrimaryID">doi:10.6084/m9.figshare.1425030.v1</jsonx:string>
  </jsonx:object>
</jsonx:object>

Using another XSLT stylesheet jsonx2json.xsl, the JSONx is converted to a JSON file.

xsltproc jsonx2json.xsl VcfHead.jsonx > VcfHead.json

the JSON file:

{
    "accessibility": "Public", 
    "affiliation": "univ-nantes.fr", 
    "collection": [
        "jvarkit"
    ], 
    "contact": [
        {
            "contactName": "Pierre Lindenbaum", 
            "contactRole": [
                "Developer", 
                "Maintainer", 
                "Helpdesk"
            ], 
            "contactURL": "https://github.com/lindenb"
        }
    ], 
    "cost": "Free", 
    "credits": {
        "creditsDeveloper": [
            "Pierre Lindenbaum"
        ], 
        "creditsInstitution": [
            "Institut du Thorax, Nantes, France"
        ]
    }, 
    "description": "Print the first variants of a VCF.", 
    "docs": {
        "docsCitationInstructions": "https://github.com/lindenb/jvarkit/wiki/Citing", 
        "docsDownload": "https://github.com/lindenb/jvarkit/archive/master.zip", 
        "docsDownloadSource": "https://github.com/lindenb/jvarkit/archive/master.zip", 
        "docsGithub": "https://github.com/lindenb/jvarkit/wiki/VcfHead", 
        "docsHome": "https://github.com/lindenb/jvarkit/wiki/VcfHead", 
        "docsTermsOfUse": "https://opensource.org/licenses/MIT"
    }, 
    "function": [
        {
            "functionDescription": "Print the first variants of a VCF.", 
            "functionName": [
                {
                    "term": "Formatting", 
                    "uri": "http://edamontology.org/operation_0335"
                }
            ], 
            "input": [
                {
                    "dataFormat": [
                        {
                            "term": "VCF", 
                            "uri": "http://edamontology.org/format_3016"
                        }
                    ], 
                    "dataType": {
                        "term": "File name", 
                        "uri": "http://edamontology.org/data_1050"
                    }
                }
            ], 
            "output": [
                {
                    "dataDescription": "any format", 
                    "dataFormat": [
                        {
                            "term": "VCF", 
                            "uri": "http://edamontology.org/format_3016"
                        }
                    ], 
                    "dataType": {
                        "term": "File name", 
                        "uri": "http://edamontology.org/data_1050"
                    }
                }
            ]
        }
    ], 
    "homepage": "https://github.com/lindenb/jvarkit/wiki/VcfHead", 
    "interface": [
        {
            "interfaceType": "Command line"
        }
    ], 
    "language": [
        "Java"
    ], 
    "license": "MIT License", 
    "maturity": "Stable", 
    "name": "VcfHead", 
    "platform": [
        "Linux", 
        "Mac"
    ], 
    "publications": {
        "publicationsPrimaryID": "doi:10.6084/m9.figshare.1425030.v1"
    }, 
    "resourceType": [
        "Tool"
    ], 
    "topic": [
        {
            "term": "Omics", 
            "uri": "http://edamontology.org/topic_3391"
        }
    ], 
    "version": "1.0"
}

Registering the tool

Now we have the Token and the json descriptor we can add VcfHead to Elixir using curl:

curl  -H "Content-type: application/json" \
 -H "Authorization: Token 74dedea023dbad8ecda49ac57bb1074acd794f" \
 -X POST \
 -d  @path/to/VcfHead.json \
 "https://bio.tools/api/tool" |\
 python -m json.tool

output:

{
    "accessibility": "Public", 
    "additionDate": "2016-02-24T11:37:17.458Z", 
    "affiliation": "univ-nantes.fr", 
    "collection": [
        "jvarkit"
    ], 
    "contact": [
        {
            "contactName": "Pierre Lindenbaum", 
            "contactRole": [
                "Developer", 
                "Maintainer", 
                "Helpdesk"
            ], 
            "contactURL": "https://github.com/lindenb"
(...)

VCfhead is now visible from the Elixir Registry at https://bio.tools/tool/univ-nantes.fr/VcfHead/1.0.

That's it,
Pierre.

Using the flickr XML/API as a source of RSS feeds.

You may know that I seek from time to time royalty free pictures  on flickr.com for my other personal blog. The flickr API can be used to search for these images but it is currently not possible to generate a RSS feed to be alerted when a new image is posted on flickr.

The following XSLT stylesheet transforms the XML returned by www.flickr.com to a RSS feed (latest source: https://github.com/lindenb/xslt-sandbox/blob/master/stylesheets/flickr/flickr2rss.xsl ):

<xsl:stylesheet xmlns:xsl="http://www.w3.org/1999/XSL/Transform"
xmlns:date="http://exslt.org/dates-and-times"
xmlns:dc="http://purl.org/dc/elements/1.1/" version="1.0">

<xsl:output method="xml" indent="yes" encoding="UTF-8"/>

<xsl:param name="title">No Title</xsl:param>


<xsl:template match="/">
<rss version="2.0">
<channel>

<title><xsl:value-of select="$title"/></title>
<link>http://www.flickr.com</link>
<description><xsl:value-of select="$title"/></description>
<pubDate><xsl:value-of select="date:date-time()"/></pubDate>
<lastBuildDate><xsl:value-of select="date:date-time()"/></lastBuildDate>
<generator>http://www.flickr.com/</generator>
<xsl:apply-templates select="rsp/photos/photo"/>
</channel>
</rss>
</xsl:template>

<xsl:template match="photo">
<item>
<title><xsl:value-of select="@title"/> : <xsl:value-of select="$title"/></title>
<link>http://www.flickr.com/photos/<xsl:value-of select="@owner"/>/<xsl:value-of select="@id"/>/</link>
<pubDate><xsl:value-of select="@datetaken"/></pubDate>

<author>
<xsl:choose>
<xsl:when test="@ownername"><xsl:value-of select="@ownername"/></xsl:when>
<xsl:otherwise><xsl:value-of select="@owner"/></xsl:otherwise>
</xsl:choose>
</author>
<guid isPermaLink="false">http://www.flickr.com/photos/<xsl:value-of select="@owner"/>/<xsl:value-of select="@id"/>/</guid>
<description>
<xsl:text><p><img </xsl:text>
<xsl:choose>
<xsl:when test="@height_s and @width_s">
<xsl:text> width='</xsl:text>
<xsl:value-of select="@width_s"/>
<xsl:text>' height='</xsl:text>
<xsl:value-of select="@height_s"/>
<xsl:text>' </xsl:text>
</xsl:when>
<xsl:when test="@height_m and @width_m">
<xsl:text> width='</xsl:text>
<xsl:value-of select="@width_m"/>
<xsl:text>' height='</xsl:text>
<xsl:value-of select="@height_m"/>
<xsl:text>' </xsl:text>
</xsl:when>
</xsl:choose>
<xsl:text> src='</xsl:text>
<xsl:choose>
<xsl:when test="@url_s"><xsl:value-of select="@url_s"/></xsl:when>
<xsl:otherwise>http://farm<xsl:value-of select="@farm"/>.staticflickr.com/<xsl:value-of select="@server"/>/<xsl:value-of select="@id"/>_<xsl:value-of select="@secret"/>_s.jpg</xsl:otherwise>
</xsl:choose>
<xsl:text>' /></p></xsl:text>
</description>
</item>
</xsl:template>

</xsl:stylesheet>

To invoke this stylesheet and generate the RSS feed, I wrote the following small quick'n dirty cgi-script "flickr.cgi". The script was made executable, installed into my public_html/cgi-bin directory. It also requires to get an API key from flickr.

#!/bin/sh
echo "Content-Type: application/rss+xml"
echo 

APIKEY=12345678910
TAGS=`echo ${QUERY_STRING}|tr "?&" "\n" | egrep '^tags=' | cut -d '=' -f 2 | sed 's/,/%2C/g'`
TEXT=`echo ${QUERY_STRING}|tr "?&" "\n" | egrep '^text=' | cut -d '=' -f 2 | tr " " "+"`

curl -s "http://api.flickr.com/services/rest/?method=flickr.photos.search&api_key=${APIKEY}&tags=${TAGS}&format=rest&extras=url_s,date_upload,date_taken,icon_server,owner_name&tag_mode=all&per_page=20&license=2,4,1,5,7&text=${TEXT}" |
xsltproc --novalid --stringparam title "${TAGS} ${TEXT}" flickr2rss.xsl -

I can know add some new RSS feeds into thunderbird and receive the new items: for example http://localhost/~me/cgi-bin/flickr.cgi?tags=science

<rss version="2.0" xmlns:date="http://exslt.org/dates-and-times" xmlns:dc="http://purl.org/dc/elements/1.1/" version="2.0" >
<channel>
<title>science </title>
<link>http://www.flickr.com</link>
<description>science </description>
<pubDate>2012-07-09T23:09:58+02:00</pubDate>
<lastBuildDate>2012-07-09T23:09:58+02:00</lastBuildDate>
<generator>http://www.flickr.com/</generator>
<item>
<title>2012 NOAA HABs Forecast : science </title>
<link>http://www.flickr.com/photos/41398337@N07/7537830696/</link>
<pubDate>2012-07-05 10:16:56</pubDate>
<author>Ohio Sea Grant and Stone Laboratory</author>
<guid isPermaLink="false">http://www.flickr.com/photos/41398337@N07/7537830696/</guid>
<description><p><img width='240' height='160' src='http://farm8.staticflickr.com/7252/7537830696_2a66783e70_m.jpg' /></p></description>
</item>
<item>
<title>2012 NOAA HABs Forecast : science </title>
<link>http://www.flickr.com/photos/41398337@N07/7537829638/</link>
<pubDate>2012-07-05 10:18:37</pubDate>
<author>Ohio Sea Grant and Stone Laboratory</author>
<guid isPermaLink="false">http://www.flickr.com/photos/41398337@N07/7537829638/</guid>
<description><p><img width='240' height='160' src='http://farm8.staticflickr.com/7252/7537829638_4c6dd535b4_m.jpg' /></p></description>
</item>
<item>
(...)
</item>
</channel>
</rss>

That's it,

Pierre

Apache SOLR and GeneOntology: Creating the JQUERY-UI client (with autocompletion)

In the previous post I showed how to use Apache SOLR to index the content of gene ontology. In this post I will show how the SOLR server can be used as a source of data for a "JQuery-UI autocomplete" HTML input.

The JQuery autocomplete Widget expects an JSON array of Objects with label and value properties :

[ { label: "Choice1", value: "value1" }, ... ]

We need to tell SOLR to return the GO terms formatted this way.

Fortunately SOLR provides a mechanism to captures the output of the normal XML response and applies a XSLT transformation to it.

The following XSLT stylesheet transforms the XML response to JSON, it is created in the conf/xslt directory (apache-solr-3.5.0/example/solr/conf/xslt/solr2json.xsl)


We can test the stylesheet by adding the tr parameter in the query URL:

$ curl "http://localhost:8983/solr/select?q=go_definition:cancer&wt=xslt&tr=solr2json.xsl"

[
    {
        "label": "Z-phenylacetaldoxime metabolic process",
        "value": "GO:0018983"
    },
    {
        "label": "epothilone metabolic process",
        "value": "GO:0050813"
    },
    {
        "label": "epothilone biosynthetic process",
        "value": "GO:0050814"
    },
    {
        "label": "aflatoxin biosynthetic process",
        "value": "GO:0045122"
    },
    {
        "label": "aflatoxin metabolic process",
        "value": "GO:0046222"
    },
    {
        "label": "aflatoxin catabolic process",
        "value": "GO:0046223"
    }
]

The jquery client

Let's create the HTML/JQuery Client, I have downloaded the component for jquery and jquery-ui in the apache-solr-3.5.0/example/work/Jetty_xxxx_solr.war__solr__xxxx/webapp/ (and that's a bad idea, everything will be deleted with the SOLR server will be stopped or re-deployed).

I've also created the following HTML page 'solr.html' in the same directory: Let's visit http://localhost:8983/solr/solr.html to test the page:


(image via openwetware)

 That worked ! :-)

That's it,

Pierre