Freebase Wikipedia Extraction (WEX)

Via the Freebase blog.

The Freebase Wikipedia Extraction (WEX) http://download.freebase.com/wex/ is a processed dump of the English language Wikipedia. The wiki markup for each article is transformed into machine-readable XML, and common relational features such as templates, infoboxes, categories, article sections, and redirects are extracted in tabular form.

Freebase WEX is provided as a set of database tables in TSV format for PostgreSQL, along with tables providing mappings between Wikipedia articles and Freebase topics, and corresponding Freebase Types

See also:

• DBPedia (Download)
• Wikipedia dumps

Pierre

Protocols & History

In Standards for biological protocols Dror, starts to discuss, how he thinks that protocol standards should look like, and starts building examples of well known procedures, I suggested him to look after RDFa or hGRDDL. However I've never seen a file format describing a protocol and that could be interpreted by a computer.

Hal9000: Hello Dave, how mini-preps do you want to process with the protocol urn:protocol:miniprep:AU87687 today ?
David Bowman: Hello Hal, I wish I could process 96 mini prep's
Hal9000: Dave, you'll need 192 tubes, 400 ml NaOH. The extraction will last 2H00 and will cost 10$. If you start now you will be able to lunch before the canteen is closed.
David Bowman: Thank you Hal, I'll use the centrifuge n°2
Hal9000: Dave, I detect problem with the centrifuge n°2. It should be repaired.
David Bowman: What ??? But we already changed it last week !!


Two blogs about digital history dealt with a parallel of this subject with bioinformatics: here and here. Handling historical data should be as fun as playing with biological data.

When that happens, the only way that historians are going to be able to grapple with this ocean of content will be something that we might call computational history–Bill likens this concept to bioinformatics.


I've already played with the RDF statements from DBPedia to generate an interactive time line about the History of Sciences but I guess there are many other things to do :-)

A Brief History Of Sciences

(Introduction copied from DBPedia): Wikipedia is the by far largest available encyclopedia on the Web. Wikipedia has the problem that its search capabilities are limited to full-text search, which only allows very limited access to this valuable knowledge-base.Semantic Web technologies enable expressive queries against structured information on the Web. The Semantic Web has the problem that there is not much RDF data online yet and that up-to-date terms and ontologies are missing for many application domains. The DBPedia project approaches both problems by extracting structured information from Wikipedia and by making this information available on the Semantic Web. A major feature of DBPedia is to enable sophisticated queries against Wikipedia using SPARQL. Extracting structured information from Wikipedia leads to quite astonishing query answering possibilities.

In February, the dbpedia dataset was available for download.

For example, here are the N3 statements about Francis Crick that were extracted from wikipeda with dbpedia.

<http://en.wikipedia.org/wiki/Francis_Crick>      <http://3ba.se/wikipedia/attributes/name> "Francis Crick" .
<http://en.wikipedia.org/wiki/Francis_Crick>      <http://3ba.se/wikipedia/attributes/image>        "FrancisHarryComptonCrick.jpg" .
<http://en.wikipedia.org/wiki/Francis_Crick>      <http://3ba.se/wikipedia/attributes/caption>      "Francis Harry Compton Crick" .
<http://en.wikipedia.org/wiki/Francis_Crick>      <http://3ba.se/wikipedia/attributes/birth_date>   "1916068"^^<http://www.w3.org/2001/XMLSchema#Date> .
<http://en.wikipedia.org/wiki/Francis_Crick>      <http://3ba.se/wikipedia/attributes/birth_place>  <http://en.wikipedia.org/wiki/Weston_Favell> .
<http://en.wikipedia.org/wiki/Francis_Crick>      <http://3ba.se/wikipedia/attributes/birth_place>  <http://en.wikipedia.org/wiki/Northamptonshire> .
<http://en.wikipedia.org/wiki/Francis_Crick>      <http://3ba.se/wikipedia/attributes/birth_place>  <http://en.wikipedia.org/wiki/UK> .
<http://en.wikipedia.org/wiki/Francis_Crick>      <http://3ba.se/wikipedia/attributes/residence>    <http://upload.wikimedia.org/wikipedia/commons/a/ae/Flag_of_the_United_Kingdom.svg> .
<http://en.wikipedia.org/wiki/Francis_Crick>      <http://3ba.se/wikipedia/attributes/residence>    <http://en.wikipedia.org/wiki/UK> .
<http://en.wikipedia.org/wiki/Francis_Crick>      <http://3ba.se/wikipedia/attributes/residence>    <http://upload.wikimedia.org/wikipedia/commons/a/a4/Flag_of_the_United_States.svg> .
<http://en.wikipedia.org/wiki/Francis_Crick>      <http://3ba.se/wikipedia/attributes/residence>    <http://en.wikipedia.org/wiki/USA> .
<http://en.wikipedia.org/wiki/Francis_Crick>      <http://3ba.se/wikipedia/attributes/nationality>  "[[Image:Flag_of_the_United_Kingdom.svg|20px|]]  [[England|English]]" .
<http://en.wikipedia.org/wiki/Francis_Crick>      <http://3ba.se/wikipedia/attributes/death_date>   "20040728"^^<http://www.w3.org/2001/XMLSchema#Date> .
<http://en.wikipedia.org/wiki/Francis_Crick>      <http://3ba.se/wikipedia/attributes/death_place>  <http://en.wikipedia.org/wiki/San_Diego> .
<http://en.wikipedia.org/wiki/Francis_Crick>      <http://3ba.se/wikipedia/attributes/death_place>  <http://en.wikipedia.org/wiki/California> .
<http://en.wikipedia.org/wiki/Francis_Crick>      <http://3ba.se/wikipedia/attributes/death_place>  <http://en.wikipedia.org/wiki/USA> .
<http://en.wikipedia.org/wiki/Francis_Crick>      <http://3ba.se/wikipedia/attributes/field>        <http://en.wikipedia.org/wiki/Biophysics> .
<http://en.wikipedia.org/wiki/Francis_Crick>      <http://3ba.se/wikipedia/attributes/work_institution>     <http://en.wikipedia.org/wiki/Salk_Institute> .
<http://en.wikipedia.org/wiki/Francis_Crick>      <http://3ba.se/wikipedia/attributes/alma_mater>   "[[University College London]][[University of Cambridge]]" .
<http://en.wikipedia.org/wiki/Francis_Crick>      <http://3ba.se/wikipedia/attributes/doctoral_advisor>     <http://en.wikipedia.org/wiki/Max_Perutz> .
<http://en.wikipedia.org/wiki/Francis_Crick>      <http://3ba.se/wikipedia/attributes/doctoral_students>    "None" .
<http://en.wikipedia.org/wiki/Francis_Crick>      <http://3ba.se/wikipedia/attributes/known_for>    "[[DNA|DNA structure]], [[consciousness]]" .
<http://en.wikipedia.org/wiki/Francis_Crick>      <http://3ba.se/wikipedia/attributes/prizes>       "[[Image:Nobel.png|20px]] [[Nobel Prize for Physiology or Medicine|Nobel Prize]] (1962)" .
<http://en.wikipedia.org/wiki/Francis_Crick>      <http://www.w3.org/1999/02/22-rdf-syntax-ns#type> <http://en.wikipedia.org/wiki/Template:infobox_scientist> .
<http://en.wikipedia.org/wiki/Francis_Crick>      <http://www.w3.org/1999/02/22-rdf-syntax-ns#label>        "Francis Crick" .
<http://en.wikipedia.org/wiki/Francis_Crick>      <http://www.w3.org/1999/02/22-rdf-syntax-ns#type> <http://en.wikipedia.org/wiki/Category:Nobel_laureates_in_Physiology_or_Medicine> .
<http://en.wikipedia.org/wiki/Francis_Crick>      <http://www.w3.org/1999/02/22-rdf-syntax-ns#type> <http://en.wikipedia.org/wiki/Category:English_neuroscientists> .
<http://en.wikipedia.org/wiki/Francis_Crick>      <http://www.w3.org/1999/02/22-rdf-syntax-ns#type> <http://en.wikipedia.org/wiki/Category:English_humanists> .

Iv' been playing with this source of data to build a dynamic timeline of scientists. The person choosen was the one where I could find a date of birth/death (some people didn't have this information or the date was not parseable (e.g. "[[March 11]], [[1922]] ([[Istanbul]], [[Turkey]])") ord I only used the predicates that could be used to find the date. For example here are a few predicates that could be used to find the date of birth:

http://3ba.se/wikipedia/attributes/birth_date
http://3ba.se/wikipedia/attributes/birthdate
http://3ba.se/wikipedia/attributes/Birthdate
http://3ba.se/wikipedia/attributes/date_birth
http://3ba.se/wikipedia/attributes/datebirth
http://3ba.se/wikipedia/attributes/date_of_birth
http://3ba.se/wikipedia/attributes/dateofbirth
http://3ba.se/wikipedia/attributes/DateOfBirth
http://3ba.se/wikipedia/attributes/DATE_OF_BIRTH
http://en.wikipedia.org/wiki/Category:1945_births>
(...)

I also used relationships between persons that could be found in the database. e.g.:

<http://en.wikipedia.org/wiki/Hermann_Joseph_Muller>    <http://3ba.se/wikipedia/attributes/teachers>   <http://en.wikipedia.org/wiki/Thomas_Hunt_Morgan> .

The result is a java application called WikiStory.
It requires JAVA Webstart 1.6. You can use the following command line:

javaws http://www.urbigene.com/wikistory/wikistory.jnlp

Selecting one or more category will select all the people that belong to it.
Clicking on a person in the timeline will load the page from wikipedia.


The timeline can also be saved as a SVG picture. If you're using Firefox, you can see an example Here

See also:

http://www.wikitimescale.org/index.php
http://meta.wikimedia.org/wiki/EasyTimeline
http://www.futureswatch.org/Timeline.htm
http://www.todayinsci.com/
http://www.perseus.tufts.edu/
http://dandelife.com/
http://www.timelineindex.com/content/home.php








Pierre