I suggested to download from pubmed all the articles published in an annual issue of NAR , to extract the URLs from the abstract and to check if they were still active. I just wrote a java program doing this job (it is available on github at https://github.com/lindenb/jsandbox/blob/master/src/sandbox/NucleicAcidsResearch404.java)
A few comments:
- The connection timeout was fixed to 10 seconds.
- Some URLs are poorly written e.g: http://www.ncbi.nlm.nih.gov/pubmed/14681415
- An abstract can contain more than one URL
- There can be different URLs for the same database
- getting a HTTP:404 error doesn't mean that the database has really been discontinued.
- getting a status HTTP:200 doesn't mean that the database is still active and/or maintained
- 1155 URLs have been extracted from this pubmed query `"Nucleic Acids Res"[JOUR] "Database issue"[ISS]` (as far as I can see , this query only goes to 2004) Edit:ok, that was because NCBI eFetch is limited to 10K records
| Year | Count(URL) | count(Active) | % |
|---|---|---|---|
| 2004 | 157 | 100 | 63 |
| 2005 | 162 | 114 | 70 |
| 2006 | 186 | 147 | 79 |
| 2007 | 194 | 158 | 81 |
| 2008 | 206 | 180 | 87 |
| 2009 | 208 | 186 | 89 |
| 2010 | 147 | 136 | 92 |
| 2011 | 200 | 193 | 96 |
... a snapshot of the output...
(...)
http://bioinfo
Exception:java.net.UnknownHostException
http://bioinformatics.med.ohio-state.edu/MPromDb
Exception:java.net.SocketTimeoutException
(...)
Credit for the Title: Neil Saunders ;-)
Update:
It seems that the URLs in the abstracts are broken where they were cut in the PDF !
That's it,
Pierre
You should publish a paper on this Pierre
ReplyDeleteThat will teach me to post my good ideas as comments before going to bed :-) Seriously, nice work and this would make a useful web application. I'm getting to work on that part right now.
ReplyDeleteSorry Neil, I didn't want to steal your idea :-)
ReplyDeleteIs there a FigShare for that table?
ReplyDeleteThis is definitely a good initiative. It will be very useful to have a web server (yes, another one) that lists all published web servers and their state (active / inactive / up / down). For better statistics, it will be interesting to re-run Pierre's script regularly (to eventually produce a percentage of accessibility).
ReplyDeletePierre and Neil, your idea also reminds me this article "Databases, data tombs and dust in the wind", by Wren and Bateman, Bioinformatics, 24: 2127 (2008) http://bioinformatics.oxfordjournals.org/content/24/19/2127
This is a great service! Did you email all the authors of the 'dead' resources? It would be very interesting to know the cause of their demise.
ReplyDeleteWren & Bateman's editorial to the 2008 NAR databases issue also deserves a mention in this context: "Databases, data tombs and dust in the wind"
http://bioinformatics.oxfordjournals.org/content/24/19/2127.long
Regarding the observation that the query only goes to 2004: this is not because of limits. It's because prior to that date, the issue (ISS) was not named "Database Issue" - it just had a volume number, like regular issues.
ReplyDeleteSome databases might move to a different location (moving lab from one university to another), hence change in the URL. NAR updates these links. For example, MPromDb is no longer at OSU. It is now available from the following link http://mpromdb.wistar.upenn.edu/
ReplyDeleteDoesn't http://biocatalogue.org already do this kind of service tracking?
ReplyDelete