GNU Make: saving the versions of the tools using 'order-only-prerequisites' : my notebook
Rule 3 of "Ten Simple Rules for Reproducible Computational Research". is
:
I work with Makefile-based workflows: how can I save the version of each software used when I invoke 'make', whatever the target is ? A naive solution is to add a dependency to each target. For example, the following makefile takes a simple SAM file, convert it to BAM, sort and index. For each target, I've added a dependency named "dump_params" that append the version of samtools to a file
"config.txt"
..PHONY: dump_params all clean | |
.SHELL=/bin/bash | |
all: sorted.bam.bai dump_params | |
sorted.bam.bai: sorted.bam dump_params | |
samtools index $< | |
sorted.bam: unsorted.bam dump_params | |
samtools sort $< $(basename $@) | |
unsorted.bam : samtools-0.1.18/examples/toy.sam dump_params | |
samtools view -Sb $< > $@ | |
dump_params: | |
date >> config.txt && \ | |
echo -n "Samtools " >> config.txt && \ | |
samtools 2>&1 | grep Version >> config.txt | |
clean: | |
rm -f sorted.bam.bai sorted.bam unsorted.bam |
But that solution doesn't work because make re-builds all targets even if the top target already exists.
$ make date << config.txt && \ echo -n "Samtools " << config.txt && \ samtools 2<&1 | grep Version << config.txt samtools view -Sb samtools-0.1.18/examples/toy.sam < unsorted.bam [samopen] SAM header is present: 2 sequences. samtools sort unsorted.bam sorted samtools index sorted.bam $ make date << config.txt && \ echo -n "Samtools " << config.txt && \ samtools 2<&1 | grep Version << config.txt samtools view -Sb samtools-0.1.18/examples/toy.sam < unsorted.bam [samopen] SAM header is present: 2 sequences. samtools sort unsorted.bam sorted samtools index sorted.bam
The solution I got via Stackoverflow is to use a order-only-prerequisites: "Order-only prerequisites can be specified by placing a pipe symbol (|) in the prerequisites list: any prerequisites to the left of the pipe symbol are normal; any prerequisites to the right are order-only... (...) Note that if you declare the same file to be both a normal and an order-only prerequisite, the normal prerequisite takes precedence (...)". The makefile with the 'order-only-prerequisites' is now:
.PHONY: dump_params all clean | |
.SHELL=/bin/bash | |
all: sorted.bam.bai dump_params | |
sorted.bam.bai: sorted.bam | dump_params | |
samtools index $< | |
sorted.bam: unsorted.bam | dump_params | |
samtools sort $< $(basename $@) | |
unsorted.bam : samtools-0.1.18/examples/toy.sam | dump_params | |
samtools view -Sb $< > $@ | |
dump_params: | |
date >> config.txt && \ | |
echo -n "Samtools " >> config.txt && \ | |
samtools 2>&1 | grep Version >> config.txt | |
clean: | |
rm -f sorted.bam.bai sorted.bam unsorted.bam |
And that works ! the final target is generated only once, but the file 'config.txt' is always generated.
$ make date << config.txt && \ echo -n "Samtools " << config.txt && \ samtools 2<&1 | grep Version << config.txt samtools view -Sb samtools-0.1.18/examples/toy.sam < unsorted.bam [samopen] SAM header is present: 2 sequences. samtools sort unsorted.bam sorted samtools index sorted.bam $ make date << config.txt && \ echo -n "Samtools " << config.txt && \ samtools 2<&1 | grep Version << config.txt $ make date << config.txt && \ echo -n "Samtools " << config.txt && \ samtools 2<&1 | grep Version << config.txtThat's it,
Pierre
Update :another solution
Citing MadScientist's answer on stackoverflow : Another option is to use immediately expanded shell functions, like:__dummy := $(shell echo "Makefile was run." >> config.txt)Since it's immediately expanded the shell script will be invoked once, as the makefile is read in. There's no need to define a dump_params target or include it as a prerequisite. This is more old-school, but has the advantage that it will run for every invocation of make, without having to go through and ensure every target has the proper order-only prerequisite defined.