Hello WDL ( Workflow Description Language )

This is a quick note about my first WDL workflow (Workflow Description Language) https://software.broadinstitute.org/wdl/.

As a Makefile, my workflow would be the following one:

NAME?=world
$(NAME)_sed.txt : $(NAME).txt
 sed 's/Hello/Goodbye/' $< > $@
$(NAME).txt:
 echo "Hello $(NAME)" > $@

Executed as:

$ make NAME=WORLD

echo "Hello WORLD" > WORLD.txt
sed 's/Hello/Goodbye/' WORLD.txt > WORLD_sed.txt

As a WDL, the workflow is described as below:

/**

print "Hello (message)" into a file

*/
task echotask {
 String outfile
 String ? message


 command {
  echo "Hello ${default="world" message}" > ${outfile}
  }
  
 output {
  File out = "${outfile}"
  }
}

/**

replace Hello with goodbye

*/
task sedtask {
 String outfile
 File infile

 
 command {
        sed 's/Hello/Goodbye/' ${infile}  > ${outfile}
  }
 
 output {
       File out = "${outfile}"
  }
}


workflow basicworkflow {
String message

call echotask {
 input: outfile="${message}.txt", message = "${message}"
 }
call sedtask {
 input: infile = echotask.out , outfile  = "${message}_sed.txt"
 }
}

I've downloaded cromwell-0.22.jar and wdltool-0.4.jar from github.


Generate a JSON describing the input parameters:

$ java -jar wdltool-0.4.jar inputs test.wdl  > inputs.json
$ cat inputs.json
{
  "basicworkflow.message": "String"
}

Customize the basicworkflow.message in "inputs.json":

$ cat inputs.json
{
  "basicworkflow.message": "WORLD"
}

Execute the workflow:

$ java -jar cromwell-0.22.jar run test.wdl inputs.json

[2016-10-27 12:36:29,69] [info] Slf4jLogger started
[2016-10-27 12:36:29,73] [info] RUN sub-command
[2016-10-27 12:36:29,73] [info]   WDL file: /home/lindenb/tmp/WDL/test.wdl
[2016-10-27 12:36:29,73] [info]   Inputs: /home/lindenb/tmp/WDL/inputs.json
[2016-10-27 12:36:29,76] [info] SingleWorkflowRunnerActor: Submitting workflow
[2016-10-27 12:36:30,43] [info] Running with database db.url = jdbc:hsqldb:mem:a402b714-c86f-414a-b03f-8e6ecb533e33;shutdown=false;hsqldb.tx=mvcc
[2016-10-27 12:36:37,67] [info] Metadata summary refreshing every 2 seconds.
[2016-10-27 12:36:38,20] [info] Workflow 172e1061-82fc-4240-9e80-e08aaafd6f3f submitted.
[2016-10-27 12:36:38,20] [info] SingleWorkflowRunnerActor: Workflow submitted 172e1061-82fc-4240-9e80-e08aaafd6f3f
[2016-10-27 12:36:38,38] [info] 1 new workflows fetched
[2016-10-27 12:36:38,38] [info] WorkflowManagerActor Starting workflow 172e1061-82fc-4240-9e80-e08aaafd6f3f
[2016-10-27 12:36:38,39] [info] WorkflowManagerActor Successfully started WorkflowActor-172e1061-82fc-4240-9e80-e08aaafd6f3f
[2016-10-27 12:36:38,39] [info] Retrieved 1 workflows from the WorkflowStoreActor
[2016-10-27 12:36:38,76] [info] MaterializeWorkflowDescriptorActor [172e1061]: Call-to-Backend assignments: basicworkflow.echotask -> Local, basicworkflow.sedtask -> Local
[2016-10-27 12:36:38,96] [info] WorkflowExecutionActor-172e1061-82fc-4240-9e80-e08aaafd6f3f [172e1061]: Starting calls: basicworkflow.echotask:NA:1
[2016-10-27 12:36:39,10] [info] SharedFileSystemAsyncJobExecutionActor [172e1061basicworkflow.echotask:NA:1]: echo "Hello WORLD" > WORLD.txt
[2016-10-27 12:36:39,10] [info] SharedFileSystemAsyncJobExecutionActor [172e1061basicworkflow.echotask:NA:1]: executing: /bin/bash /home/lindenb/tmp/WDL/cromwell-executions/basicworkflow/172e1061-82fc-4240-9e80-e08aaafd6f3f/call-echotask/execution/script
[2016-10-27 12:36:39,11] [info] SharedFileSystemAsyncJobExecutionActor [172e1061basicworkflow.echotask:NA:1]: command: "/bin/bash" "/home/lindenb/tmp/WDL/cromwell-executions/basicworkflow/172e1061-82fc-4240-9e80-e08aaafd6f3f/call-echotask/execution/script.submit"
[2016-10-27 12:36:39,14] [info] SharedFileSystemAsyncJobExecutionActor [172e1061basicworkflow.echotask:NA:1]: job id: 6056
[2016-10-27 12:36:39,20] [info] WorkflowExecutionActor-172e1061-82fc-4240-9e80-e08aaafd6f3f [172e1061]: Starting calls: basicworkflow.sedtask:NA:1
[2016-10-27 12:36:39,27] [info] SharedFileSystemAsyncJobExecutionActor [172e1061basicworkflow.sedtask:NA:1]: sed 's/Hello/Goodbye/' /home/lindenb/tmp/WDL/cromwell-executions/basicworkflow/172e1061-82fc-4240-9e80-e08aaafd6f3f/call-sedtask/inputs/home/lindenb/tmp/WDL/cromwell-executions/basicworkflow/172e1061-82fc-4240-9e80-e08aaafd6f3f/call-echotask/execution/WORLD.txt  > WORLD_sed.txt
[2016-10-27 12:36:39,27] [info] SharedFileSystemAsyncJobExecutionActor [172e1061basicworkflow.sedtask:NA:1]: executing: /bin/bash /home/lindenb/tmp/WDL/cromwell-executions/basicworkflow/172e1061-82fc-4240-9e80-e08aaafd6f3f/call-sedtask/execution/script
[2016-10-27 12:36:39,27] [info] SharedFileSystemAsyncJobExecutionActor [172e1061basicworkflow.sedtask:NA:1]: command: "/bin/bash" "/home/lindenb/tmp/WDL/cromwell-executions/basicworkflow/172e1061-82fc-4240-9e80-e08aaafd6f3f/call-sedtask/execution/script.submit"
[2016-10-27 12:36:39,28] [info] SharedFileSystemAsyncJobExecutionActor [172e1061basicworkflow.sedtask:NA:1]: job id: 6063
[2016-10-27 12:36:40,57] [info] WorkflowExecutionActor-172e1061-82fc-4240-9e80-e08aaafd6f3f [172e1061]: Workflow complete. Final Outputs: 
{
  "basicworkflow.echotask.out": "/home/lindenb/tmp/WDL/cromwell-executions/basicworkflow/172e1061-82fc-4240-9e80-e08aaafd6f3f/call-echotask/execution/WORLD.txt",
  "basicworkflow.sedtask.out": "/home/lindenb/tmp/WDL/cromwell-executions/basicworkflow/172e1061-82fc-4240-9e80-e08aaafd6f3f/call-sedtask/execution/WORLD_sed.txt"
}
[2016-10-27 12:36:40,61] [info] WorkflowManagerActor WorkflowActor-172e1061-82fc-4240-9e80-e08aaafd6f3f is in a terminal state: WorkflowSucceededState
{
  "outputs": {
    "basicworkflow.sedtask.out": "/home/lindenb/tmp/WDL/cromwell-executions/basicworkflow/172e1061-82fc-4240-9e80-e08aaafd6f3f/call-sedtask/execution/WORLD_sed.txt",
    "basicworkflow.echotask.out": "/home/lindenb/tmp/WDL/cromwell-executions/basicworkflow/172e1061-82fc-4240-9e80-e08aaafd6f3f/call-echotask/execution/WORLD.txt"
  },
  "id": "172e1061-82fc-4240-9e80-e08aaafd6f3f"
}
[2016-10-27 12:36:43,43] [info] SingleWorkflowRunnerActor workflow finished with status 'Succeeded'.

Check the final output:

$ cat ./cromwell-executions/basicworkflow/172e1061-82fc-4240-9e80-e08aaafd6f3f/call-sedtask/execution/WORLD_sed.txt
Goodbye WORLD

That's it,

Pierre