path: root/doc/ccwl.texi

\input texinfo
@settitle Concise Common Workflow Language

@copying
Copyright @copyright{} 2021 Arun Isaac@*

Permission is granted to copy, distribute and/or modify this document
under the terms of the GNU Free Documentation License, Version 1.3 or
any later version published by the Free Software Foundation; with no
Invariant Sections, no Front-Cover Texts, and no Back-Cover Texts.  A
copy of the license is included in the section entitled ``GNU Free
Documentation License''.
@end copying

@ifnottex
@node Top
@top Concise Common Workflow Language
@end ifnottex

This manual documents the development version of @abbr{ccwl, Concise
Common Workflow Language}, a concise syntax to express @abbr{CWL, Common
Workflow Language} workflows.

@menu
* Introduction:: What is ccwl?
* Tutorial:: A quick tutorial to get started with ccwl
* Contributing:: Contributing
@end menu

@node Introduction
@chapter Introduction

@abbr{CWL, Common Workflow Language} is an open standard for describing
analysis workflows and tools in a way that makes them portable and
scalable across a variety of software and hardware environments.

@abbr{ccwl, Concise Common Workflow Language} is a concise syntax to
express CWL workflows. It is implemented as an @abbr{EDSL, Embedded
Domain Specific Language} in the Scheme programming language, a
minimalist dialect of the Lisp family of programming languages.

ccwl is a compiler to generate CWL workflows from concise descriptions
in ccwl. In the future, ccwl will also have a runtime whereby users can
interactively execute workflows while developing them.

@node Tutorial
@chapter Tutorial

This tutorial will introduce you to writing workflows in ccwl. Some
knowledge of CWL is assumed. To learn about CWL, please see the
@url{https://www.commonwl.org/user_guide/, Common Workflow Language User
Guide}.

@menu
* Important concepts:: Static typing, multiple named inputs and outputs
* First example:: Our first ccwl workflow
* Capturing stdout:: Capturing the standard output stream of a command
* Capturing output files:: Capturing output files produced by a command
@end menu

@node Important concepts
@section The CWL and ccwl workflow languages

The CWL and ccwl workflow languages are statically typed programming
languages where functions accept multiple named inputs and return
multiple named outputs. Let's break down what that means.

@subsection Static typing

In CWL, the type of arguments accepted by a function and the type of
outputs returned by that function are specified explicitly by the
programmer, and are known at compile time even before the code has been
run. Hence, we say that it is statically typed.

@subsection Positional arguments and named arguments

In many languages, the order of arguments passed to a function is
significant. The position of each argument determines which formal
argument it gets mapped to. For example, passing positional arguments in
Scheme looks like

@lisp
(foo 1 2)
@end lisp

In a language that supports named arguments, the order of arguments is
not significant. Each argument explicitly names the formal argument it
gets mapped to. For example, in Scheme, passing named arguments may look
like

@lisp
(foo #:bar 1 #:baz 2)
@end lisp

@subsection Multiple function arguments and return values

In most languages, functions accept multiple input arguments but only
return a single output value. However, in CWL, a function can return
multiple output values as well. These multiple outputs are unordered and
are each addressed by a unique name.

@node First example
@section First example

As is tradition, let us start with a simple ``Hello World'' workflow in
ccwl. This workflow accepts a string input and prints that string.

@lisp
(define print
  (command #:run "echo" (input 'message #:type 'string)))

(workflow ((message #:type string))
  (print #:message message))
@end lisp

The first form in this code defines the @code{print} command. This form
is the equivalent of defining a @code{CommandLineTool} class workflow in
CWL. All arguments after @code{#:run} specify the command that will be
run. One of those arguments @code{(input 'message #:type 'string)}
refers to a @code{string} type input named @code{message}. Notice how
the command definition is very close to a shell command, only that it is
slightly annotated with inputs and their types.

The second form describes the actual workflow and is the equivalent of
defining a @code{Workflow} class workflow in CWL. The form
@code{((message #:type string))} specifies the inputs of the
workflow. In this case, there is only one input---@code{message} of type
@code{string}. The body of the workflow specifies the commands that will
be executed. The body of this workflow executes only a single
command---the @code{print} command---passing the @code{message} input of
the workflow as the @code{message} input to the @code{print} command.

If this workflow is written to a file @file{hello-world.scm}, we may
compile it to CWL by running

@example
$ ccwl compile hello-world.scm
@end example

This prints a big chunk of generated CWL to standard output. We have
achieved quite a lot of concision already! We write the generated CWL to
a file and execute it using @code{cwltool} as follows. The expected
output is also shown.

@example
$ ccwl compile hello-world.scm > hello-world.cwl
$ cwltool hello-world.cwl --message "Hello World!"
[workflow ] start
[workflow ] starting step echo
[step echo] start
[job echo] /tmp/zprgn3x0$ echo \
    'Hello World!'
Hello World!
[job echo] completed success
[step echo] completed success
[workflow ] completed success
@{@}
Final process status is success
@end example

@node Capturing stdout
@section Capturing the standard output stream of a command

Let us return to the ``Hello World'' example in the previous
section. But now, let us capture the standard output of the @code{print}
command in an output object. The ccwl code is the same as earlier with
only the addition of an @code{stdout} type output object to the command
definition.

@lisp
(define print
  (command #:run "echo" (input 'message #:type 'string)
           #:outputs (output 'printed-message #:type 'stdout)))

(workflow ((message #:type string))
  (print #:message message))
@end lisp

Let's write this code to a file @file{capture-stdout.scm}, generate CWL,
write the generated CWL to @file{capture-stdout.cwl}, and run it using
@code{cwltool}. We might expect something like the output below. Notice
how the standard output of the @code{print} command has been captured in
the file @file{51fe79d15e7790a9ded795304220d7a44aa84b48}.

@example
$ ccwl compile capture-stdout.scm > capture-stdout.cwl
$ cwltool capture-stdout.cwl --message "Hello World!"
[workflow ] start
[workflow ] starting step print
[step print] start
[job print] /tmp/7zksx3xm$ echo \
    'Hello World!' > /tmp/7zksx3xm/51fe79d15e7790a9ded795304220d7a44aa84b48
[job print] completed success
[step print] completed success
[workflow ] completed success
@{
    "printed-message": @{
        "location": "file:///home/manimekalai/51fe79d15e7790a9ded795304220d7a44aa84b48",
        "basename": "51fe79d15e7790a9ded795304220d7a44aa84b48",
        "class": "File",
        "checksum": "sha1$a0b65939670bc2c010f4d5d6a0b3e4e4590fb92b",
        "size": 13,
        "path": "/home/manimekalai/51fe79d15e7790a9ded795304220d7a44aa84b48"
    @}
@}
Final process status is success
@end example

@node Capturing output files
@section Capturing output files

In the previous section, we captured the standard output stream of a
command. But, how do we capture any output files created by a command?
Let us see.

Consider a tar archive @file{hello.tar} containing a file
@file{hello.txt}.

@example
$ tar --list --file hello.tar
hello.txt
@end example

Let us write a workflow to extract the file @file{hello.txt} from the
archive. Everything in the following workflow except the
@code{#:binding} parameter will already be familiar to you. The
@code{#:binding} parameter sets the @code{outputBinding} field in the
generated CWL. In the example below, we set the @code{glob} field to
look for a file named @file{hello.txt}.

@lisp
(define extract
  (command #:run "tar" "--extract" "--file" (input 'archive #:type 'File)
           #:outputs (output 'extracted-file
                             #:type 'File
                             #:binding '((glob . "hello.txt")))))

(workflow ((archive #:type File))
  (extract #:archive archive))
@end lisp

Writing this workflow to @file{capture-output-file.scm}, compiling and
running it gives us the following output. Notice that the file
@file{hello.txt} has been captured and is now present in our current
working directory.

@example
$ ccwl compile capture-output-file.scm > capture-output-file.cwl
$ cwltool capture-output-file.cwl --archive hello.tar
[workflow ] start
[workflow ] starting step extract
[step extract] start
[job extract] /tmp/nrolttex$ tar \
    --extract \
    --file \
    /tmp/z7pp7qwh/stg3ac272aa-3459-4f20-a033-86f53ba72caf/hello.tar
[job extract] completed success
[step extract] completed success
[workflow ] completed success
@{
    "extracted-file": @{
        "location": "file:///home/manimekalai/hello.txt",
        "basename": "hello.txt",
        "class": "File",
        "checksum": "sha1$a0b65939670bc2c010f4d5d6a0b3e4e4590fb92b",
        "size": 13,
        "path": "/home/manimekalai/hello.txt"
    @}
@}
Final process status is success
@end example

The above workflow is not awfully flexible. The name of the file to
extract is hardcoded into the workflow. Let us modify the workflow to
accept the name of the file to extract. We introduce @code{extractfile},
a @code{string} type input that is passed to @command{tar} and is
referenced in the @code{glob} field.

@lisp
(define extract-specific-file
  (command #:run "tar" "--extract" "--file" (input 'archive #:type 'File)
                                            (input 'extractfile #:type 'string)
           #:outputs (output 'extracted-file
                             #:type 'File
                             #:binding '((glob . "$(inputs.extractfile)")))))

(workflow ((archive #:type File) (extractfile #:type string))
  (extract-specific-file #:archive archive #:extractfile extractfile))
@end lisp

Compiling and running this workflow gives us the following output.

@example
$ ccwl compile capture-output-file.scm > capture-output-file.cwl
$ cwltool capture-output-file.cwl --archive hello.tar --extractfile hello.txt
[workflow ] start
[workflow ] starting step extract-specific-file
[step extract-specific-file] start
[job extract-specific-file] /tmp/751nydd1$ tar \
    --extract \
    --file \
    /tmp/1zzw2n6m/stgc851e003-b5bd-437e-844b-311f6f66a7f1/hello.tar \
    hello.txt
[job extract-specific-file] completed success
[step extract-specific-file] completed success
[workflow ] completed success
@{
    "extracted-file": @{
        "location": "file:///home/manimekalai/hello.txt",
        "basename": "hello.txt",
        "class": "File",
        "checksum": "sha1$a0b65939670bc2c010f4d5d6a0b3e4e4590fb92b",
        "size": 13,
        "path": "/home/manimekalai/hello.txt"
    @}
@}
Final process status is success
@end example

@node Contributing
@chapter Contributing

ccwl is developed on GitHub at
@url{https://github.com/arunisaac/ccwl}. Feedback, suggestions, feature
requests, bug reports and pull requests are all welcome. Unclear and
unspecific error messages are considered a bug. Do report them!

@bye