From 6eef898f8080e64a2eab9b60f54cacbd419c279e Mon Sep 17 00:00:00 2001 From: Pjotr Prins Date: Tue, 10 Nov 2020 11:08:26 +0000 Subject: Document Arvados runner --- doc/blog/using-covid-19-pubseq-part2.html | 127 +++++++++++++++--------------- doc/blog/using-covid-19-pubseq-part2.org | 21 +++++ 2 files changed, 84 insertions(+), 64 deletions(-) diff --git a/doc/blog/using-covid-19-pubseq-part2.html b/doc/blog/using-covid-19-pubseq-part2.html index 567980d..eff6fcd 100644 --- a/doc/blog/using-covid-19-pubseq-part2.html +++ b/doc/blog/using-covid-19-pubseq-part2.html @@ -3,7 +3,7 @@ "http://www.w3.org/TR/xhtml1/DTD/xhtml1-strict.dtd"> - + COVID-19 PubSeq - Arvados @@ -40,7 +40,7 @@ } pre.src { position: relative; - overflow: visible; + overflow: auto; padding-top: 1.2em; } pre.src:before { @@ -195,50 +195,26 @@ @@ -252,18 +228,18 @@ for the JavaScript code in this tag.

Table of Contents

-
-

1 The Arvados Web Server

+
+

1 The Arvados Web Server

We are using Arvados to run common workflow language (CWL) pipelines. @@ -283,8 +259,8 @@ workflows and the output of analysis pipelines (here CWL workflows).

-
-

2 The Arvados file interface

+
+

2 The Arvados file interface

Arvados has the web server, but it also has a REST API and associated @@ -361,8 +337,8 @@ arv-get 2be6af7b4741f2a5c5f8ff2bc6152d73+1955623+Ab9ad65d7fe958a053b3a57d545839d

-
-

3 The PubSeq Arvados shell

+
+

3 The PubSeq Arvados shell

When you login to Arvados (you can request permission from us) it is @@ -414,11 +390,34 @@ the git repo and starts a new run calling into /data/pubseq/bh20-seq-resource/venv3/bin/bh20-seq-analyzer which is essentially monitoring for uploads.

+ +

+On run --help +

+ +
+optional arguments:
+  -h, --help            show this help message and exit
+  --uploader-project UPLOADER_PROJECT
+  --pangenome-analysis-project PANGENOME_ANALYSIS_PROJECT
+  --fastq-project FASTQ_PROJECT
+  --validated-project VALIDATED_PROJECT
+  --workflow-def-project WORKFLOW_DEF_PROJECT
+  --pangenome-workflow-uuid PANGENOME_WORKFLOW_UUID
+  --fastq-workflow-uuid FASTQ_WORKFLOW_UUID
+  --exclude-list EXCLUDE_LIST
+  --latest-result-collection LATEST_RESULT_COLLECTION
+  --kickoff
+  --no-start-analysis
+  --once
+  --print-status PRINT_STATUS
+  --revalidate
+
-
-

4 Wiring up CWL

+
+

4 Wiring up CWL

In above script bh20-seq-analyzer you can see that the Common @@ -459,8 +458,8 @@ For more see -

5 Using the Arvados API

+
+

5 Using the Arvados API

Arvados provides a rich API for accessing internals of the Cloud @@ -476,8 +475,8 @@ get a list of -

6 Troubleshooting

+
+

6 Troubleshooting

When workflows have errors we should check the logs in Arvados. @@ -494,7 +493,7 @@ see what parts failed.

-
Created by Pjotr Prins (pjotr.public768 at thebird 'dot' nl) using Emacs org-mode and a healthy dose of Lisp!
Modified 2020-08-26 Wed 05:01. +
Created by Pjotr Prins (pjotr.public768 at thebird 'dot' nl) using Emacs org-mode and a healthy dose of Lisp!
Modified 2020-11-09 Mon 01:20.
diff --git a/doc/blog/using-covid-19-pubseq-part2.org b/doc/blog/using-covid-19-pubseq-part2.org index 4b827f5..d7816ba 100644 --- a/doc/blog/using-covid-19-pubseq-part2.org +++ b/doc/blog/using-covid-19-pubseq-part2.org @@ -96,6 +96,27 @@ the git repo and starts a new run calling into /data/pubseq/bh20-seq-resource/venv3/bin/bh20-seq-analyzer which is essentially [[https://github.com/arvados/bh20-seq-resource/blob/2baa88b766ec540bd34b96599014dd16e393af39/bh20seqanalyzer/main.py#L354][monitoring]] for uploads. +On ~run --help~ + +#+begin_example +optional arguments: + -h, --help show this help message and exit + --uploader-project UPLOADER_PROJECT + --pangenome-analysis-project PANGENOME_ANALYSIS_PROJECT + --fastq-project FASTQ_PROJECT + --validated-project VALIDATED_PROJECT + --workflow-def-project WORKFLOW_DEF_PROJECT + --pangenome-workflow-uuid PANGENOME_WORKFLOW_UUID + --fastq-workflow-uuid FASTQ_WORKFLOW_UUID + --exclude-list EXCLUDE_LIST + --latest-result-collection LATEST_RESULT_COLLECTION + --kickoff + --no-start-analysis + --once + --print-status PRINT_STATUS + --revalidate +#+end_example + * Wiring up CWL In above script ~bh20-seq-analyzer~ you can see that the [[https://www.commonwl.org/][Common -- cgit v1.2.3