From 8e052cb7355eed7ce4d7075b23c9b0439285f84e Mon Sep 17 00:00:00 2001 From: AndreaGuarracino Date: Sun, 26 Jul 2020 18:02:41 +0200 Subject: updated org files, removing unuseful information and adding the BCC2020 video talk and poster links in the about page --- doc/blog/using-covid-19-pubseq-part2.html | 33 +- doc/blog/using-covid-19-pubseq-part2.org | 25 +- doc/blog/using-covid-19-pubseq-part3.html | 2 +- doc/blog/using-covid-19-pubseq-part3.org | 2 +- doc/web/about.html | 1489 ++++++++++++++++++----------- doc/web/about.org | 2 + 6 files changed, 960 insertions(+), 593 deletions(-) diff --git a/doc/blog/using-covid-19-pubseq-part2.html b/doc/blog/using-covid-19-pubseq-part2.html index c047441..c041ebe 100644 --- a/doc/blog/using-covid-19-pubseq-part2.html +++ b/doc/blog/using-covid-19-pubseq-part2.html @@ -259,39 +259,12 @@ for the JavaScript code in this tag. -

-As part of the COVID-19 Biohackathon 2020 we formed a working group to -create a COVID-19 Public Sequence Resource (COVID-19 PubSeq) for -Corona virus sequences. The general idea is to create a repository -that has a low barrier to entry for uploading sequence data using best -practices. I.e., data published with a creative commons 4.0 (CC-4.0) -license with metadata using state-of-the art standards and, perhaps -most importantly, providing standardised workflows that get triggered -on upload, so that results are immediately available in standardised -data formats. -

1 Finding output of workflows

-

-As part of the COVID-19 Biohackathon 2020 we formed a working group to -create a COVID-19 Public Sequence Resource (COVID-19 PubSeq) for -Corona virus sequences. The general idea is to create a repository -that has a low barrier to entry for uploading sequence data using best -practices. I.e., data published with a creative commons 4.0 (CC-4.0) -license with metadata using state-of-the art standards and, perhaps -most importantly, providing standardised workflows that get triggered -on upload, so that results are immediately available in standardised -data formats. -

-
-
-
-

2 Introduction

-
-

+

We are using Arvados to run common workflow language (CWL) pipelines. The most recent output is on display on a web page (with time stamp) and a full list is generated here. It is nice to start up, but for @@ -302,7 +275,7 @@ want to wade through thousands of output files!

-

3 The Arvados file interface

+

2 The Arvados file interface

Arvados has the web server, but it also has a REST API and associated @@ -384,7 +357,7 @@ arv-get 2be6af7b4741f2a5c5f8ff2bc6152d73+1955623+Ab9ad65d7fe958a053b3a57d545839d

-

4 Using the Arvados API

+

3 TODO Using the Arvados API

diff --git a/doc/blog/using-covid-19-pubseq-part2.org b/doc/blog/using-covid-19-pubseq-part2.org index d2a1cbc..349fd06 100644 --- a/doc/blog/using-covid-19-pubseq-part2.org +++ b/doc/blog/using-covid-19-pubseq-part2.org @@ -8,36 +8,13 @@ #+HTML_LINK_HOME: http://covid19.genenetwork.org #+HTML_HEAD: -As part of the COVID-19 Biohackathon 2020 we formed a working group to -create a COVID-19 Public Sequence Resource (COVID-19 PubSeq) for -Corona virus sequences. The general idea is to create a repository -that has a low barrier to entry for uploading sequence data using best -practices. I.e., data published with a creative commons 4.0 (CC-4.0) -license with metadata using state-of-the art standards and, perhaps -most importantly, providing standardised workflows that get triggered -on upload, so that results are immediately available in standardised -data formats. - * Table of Contents :TOC:noexport: - [[#finding-output-of-workflows][Finding output of workflows]] - - [[#introduction][Introduction]] - [[#the-arvados-file-interface][The Arvados file interface]] - [[#using-the-arvados-api][Using the Arvados API]] * Finding output of workflows -As part of the COVID-19 Biohackathon 2020 we formed a working group to -create a COVID-19 Public Sequence Resource (COVID-19 PubSeq) for -Corona virus sequences. The general idea is to create a repository -that has a low barrier to entry for uploading sequence data using best -practices. I.e., data published with a creative commons 4.0 (CC-4.0) -license with metadata using state-of-the art standards and, perhaps -most importantly, providing standardised workflows that get triggered -on upload, so that results are immediately available in standardised -data formats. - -* Introduction - We are using Arvados to run common workflow language (CWL) pipelines. The most recent output is on display on a [[https://workbench.lugli.arvadosapi.com/collections/lugli-4zz18-z513nlpqm03hpca][web page]] (with time stamp) and a full list is generated [[https://collections.lugli.arvadosapi.com/c=lugli-4zz18-z513nlpqm03hpca/][here]]. It is nice to start up, but for @@ -81,4 +58,4 @@ its listed UUID: : arv-get 2be6af7b4741f2a5c5f8ff2bc6152d73+1955623+Ab9ad65d7fe958a053b3a57d545839de18290843a@5ed7f3c5 -* Using the Arvados API +* TODO Using the Arvados API diff --git a/doc/blog/using-covid-19-pubseq-part3.html b/doc/blog/using-covid-19-pubseq-part3.html index 91879b0..df4a286 100644 --- a/doc/blog/using-covid-19-pubseq-part3.html +++ b/doc/blog/using-covid-19-pubseq-part3.html @@ -625,7 +625,7 @@ The web interface using this exact same script so it should just work

6.2 Example: uploading bulk GenBank sequences

-We also use above script to bulk upload GenBank sequences with a FASTA +We also use above script to bulk upload GenBank sequences with a FASTA and YAML extractor specific for GenBank. This means that the steps we took above for uploading a GenBank sequence are already automated.

diff --git a/doc/blog/using-covid-19-pubseq-part3.org b/doc/blog/using-covid-19-pubseq-part3.org index 03f37ab..e8fee36 100644 --- a/doc/blog/using-covid-19-pubseq-part3.org +++ b/doc/blog/using-covid-19-pubseq-part3.org @@ -234,6 +234,6 @@ The web interface using this exact same script so it should just work ** Example: uploading bulk GenBank sequences -We also use above script to bulk upload GenBank sequences with a [[https://github.com/arvados/bh20-seq-resource/blob/master/scripts/from_genbank_to_fasta_and_yaml.py][FASTA +We also use above script to bulk upload GenBank sequences with a [[https://github.com/arvados/bh20-seq-resource/blob/master/scripts/download_genbank_data/from_genbank_to_fasta_and_yaml.py][FASTA and YAML]] extractor specific for GenBank. This means that the steps we took above for uploading a GenBank sequence are already automated. diff --git a/doc/web/about.html b/doc/web/about.html index dfd4252..c971a4e 100644 --- a/doc/web/about.html +++ b/doc/web/about.html @@ -1,549 +1,964 @@ + "http://www.w3.org/TR/xhtml1/DTD/xhtml1-strict.dtd"> - - - -About/FAQ - - - - + + + + About/FAQ + + + +
-

About/FAQ

- - -
-

1 What is the 'public sequence resource' about?

-
-

-The public sequence resource aims to provide a generic and useful -resource for COVID-19 research. The focus is on providing the best -possible sequence data with associated metadata that can be used for -sequence comparison and protein prediction. -

-
-
- -
-

2 Who created the public sequence resource?

-
-

-The public sequence resource is an initiative by bioinformatics and -ontology experts who want to create something agile and useful for the -wider research community. The initiative started at the COVID-19 -biohackathon in April 2020 and is ongoing. The main project drivers -are Pjotr Prins (UTHSC), Peter Amstutz (Curii), Andrea Guarracino -(University of Rome Tor Vergata), Michael Crusoe (Common Workflow -Language), Thomas Liener (consultant, formerly EBI), Erik Garrison -(UCSC) and Jerven Bolleman (Swiss Institute of Bioinformatics). -

- -

-Notably, as this is a free software initiative, the project represents -major work by hundreds of software developers and ontology and data -wrangling experts. Thank you everyone! -

-
-
- -
-

3 How does the public sequence resource compare to other data resources?

-
-

-The short version is that we use state-of-the-art practices in -bioinformatics using agile methods. Unlike the resources from large -institutes we can improve things on a dime and anyone can contribute -to building out this resource! Sequences from GenBank, EBI/ENA and -others are regularly added to PubSeq. We encourage people to everyone -to submit on PubSeq because of its superior live tooling and metadata -support (see the next question). -

- -

-Importantly: all data is published under either the Creative Commons -4.0 attribution license or the CC0 “No Rights Reserved” license which -means it data can be published and workflows can run in public -environments allowing for improved access for research and -reproducible results. This contrasts with some other public resources, -such as GISAID. -

-
-
- -
-

4 Why should I upload my data here?

-
-
    -
  1. We champion truly shareable data without licensing restrictions - with proper -attribution
  2. -
  3. We provide full metadata support using state-of-the-art ontology's
  4. -
  5. We provide a web-based sequence uploader and a command-line version -for bulk uploads
  6. -
  7. We provide a live SPARQL end-point for all metadata
  8. -
  9. We provide free data analysis and sequence comparison triggered on data upload
  10. -
  11. We do real work for you, with this link you can see the last -run took 5.5 hours!
  12. -
  13. We provide free downloads of all computed output
  14. -
  15. There is no need to set up pipelines and/or compute clusters
  16. -
  17. All workflows get triggered on uploading a new sequence
  18. -
  19. When someone (you?) improves the software/workflows and everyone benefits
  20. -
  21. Your data gets automatically integrated with the Swiss Institure of -Bioinformatics COVID-19 knowledge base -https://covid-19-sparql.expasy.org/ (Elixir Switzerland)
  22. -
  23. Your data will be used to develop drug targets
  24. -
- -

-Finally, if you upload your data here we have workflows that output -formatted data suitable for uploading to EBI resources (and soon -others). Uploading your data here get your data ready for upload to -multiple resources. -

-
-
- -
-

5 Why should I not upload by data here?

-
-

-Funny question. There are only good reasons to upload your data here -and make it available to the widest audience possible. -

- -

-In fact, you can upload your data here as well as to other -resources. It is your data after all. No one can prevent you from -uploading your data to multiple resources. -

- -

-We recommend uploading to EBI and NCBI resources using our data -conversion tools. It means you only enter data once and make the -process smooth. You can also use our command line data uploader -for bulk uploads! -

-
-
- -
-

6 How does the public sequence resource work?

-
-

-On uploading a sequence with metadata it will automatically be -processed and incorporated into the public pangenome with metadata -using workflows from the High Performance Open Biology Lab defined -here. -

-
-
- -
-

7 Who uses the public sequence resource?

-
-

-The Swiss Institute of Bioinformatics has included this data in -https://covid-19-sparql.expasy.org/ and made it part of Uniprot. -

- -

-The Pantograph viewer uses PubSeq data for their visualisations. -

- -

-UTHSC (USA), ESR (New Zealand) and ORNL (USA) use COVID-19 PubSeq data -for monitoring, protein prediction and drug development. -

-
-
- -
-

8 How can I contribute?

-
-

-You can contribute by submitting sequences, updating metadata, submit -issues on our issue tracker, and more importantly add functionality. -See 'How do I change the source code' below. Read through our online -documentation at http://covid19.genenetwork.org/blog as a starting -point. -

-
-
- -
-

9 Is this about open data?

-
-

-All data is published under a Creative Commons 4.0 attribution license -(CC-BY-4.0). You can download the raw and published (GFA/RDF/FASTA) -data and store it for further processing. -

-
-
- -
-

10 Is this about free software?

-
-

-Absolutely. Free software allows for fully reproducible pipelines. You -can take our workflows and data and run it elsewhere! -

-
-
- -
-

11 How do I upload raw data?

-
-

-We are preparing raw sequence data pipelines (fastq and BAM). The -reason is that we want the best data possible for downstream analysis -(including protein prediction and test development). The current -approach where people publish final sequences of SARS-CoV-2 is lacking -because it hides how this sequence was created. For reasons of -reproducible and improved results we want/need to work with the raw -sequence reads (both short reads and long reads) and take alternative -assembly variations into consideration. This is all work in progress. -

-
-
- -
-

12 How do I change metadata?

- -
- -
-

13 How do I change the work flows?

-
-

-Workflows are on github and can be modified. See also the BLOG -http://covid19.genenetwork.org/blog on workflows. -

-
-
- -
-

14 How do I change the source code?

-
-

-Go to our source code repositories, fork/clone the repository, change -something and submit a pull request (PR). That easy! Check out how -many PRs we already merged. -

-
-
- -
-

15 Should I choose CC-BY or CC0?

-
-

-Restrictive data licenses are hampering data sharing and reproducible -research. CC0 is the preferred license because it gives researchers -the most freedom. Since we provide metadata there is no reason for -others not to honour your work. We also provide CC-BY as an option -because we know people like the attribution clause. -

- -

-In all honesty: we prefer both data and software to be free. -

-
-
- -
-

16 How do I deal with private data and privacy?

-
-

-A public sequence resource is about public data. Metadata can refer to -private data. You can use your own (anonymous) identifiers. We also -plan to combine identifiers with clinical data stored securely at -REDCap. See the relevant tracker for more information and contributing. -

-
-
- -
-

17 How do I communicate with you?

-
-

-We use a gitter channel you can join. -

-
-
- -
-

18 Who are the sponsors?

-
-

-The main sponsors are listed in the footer. In addition to the time -generously donated by many contributors we also acknowledge Amazon AWS -for donating COVID-19 related compute time. -

-
-
+

About/FAQ

+ + +
+

1 What is the 'public sequence resource' about?

+
+

+ The public sequence resource aims to provide a generic and useful + resource for COVID-19 research. The focus is on providing the best + possible sequence data with associated metadata that can be used for + sequence comparison and protein prediction. +

+

+ We were at the Bioinformatics Community Conference 2020! Have a look at the + video talk + (alternative link) + and the poster. +

+
+
+ +
+

2 Who created the public sequence resource?

+
+

+ The public sequence resource is an initiative by bioinformatics and + ontology experts who want to create something agile and useful for the + wider research community. The initiative started at the COVID-19 + biohackathon in April 2020 and is ongoing. The main project drivers + are Pjotr Prins (UTHSC), Peter Amstutz (Curii), Andrea Guarracino + (University of Rome Tor Vergata), Michael Crusoe (Common Workflow + Language), Thomas Liener (consultant, formerly EBI), Erik Garrison + (UCSC) and Jerven Bolleman (Swiss Institute of Bioinformatics). +

+ +

+ Notably, as this is a free software initiative, the project represents + major work by hundreds of software developers and ontology and data + wrangling experts. Thank you everyone! +

+
+
+ +
+

3 How does the public sequence resource compare to + other data resources?

+
+

+ The short version is that we use state-of-the-art practices in + bioinformatics using agile methods. Unlike the resources from large + institutes we can improve things on a dime and anyone can contribute + to building out this resource! Sequences from GenBank, EBI/ENA and + others are regularly added to PubSeq. We encourage people to everyone + to submit on PubSeq because of its superior live tooling and metadata + support (see the next question). +

+ +

+ Importantly: all data is published under either the Creative Commons + 4.0 attribution license or the CC0 “No Rights Reserved” + license which + means it data can be published and workflows can run in public + environments allowing for improved access for research and + reproducible results. This contrasts with some other public resources, + such as GISAID. +

+
+
+ +
+

4 Why should I upload my data here?

+
+
    +
  1. We champion truly shareable data without licensing restrictions - with proper + attribution +
  2. +
  3. We provide full metadata support using state-of-the-art ontology's
  4. +
  5. We provide a web-based sequence uploader and a command-line version + for bulk uploads +
  6. +
  7. We provide a live SPARQL end-point for all metadata
  8. +
  9. We provide free data analysis and sequence comparison triggered on data upload
  10. +
  11. We do real work for you, with this link + you can see the last + run took 5.5 hours! +
  12. +
  13. We provide free downloads of all computed output
  14. +
  15. There is no need to set up pipelines and/or compute clusters
  16. +
  17. All workflows get triggered on uploading a new sequence
  18. +
  19. When someone (you?) improves the software/workflows and everyone benefits
  20. +
  21. Your data gets automatically integrated with the Swiss Institure of + Bioinformatics COVID-19 knowledge base + https://covid-19-sparql.expasy.org/ (Elixir + Switzerland) +
  22. +
  23. Your data will be used to develop drug targets
  24. +
+ +

+ Finally, if you upload your data here we have workflows that output + formatted data suitable for uploading to EBI + resources (and soon + others). Uploading your data here get your data ready for upload to + multiple resources. +

+
+
+ +
+

5 Why should I not upload by data here?

+
+

+ Funny question. There are only good reasons to upload your data here + and make it available to the widest audience possible. +

+ +

+ In fact, you can upload your data here as well as to other + resources. It is your data after all. No one can prevent you from + uploading your data to multiple resources. +

+ +

+ We recommend uploading to EBI and NCBI resources using our data + conversion tools. It means you only enter data once and make the + process smooth. You can also use our command line data uploader + for bulk uploads! +

+
+
+ +
+

6 How does the public sequence resource work?

+
+

+ On uploading a sequence with metadata it will automatically be + processed and incorporated into the public pangenome with metadata + using workflows from the High Performance Open Biology Lab defined + here. +

+
+
+ +
+

7 Who uses the public sequence resource?

+
+

+ The Swiss Institute of Bioinformatics has included this data in + https://covid-19-sparql.expasy.org/ and made it part + of Uniprot. +

+ +

+ The Pantograph viewer uses PubSeq data for their + visualisations. +

+ +

+ UTHSC (USA), ESR (New Zealand) and + ORNL (USA) use COVID-19 PubSeq data + for monitoring, protein prediction and drug development. +

+
+
+ +
+

8 How can I contribute?

+
+

+ You can contribute by submitting sequences, updating metadata, submit + issues on our issue tracker, and more importantly add functionality. + See 'How do I change the source code' below. Read through our online + documentation at http://covid19.genenetwork.org/blog + as a starting + point. +

+
+
+ +
+

9 Is this about open data?

+
+

+ All data is published under a Creative Commons + 4.0 attribution license + (CC-BY-4.0). You can download the raw and published (GFA/RDF/FASTA) + data and store it for further processing. +

+
+
+ +
+

10 Is this about free software?

+
+

+ Absolutely. Free software allows for fully reproducible pipelines. You + can take our workflows and data and run it elsewhere! +

+
+
+ +
+

11 How do I upload raw data?

+
+

+ We are preparing raw sequence data pipelines (fastq and BAM). The + reason is that we want the best data possible for downstream analysis + (including protein prediction and test development). The current + approach where people publish final sequences of SARS-CoV-2 is lacking + because it hides how this sequence was created. For reasons of + reproducible and improved results we want/need to work with the raw + sequence reads (both short reads and long reads) and take alternative + assembly variations into consideration. This is all work in progress. +

+
+
+ +
+

12 How do I change metadata?

+ +
+ +
+

13 How do I change the work flows?

+
+

+ Workflows are on github + and can be modified. See also the BLOG + http://covid19.genenetwork.org/blog on workflows. +

+
+
+ +
+

14 How do I change the source code?

+
+

+ Go to our source code repositories, + fork/clone the repository, change + something and submit a pull request + (PR). That easy! Check out how + many PRs we already merged. +

+
+
+ +
+

15 Should I choose CC-BY or CC0?

+
+

+ Restrictive data licenses are hampering data sharing and reproducible + research. CC0 is the preferred license because it gives researchers + the most freedom. Since we provide metadata there is no reason for + others not to honour your work. We also provide CC-BY as an option + because we know people like the attribution clause. +

+ +

+ In all honesty: we prefer both data and software to be free. +

+
+
+ +
+

16 How do I deal with private data and privacy?

+
+

+ A public sequence resource is about public data. Metadata can refer to + private data. You can use your own (anonymous) identifiers. We also + plan to combine identifiers with clinical data stored securely at + REDCap. See the relevant tracker for more information and + contributing. +

+
+
+ +
+

17 How do I communicate with you?

+
+

+ We use a gitter + channel you can join. +

+
+
+ +
+

18 Who are the sponsors?

+
+

+ The main sponsors are listed in the footer. In addition to the time + generously donated by many contributors we also acknowledge Amazon AWS + for donating COVID-19 related compute time. +

+
+
-
Created by Pjotr Prins (pjotr.public768 at thebird 'dot' nl) using Emacs org-mode and a healthy dose of Lisp!
Modified 2020-07-18 Sat 03:27
. +
+ Created by Pjotr Prins (pjotr.public768 at thebird 'dot' nl) using Emacs + org-mode and a healthy dose of Lisp!
Modified 2020-07-18 Sat 03:27
.
diff --git a/doc/web/about.org b/doc/web/about.org index 39fb667..8a954bb 100644 --- a/doc/web/about.org +++ b/doc/web/about.org @@ -28,6 +28,8 @@ resource for COVID-19 research. The focus is on providing the best possible sequence data with associated metadata that can be used for sequence comparison and protein prediction. +We were at the *Bioinformatics Community Conference 2020*! Have a look at the [[https://bcc2020.sched.com/event/coLw]][video talk] ([[https://drive.google.com/file/d/1skXHwVKM_gl73-_4giYIOQ1IlC5X5uBo/view?usp=sharing]][alternative link]) and the [[https://drive.google.com/file/d/1vyEgfvSqhM9yIwWZ6Iys-QxhxtVxPSdp/view?usp=sharing]][poster]. + * Who created the public sequence resource? The *public sequence resource* is an initiative by [[https://github.com/arvados/bh20-seq-resource/graphs/contributors][bioinformatics]] and -- cgit v1.2.3