Genbank upload

author: Pjotr Prins 2020-08-24 10:31:24 +0100
committer: Pjotr Prins 2020-08-24 10:31:24 +0100
commit: 7fabc4f9427856600e237c6cacd710f49b88d45d (patch)
tree: 23fc807559c84ca618ec0145233f76bc6ba64fd5 /doc/blog
parent: b50c13cab729327490a9976727a95af81d629f23 (diff)
download: bh20-seq-resource-7fabc4f9427856600e237c6cacd710f49b88d45d.tar.gz
bh20-seq-resource-7fabc4f9427856600e237c6cacd710f49b88d45d.tar.lz
bh20-seq-resource-7fabc4f9427856600e237c6cacd710f49b88d45d.zip
2 files changed, 92 insertions, 65 deletions
diff --git a/doc/blog/using-covid-19-pubseq-part3.html b/doc/blog/using-covid-19-pubseq-part3.html
index 80304c3..718b10f 100644
--- a/doc/blog/using-covid-19-pubseq-part3.html
+++ b/doc/blog/using-covid-19-pubseq-part3.html
@@ -3,7 +3,7 @@
 "http://www.w3.org/TR/xhtml1/DTD/xhtml1-strict.dtd">
 <html xmlns="http://www.w3.org/1999/xhtml" lang="en" xml:lang="en">
 <head>
-<!-- 2020-08-22 Sat 07:43 -->
+<!-- 2020-08-24 Mon 04:31 -->
 <meta http-equiv="Content-Type" content="text/html;charset=utf-8" />
 <meta name="viewport" content="width=device-width, initial-scale=1" />
 <title>COVID-19 PubSeq Uploading Data (part 3)</title>
@@ -248,40 +248,40 @@ for the JavaScript code in this tag.
 <h2>Table of Contents</h2>
 <div id="text-table-of-contents">
 <ul>
-<li><a href="#org4419df1">1. Uploading Data</a></li>
-<li><a href="#org53a95df">2. Step 1: Upload sequence</a></li>
-<li><a href="#orga007292">3. Step 2: Add metadata</a>
+<li><a href="#orgdaec996">1. Uploading Data</a></li>
+<li><a href="#org8472a05">2. Step 1: Upload sequence</a></li>
+<li><a href="#org668a46d">3. Step 2: Add metadata</a>
 <ul>
-<li><a href="#org645875f">3.1. Obligatory fields</a>
+<li><a href="#orga044bef">3.1. Obligatory fields</a>
 <ul>
-<li><a href="#org6e395e3">3.1.1. Sample ID (sample<sub>id</sub>)</a></li>
-<li><a href="#org350066d">3.1.2. Collection date</a></li>
-<li><a href="#org336b4af">3.1.3. Collection location</a></li>
-<li><a href="#org66f0f8e">3.1.4. Sequencing technology</a></li>
-<li><a href="#orgb370da6">3.1.5. Authors</a></li>
+<li><a href="#org8e17492">3.1.1. Sample ID (sample<sub>id</sub>)</a></li>
+<li><a href="#orgd9805db">3.1.2. Collection date</a></li>
+<li><a href="#org3bd4901">3.1.3. Collection location</a></li>
+<li><a href="#org921de27">3.1.4. Sequencing technology</a></li>
+<li><a href="#org39fa678">3.1.5. Authors</a></li>
 </ul>
 </li>
-<li><a href="#orgaafe83d">3.2. Optional fields</a>
+<li><a href="#org5315804">3.2. Optional fields</a>
 <ul>
-<li><a href="#org9708c5c">3.2.1. Host information</a></li>
-<li><a href="#org5c6c880">3.2.2. Collecting institution</a></li>
-<li><a href="#org1754a88">3.2.3. Specimen source</a></li>
-<li><a href="#org00806b9">3.2.4. Source database accession</a></li>
-<li><a href="#orga69b584">3.2.5. Strain name</a></li>
+<li><a href="#orgf2b82d9">3.2.1. Host information</a></li>
+<li><a href="#org8986ca7">3.2.2. Collecting institution</a></li>
+<li><a href="#orge03eb0c">3.2.3. Specimen source</a></li>
+<li><a href="#org6815a6e">3.2.4. Source database accession</a></li>
+<li><a href="#org51b37e8">3.2.5. Strain name</a></li>
 </ul>
 </li>
 </ul>
 </li>
-<li><a href="#org34a76cf">4. Step 3: Submit to COVID-19 PubSeq</a>
+<li><a href="#org5778da6">4. Step 3: Submit to COVID-19 PubSeq</a>
 <ul>
-<li><a href="#orge4dfc55">4.1. Trouble shooting</a></li>
+<li><a href="#orge803d65">4.1. Trouble shooting</a></li>
 </ul>
 </li>
-<li><a href="#org0bbb27f">5. Step 4: Check output</a></li>
-<li><a href="#orged64f4e">6. Bulk sequence uploader</a>
+<li><a href="#org540cfdf">5. Step 4: Check output</a></li>
+<li><a href="#org6c43ab3">6. Bulk sequence uploader</a>
 <ul>
-<li><a href="#orge1c3625">6.1. Run the uploader (CLI)</a></li>
-<li><a href="#org07f04d9">6.2. Example: uploading bulk GenBank sequences</a></li>
+<li><a href="#org99bb8b7">6.1. Run the uploader (CLI)</a></li>
+<li><a href="#orga88593f">6.2. Example: uploading bulk GenBank sequences</a></li>
 </ul>
 </li>
 </ul>
@@ -290,8 +290,8 @@ for the JavaScript code in this tag.
 
 
 
-<div id="outline-container-org4419df1" class="outline-2">
-<h2 id="org4419df1"><span class="section-number-2">1</span> Uploading Data</h2>
+<div id="outline-container-orgdaec996" class="outline-2">
+<h2 id="orgdaec996"><span class="section-number-2">1</span> Uploading Data</h2>
 <div class="outline-text-2" id="text-1">
 <p>
 The COVID-19 PubSeq allows you to upload your SARS-Cov-2 strains to a
@@ -301,8 +301,8 @@ gets triggered on upload. Read the <a href="./about">ABOUT</a> page for more inf
 </div>
 </div>
 
-<div id="outline-container-org53a95df" class="outline-2">
-<h2 id="org53a95df"><span class="section-number-2">2</span> Step 1: Upload sequence</h2>
+<div id="outline-container-org8472a05" class="outline-2">
+<h2 id="org8472a05"><span class="section-number-2">2</span> Step 1: Upload sequence</h2>
 <div class="outline-text-2" id="text-2">
 <p>
 To upload a sequence in the <a href="http://covid19.genenetwork.org/">web upload page</a> hit the browse button and
@@ -330,8 +330,8 @@ an improved pangenome.
 </div>
 </div>
 
-<div id="outline-container-orga007292" class="outline-2">
-<h2 id="orga007292"><span class="section-number-2">3</span> Step 2: Add metadata</h2>
+<div id="outline-container-org668a46d" class="outline-2">
+<h2 id="org668a46d"><span class="section-number-2">3</span> Step 2: Add metadata</h2>
 <div class="outline-text-2" id="text-3">
 <p>
 The <a href="./">web upload page</a> contains fields for adding metadata. Metadata is
@@ -357,12 +357,12 @@ the web form. Here we add some extra information.
 </p>
 </div>
 
-<div id="outline-container-org645875f" class="outline-3">
-<h3 id="org645875f"><span class="section-number-3">3.1</span> Obligatory fields</h3>
+<div id="outline-container-orga044bef" class="outline-3">
+<h3 id="orga044bef"><span class="section-number-3">3.1</span> Obligatory fields</h3>
 <div class="outline-text-3" id="text-3-1">
 </div>
-<div id="outline-container-org6e395e3" class="outline-4">
-<h4 id="org6e395e3"><span class="section-number-4">3.1.1</span> Sample ID (sample<sub>id</sub>)</h4>
+<div id="outline-container-org8e17492" class="outline-4">
+<h4 id="org8e17492"><span class="section-number-4">3.1.1</span> Sample ID (sample<sub>id</sub>)</h4>
 <div class="outline-text-4" id="text-3-1-1">
 <p>
 This is a string field that defines a unique sample identifier by the
@@ -380,8 +380,8 @@ Here we add the GenBank ID MT536190.1.
 </div>
 </div>
 
-<div id="outline-container-org350066d" class="outline-4">
-<h4 id="org350066d"><span class="section-number-4">3.1.2</span> Collection date</h4>
+<div id="outline-container-orgd9805db" class="outline-4">
+<h4 id="orgd9805db"><span class="section-number-4">3.1.2</span> Collection date</h4>
 <div class="outline-text-4" id="text-3-1-2">
 <p>
 Estimated collection date. The GenBank page says April 6, 2020.
@@ -389,8 +389,8 @@ Estimated collection date. The GenBank page says April 6, 2020.
 </div>
 </div>
 
-<div id="outline-container-org336b4af" class="outline-4">
-<h4 id="org336b4af"><span class="section-number-4">3.1.3</span> Collection location</h4>
+<div id="outline-container-org3bd4901" class="outline-4">
+<h4 id="org3bd4901"><span class="section-number-4">3.1.3</span> Collection location</h4>
 <div class="outline-text-4" id="text-3-1-3">
 <p>
 A search on wikidata says Los Angeles is
@@ -399,8 +399,8 @@ A search on wikidata says Los Angeles is
 </div>
 </div>
 
-<div id="outline-container-org66f0f8e" class="outline-4">
-<h4 id="org66f0f8e"><span class="section-number-4">3.1.4</span> Sequencing technology</h4>
+<div id="outline-container-org921de27" class="outline-4">
+<h4 id="org921de27"><span class="section-number-4">3.1.4</span> Sequencing technology</h4>
 <div class="outline-text-4" id="text-3-1-4">
 <p>
 GenBank entry says Illumina, so we can fill that in
@@ -408,8 +408,8 @@ GenBank entry says Illumina, so we can fill that in
 </div>
 </div>
 
-<div id="outline-container-orgb370da6" class="outline-4">
-<h4 id="orgb370da6"><span class="section-number-4">3.1.5</span> Authors</h4>
+<div id="outline-container-org39fa678" class="outline-4">
+<h4 id="org39fa678"><span class="section-number-4">3.1.5</span> Authors</h4>
 <div class="outline-text-4" id="text-3-1-5">
 <p>
 GenBank entry says 'Lamers,S., Nolan,D.J., Rose,R., Cross,S., Moraga
@@ -420,16 +420,16 @@ Freehan,A. and Garcia-Diaz,J.', so we can fill that in.
 </div>
 </div>
 
-<div id="outline-container-orgaafe83d" class="outline-3">
-<h3 id="orgaafe83d"><span class="section-number-3">3.2</span> Optional fields</h3>
+<div id="outline-container-org5315804" class="outline-3">
+<h3 id="org5315804"><span class="section-number-3">3.2</span> Optional fields</h3>
 <div class="outline-text-3" id="text-3-2">
 <p>
 All other fields are optional. But let's see what we can add.
 </p>
 </div>
 
-<div id="outline-container-org9708c5c" class="outline-4">
-<h4 id="org9708c5c"><span class="section-number-4">3.2.1</span> Host information</h4>
+<div id="outline-container-orgf2b82d9" class="outline-4">
+<h4 id="orgf2b82d9"><span class="section-number-4">3.2.1</span> Host information</h4>
 <div class="outline-text-4" id="text-3-2-1">
 <p>
 Sadly, not much is known about the host from GenBank. A little
@@ -443,8 +443,8 @@ did to the person and what the person was like (say age group).
 </div>
 </div>
 
-<div id="outline-container-org5c6c880" class="outline-4">
-<h4 id="org5c6c880"><span class="section-number-4">3.2.2</span> Collecting institution</h4>
+<div id="outline-container-org8986ca7" class="outline-4">
+<h4 id="org8986ca7"><span class="section-number-4">3.2.2</span> Collecting institution</h4>
 <div class="outline-text-4" id="text-3-2-2">
 <p>
 We can fill that in.
@@ -452,8 +452,8 @@ We can fill that in.
 </div>
 </div>
 
-<div id="outline-container-org1754a88" class="outline-4">
-<h4 id="org1754a88"><span class="section-number-4">3.2.3</span> Specimen source</h4>
+<div id="outline-container-orge03eb0c" class="outline-4">
+<h4 id="orge03eb0c"><span class="section-number-4">3.2.3</span> Specimen source</h4>
 <div class="outline-text-4" id="text-3-2-3">
 <p>
 We have that: nasopharyngeal swab
@@ -461,8 +461,8 @@ We have that: nasopharyngeal swab
 </div>
 </div>
 
-<div id="outline-container-org00806b9" class="outline-4">
-<h4 id="org00806b9"><span class="section-number-4">3.2.4</span> Source database accession</h4>
+<div id="outline-container-org6815a6e" class="outline-4">
+<h4 id="org6815a6e"><span class="section-number-4">3.2.4</span> Source database accession</h4>
 <div class="outline-text-4" id="text-3-2-4">
 <p>
 Genbank which is <a href="http://identifiers.org/insdc/MT536190.1#sequence">http://identifiers.org/insdc/MT536190.1#sequence</a>.
@@ -471,8 +471,8 @@ Note we plug in our own identifier MT536190.1.
 </div>
 </div>
 
-<div id="outline-container-orga69b584" class="outline-4">
-<h4 id="orga69b584"><span class="section-number-4">3.2.5</span> Strain name</h4>
+<div id="outline-container-org51b37e8" class="outline-4">
+<h4 id="org51b37e8"><span class="section-number-4">3.2.5</span> Strain name</h4>
 <div class="outline-text-4" id="text-3-2-5">
 <p>
 SARS-CoV-2/human/USA/LA-BIE-070/2020
@@ -482,8 +482,8 @@ SARS-CoV-2/human/USA/LA-BIE-070/2020
 </div>
 </div>
 
-<div id="outline-container-org34a76cf" class="outline-2">
-<h2 id="org34a76cf"><span class="section-number-2">4</span> Step 3: Submit to COVID-19 PubSeq</h2>
+<div id="outline-container-org5778da6" class="outline-2">
+<h2 id="org5778da6"><span class="section-number-2">4</span> Step 3: Submit to COVID-19 PubSeq</h2>
 <div class="outline-text-2" id="text-4">
 <p>
 Once you have the sequence and the metadata together, hit
@@ -493,8 +493,8 @@ submitted and the workflows should kick in!
 </div>
 
 
-<div id="outline-container-orge4dfc55" class="outline-3">
-<h3 id="orge4dfc55"><span class="section-number-3">4.1</span> Trouble shooting</h3>
+<div id="outline-container-orge803d65" class="outline-3">
+<h3 id="orge803d65"><span class="section-number-3">4.1</span> Trouble shooting</h3>
 <div class="outline-text-3" id="text-4-1">
 <p>
 We got an error saying: {"stem": "<a href="http://www.wikidata.org/entity/">http://www.wikidata.org/entity/</a>",&#x2026;
@@ -508,8 +508,8 @@ submit button.
 </div>
 </div>
 
-<div id="outline-container-org0bbb27f" class="outline-2">
-<h2 id="org0bbb27f"><span class="section-number-2">5</span> Step 4: Check output</h2>
+<div id="outline-container-org540cfdf" class="outline-2">
+<h2 id="org540cfdf"><span class="section-number-2">5</span> Step 4: Check output</h2>
 <div class="outline-text-2" id="text-5">
 <p>
 The current pipeline takes 5.5 hours to complete! Once it completes
@@ -520,8 +520,8 @@ in.
 </div>
 </div>
 
-<div id="outline-container-orged64f4e" class="outline-2">
-<h2 id="orged64f4e"><span class="section-number-2">6</span> Bulk sequence uploader</h2>
+<div id="outline-container-org6c43ab3" class="outline-2">
+<h2 id="org6c43ab3"><span class="section-number-2">6</span> Bulk sequence uploader</h2>
 <div class="outline-text-2" id="text-6">
 <p>
 Above steps require a manual upload of one sequence with metadata.
@@ -584,8 +584,8 @@ submitter:
 </div>
 </div>
 
-<div id="outline-container-orge1c3625" class="outline-3">
-<h3 id="orge1c3625"><span class="section-number-3">6.1</span> Run the uploader (CLI)</h3>
+<div id="outline-container-org99bb8b7" class="outline-3">
+<h3 id="org99bb8b7"><span class="section-number-3">6.1</span> Run the uploader (CLI)</h3>
 <div class="outline-text-3" id="text-6-1">
 <p>
 Installing with pip you should be
@@ -620,20 +620,35 @@ The web interface using this exact same script so it should just work
 </div>
 </div>
 
-<div id="outline-container-org07f04d9" class="outline-3">
-<h3 id="org07f04d9"><span class="section-number-3">6.2</span> Example: uploading bulk GenBank sequences</h3>
+<div id="outline-container-orga88593f" class="outline-3">
+<h3 id="orga88593f"><span class="section-number-3">6.2</span> Example: uploading bulk GenBank sequences</h3>
 <div class="outline-text-3" id="text-6-2">
 <p>
 We also use above script to bulk upload GenBank sequences with a <a href="https://github.com/arvados/bh20-seq-resource/blob/master/scripts/download_genbank_data/from_genbank_to_fasta_and_yaml.py">FASTA
 and YAML</a> extractor specific for GenBank. This means that the steps we
 took above for uploading a GenBank sequence are already automated.
 </p>
+
+<p>
+The steps are: from the
+<code>bh20-seq-resource/scripts/download_genbank_data/</code> directory
+</p>
+
+<div class="org-src-container">
+<pre class="src src-sh">python3 from_genbank_to_fasta_and_yaml.py
+<span style="color: #ffcc80;">dir_fasta_and_yaml</span>=~/bh20-seq-resource/scripts/download_genbank_data/fasta_and_yaml
+ls $<span style="color: #ffcc80;">dir_fasta_and_yaml</span>/*.yaml | <span style="color: #fff59d;">while </span><span style="color: #ff8A65;">read</span> path_code_yaml; <span style="color: #fff59d;">do</span>
+   <span style="color: #ffcc80;">path_code_fasta</span>=${<span style="color: #ffcc80;">path_code_yaml</span>%.*}.fasta
+   bh20-seq-uploader --skip-qc $<span style="color: #ffcc80;">path_code_yaml</span> $<span style="color: #ffcc80;">path_code_fasta</span>
+<span style="color: #fff59d;">done</span>
+</pre>
+</div>
 </div>
 </div>
 </div>
 </div>
 <div id="postamble" class="status">
-<hr><small>Created by <a href="http://thebird.nl/">Pjotr Prins</a> (pjotr.public768 at thebird 'dot' nl) using Emacs org-mode and a healthy dose of Lisp!<br />Modified 2020-08-22 Sat 07:43</small>.
+<hr><small>Created by <a href="http://thebird.nl/">Pjotr Prins</a> (pjotr.public768 at thebird 'dot' nl) using Emacs org-mode and a healthy dose of Lisp!<br />Modified 2020-08-24 Mon 04:31</small>.
 </div>
 </body>
 </html>
diff --git a/doc/blog/using-covid-19-pubseq-part3.org b/doc/blog/using-covid-19-pubseq-part3.org
index b1ab90d..fda7be8 100644
--- a/doc/blog/using-covid-19-pubseq-part3.org
+++ b/doc/blog/using-covid-19-pubseq-part3.org
@@ -236,3 +236,15 @@ The web interface using this exact same script so it should just work
 We also use above script to bulk upload GenBank sequences with a [[https://github.com/arvados/bh20-seq-resource/blob/master/scripts/download_genbank_data/from_genbank_to_fasta_and_yaml.py][FASTA
 and YAML]] extractor specific for GenBank. This means that the steps we
 took above for uploading a GenBank sequence are already automated.
+
+The steps are: from the
+~bh20-seq-resource/scripts/download_genbank_data/~ directory
+
+#+BEGIN_SRC sh
+python3 from_genbank_to_fasta_and_yaml.py
+dir_fasta_and_yaml=~/bh20-seq-resource/scripts/download_genbank_data/fasta_and_yaml
+ls $dir_fasta_and_yaml/*.yaml | while read path_code_yaml; do
+   path_code_fasta=${path_code_yaml%.*}.fasta
+   bh20-seq-uploader --skip-qc $path_code_yaml $path_code_fasta
+done
+#+END_SRC
author	Pjotr Prins	2020-08-24 10:31:24 +0100
committer	Pjotr Prins	2020-08-24 10:31:24 +0100
commit	7fabc4f9427856600e237c6cacd710f49b88d45d (patch)
tree	23fc807559c84ca618ec0145233f76bc6ba64fd5 /doc/blog
parent	b50c13cab729327490a9976727a95af81d629f23 (diff)
download	bh20-seq-resource-7fabc4f9427856600e237c6cacd710f49b88d45d.tar.gz bh20-seq-resource-7fabc4f9427856600e237c6cacd710f49b88d45d.tar.lz bh20-seq-resource-7fabc4f9427856600e237c6cacd710f49b88d45d.zip