aboutsummaryrefslogtreecommitdiff
AgeCommit message (Collapse)Author
2024-10-02README: Mention "work reuse".Arun Isaac
"Work reuse" is the term used by the CWL specification for caching. Use it to keep things familiar. * README.md: Mention "work reuse" in introduction.
2024-10-02command-line-tool: Promote find-requirement to top level.Arun Isaac
* ravanan/command-line-tool.scm (build-command-line-tool-script)[find-requirement]: Promote to top level.
2024-10-02slurm-api: Allow requesting a certain number of CPUs.Arun Isaac
* ravanan/slurm-api.scm (submit-job): Accept cpus argument. * ravanan/command-line-tool.scm (run-command-line-tool): Pass cpus argument. (build-command-line-tool-script)[cores]: New function. Set runtime.cores based on CPU allocation.
2024-10-02propnet: Explain in detail why we never cancel running propagators.Arun Isaac
* ravanan/propnet.scm (poll-propnet): Add detailed comment elaborating on why we never cancel or forget about running propagators.
2024-10-02command-line-tool: Allow per-batch system requirements support.Arun Isaac
* ravanan/command-line-tool.scm (command-line-tool-supported-requirements): New public function. (check-requirements): Accept batch-system and supported-requirements-for-batch-system arguments. * ravanan/command-line-tool.scm (run-command-line-tool): Pass batch-system to build-command-line-tool-script. (build-command-line-tool-script): Accept batch-system argument. Update calls to check-requirements. * ravanan/workflow.scm (%workflow-only-requirements): New variable. (%supported-requirements): Use %workflow-only-requirements. (supported-requirements): New function. (workflow-class->propnet): Accept batch-system argument. Update calls to check-requirements. (workflow-scheduler)[schedule]: Pass batch-system to workflow-class->propnet.
2024-10-02utils: Remove unused (ice-9 match) import.Arun Isaac
* ravanan/utils.scm: Do not import (ice-9 match).
2024-10-02command-line-tool: Remove unused imports.Arun Isaac
* ravanan/command-line-tool.scm: Do not import (srfi srfi-171), (ice-9 popen), (gcrypt base16), (gcrypt hash), (guix describe), (yaml), (ravanan glob) and (ravanan reader).
2024-10-02propnet: Remove unused (srfi srfi-26) import.Arun Isaac
* ravanan/propnet.scm: Do not import (srfi srfi-26).
2024-10-02workflow: Remove unused (ravanan reader) import.Arun Isaac
* ravanan/workflow.scm: Do not import (ravanan reader).
2024-10-02command-line-tool: Remove redundant newline after warning.Arun Isaac
* ravanan/command-line-tool.scm (check-requirements): Remove redundant newline after warning.
2024-10-02command-line-tool: Canonicalize inputs before embedding in G-exp.Arun Isaac
* ravanan/command-line-tool.scm (canonicalize-json): New function. (build-command-line-tool-script): Use canonicalize-json.
2024-10-02workflow: Support subworkflows recursively.Arun Isaac
Implementing subworkflows recursively as sub-propnets is the correct and elegant ouroboros way. Earlier, we were flattening workflows into their constituent CommandLineTools steps. Among many other issues, this approach ran into problems when scattering over subworkflows. * ravanan/workflow.scm: Import assertion-violation from (rnrs base). (<command-line-tool-state>, <workflow-state>): New record types. (workflow-class->propnet): New function. (workflow-class->propagators, workflow->command-line-tool-steps): Delete functions. (workflow-scheduler): Represent state using <command-line-tool-state> and <workflow-state>. Recursively evaluate compound propagators as propnets. (run-workflow): Use capture-output and poll from scheduler.
2024-10-01workflow: Pass a list, not two vectors, to fold.Arun Isaac
* ravanan/workflow.scm (inherit-requirements-and-hints): Pass a list, not two vectors, to fold.
2024-10-01propnet: Pass on scheduler to schedule calls.Arun Isaac
This enables sub-propnets (or compound propagators) to share the same scheduler. * ravanan/propnet.scm (activate-propagator): Accept scheduler as argument instead of the schedule function. (poll-propnet): Pass scheduler as argument instead of its schedule function.
2024-10-01command-line-tool: Move inputs resolution to (ravanan workflow).Arun Isaac
We resolve inputs in run-workflow before doing anything else. We thus avoid bugs due to partially or insufficiently resolved inputs. * ravanan/command-line-tool.scm: Do not import (web uri). (%store-files-directory, %store-data-directory, %store-logs-directory): Export. (location->path, resolve-inputs, intern-file): Move to (ravanan workflow). (build-command-line-tool-script): Do not call resolve-inputs. * ravanan/workflow.scm: Import (web uri) and (ravanan work types). (run-workflow): Call resolve-inputs.
2024-10-01job-state: Add that the job-state records are for batch systems.Arun Isaac
* ravanan/job-state.scm: Add to the commentary that the job-state records are per batch system.
2024-10-01vectors: Add vector-mapn.Arun Isaac
* ravanan/work/vectors.scm: Import (srfi srfi-26). (vector-mapn): New public function.
2024-10-01vectors: Add unindexed variant of vector-fold.Arun Isaac
* ravanan/work/vectors.scm: Import vector-fold from (srfi srfi-43) with prefix. (vector-fold): New public function. (vector-map->list, vector-append-map, vector-append-map->list): Use vector-fold.
2024-10-01job-state: Re-indent.Arun Isaac
* ravanan/job-state.scm (job-state-script): Re-indent.
2024-10-01command-line-tool: Copy path, not location.Arun Isaac
location is an implementation-dependent URI. path is a real filesystem path. * ravanan/command-line-tool.scm (copy-input-files-gexp): Copy path, not location.
2024-10-01vectors: Add vector-member.Arun Isaac
* ravanan/work/vectors.scm (vector-member): New public function.
2024-10-01workflow: Use bound variables instead of calling getters again.Arun Isaac
* ravanan/workflow.scm (workflow-scheduler)[schedule]: Use bound variables name and cwl instead of calling getters scheduler-proc-name and scheduler-proc-cwl again.
2024-10-01command-line-tool: Move scheduler to (ravanan workflow).Arun Isaac
* ravanan/command-line-tool.scm: Do not import (rnrs conditions), (rnrs exceptions) and (ravanan propnet). (command-line-tool-scheduler): Move to (ravanan workflow) as workflow-scheduler. (&job-failure, <scheduler-proc>): Move to (ravanan workflow). (script->store-stdout-file, script->store-stderr-file, capture-command-line-tool-output): Export functions. * ravanan/workflow.scm: Import define-condition-type from (rnrs conditions), (rnrs exceptions), (srfi srfi-9 gnu) and (ravanan job-state). (run-workflow): Call workflow-scheduler instead of command-line-tool-scheduler.
2024-10-01propnet: Allow propagators to update their state.Arun Isaac
This will come in handy later when we implement compound propagators (propagators that are themselves a propagator network). * ravanan/command-line-tool.scm (command-line-tool-scheduler)[poll]: Return two values---the status and the updated state. * ravanan/job-state.scm (job-state-status): Return two values---the status and the updated state. * ravanan/propnet.scm (partition-map): New function. (poll-propnet): Update state of a propagator in flight.
2024-10-01propnet: Present a polling interface to callers.Arun Isaac
(ravanan propnet) now presents a polling interface to the outside world. Instead of computing the entire propagator network in one function call, it needs to be polled repeatedly until it is done. Hence, the polling logic (such as the polling interval) needs to be outside (ravanan propnet). * ravanan/propnet.scm (<scheduler>)[poll-interval]: Delete field. (<propnet-state>): New record type. (schedule-propnet, capture-propnet-output): New public functions. (run-propnet): Rename to poll-propnet; return status and state values instead of fulling computing the propnet. * ravanan/command-line-tool.scm (%job-poll-interval): Move to (ravanan workflow). (command-line-tool-scheduler): Do not initialize poll-interval in scheduler. * ravanan/workflow.scm: Import (srfi srfi-71). (run-workflow): Use the new polling interface to the propnet.
2024-09-23command-line-tool: Update path to staged input files.Arun Isaac
* ravanan/command-line-tool.scm (build-command-line-tool-script)[stage-files, set-staged-path]: New functions. After staging, update path to staged input files.
2024-09-23command-line-tool: Defer access of input values to runtime.Arun Isaac
* ravanan/command-line-tool.scm (build-command): Pass id argument to collect-bindings. Defer access of input values to runtime. [collect-bindings]: Accept id argument. [input+type-tree+binding->command-line-binding]: Rename to id+input+type-tree+binding->command-line-binding. Accept id argument and return G-expressions for command-line-binding values. (command-line-binding->args): Defer access of input values to runtime.
2024-09-23command-line-tool: Separate file copying from canonization.Arun Isaac
* ravanan/command-line-tool.scm (build-command-line-tool-script)[canonicalize-file-value]: Separate out file copying into ... [copy-file-value]: ... new function. [capture-outputs-gexp, capture-secondary-file, path->value]: Call copy-file-value after canonicalize-file-value.
2024-09-23utils: Move json-ref to (ravanan work utils).Arun Isaac
* ravanan/utils.scm: Move to ... * ravanan/work/utils.scm (json-ref): ... here. * ravanan/slurm-api.scm: Import (ravanan work utils) instead of (ravanan utils).
2024-09-20command-line-tool: Add missing argument to canonicalize-file-value.Arun Isaac
* ravanan/command-line-tool.scm (build-command-line-tool-script): Add missing workflow-output-directory argument to canonicalize-file-value.
2024-09-19work/utils: Add alist=?.Arun Isaac
* ravanan/work/utils.scm (alist=?): New public function.
2024-09-19propnet: Describe state machine implementation.Arun Isaac
* ravanan/propnet.scm (run-propnet): Add comment describing state machine implementation.
2024-09-16bin: Explain why we support the SLURM_JWT=token format.Arun Isaac
* bin/ravanan (read-jwt): Explain why we support the SLURM_JWT=token format.
2024-09-13bin: Support SLURM_JWT=token format in JWT files.Arun Isaac
* bin/ravanan: Import (srfi srfi-26) and (ravanan utils). (read-jwt): Support SLURM_JWT=token format.
2024-09-13bin: Refactor JWT reading into separate function.Arun Isaac
* bin/ravanan (read-jwt): New function. (main): Call read-jwt.
2024-09-13utils: Add string-trim-prefix.Arun Isaac
* ravanan/utils.scm (string-trim-prefix): New public function.
2024-09-13command-line-tool: Add nameroot and nameext fields.Arun Isaac
* ravanan/command-line-tool.scm (resolve-inputs)[canonicalize-file-value], (build-command-line-tool-script)[canonicalize-file-value]: Add nameroot and nameext fields.
2024-09-13command-line-tool: Remove unused collect-input-files function.Arun Isaac
* ravanan/command-line-tool.scm (collect-input-files): Delete function.
2024-09-13command-line-tool: Do not embed stdout-filename outside main body.Arun Isaac
stdout-output->value cannot access the inputs variable, and thus will fail when stdout-filename is a parameter reference that needs to be evaluated. To avoid this, pass it in as an argument. * ravanan/command-line-tool.scm (build-command-line-tool-script)[stdout-output->value]: Accept stdout-filename as argument instead of G-expression embedding. Pass stdout-filename as argument to stdout-output->value.
2024-09-13command-line-tool: Create vectors of secondary files for recursion.Arun Isaac
* ravanan/command-line-tool.scm (resolve-inputs)[resolve]: When recursing, create a vector of secondary files.
2024-09-13job-state: Pass on slurm API endpoint and jwt recursively.Arun Isaac
* ravanan/job-state.scm (job-state-status): Pass on slurm API endpoint and jwt recursively.
2024-09-13config: Remove.Arun Isaac
* Makefile (config_file): Delete variable. (sources): Remove $(config_file). (all, %.go): Do not depend on $(config_file). (clean): Do not delete $(config_file). * ravanan/config.scm, ravanan/config.scm.in: Delete files. * ravanan/command-line-tool.scm: Do not import (ravanan config).
2024-09-13command-line-tool: Support secondary files.Arun Isaac
* ravanan/command-line-tool.scm (resolve-inputs)[canonicalize-file-input, match-secondary-file-pattern, check-secondary-files]: New functions. [resolve]: Accept maybe-secondary-files argument. Check that secondary files are provided with File type inputs. (copy-input-files-gexp)[copy-input-files]: New function. Copy secondary files to inputs directory. (build-command-line-tool-script)[capture-outputs-gexp]: Pass maybe-secondary-files argument to other-output->value. [capture-secondary-file]: New function. [path->value]: Call capture-secondary-file. Accept maybe-secondary-files argument. [stdout-output->value]: Pass maybe-secondary-files argument to path->value. [other-output->value]: Accept maybe-secondary-files argument and pass it on to path->value. Add TODO note about updating paths of staged files in inputs objects.
2024-09-13vectors: Add vector-filter-map.Arun Isaac
* ravanan/work/vectors.scm (vector-filter-map): New public function.
2024-09-13work/command-line-tool: Comment on match-type clauses.Arun Isaac
* ravanan/work/command-line-tool.scm (match-type): Comment on cond ladder clauses.
2024-09-13work/command-line-tool: Comment on formal-parameter-type clauses.Arun Isaac
* ravanan/work/command-line-tool.scm (formal-parameter-type): Comment on cond ladder clauses.
2024-09-13vectors: Rename module to (ravanan work vectors).Arun Isaac
* ravanan/vectors.scm: Move to ravanan/work/vectors.scm. * ravanan/command-line-tool.scm, ravanan/job-state.scm, ravanan/reader.scm, ravanan/workflow.scm: Import (ravanan work vectors) instead of (ravanan vectors).
2024-09-13monads: Rename module to (ravanan work monads).Arun Isaac
* ravanan/monads.scm: Move to ravanan/work/monads.scm. * ravanan/command-line-tool.scm, ravanan/propnet.scm, ravanan/reader.scm, ravanan/workflow.scm: Import (ravanan work vectors) instead of (ravanan vectors).
2024-09-13utils: Move assoc-ref* to (ravanan work utils).Arun Isaac
* ravanan/utils.scm: Do not import (rnrs base). Move assoc-ref* to ... * ravanan/work/utils.scm: ... here. Import assertion-violation from (rnrs base). * ravanan/command-line-tool.scm, ravanan/workflow.scm: Do not import (ravanan utils).
2024-09-13command-line-tool: Do not re-intern already interned files.Arun Isaac
* ravanan/command-line-tool.scm (intern-file): Intern file only if it is not already in the store.
generated by cgit v1.2.3 (git 2.45.2) at 2024-10-16 21:24:33 +0000