aboutsummaryrefslogtreecommitdiff
AgeCommit message (Expand)Author
2024-10-02command-line-tool: Make runtime.cores a number.•••* ravanan/command-line-tool.scm (build-command-line-tool-script)[cores]: Return a number on the slurm-api batch system. Arun Isaac
2024-10-02command-line-tool: Support coresMin ResourceRequirement.•••* ravanan/command-line-tool.scm: Import (ravanan reader). (%command-line-tool-supported-requirements): Add ResouceRequirement. (command-line-tool-supported-requirements): Remove ResourceRequirement for the single-machine batch system. * ravanan/command-line-tool.scm (run-command-line-tool): Request the number of CPUs specified in coresMin of the ResourceRequirement. Arun Isaac
2024-10-02reader: Support coercion to number.•••* ravanan/reader.scm (coerce-type): Support coercion to number. Arun Isaac
2024-10-02workflow: Move coerce-type to (ravanan reader).•••* ravanan/workflow.scm (coerce-type): Move to ... * ravanan/reader.scm (coerce-type): ... here. Arun Isaac
2024-10-02README: Mention "work reuse".•••"Work reuse" is the term used by the CWL specification for caching. Use it to keep things familiar. * README.md: Mention "work reuse" in introduction. Arun Isaac
2024-10-02command-line-tool: Promote find-requirement to top level.•••* ravanan/command-line-tool.scm (build-command-line-tool-script)[find-requirement]: Promote to top level. Arun Isaac
2024-10-02slurm-api: Allow requesting a certain number of CPUs.•••* ravanan/slurm-api.scm (submit-job): Accept cpus argument. * ravanan/command-line-tool.scm (run-command-line-tool): Pass cpus argument. (build-command-line-tool-script)[cores]: New function. Set runtime.cores based on CPU allocation. Arun Isaac
2024-10-02propnet: Explain in detail why we never cancel running propagators.•••* ravanan/propnet.scm (poll-propnet): Add detailed comment elaborating on why we never cancel or forget about running propagators. Arun Isaac
2024-10-02command-line-tool: Allow per-batch system requirements support.•••* ravanan/command-line-tool.scm (command-line-tool-supported-requirements): New public function. (check-requirements): Accept batch-system and supported-requirements-for-batch-system arguments. * ravanan/command-line-tool.scm (run-command-line-tool): Pass batch-system to build-command-line-tool-script. (build-command-line-tool-script): Accept batch-system argument. Update calls to check-requirements. * ravanan/workflow.scm (%workflow-only-requirements): New variable. (%supported-requirements): Use %workflow-only-requirements. (supported-requirements): New function. (workflow-class->propnet): Accept batch-system argument. Update calls to check-requirements. (workflow-scheduler)[schedule]: Pass batch-system to workflow-class->propnet. Arun Isaac
2024-10-02utils: Remove unused (ice-9 match) import.•••* ravanan/utils.scm: Do not import (ice-9 match). Arun Isaac
2024-10-02command-line-tool: Remove unused imports.•••* ravanan/command-line-tool.scm: Do not import (srfi srfi-171), (ice-9 popen), (gcrypt base16), (gcrypt hash), (guix describe), (yaml), (ravanan glob) and (ravanan reader). Arun Isaac
2024-10-02propnet: Remove unused (srfi srfi-26) import.•••* ravanan/propnet.scm: Do not import (srfi srfi-26). Arun Isaac
2024-10-02workflow: Remove unused (ravanan reader) import.•••* ravanan/workflow.scm: Do not import (ravanan reader). Arun Isaac
2024-10-02command-line-tool: Remove redundant newline after warning.•••* ravanan/command-line-tool.scm (check-requirements): Remove redundant newline after warning. Arun Isaac
2024-10-02command-line-tool: Canonicalize inputs before embedding in G-exp.•••* ravanan/command-line-tool.scm (canonicalize-json): New function. (build-command-line-tool-script): Use canonicalize-json. Arun Isaac
2024-10-02workflow: Support subworkflows recursively.•••Implementing subworkflows recursively as sub-propnets is the correct and elegant ouroboros way. Earlier, we were flattening workflows into their constituent CommandLineTools steps. Among many other issues, this approach ran into problems when scattering over subworkflows. * ravanan/workflow.scm: Import assertion-violation from (rnrs base). (<command-line-tool-state>, <workflow-state>): New record types. (workflow-class->propnet): New function. (workflow-class->propagators, workflow->command-line-tool-steps): Delete functions. (workflow-scheduler): Represent state using <command-line-tool-state> and <workflow-state>. Recursively evaluate compound propagators as propnets. (run-workflow): Use capture-output and poll from scheduler. Arun Isaac
2024-10-01workflow: Pass a list, not two vectors, to fold.•••* ravanan/workflow.scm (inherit-requirements-and-hints): Pass a list, not two vectors, to fold. Arun Isaac
2024-10-01propnet: Pass on scheduler to schedule calls.•••This enables sub-propnets (or compound propagators) to share the same scheduler. * ravanan/propnet.scm (activate-propagator): Accept scheduler as argument instead of the schedule function. (poll-propnet): Pass scheduler as argument instead of its schedule function. Arun Isaac
2024-10-01command-line-tool: Move inputs resolution to (ravanan workflow).•••We resolve inputs in run-workflow before doing anything else. We thus avoid bugs due to partially or insufficiently resolved inputs. * ravanan/command-line-tool.scm: Do not import (web uri). (%store-files-directory, %store-data-directory, %store-logs-directory): Export. (location->path, resolve-inputs, intern-file): Move to (ravanan workflow). (build-command-line-tool-script): Do not call resolve-inputs. * ravanan/workflow.scm: Import (web uri) and (ravanan work types). (run-workflow): Call resolve-inputs. Arun Isaac
2024-10-01job-state: Add that the job-state records are for batch systems.•••* ravanan/job-state.scm: Add to the commentary that the job-state records are per batch system. Arun Isaac
2024-10-01vectors: Add vector-mapn.•••* ravanan/work/vectors.scm: Import (srfi srfi-26). (vector-mapn): New public function. Arun Isaac
2024-10-01vectors: Add unindexed variant of vector-fold.•••* ravanan/work/vectors.scm: Import vector-fold from (srfi srfi-43) with prefix. (vector-fold): New public function. (vector-map->list, vector-append-map, vector-append-map->list): Use vector-fold. Arun Isaac
2024-10-01job-state: Re-indent.•••* ravanan/job-state.scm (job-state-script): Re-indent. Arun Isaac
2024-10-01command-line-tool: Copy path, not location.•••location is an implementation-dependent URI. path is a real filesystem path. * ravanan/command-line-tool.scm (copy-input-files-gexp): Copy path, not location. Arun Isaac
2024-10-01vectors: Add vector-member.•••* ravanan/work/vectors.scm (vector-member): New public function. Arun Isaac
2024-10-01workflow: Use bound variables instead of calling getters again.•••* ravanan/workflow.scm (workflow-scheduler)[schedule]: Use bound variables name and cwl instead of calling getters scheduler-proc-name and scheduler-proc-cwl again. Arun Isaac
2024-10-01command-line-tool: Move scheduler to (ravanan workflow).•••* ravanan/command-line-tool.scm: Do not import (rnrs conditions), (rnrs exceptions) and (ravanan propnet). (command-line-tool-scheduler): Move to (ravanan workflow) as workflow-scheduler. (&job-failure, <scheduler-proc>): Move to (ravanan workflow). (script->store-stdout-file, script->store-stderr-file, capture-command-line-tool-output): Export functions. * ravanan/workflow.scm: Import define-condition-type from (rnrs conditions), (rnrs exceptions), (srfi srfi-9 gnu) and (ravanan job-state). (run-workflow): Call workflow-scheduler instead of command-line-tool-scheduler. Arun Isaac
2024-10-01propnet: Allow propagators to update their state.•••This will come in handy later when we implement compound propagators (propagators that are themselves a propagator network). * ravanan/command-line-tool.scm (command-line-tool-scheduler)[poll]: Return two values---the status and the updated state. * ravanan/job-state.scm (job-state-status): Return two values---the status and the updated state. * ravanan/propnet.scm (partition-map): New function. (poll-propnet): Update state of a propagator in flight. Arun Isaac
2024-10-01propnet: Present a polling interface to callers.•••(ravanan propnet) now presents a polling interface to the outside world. Instead of computing the entire propagator network in one function call, it needs to be polled repeatedly until it is done. Hence, the polling logic (such as the polling interval) needs to be outside (ravanan propnet). * ravanan/propnet.scm (<scheduler>)[poll-interval]: Delete field. (<propnet-state>): New record type. (schedule-propnet, capture-propnet-output): New public functions. (run-propnet): Rename to poll-propnet; return status and state values instead of fulling computing the propnet. * ravanan/command-line-tool.scm (%job-poll-interval): Move to (ravanan workflow). (command-line-tool-scheduler): Do not initialize poll-interval in scheduler. * ravanan/workflow.scm: Import (srfi srfi-71). (run-workflow): Use the new polling interface to the propnet. Arun Isaac
2024-09-23command-line-tool: Update path to staged input files.•••* ravanan/command-line-tool.scm (build-command-line-tool-script)[stage-files, set-staged-path]: New functions. After staging, update path to staged input files. Arun Isaac
2024-09-23command-line-tool: Defer access of input values to runtime.•••* ravanan/command-line-tool.scm (build-command): Pass id argument to collect-bindings. Defer access of input values to runtime. [collect-bindings]: Accept id argument. [input+type-tree+binding->command-line-binding]: Rename to id+input+type-tree+binding->command-line-binding. Accept id argument and return G-expressions for command-line-binding values. (command-line-binding->args): Defer access of input values to runtime. Arun Isaac
2024-09-23command-line-tool: Separate file copying from canonization.•••* ravanan/command-line-tool.scm (build-command-line-tool-script)[canonicalize-file-value]: Separate out file copying into ... [copy-file-value]: ... new function. [capture-outputs-gexp, capture-secondary-file, path->value]: Call copy-file-value after canonicalize-file-value. Arun Isaac
2024-09-23utils: Move json-ref to (ravanan work utils).•••* ravanan/utils.scm: Move to ... * ravanan/work/utils.scm (json-ref): ... here. * ravanan/slurm-api.scm: Import (ravanan work utils) instead of (ravanan utils). Arun Isaac
2024-09-20command-line-tool: Add missing argument to canonicalize-file-value.•••* ravanan/command-line-tool.scm (build-command-line-tool-script): Add missing workflow-output-directory argument to canonicalize-file-value. Arun Isaac
2024-09-19work/utils: Add alist=?.•••* ravanan/work/utils.scm (alist=?): New public function. Arun Isaac
2024-09-19propnet: Describe state machine implementation.•••* ravanan/propnet.scm (run-propnet): Add comment describing state machine implementation. Arun Isaac
2024-09-16bin: Explain why we support the SLURM_JWT=token format.•••* bin/ravanan (read-jwt): Explain why we support the SLURM_JWT=token format. Arun Isaac
2024-09-13bin: Support SLURM_JWT=token format in JWT files.•••* bin/ravanan: Import (srfi srfi-26) and (ravanan utils). (read-jwt): Support SLURM_JWT=token format. Arun Isaac
2024-09-13bin: Refactor JWT reading into separate function.•••* bin/ravanan (read-jwt): New function. (main): Call read-jwt. Arun Isaac
2024-09-13utils: Add string-trim-prefix.•••* ravanan/utils.scm (string-trim-prefix): New public function. Arun Isaac
2024-09-13command-line-tool: Add nameroot and nameext fields.•••* ravanan/command-line-tool.scm (resolve-inputs)[canonicalize-file-value], (build-command-line-tool-script)[canonicalize-file-value]: Add nameroot and nameext fields. Arun Isaac
2024-09-13command-line-tool: Remove unused collect-input-files function.•••* ravanan/command-line-tool.scm (collect-input-files): Delete function. Arun Isaac
2024-09-13command-line-tool: Do not embed stdout-filename outside main body.•••stdout-output->value cannot access the inputs variable, and thus will fail when stdout-filename is a parameter reference that needs to be evaluated. To avoid this, pass it in as an argument. * ravanan/command-line-tool.scm (build-command-line-tool-script)[stdout-output->value]: Accept stdout-filename as argument instead of G-expression embedding. Pass stdout-filename as argument to stdout-output->value. Arun Isaac
2024-09-13command-line-tool: Create vectors of secondary files for recursion.•••* ravanan/command-line-tool.scm (resolve-inputs)[resolve]: When recursing, create a vector of secondary files. Arun Isaac
2024-09-13job-state: Pass on slurm API endpoint and jwt recursively.•••* ravanan/job-state.scm (job-state-status): Pass on slurm API endpoint and jwt recursively. Arun Isaac
2024-09-13config: Remove.•••* Makefile (config_file): Delete variable. (sources): Remove $(config_file). (all, %.go): Do not depend on $(config_file). (clean): Do not delete $(config_file). * ravanan/config.scm, ravanan/config.scm.in: Delete files. * ravanan/command-line-tool.scm: Do not import (ravanan config). Arun Isaac
2024-09-13command-line-tool: Support secondary files.•••* ravanan/command-line-tool.scm (resolve-inputs)[canonicalize-file-input, match-secondary-file-pattern, check-secondary-files]: New functions. [resolve]: Accept maybe-secondary-files argument. Check that secondary files are provided with File type inputs. (copy-input-files-gexp)[copy-input-files]: New function. Copy secondary files to inputs directory. (build-command-line-tool-script)[capture-outputs-gexp]: Pass maybe-secondary-files argument to other-output->value. [capture-secondary-file]: New function. [path->value]: Call capture-secondary-file. Accept maybe-secondary-files argument. [stdout-output->value]: Pass maybe-secondary-files argument to path->value. [other-output->value]: Accept maybe-secondary-files argument and pass it on to path->value. Add TODO note about updating paths of staged files in inputs objects. Arun Isaac
2024-09-13vectors: Add vector-filter-map.•••* ravanan/work/vectors.scm (vector-filter-map): New public function. Arun Isaac
2024-09-13work/command-line-tool: Comment on match-type clauses.•••* ravanan/work/command-line-tool.scm (match-type): Comment on cond ladder clauses. Arun Isaac
2024-09-13work/command-line-tool: Comment on formal-parameter-type clauses.•••* ravanan/work/command-line-tool.scm (formal-parameter-type): Comment on cond ladder clauses. Arun Isaac