aboutsummaryrefslogtreecommitdiff
AgeCommit message (Collapse)Author
2024-10-04command-line-tool: Move canonicalize-json to (ravanan work utils).Arun Isaac
* ravanan/command-line-tool.scm (canonicalize-json): Move to (ravanan work utils). Move comment about G-expressions to build-command-line-tool-script. * ravanan/work/utils.scm (ravanan): Import (ravanan work vectors).
2024-10-04command-line-tool: Merge local and remote coerce-expression.Arun Isaac
coerce-expression and coerce-expression-local share most of their code. So, it makes sense to merge them. * ravanan/command-line-tool.scm (coerce-expression): Add optional context argument. (coerce-expression-local): Delete function. (run-command-line-tool): Call coerce-expression instead of coerce-expression-local.
2024-10-02command-line-tool: Embed int value into G-expression.Arun Isaac
* ravanan/command-line-tool.scm (command-line-binding->args): Embed int value into G-expression instead of evaluating it immediately.
2024-10-02command-line-tool: Coerce coresMin expressions.Arun Isaac
* ravanan/command-line-tool.scm (run-command-line-tool): Coerce coresMin expressions.
2024-10-02command-line-tool: Re-indent coerce-expression.Arun Isaac
* ravanan/command-line-tool.scm (coerce-expression): Re-indent.
2024-10-02command-line-tool: Add local version of coerce-expression.Arun Isaac
* ravanan/command-line-tool.scm (coerce-expression-local): New function.
2024-10-02reader: Handle coercion of number to number.Arun Isaac
* ravanan/reader.scm (coerce-type): Handle coercion of number to number. * tests/reader.scm: New file.
2024-10-02config: Remove generated source file.Arun Isaac
config.scm is generated from config.scm.in. Therefore, config.scm must not be committed. * ravanan/config.scm: Delete file.
2024-10-02Revert "config: Remove."Arun Isaac
This reverts commit d611d990b297b3d573d7d5af83e9376744d51dd5. We need to bring back a local node to evaluate javascript expressions on the head node.
2024-10-02workflow: Add default inputs before scheduling.Arun Isaac
We add default input values and resolve them before scheduling. Now that we do that, we no longer have to resolve inputs or check for required inputs in run-workflow. * ravanan/workflow.scm (add-defaults): New function. (workflow-scheduler)[schedule]: Add default inputs and resolve them before scheduling. (run-workflow): Do not resolve inputs or check for required inputs.
2024-10-02command-line-tool: Make runtime.cores a number.Arun Isaac
* ravanan/command-line-tool.scm (build-command-line-tool-script)[cores]: Return a number on the slurm-api batch system.
2024-10-02command-line-tool: Support coresMin ResourceRequirement.Arun Isaac
* ravanan/command-line-tool.scm: Import (ravanan reader). (%command-line-tool-supported-requirements): Add ResouceRequirement. (command-line-tool-supported-requirements): Remove ResourceRequirement for the single-machine batch system. * ravanan/command-line-tool.scm (run-command-line-tool): Request the number of CPUs specified in coresMin of the ResourceRequirement.
2024-10-02reader: Support coercion to number.Arun Isaac
* ravanan/reader.scm (coerce-type): Support coercion to number.
2024-10-02workflow: Move coerce-type to (ravanan reader).Arun Isaac
* ravanan/workflow.scm (coerce-type): Move to ... * ravanan/reader.scm (coerce-type): ... here.
2024-10-02README: Mention "work reuse".Arun Isaac
"Work reuse" is the term used by the CWL specification for caching. Use it to keep things familiar. * README.md: Mention "work reuse" in introduction.
2024-10-02command-line-tool: Promote find-requirement to top level.Arun Isaac
* ravanan/command-line-tool.scm (build-command-line-tool-script)[find-requirement]: Promote to top level.
2024-10-02slurm-api: Allow requesting a certain number of CPUs.Arun Isaac
* ravanan/slurm-api.scm (submit-job): Accept cpus argument. * ravanan/command-line-tool.scm (run-command-line-tool): Pass cpus argument. (build-command-line-tool-script)[cores]: New function. Set runtime.cores based on CPU allocation.
2024-10-02propnet: Explain in detail why we never cancel running propagators.Arun Isaac
* ravanan/propnet.scm (poll-propnet): Add detailed comment elaborating on why we never cancel or forget about running propagators.
2024-10-02command-line-tool: Allow per-batch system requirements support.Arun Isaac
* ravanan/command-line-tool.scm (command-line-tool-supported-requirements): New public function. (check-requirements): Accept batch-system and supported-requirements-for-batch-system arguments. * ravanan/command-line-tool.scm (run-command-line-tool): Pass batch-system to build-command-line-tool-script. (build-command-line-tool-script): Accept batch-system argument. Update calls to check-requirements. * ravanan/workflow.scm (%workflow-only-requirements): New variable. (%supported-requirements): Use %workflow-only-requirements. (supported-requirements): New function. (workflow-class->propnet): Accept batch-system argument. Update calls to check-requirements. (workflow-scheduler)[schedule]: Pass batch-system to workflow-class->propnet.
2024-10-02utils: Remove unused (ice-9 match) import.Arun Isaac
* ravanan/utils.scm: Do not import (ice-9 match).
2024-10-02command-line-tool: Remove unused imports.Arun Isaac
* ravanan/command-line-tool.scm: Do not import (srfi srfi-171), (ice-9 popen), (gcrypt base16), (gcrypt hash), (guix describe), (yaml), (ravanan glob) and (ravanan reader).
2024-10-02propnet: Remove unused (srfi srfi-26) import.Arun Isaac
* ravanan/propnet.scm: Do not import (srfi srfi-26).
2024-10-02workflow: Remove unused (ravanan reader) import.Arun Isaac
* ravanan/workflow.scm: Do not import (ravanan reader).
2024-10-02command-line-tool: Remove redundant newline after warning.Arun Isaac
* ravanan/command-line-tool.scm (check-requirements): Remove redundant newline after warning.
2024-10-02command-line-tool: Canonicalize inputs before embedding in G-exp.Arun Isaac
* ravanan/command-line-tool.scm (canonicalize-json): New function. (build-command-line-tool-script): Use canonicalize-json.
2024-10-02workflow: Support subworkflows recursively.Arun Isaac
Implementing subworkflows recursively as sub-propnets is the correct and elegant ouroboros way. Earlier, we were flattening workflows into their constituent CommandLineTools steps. Among many other issues, this approach ran into problems when scattering over subworkflows. * ravanan/workflow.scm: Import assertion-violation from (rnrs base). (<command-line-tool-state>, <workflow-state>): New record types. (workflow-class->propnet): New function. (workflow-class->propagators, workflow->command-line-tool-steps): Delete functions. (workflow-scheduler): Represent state using <command-line-tool-state> and <workflow-state>. Recursively evaluate compound propagators as propnets. (run-workflow): Use capture-output and poll from scheduler.
2024-10-01workflow: Pass a list, not two vectors, to fold.Arun Isaac
* ravanan/workflow.scm (inherit-requirements-and-hints): Pass a list, not two vectors, to fold.
2024-10-01propnet: Pass on scheduler to schedule calls.Arun Isaac
This enables sub-propnets (or compound propagators) to share the same scheduler. * ravanan/propnet.scm (activate-propagator): Accept scheduler as argument instead of the schedule function. (poll-propnet): Pass scheduler as argument instead of its schedule function.
2024-10-01command-line-tool: Move inputs resolution to (ravanan workflow).Arun Isaac
We resolve inputs in run-workflow before doing anything else. We thus avoid bugs due to partially or insufficiently resolved inputs. * ravanan/command-line-tool.scm: Do not import (web uri). (%store-files-directory, %store-data-directory, %store-logs-directory): Export. (location->path, resolve-inputs, intern-file): Move to (ravanan workflow). (build-command-line-tool-script): Do not call resolve-inputs. * ravanan/workflow.scm: Import (web uri) and (ravanan work types). (run-workflow): Call resolve-inputs.
2024-10-01job-state: Add that the job-state records are for batch systems.Arun Isaac
* ravanan/job-state.scm: Add to the commentary that the job-state records are per batch system.
2024-10-01vectors: Add vector-mapn.Arun Isaac
* ravanan/work/vectors.scm: Import (srfi srfi-26). (vector-mapn): New public function.
2024-10-01vectors: Add unindexed variant of vector-fold.Arun Isaac
* ravanan/work/vectors.scm: Import vector-fold from (srfi srfi-43) with prefix. (vector-fold): New public function. (vector-map->list, vector-append-map, vector-append-map->list): Use vector-fold.
2024-10-01job-state: Re-indent.Arun Isaac
* ravanan/job-state.scm (job-state-script): Re-indent.
2024-10-01command-line-tool: Copy path, not location.Arun Isaac
location is an implementation-dependent URI. path is a real filesystem path. * ravanan/command-line-tool.scm (copy-input-files-gexp): Copy path, not location.
2024-10-01vectors: Add vector-member.Arun Isaac
* ravanan/work/vectors.scm (vector-member): New public function.
2024-10-01workflow: Use bound variables instead of calling getters again.Arun Isaac
* ravanan/workflow.scm (workflow-scheduler)[schedule]: Use bound variables name and cwl instead of calling getters scheduler-proc-name and scheduler-proc-cwl again.
2024-10-01command-line-tool: Move scheduler to (ravanan workflow).Arun Isaac
* ravanan/command-line-tool.scm: Do not import (rnrs conditions), (rnrs exceptions) and (ravanan propnet). (command-line-tool-scheduler): Move to (ravanan workflow) as workflow-scheduler. (&job-failure, <scheduler-proc>): Move to (ravanan workflow). (script->store-stdout-file, script->store-stderr-file, capture-command-line-tool-output): Export functions. * ravanan/workflow.scm: Import define-condition-type from (rnrs conditions), (rnrs exceptions), (srfi srfi-9 gnu) and (ravanan job-state). (run-workflow): Call workflow-scheduler instead of command-line-tool-scheduler.
2024-10-01propnet: Allow propagators to update their state.Arun Isaac
This will come in handy later when we implement compound propagators (propagators that are themselves a propagator network). * ravanan/command-line-tool.scm (command-line-tool-scheduler)[poll]: Return two values---the status and the updated state. * ravanan/job-state.scm (job-state-status): Return two values---the status and the updated state. * ravanan/propnet.scm (partition-map): New function. (poll-propnet): Update state of a propagator in flight.
2024-10-01propnet: Present a polling interface to callers.Arun Isaac
(ravanan propnet) now presents a polling interface to the outside world. Instead of computing the entire propagator network in one function call, it needs to be polled repeatedly until it is done. Hence, the polling logic (such as the polling interval) needs to be outside (ravanan propnet). * ravanan/propnet.scm (<scheduler>)[poll-interval]: Delete field. (<propnet-state>): New record type. (schedule-propnet, capture-propnet-output): New public functions. (run-propnet): Rename to poll-propnet; return status and state values instead of fulling computing the propnet. * ravanan/command-line-tool.scm (%job-poll-interval): Move to (ravanan workflow). (command-line-tool-scheduler): Do not initialize poll-interval in scheduler. * ravanan/workflow.scm: Import (srfi srfi-71). (run-workflow): Use the new polling interface to the propnet.
2024-09-23command-line-tool: Update path to staged input files.Arun Isaac
* ravanan/command-line-tool.scm (build-command-line-tool-script)[stage-files, set-staged-path]: New functions. After staging, update path to staged input files.
2024-09-23command-line-tool: Defer access of input values to runtime.Arun Isaac
* ravanan/command-line-tool.scm (build-command): Pass id argument to collect-bindings. Defer access of input values to runtime. [collect-bindings]: Accept id argument. [input+type-tree+binding->command-line-binding]: Rename to id+input+type-tree+binding->command-line-binding. Accept id argument and return G-expressions for command-line-binding values. (command-line-binding->args): Defer access of input values to runtime.
2024-09-23command-line-tool: Separate file copying from canonization.Arun Isaac
* ravanan/command-line-tool.scm (build-command-line-tool-script)[canonicalize-file-value]: Separate out file copying into ... [copy-file-value]: ... new function. [capture-outputs-gexp, capture-secondary-file, path->value]: Call copy-file-value after canonicalize-file-value.
2024-09-23utils: Move json-ref to (ravanan work utils).Arun Isaac
* ravanan/utils.scm: Move to ... * ravanan/work/utils.scm (json-ref): ... here. * ravanan/slurm-api.scm: Import (ravanan work utils) instead of (ravanan utils).
2024-09-20command-line-tool: Add missing argument to canonicalize-file-value.Arun Isaac
* ravanan/command-line-tool.scm (build-command-line-tool-script): Add missing workflow-output-directory argument to canonicalize-file-value.
2024-09-19work/utils: Add alist=?.Arun Isaac
* ravanan/work/utils.scm (alist=?): New public function.
2024-09-19propnet: Describe state machine implementation.Arun Isaac
* ravanan/propnet.scm (run-propnet): Add comment describing state machine implementation.
2024-09-16bin: Explain why we support the SLURM_JWT=token format.Arun Isaac
* bin/ravanan (read-jwt): Explain why we support the SLURM_JWT=token format.
2024-09-13bin: Support SLURM_JWT=token format in JWT files.Arun Isaac
* bin/ravanan: Import (srfi srfi-26) and (ravanan utils). (read-jwt): Support SLURM_JWT=token format.
2024-09-13bin: Refactor JWT reading into separate function.Arun Isaac
* bin/ravanan (read-jwt): New function. (main): Call read-jwt.
2024-09-13utils: Add string-trim-prefix.Arun Isaac
* ravanan/utils.scm (string-trim-prefix): New public function.
generated by cgit v1.2.3 (git 2.45.2) at 2024-12-23 05:52:06 +0000