aboutsummaryrefslogtreecommitdiff
AgeCommit message (Collapse)Author
2024-10-08javascript: Reference inputs, self or runtime symbols; not strings.Arun Isaac
* ravanan/javascript.scm (evaluate-parameter-reference-1): Reference inputs, self or runtime symbols; not strings.
2024-10-08tests: Add tests for javascript G-expressions.Arun Isaac
* tests/javascript.scm: Import (guix gexp) and (ice-9 match). (gexp->sexp-rec): New function. ("evaluate parameter reference (without context)", "evaluate parameter reference with string interpolation (without context)", "evaluate parameter reference with string interpolation of JSON trees (without context)", "evaluate parameter reference with node (without context)", "evaluate parameter reference with string interpolation using node (without context)", "evaluate parameter reference with string interpolation of JSON trees using node (without context)"): New tests.
2024-10-07command-line-tool: Implement GuixManifestRequirement.Arun Isaac
* ravanan/command-line-tool.scm (%command-line-tool-supported-requirements): Add GuixManifestRequirement. (build-command-line-tool-script): Implement GuixManifestRequirement.
2024-10-07bin: Pass manifest file paths, not manifests to other functions.Arun Isaac
* bin/ravanan: Move to (ravanan command-line-tool). Add comment about declarative modules. (main): Pass manifest file path to run-workflow. * ravanan/command-line-tool.scm (run-command-line-tool, build-command-line-tool-script): Accept manifest file path instead of manifest. * ravanan/workflow.scm (workflow-scheduler, run-workflow): Accept manifest file path instead of manifest.
2024-10-07bin: Load manifests without their definitions leaking out.Arun Isaac
* bin/ravanan: Do not import (gnu packages) and (guix profiles). (load-manifest): New function. (main): Use load-manifest.
2024-10-04javascript: Unify simple parameter reference and node fallback.Arun Isaac
* ravanan/command-line-tool.scm: Do not import (ravanan config), and node from (gnu packages node). (%worker-node): Move to (ravanan javascript). (interpolate-parameter-references): Delete function. (coerce-expression): Always call evaluate-parameter-reference from (ravanan javascript). * ravanan/work/command-line-tool.scm (evaluate-parameter-reference): Delete function. * ravanan/javascript.scm: Mention fallback to node in module commentary. Import (ravanan config), (ravanan work command-line-tool), and node from (gnu packages node). (evaluate-simple-parameter-reference): Rename to evaluate-parameter-reference. (evaluate-parameter-reference-1, evaluate-parameter-reference): Add expression-lib argument. Return regular values, not maybe-monadic values. (evaluate-using-node): New function. * tests/javascript.scm ("evaluate parameter reference", "evaluate parameter reference with string interpolation", "evaluate parameter reference with string interpolation of JSON trees"): Update tests. ("evaluate parameter reference with node", "evaluate parameter reference with string interpolation using node", "evaluate parameter reference with string interpolation of JSON trees using node"): New tests.
2024-10-04javascript: Remove "alternating" from tokenizing docstring.Arun Isaac
tokenize-parameter-references need not return an "alternating" sequence of tokens. There can be consecutive parameter reference tokens or literal strings. * ravanan/javascript.scm (tokenize-parameter-references): Remove "alternating" from docstring.
2024-10-04monads: Use write, not display, to print Just values.Arun Isaac
* ravanan/work/monads.scm (<maybe>): Use write, not display, to print Just values.
2024-10-04javascript: Implement parameter reference javascript subset language.Arun Isaac
* ravanan/javascript.scm, tests/javascript.scm: New files. * ravanan/command-line-tool.scm: Import (ravanan javascript). (coerce-expression): Use evaluate-simple-parameter-reference. (interpolate-parameter-references)[tokenize]: Move to (ravanan javascript) as tokenize-parameter-references.
2024-10-04command-line-tool: Move canonicalize-json to (ravanan work utils).Arun Isaac
* ravanan/command-line-tool.scm (canonicalize-json): Move to (ravanan work utils). Move comment about G-expressions to build-command-line-tool-script. * ravanan/work/utils.scm (ravanan): Import (ravanan work vectors).
2024-10-04command-line-tool: Merge local and remote coerce-expression.Arun Isaac
coerce-expression and coerce-expression-local share most of their code. So, it makes sense to merge them. * ravanan/command-line-tool.scm (coerce-expression): Add optional context argument. (coerce-expression-local): Delete function. (run-command-line-tool): Call coerce-expression instead of coerce-expression-local.
2024-10-02command-line-tool: Embed int value into G-expression.Arun Isaac
* ravanan/command-line-tool.scm (command-line-binding->args): Embed int value into G-expression instead of evaluating it immediately.
2024-10-02command-line-tool: Coerce coresMin expressions.Arun Isaac
* ravanan/command-line-tool.scm (run-command-line-tool): Coerce coresMin expressions.
2024-10-02command-line-tool: Re-indent coerce-expression.Arun Isaac
* ravanan/command-line-tool.scm (coerce-expression): Re-indent.
2024-10-02command-line-tool: Add local version of coerce-expression.Arun Isaac
* ravanan/command-line-tool.scm (coerce-expression-local): New function.
2024-10-02reader: Handle coercion of number to number.Arun Isaac
* ravanan/reader.scm (coerce-type): Handle coercion of number to number. * tests/reader.scm: New file.
2024-10-02config: Remove generated source file.Arun Isaac
config.scm is generated from config.scm.in. Therefore, config.scm must not be committed. * ravanan/config.scm: Delete file.
2024-10-02Revert "config: Remove."Arun Isaac
This reverts commit d611d990b297b3d573d7d5af83e9376744d51dd5. We need to bring back a local node to evaluate javascript expressions on the head node.
2024-10-02workflow: Add default inputs before scheduling.Arun Isaac
We add default input values and resolve them before scheduling. Now that we do that, we no longer have to resolve inputs or check for required inputs in run-workflow. * ravanan/workflow.scm (add-defaults): New function. (workflow-scheduler)[schedule]: Add default inputs and resolve them before scheduling. (run-workflow): Do not resolve inputs or check for required inputs.
2024-10-02command-line-tool: Make runtime.cores a number.Arun Isaac
* ravanan/command-line-tool.scm (build-command-line-tool-script)[cores]: Return a number on the slurm-api batch system.
2024-10-02command-line-tool: Support coresMin ResourceRequirement.Arun Isaac
* ravanan/command-line-tool.scm: Import (ravanan reader). (%command-line-tool-supported-requirements): Add ResouceRequirement. (command-line-tool-supported-requirements): Remove ResourceRequirement for the single-machine batch system. * ravanan/command-line-tool.scm (run-command-line-tool): Request the number of CPUs specified in coresMin of the ResourceRequirement.
2024-10-02reader: Support coercion to number.Arun Isaac
* ravanan/reader.scm (coerce-type): Support coercion to number.
2024-10-02workflow: Move coerce-type to (ravanan reader).Arun Isaac
* ravanan/workflow.scm (coerce-type): Move to ... * ravanan/reader.scm (coerce-type): ... here.
2024-10-02README: Mention "work reuse".Arun Isaac
"Work reuse" is the term used by the CWL specification for caching. Use it to keep things familiar. * README.md: Mention "work reuse" in introduction.
2024-10-02command-line-tool: Promote find-requirement to top level.Arun Isaac
* ravanan/command-line-tool.scm (build-command-line-tool-script)[find-requirement]: Promote to top level.
2024-10-02slurm-api: Allow requesting a certain number of CPUs.Arun Isaac
* ravanan/slurm-api.scm (submit-job): Accept cpus argument. * ravanan/command-line-tool.scm (run-command-line-tool): Pass cpus argument. (build-command-line-tool-script)[cores]: New function. Set runtime.cores based on CPU allocation.
2024-10-02propnet: Explain in detail why we never cancel running propagators.Arun Isaac
* ravanan/propnet.scm (poll-propnet): Add detailed comment elaborating on why we never cancel or forget about running propagators.
2024-10-02command-line-tool: Allow per-batch system requirements support.Arun Isaac
* ravanan/command-line-tool.scm (command-line-tool-supported-requirements): New public function. (check-requirements): Accept batch-system and supported-requirements-for-batch-system arguments. * ravanan/command-line-tool.scm (run-command-line-tool): Pass batch-system to build-command-line-tool-script. (build-command-line-tool-script): Accept batch-system argument. Update calls to check-requirements. * ravanan/workflow.scm (%workflow-only-requirements): New variable. (%supported-requirements): Use %workflow-only-requirements. (supported-requirements): New function. (workflow-class->propnet): Accept batch-system argument. Update calls to check-requirements. (workflow-scheduler)[schedule]: Pass batch-system to workflow-class->propnet.
2024-10-02utils: Remove unused (ice-9 match) import.Arun Isaac
* ravanan/utils.scm: Do not import (ice-9 match).
2024-10-02command-line-tool: Remove unused imports.Arun Isaac
* ravanan/command-line-tool.scm: Do not import (srfi srfi-171), (ice-9 popen), (gcrypt base16), (gcrypt hash), (guix describe), (yaml), (ravanan glob) and (ravanan reader).
2024-10-02propnet: Remove unused (srfi srfi-26) import.Arun Isaac
* ravanan/propnet.scm: Do not import (srfi srfi-26).
2024-10-02workflow: Remove unused (ravanan reader) import.Arun Isaac
* ravanan/workflow.scm: Do not import (ravanan reader).
2024-10-02command-line-tool: Remove redundant newline after warning.Arun Isaac
* ravanan/command-line-tool.scm (check-requirements): Remove redundant newline after warning.
2024-10-02command-line-tool: Canonicalize inputs before embedding in G-exp.Arun Isaac
* ravanan/command-line-tool.scm (canonicalize-json): New function. (build-command-line-tool-script): Use canonicalize-json.
2024-10-02workflow: Support subworkflows recursively.Arun Isaac
Implementing subworkflows recursively as sub-propnets is the correct and elegant ouroboros way. Earlier, we were flattening workflows into their constituent CommandLineTools steps. Among many other issues, this approach ran into problems when scattering over subworkflows. * ravanan/workflow.scm: Import assertion-violation from (rnrs base). (<command-line-tool-state>, <workflow-state>): New record types. (workflow-class->propnet): New function. (workflow-class->propagators, workflow->command-line-tool-steps): Delete functions. (workflow-scheduler): Represent state using <command-line-tool-state> and <workflow-state>. Recursively evaluate compound propagators as propnets. (run-workflow): Use capture-output and poll from scheduler.
2024-10-01workflow: Pass a list, not two vectors, to fold.Arun Isaac
* ravanan/workflow.scm (inherit-requirements-and-hints): Pass a list, not two vectors, to fold.
2024-10-01propnet: Pass on scheduler to schedule calls.Arun Isaac
This enables sub-propnets (or compound propagators) to share the same scheduler. * ravanan/propnet.scm (activate-propagator): Accept scheduler as argument instead of the schedule function. (poll-propnet): Pass scheduler as argument instead of its schedule function.
2024-10-01command-line-tool: Move inputs resolution to (ravanan workflow).Arun Isaac
We resolve inputs in run-workflow before doing anything else. We thus avoid bugs due to partially or insufficiently resolved inputs. * ravanan/command-line-tool.scm: Do not import (web uri). (%store-files-directory, %store-data-directory, %store-logs-directory): Export. (location->path, resolve-inputs, intern-file): Move to (ravanan workflow). (build-command-line-tool-script): Do not call resolve-inputs. * ravanan/workflow.scm: Import (web uri) and (ravanan work types). (run-workflow): Call resolve-inputs.
2024-10-01job-state: Add that the job-state records are for batch systems.Arun Isaac
* ravanan/job-state.scm: Add to the commentary that the job-state records are per batch system.
2024-10-01vectors: Add vector-mapn.Arun Isaac
* ravanan/work/vectors.scm: Import (srfi srfi-26). (vector-mapn): New public function.
2024-10-01vectors: Add unindexed variant of vector-fold.Arun Isaac
* ravanan/work/vectors.scm: Import vector-fold from (srfi srfi-43) with prefix. (vector-fold): New public function. (vector-map->list, vector-append-map, vector-append-map->list): Use vector-fold.
2024-10-01job-state: Re-indent.Arun Isaac
* ravanan/job-state.scm (job-state-script): Re-indent.
2024-10-01command-line-tool: Copy path, not location.Arun Isaac
location is an implementation-dependent URI. path is a real filesystem path. * ravanan/command-line-tool.scm (copy-input-files-gexp): Copy path, not location.
2024-10-01vectors: Add vector-member.Arun Isaac
* ravanan/work/vectors.scm (vector-member): New public function.
2024-10-01workflow: Use bound variables instead of calling getters again.Arun Isaac
* ravanan/workflow.scm (workflow-scheduler)[schedule]: Use bound variables name and cwl instead of calling getters scheduler-proc-name and scheduler-proc-cwl again.
2024-10-01command-line-tool: Move scheduler to (ravanan workflow).Arun Isaac
* ravanan/command-line-tool.scm: Do not import (rnrs conditions), (rnrs exceptions) and (ravanan propnet). (command-line-tool-scheduler): Move to (ravanan workflow) as workflow-scheduler. (&job-failure, <scheduler-proc>): Move to (ravanan workflow). (script->store-stdout-file, script->store-stderr-file, capture-command-line-tool-output): Export functions. * ravanan/workflow.scm: Import define-condition-type from (rnrs conditions), (rnrs exceptions), (srfi srfi-9 gnu) and (ravanan job-state). (run-workflow): Call workflow-scheduler instead of command-line-tool-scheduler.
2024-10-01propnet: Allow propagators to update their state.Arun Isaac
This will come in handy later when we implement compound propagators (propagators that are themselves a propagator network). * ravanan/command-line-tool.scm (command-line-tool-scheduler)[poll]: Return two values---the status and the updated state. * ravanan/job-state.scm (job-state-status): Return two values---the status and the updated state. * ravanan/propnet.scm (partition-map): New function. (poll-propnet): Update state of a propagator in flight.
2024-10-01propnet: Present a polling interface to callers.Arun Isaac
(ravanan propnet) now presents a polling interface to the outside world. Instead of computing the entire propagator network in one function call, it needs to be polled repeatedly until it is done. Hence, the polling logic (such as the polling interval) needs to be outside (ravanan propnet). * ravanan/propnet.scm (<scheduler>)[poll-interval]: Delete field. (<propnet-state>): New record type. (schedule-propnet, capture-propnet-output): New public functions. (run-propnet): Rename to poll-propnet; return status and state values instead of fulling computing the propnet. * ravanan/command-line-tool.scm (%job-poll-interval): Move to (ravanan workflow). (command-line-tool-scheduler): Do not initialize poll-interval in scheduler. * ravanan/workflow.scm: Import (srfi srfi-71). (run-workflow): Use the new polling interface to the propnet.
2024-09-23command-line-tool: Update path to staged input files.Arun Isaac
* ravanan/command-line-tool.scm (build-command-line-tool-script)[stage-files, set-staged-path]: New functions. After staging, update path to staged input files.
2024-09-23command-line-tool: Defer access of input values to runtime.Arun Isaac
* ravanan/command-line-tool.scm (build-command): Pass id argument to collect-bindings. Defer access of input values to runtime. [collect-bindings]: Accept id argument. [input+type-tree+binding->command-line-binding]: Rename to id+input+type-tree+binding->command-line-binding. Accept id argument and return G-expressions for command-line-binding values. (command-line-binding->args): Defer access of input values to runtime.
generated by cgit v1.2.3 (git 2.45.2) at 2024-12-23 04:47:24 +0000