aboutsummaryrefslogtreecommitdiff
AgeCommit message (Collapse)Author
2024-10-09batch-system: Introduce <slurm-api-batch-system> record type.Arun Isaac
A <slurm-api-batch-system> record type allows us an easy way to group and pass along parameters required by the slurm-api batch system. * ravanan/batch-system.scm: New file. * bin/ravanan: Import (ravanan batch-system). (main): Create <slurm-api-batch-system> object for batch-system argument. Do not pass #:slurm-api-endpoint and #:slurm-jwt arguments. * ravanan/command-line-tool.scm: Import (ravanan batch-system). (command-line-tool-supported-requirements): Check for slurm-api batch system using slurm-api-batch-system?. (run-command-line-tool): Check for slurm-api batch system using slurm-api-batch-system?. Do not accept #:slurm-api-endpoint and #:slurm-jwt arguments. * ravanan/workflow.scm: Import (ravanan batch-system). (workflow-scheduler): Do not accept #:slurm-api-endpoint and #:slurm-jwt arguments. Pass batch-system to job-state-status. (run-workflow): Check for slurm-api batch system using slurm-api-batch-system?. Do not accept #:slurm-api-endpoint and #:slurm-jwt arguments. * ravanan/job-state.scm: Import (ravanan batch-system). (job-state-status): Do not accept #:slurm-api-endpoint and #:slurm-jwt arguments. Accept batch-system argument.
2024-10-08command-line-tool: Import (guix gexp) into manifest files.Arun Isaac
* ravanan/command-line-tool.scm (load-manifest): Import (guix gexp) into manifest files.
2024-10-08README: Add CI badge.Arun Isaac
* README.md: Add CI badge.
2024-10-08command-line-tool: Do not auto-compile manifest files.Arun Isaac
* ravanan/command-line-tool.scm (load-manifest): Do not auto-compile manifest files.
2024-10-08javascript: Do not serialize JSON if not interpolating string.Arun Isaac
* ravanan/javascript.scm (evaluate-parameter-reference): Do not serialize JSON tree if not interpolating parameter reference. * tests/javascript.scm ("evaluate parameter reference to JSON object", "evaluate parameter reference to JSON object using node"): New tests. ("evaluate parameter reference with node", "evaluate parameter reference (without context)", "evaluate parameter reference with node (without context)"): Update tests.
2024-10-08javascript: Handle node calls without context.Arun Isaac
* ravanan/javascript.scm (evaluate-using-node): Handle cases when context is #f.
2024-10-08javascript: Reference inputs, self or runtime symbols; not strings.Arun Isaac
* ravanan/javascript.scm (evaluate-parameter-reference-1): Reference inputs, self or runtime symbols; not strings.
2024-10-08tests: Add tests for javascript G-expressions.Arun Isaac
* tests/javascript.scm: Import (guix gexp) and (ice-9 match). (gexp->sexp-rec): New function. ("evaluate parameter reference (without context)", "evaluate parameter reference with string interpolation (without context)", "evaluate parameter reference with string interpolation of JSON trees (without context)", "evaluate parameter reference with node (without context)", "evaluate parameter reference with string interpolation using node (without context)", "evaluate parameter reference with string interpolation of JSON trees using node (without context)"): New tests.
2024-10-07command-line-tool: Implement GuixManifestRequirement.Arun Isaac
* ravanan/command-line-tool.scm (%command-line-tool-supported-requirements): Add GuixManifestRequirement. (build-command-line-tool-script): Implement GuixManifestRequirement.
2024-10-07bin: Pass manifest file paths, not manifests to other functions.Arun Isaac
* bin/ravanan: Move to (ravanan command-line-tool). Add comment about declarative modules. (main): Pass manifest file path to run-workflow. * ravanan/command-line-tool.scm (run-command-line-tool, build-command-line-tool-script): Accept manifest file path instead of manifest. * ravanan/workflow.scm (workflow-scheduler, run-workflow): Accept manifest file path instead of manifest.
2024-10-07bin: Load manifests without their definitions leaking out.Arun Isaac
* bin/ravanan: Do not import (gnu packages) and (guix profiles). (load-manifest): New function. (main): Use load-manifest.
2024-10-04javascript: Unify simple parameter reference and node fallback.Arun Isaac
* ravanan/command-line-tool.scm: Do not import (ravanan config), and node from (gnu packages node). (%worker-node): Move to (ravanan javascript). (interpolate-parameter-references): Delete function. (coerce-expression): Always call evaluate-parameter-reference from (ravanan javascript). * ravanan/work/command-line-tool.scm (evaluate-parameter-reference): Delete function. * ravanan/javascript.scm: Mention fallback to node in module commentary. Import (ravanan config), (ravanan work command-line-tool), and node from (gnu packages node). (evaluate-simple-parameter-reference): Rename to evaluate-parameter-reference. (evaluate-parameter-reference-1, evaluate-parameter-reference): Add expression-lib argument. Return regular values, not maybe-monadic values. (evaluate-using-node): New function. * tests/javascript.scm ("evaluate parameter reference", "evaluate parameter reference with string interpolation", "evaluate parameter reference with string interpolation of JSON trees"): Update tests. ("evaluate parameter reference with node", "evaluate parameter reference with string interpolation using node", "evaluate parameter reference with string interpolation of JSON trees using node"): New tests.
2024-10-04javascript: Remove "alternating" from tokenizing docstring.Arun Isaac
tokenize-parameter-references need not return an "alternating" sequence of tokens. There can be consecutive parameter reference tokens or literal strings. * ravanan/javascript.scm (tokenize-parameter-references): Remove "alternating" from docstring.
2024-10-04monads: Use write, not display, to print Just values.Arun Isaac
* ravanan/work/monads.scm (<maybe>): Use write, not display, to print Just values.
2024-10-04javascript: Implement parameter reference javascript subset language.Arun Isaac
* ravanan/javascript.scm, tests/javascript.scm: New files. * ravanan/command-line-tool.scm: Import (ravanan javascript). (coerce-expression): Use evaluate-simple-parameter-reference. (interpolate-parameter-references)[tokenize]: Move to (ravanan javascript) as tokenize-parameter-references.
2024-10-04command-line-tool: Move canonicalize-json to (ravanan work utils).Arun Isaac
* ravanan/command-line-tool.scm (canonicalize-json): Move to (ravanan work utils). Move comment about G-expressions to build-command-line-tool-script. * ravanan/work/utils.scm (ravanan): Import (ravanan work vectors).
2024-10-04command-line-tool: Merge local and remote coerce-expression.Arun Isaac
coerce-expression and coerce-expression-local share most of their code. So, it makes sense to merge them. * ravanan/command-line-tool.scm (coerce-expression): Add optional context argument. (coerce-expression-local): Delete function. (run-command-line-tool): Call coerce-expression instead of coerce-expression-local.
2024-10-02command-line-tool: Embed int value into G-expression.Arun Isaac
* ravanan/command-line-tool.scm (command-line-binding->args): Embed int value into G-expression instead of evaluating it immediately.
2024-10-02command-line-tool: Coerce coresMin expressions.Arun Isaac
* ravanan/command-line-tool.scm (run-command-line-tool): Coerce coresMin expressions.
2024-10-02command-line-tool: Re-indent coerce-expression.Arun Isaac
* ravanan/command-line-tool.scm (coerce-expression): Re-indent.
2024-10-02command-line-tool: Add local version of coerce-expression.Arun Isaac
* ravanan/command-line-tool.scm (coerce-expression-local): New function.
2024-10-02reader: Handle coercion of number to number.Arun Isaac
* ravanan/reader.scm (coerce-type): Handle coercion of number to number. * tests/reader.scm: New file.
2024-10-02config: Remove generated source file.Arun Isaac
config.scm is generated from config.scm.in. Therefore, config.scm must not be committed. * ravanan/config.scm: Delete file.
2024-10-02Revert "config: Remove."Arun Isaac
This reverts commit d611d990b297b3d573d7d5af83e9376744d51dd5. We need to bring back a local node to evaluate javascript expressions on the head node.
2024-10-02workflow: Add default inputs before scheduling.Arun Isaac
We add default input values and resolve them before scheduling. Now that we do that, we no longer have to resolve inputs or check for required inputs in run-workflow. * ravanan/workflow.scm (add-defaults): New function. (workflow-scheduler)[schedule]: Add default inputs and resolve them before scheduling. (run-workflow): Do not resolve inputs or check for required inputs.
2024-10-02command-line-tool: Make runtime.cores a number.Arun Isaac
* ravanan/command-line-tool.scm (build-command-line-tool-script)[cores]: Return a number on the slurm-api batch system.
2024-10-02command-line-tool: Support coresMin ResourceRequirement.Arun Isaac
* ravanan/command-line-tool.scm: Import (ravanan reader). (%command-line-tool-supported-requirements): Add ResouceRequirement. (command-line-tool-supported-requirements): Remove ResourceRequirement for the single-machine batch system. * ravanan/command-line-tool.scm (run-command-line-tool): Request the number of CPUs specified in coresMin of the ResourceRequirement.
2024-10-02reader: Support coercion to number.Arun Isaac
* ravanan/reader.scm (coerce-type): Support coercion to number.
2024-10-02workflow: Move coerce-type to (ravanan reader).Arun Isaac
* ravanan/workflow.scm (coerce-type): Move to ... * ravanan/reader.scm (coerce-type): ... here.
2024-10-02README: Mention "work reuse".Arun Isaac
"Work reuse" is the term used by the CWL specification for caching. Use it to keep things familiar. * README.md: Mention "work reuse" in introduction.
2024-10-02command-line-tool: Promote find-requirement to top level.Arun Isaac
* ravanan/command-line-tool.scm (build-command-line-tool-script)[find-requirement]: Promote to top level.
2024-10-02slurm-api: Allow requesting a certain number of CPUs.Arun Isaac
* ravanan/slurm-api.scm (submit-job): Accept cpus argument. * ravanan/command-line-tool.scm (run-command-line-tool): Pass cpus argument. (build-command-line-tool-script)[cores]: New function. Set runtime.cores based on CPU allocation.
2024-10-02propnet: Explain in detail why we never cancel running propagators.Arun Isaac
* ravanan/propnet.scm (poll-propnet): Add detailed comment elaborating on why we never cancel or forget about running propagators.
2024-10-02command-line-tool: Allow per-batch system requirements support.Arun Isaac
* ravanan/command-line-tool.scm (command-line-tool-supported-requirements): New public function. (check-requirements): Accept batch-system and supported-requirements-for-batch-system arguments. * ravanan/command-line-tool.scm (run-command-line-tool): Pass batch-system to build-command-line-tool-script. (build-command-line-tool-script): Accept batch-system argument. Update calls to check-requirements. * ravanan/workflow.scm (%workflow-only-requirements): New variable. (%supported-requirements): Use %workflow-only-requirements. (supported-requirements): New function. (workflow-class->propnet): Accept batch-system argument. Update calls to check-requirements. (workflow-scheduler)[schedule]: Pass batch-system to workflow-class->propnet.
2024-10-02utils: Remove unused (ice-9 match) import.Arun Isaac
* ravanan/utils.scm: Do not import (ice-9 match).
2024-10-02command-line-tool: Remove unused imports.Arun Isaac
* ravanan/command-line-tool.scm: Do not import (srfi srfi-171), (ice-9 popen), (gcrypt base16), (gcrypt hash), (guix describe), (yaml), (ravanan glob) and (ravanan reader).
2024-10-02propnet: Remove unused (srfi srfi-26) import.Arun Isaac
* ravanan/propnet.scm: Do not import (srfi srfi-26).
2024-10-02workflow: Remove unused (ravanan reader) import.Arun Isaac
* ravanan/workflow.scm: Do not import (ravanan reader).
2024-10-02command-line-tool: Remove redundant newline after warning.Arun Isaac
* ravanan/command-line-tool.scm (check-requirements): Remove redundant newline after warning.
2024-10-02command-line-tool: Canonicalize inputs before embedding in G-exp.Arun Isaac
* ravanan/command-line-tool.scm (canonicalize-json): New function. (build-command-line-tool-script): Use canonicalize-json.
2024-10-02workflow: Support subworkflows recursively.Arun Isaac
Implementing subworkflows recursively as sub-propnets is the correct and elegant ouroboros way. Earlier, we were flattening workflows into their constituent CommandLineTools steps. Among many other issues, this approach ran into problems when scattering over subworkflows. * ravanan/workflow.scm: Import assertion-violation from (rnrs base). (<command-line-tool-state>, <workflow-state>): New record types. (workflow-class->propnet): New function. (workflow-class->propagators, workflow->command-line-tool-steps): Delete functions. (workflow-scheduler): Represent state using <command-line-tool-state> and <workflow-state>. Recursively evaluate compound propagators as propnets. (run-workflow): Use capture-output and poll from scheduler.
2024-10-01workflow: Pass a list, not two vectors, to fold.Arun Isaac
* ravanan/workflow.scm (inherit-requirements-and-hints): Pass a list, not two vectors, to fold.
2024-10-01propnet: Pass on scheduler to schedule calls.Arun Isaac
This enables sub-propnets (or compound propagators) to share the same scheduler. * ravanan/propnet.scm (activate-propagator): Accept scheduler as argument instead of the schedule function. (poll-propnet): Pass scheduler as argument instead of its schedule function.
2024-10-01command-line-tool: Move inputs resolution to (ravanan workflow).Arun Isaac
We resolve inputs in run-workflow before doing anything else. We thus avoid bugs due to partially or insufficiently resolved inputs. * ravanan/command-line-tool.scm: Do not import (web uri). (%store-files-directory, %store-data-directory, %store-logs-directory): Export. (location->path, resolve-inputs, intern-file): Move to (ravanan workflow). (build-command-line-tool-script): Do not call resolve-inputs. * ravanan/workflow.scm: Import (web uri) and (ravanan work types). (run-workflow): Call resolve-inputs.
2024-10-01job-state: Add that the job-state records are for batch systems.Arun Isaac
* ravanan/job-state.scm: Add to the commentary that the job-state records are per batch system.
2024-10-01vectors: Add vector-mapn.Arun Isaac
* ravanan/work/vectors.scm: Import (srfi srfi-26). (vector-mapn): New public function.
2024-10-01vectors: Add unindexed variant of vector-fold.Arun Isaac
* ravanan/work/vectors.scm: Import vector-fold from (srfi srfi-43) with prefix. (vector-fold): New public function. (vector-map->list, vector-append-map, vector-append-map->list): Use vector-fold.
2024-10-01job-state: Re-indent.Arun Isaac
* ravanan/job-state.scm (job-state-script): Re-indent.
2024-10-01command-line-tool: Copy path, not location.Arun Isaac
location is an implementation-dependent URI. path is a real filesystem path. * ravanan/command-line-tool.scm (copy-input-files-gexp): Copy path, not location.
2024-10-01vectors: Add vector-member.Arun Isaac
* ravanan/work/vectors.scm (vector-member): New public function.
generated by cgit v1.2.3 (git 2.45.2) at 2024-12-23 05:06:47 +0000