| Age | Commit message (Expand) | Author |
|---|
| 2024-10-02 | command-line-tool: Re-indent coerce-expression.•••* ravanan/command-line-tool.scm (coerce-expression): Re-indent.
| Arun Isaac |
| 2024-10-02 | command-line-tool: Add local version of coerce-expression.•••* ravanan/command-line-tool.scm (coerce-expression-local): New
function.
| Arun Isaac |
| 2024-10-02 | reader: Handle coercion of number to number.•••* ravanan/reader.scm (coerce-type): Handle coercion of number to
number.
* tests/reader.scm: New file.
| Arun Isaac |
| 2024-10-02 | config: Remove generated source file.•••config.scm is generated from config.scm.in. Therefore, config.scm must
not be committed.
* ravanan/config.scm: Delete file.
| Arun Isaac |
| 2024-10-02 | Revert "config: Remove."•••This reverts commit d611d990b297b3d573d7d5af83e9376744d51dd5. We need
to bring back a local node to evaluate javascript expressions on the
head node.
| Arun Isaac |
| 2024-10-02 | workflow: Add default inputs before scheduling.•••We add default input values and resolve them before scheduling. Now
that we do that, we no longer have to resolve inputs or check for
required inputs in run-workflow.
* ravanan/workflow.scm (add-defaults): New function.
(workflow-scheduler)[schedule]: Add default inputs and resolve them
before scheduling.
(run-workflow): Do not resolve inputs or check for required inputs.
| Arun Isaac |
| 2024-10-02 | command-line-tool: Make runtime.cores a number.•••*
ravanan/command-line-tool.scm (build-command-line-tool-script)[cores]:
Return a number on the slurm-api batch system.
| Arun Isaac |
| 2024-10-02 | command-line-tool: Support coresMin ResourceRequirement.•••* ravanan/command-line-tool.scm: Import (ravanan reader).
(%command-line-tool-supported-requirements):
Add ResouceRequirement.
(command-line-tool-supported-requirements): Remove ResourceRequirement
for the single-machine batch system.
* ravanan/command-line-tool.scm (run-command-line-tool): Request the
number of CPUs specified in coresMin of the ResourceRequirement.
| Arun Isaac |
| 2024-10-02 | reader: Support coercion to number.•••* ravanan/reader.scm (coerce-type): Support coercion to number.
| Arun Isaac |
| 2024-10-02 | workflow: Move coerce-type to (ravanan reader).•••* ravanan/workflow.scm (coerce-type): Move to ...
* ravanan/reader.scm (coerce-type): ... here.
| Arun Isaac |
| 2024-10-02 | README: Mention "work reuse".•••"Work reuse" is the term used by the CWL specification for caching.
Use it to keep things familiar.
* README.md: Mention "work reuse" in introduction.
| Arun Isaac |
| 2024-10-02 | command-line-tool: Promote find-requirement to top level.•••*
ravanan/command-line-tool.scm (build-command-line-tool-script)[find-requirement]:
Promote to top level.
| Arun Isaac |
| 2024-10-02 | slurm-api: Allow requesting a certain number of CPUs.•••* ravanan/slurm-api.scm (submit-job): Accept cpus argument.
* ravanan/command-line-tool.scm (run-command-line-tool): Pass cpus
argument.
(build-command-line-tool-script)[cores]: New function.
Set runtime.cores based on CPU allocation.
| Arun Isaac |
| 2024-10-02 | propnet: Explain in detail why we never cancel running propagators.•••* ravanan/propnet.scm (poll-propnet): Add detailed comment elaborating
on why we never cancel or forget about running propagators.
| Arun Isaac |
| 2024-10-02 | command-line-tool: Allow per-batch system requirements support.•••*
ravanan/command-line-tool.scm (command-line-tool-supported-requirements):
New public function.
(check-requirements): Accept batch-system and
supported-requirements-for-batch-system arguments.
* ravanan/command-line-tool.scm (run-command-line-tool): Pass
batch-system to build-command-line-tool-script.
(build-command-line-tool-script): Accept batch-system argument. Update
calls to check-requirements.
* ravanan/workflow.scm (%workflow-only-requirements): New variable.
(%supported-requirements): Use %workflow-only-requirements.
(supported-requirements): New function.
(workflow-class->propnet): Accept batch-system argument. Update calls
to check-requirements.
(workflow-scheduler)[schedule]: Pass batch-system to
workflow-class->propnet.
| Arun Isaac |
| 2024-10-02 | utils: Remove unused (ice-9 match) import.•••* ravanan/utils.scm: Do not import (ice-9 match).
| Arun Isaac |
| 2024-10-02 | command-line-tool: Remove unused imports.•••* ravanan/command-line-tool.scm: Do not import (srfi srfi-171), (ice-9
popen), (gcrypt base16), (gcrypt hash), (guix
describe), (yaml), (ravanan glob) and (ravanan reader).
| Arun Isaac |
| 2024-10-02 | propnet: Remove unused (srfi srfi-26) import.•••* ravanan/propnet.scm: Do not import (srfi srfi-26).
| Arun Isaac |
| 2024-10-02 | workflow: Remove unused (ravanan reader) import.•••* ravanan/workflow.scm: Do not import (ravanan reader).
| Arun Isaac |
| 2024-10-02 | command-line-tool: Remove redundant newline after warning.•••* ravanan/command-line-tool.scm (check-requirements): Remove redundant
newline after warning.
| Arun Isaac |
| 2024-10-02 | command-line-tool: Canonicalize inputs before embedding in G-exp.•••* ravanan/command-line-tool.scm (canonicalize-json): New function.
(build-command-line-tool-script): Use canonicalize-json.
| Arun Isaac |
| 2024-10-02 | workflow: Support subworkflows recursively.•••Implementing subworkflows recursively as sub-propnets is the correct
and elegant ouroboros way. Earlier, we were flattening workflows into
their constituent CommandLineTools steps. Among many other issues,
this approach ran into problems when scattering over subworkflows.
* ravanan/workflow.scm: Import assertion-violation from (rnrs base).
(<command-line-tool-state>, <workflow-state>): New record types.
(workflow-class->propnet): New function.
(workflow-class->propagators, workflow->command-line-tool-steps):
Delete functions.
(workflow-scheduler): Represent state using <command-line-tool-state>
and <workflow-state>. Recursively evaluate compound propagators as
propnets.
(run-workflow): Use capture-output and poll from scheduler.
| Arun Isaac |
| 2024-10-01 | workflow: Pass a list, not two vectors, to fold.•••* ravanan/workflow.scm (inherit-requirements-and-hints): Pass a list,
not two vectors, to fold.
| Arun Isaac |
| 2024-10-01 | propnet: Pass on scheduler to schedule calls.•••This enables sub-propnets (or compound propagators) to share the same
scheduler.
* ravanan/propnet.scm (activate-propagator): Accept scheduler as
argument instead of the schedule function.
(poll-propnet): Pass scheduler as argument instead of its schedule
function.
| Arun Isaac |
| 2024-10-01 | command-line-tool: Move inputs resolution to (ravanan workflow).•••We resolve inputs in run-workflow before doing anything else. We thus
avoid bugs due to partially or insufficiently resolved inputs.
* ravanan/command-line-tool.scm: Do not import (web uri).
(%store-files-directory, %store-data-directory,
%store-logs-directory): Export.
(location->path, resolve-inputs, intern-file): Move to (ravanan
workflow).
(build-command-line-tool-script): Do not call resolve-inputs.
* ravanan/workflow.scm: Import (web uri) and (ravanan work types).
(run-workflow): Call resolve-inputs.
| Arun Isaac |
| 2024-10-01 | job-state: Add that the job-state records are for batch systems.•••* ravanan/job-state.scm: Add to the commentary that the job-state
records are per batch system.
| Arun Isaac |
| 2024-10-01 | vectors: Add vector-mapn.•••* ravanan/work/vectors.scm: Import (srfi srfi-26).
(vector-mapn): New public function.
| Arun Isaac |
| 2024-10-01 | vectors: Add unindexed variant of vector-fold.•••* ravanan/work/vectors.scm: Import vector-fold from (srfi srfi-43)
with prefix.
(vector-fold): New public function.
(vector-map->list, vector-append-map, vector-append-map->list): Use
vector-fold.
| Arun Isaac |
| 2024-10-01 | job-state: Re-indent.•••* ravanan/job-state.scm (job-state-script): Re-indent.
| Arun Isaac |
| 2024-10-01 | command-line-tool: Copy path, not location.•••location is an implementation-dependent URI. path is a real filesystem
path.
* ravanan/command-line-tool.scm (copy-input-files-gexp): Copy path,
not location.
| Arun Isaac |
| 2024-10-01 | vectors: Add vector-member.•••* ravanan/work/vectors.scm (vector-member): New public function.
| Arun Isaac |
| 2024-10-01 | workflow: Use bound variables instead of calling getters again.•••* ravanan/workflow.scm (workflow-scheduler)[schedule]: Use bound
variables name and cwl instead of calling getters scheduler-proc-name
and scheduler-proc-cwl again.
| Arun Isaac |
| 2024-10-01 | command-line-tool: Move scheduler to (ravanan workflow).•••* ravanan/command-line-tool.scm: Do not import (rnrs
conditions), (rnrs exceptions) and (ravanan propnet).
(command-line-tool-scheduler): Move to (ravanan workflow) as
workflow-scheduler.
(&job-failure, <scheduler-proc>): Move to (ravanan workflow).
(script->store-stdout-file, script->store-stderr-file,
capture-command-line-tool-output): Export functions.
* ravanan/workflow.scm: Import define-condition-type from (rnrs
conditions), (rnrs exceptions), (srfi srfi-9 gnu) and (ravanan
job-state).
(run-workflow): Call workflow-scheduler instead of
command-line-tool-scheduler.
| Arun Isaac |
| 2024-10-01 | propnet: Allow propagators to update their state.•••This will come in handy later when we implement compound
propagators (propagators that are themselves a propagator network).
* ravanan/command-line-tool.scm (command-line-tool-scheduler)[poll]:
Return two values---the status and the updated state.
* ravanan/job-state.scm (job-state-status): Return two values---the
status and the updated state.
* ravanan/propnet.scm (partition-map): New function.
(poll-propnet): Update state of a propagator in flight.
| Arun Isaac |
| 2024-10-01 | propnet: Present a polling interface to callers.•••(ravanan propnet) now presents a polling interface to the outside
world. Instead of computing the entire propagator network in one
function call, it needs to be polled repeatedly until it is done.
Hence, the polling logic (such as the polling interval) needs to be
outside (ravanan propnet).
* ravanan/propnet.scm (<scheduler>)[poll-interval]: Delete field.
(<propnet-state>): New record type.
(schedule-propnet, capture-propnet-output): New public functions.
(run-propnet): Rename to poll-propnet; return status and state values
instead of fulling computing the propnet.
* ravanan/command-line-tool.scm (%job-poll-interval): Move to (ravanan
workflow).
(command-line-tool-scheduler): Do not initialize poll-interval in
scheduler.
* ravanan/workflow.scm: Import (srfi srfi-71).
(run-workflow): Use the new polling interface to the propnet.
| Arun Isaac |
| 2024-09-23 | command-line-tool: Update path to staged input files.•••*
ravanan/command-line-tool.scm (build-command-line-tool-script)[stage-files,
set-staged-path]: New functions.
After staging, update path to staged input files.
| Arun Isaac |
| 2024-09-23 | command-line-tool: Defer access of input values to runtime.•••* ravanan/command-line-tool.scm (build-command): Pass id argument to
collect-bindings. Defer access of input values to runtime.
[collect-bindings]: Accept id argument.
[input+type-tree+binding->command-line-binding]: Rename to
id+input+type-tree+binding->command-line-binding. Accept id argument
and return G-expressions for command-line-binding values.
(command-line-binding->args): Defer access of input values to runtime.
| Arun Isaac |
| 2024-09-23 | command-line-tool: Separate file copying from canonization.•••*
ravanan/command-line-tool.scm (build-command-line-tool-script)[canonicalize-file-value]:
Separate out file copying into ...
[copy-file-value]: ... new function.
[capture-outputs-gexp, capture-secondary-file, path->value]: Call
copy-file-value after canonicalize-file-value.
| Arun Isaac |
| 2024-09-23 | utils: Move json-ref to (ravanan work utils).•••* ravanan/utils.scm: Move to ...
* ravanan/work/utils.scm (json-ref): ... here.
* ravanan/slurm-api.scm: Import (ravanan work utils) instead
of (ravanan utils).
| Arun Isaac |
| 2024-09-20 | command-line-tool: Add missing argument to canonicalize-file-value.•••* ravanan/command-line-tool.scm (build-command-line-tool-script): Add
missing workflow-output-directory argument to canonicalize-file-value.
| Arun Isaac |
| 2024-09-19 | work/utils: Add alist=?.•••* ravanan/work/utils.scm (alist=?): New public function.
| Arun Isaac |
| 2024-09-19 | propnet: Describe state machine implementation.•••* ravanan/propnet.scm (run-propnet): Add comment describing state
machine implementation.
| Arun Isaac |
| 2024-09-16 | bin: Explain why we support the SLURM_JWT=token format.•••* bin/ravanan (read-jwt): Explain why we support the SLURM_JWT=token
format.
| Arun Isaac |
| 2024-09-13 | bin: Support SLURM_JWT=token format in JWT files.•••* bin/ravanan: Import (srfi srfi-26) and (ravanan utils).
(read-jwt): Support SLURM_JWT=token format.
| Arun Isaac |
| 2024-09-13 | bin: Refactor JWT reading into separate function.•••* bin/ravanan (read-jwt): New function.
(main): Call read-jwt.
| Arun Isaac |
| 2024-09-13 | utils: Add string-trim-prefix.•••* ravanan/utils.scm (string-trim-prefix): New public function.
| Arun Isaac |
| 2024-09-13 | command-line-tool: Add nameroot and nameext fields.•••*
ravanan/command-line-tool.scm (resolve-inputs)[canonicalize-file-value],
(build-command-line-tool-script)[canonicalize-file-value]: Add
nameroot and nameext fields.
| Arun Isaac |
| 2024-09-13 | command-line-tool: Remove unused collect-input-files function.•••* ravanan/command-line-tool.scm (collect-input-files): Delete
function.
| Arun Isaac |
| 2024-09-13 | command-line-tool: Do not embed stdout-filename outside main body.•••stdout-output->value cannot access the inputs variable, and thus
will fail when stdout-filename is a parameter reference that needs to
be evaluated. To avoid this, pass it in as an argument.
*
ravanan/command-line-tool.scm (build-command-line-tool-script)[stdout-output->value]:
Accept stdout-filename as argument instead of G-expression embedding.
Pass stdout-filename as argument to stdout-output->value.
| Arun Isaac |
| 2024-09-13 | command-line-tool: Create vectors of secondary files for recursion.•••* ravanan/command-line-tool.scm (resolve-inputs)[resolve]: When
recursing, create a vector of secondary files.
| Arun Isaac |
