| Age | Commit message (Collapse) | Author |
|---|
|
These scripts were a part of the CI server system definition, and out
of reach of ravanan maintainers. We move them here since it's easier
to keep updated and within the reach of ravanan maintainers to modify
as they please.
|
|
|
|
|
|
|
|
"more than one" is simpler English.
|
|
|
|
|
|
|
|
|
|
This will enable us to effect global control over the log level.
|
|
Implement logging system with five log levels. Rewrite warning and
user-error in terms of the new logging functions.
|
|
read-workflow and read-inputs now handle &unsupported-uri-scheme
conditions from location->path.
|
|
location->path now raises an &unsupported-uri-scheme condition on
unsupported URI schemes.
|
|
|
|
|
|
read-yaml-file is a wrapper around read-yaml-file from (yaml). It
raises the relevant exceptions when it fails.
|
|
|
|
read-json-file is a wrapper around call-with-input-file and json->scm.
It raises the relevant exceptions when it fails.
|
|
Return monadic value for the default key in normalize-formal-input.
Values returned from maybe-bind must be monadic; compose it with just.
|
|
* ravanan/workflow.scm (workflow-scheduler): Copy resource-requirement
field on scatter steps.
|
|
It is not the responsibility of load-script to canonicalize file
paths; the caller must do that.
* ravanan/utils.scm (load-script): Do not canonicalize script file
path.
|
|
* ravanan/reader.scm (normalize-software-requirement): New function.
(normalize-requirements): Use normalize-software-requirement.
|
|
* ravanan/reader.scm (normalize-env-var-requirement): Re-indent.
|
|
* ravanan/store.scm (step-store-basename): Canonicalize inputs JSON.
* tests/store.scm: New file.
|
|
*
ravanan/command-line-tool.scm (build-command-line-tool-script)[copy-input-files]:
Pass down inputs directory to recursive calls.
|
|
The comment was there earlier, but the code was missing!
* ravanan/command-line-tool.scm (manifest-file->search-path-sexps): Do
not auto-compile manifest files on inferior.
|
|
Canonicalization is unnecessary, and causes trouble when the manifest
file does not exist.
* ravanan/command-line-tool.scm (manifest-file->search-path-sexps,
manifest-file->profile-derivation): Do not canonicalize manifest file
paths.
|
|
*
ravanan/command-line-tool.scm (build-command-line-tool-script)[vector->gexp,
alist->gexp]: New functions.
[run-command-gexp]: Use vector->gexp and alist->gexp.
|
|
* ravanan/command-line-tool.scm (build-gexp-script)
(build-command-line-tool-script): Present a monadic interface.
* ravanan/workflow.scm (workflow->scheduler-proc): Run monadic value
returned by build-command-line-tool-script through the store.
(workflow->scheduler-proc, workflow-class->propnet): Update docstring
about the guix-store argument.
|
|
In the interest of consistency, return search path S-expressions from
both manifest-file->search-path-sexps and
software-packages->search-path-sexps as monadic values.
*
ravanan/command-line-tool.scm (software-packages->search-path-sexps):
Return monadic value.
* ravanan/command-line-tool.scm (build-command-line-tool-script): Use
monadic values from software-packages->search-path-sexps.
|
|
The monadic interface allows more of the building to be done in one go
when build-gexp-script is invoked.
* ravanan/command-line-tool.scm (manifest-file->environment,
software-packages->environment, manifest->environment): Delete
functions.
(inferior-meval-with-store, manifest-file->search-path-sexps,
manifest-file->profile-derivation, software-packages->manifest,
software-packages->search-path-sexps,
software-packages->profile-derivation): New functions.
(build-command-line-tool-script): Use manifest-file->search-paths,
manifest-file->profile-derivation,
software-packages->search-path-sexps and
software-packages->profile-derivation instead of
manifest-file->environment and software-packages->environment.
|
|
Re-using the connection improves performance and is cleaner than
passing around the Guix daemon socket specification.
* ravanan/command-line-tool.scm (manifest-file->environment,
software-packages->environment, manifest->environment,
build-command-line-tool-script, build-gexp-script): Work with store
connection instead of the Guix daemon socket.
* ravanan/workflow.scm: Import (guix store).
(workflow->scheduler-proc, workflow-class->propnet): Work with store
connection instead of the Guix daemon socket.
(build-workflow): Pass a store connection, not the Guix daemon socket,
to workflow->scheduler-proc.
|
|
Spawning inferiors is expensive. Do it only once and re-use the
connection.
* ravanan/command-line-tool.scm (call-with-inferior): Move to (ravanan
workflow).
(manifest-file->environment, software-packages->environment,
build-command-line-tool-script): Accept inferior instead of channels.
* ravanan/workflow.scm: Import (guix inferior).
(workflow->scheduler-proc, workflow-class->propnet): Accept inferior
instead of channels.
(build-workflow): New function.
(run-workflow): Use build-workflow instead of
workflow->scheduler-proc.
|
|
* ravanan/command-line-tool.scm (build-command-line-tool-script):
Clarify it is "disabling WorkReuse" that is unnecessary.
|
|
* ravanan/command-line-tool.scm (build-command-line-tool-script,
find-requirement): Export.
(run-command-line-tool): Accept script as argument instead of cwl. Add
resource-requirement argument. Remove manifest-file, channels, scratch
and guix-daemon-socket arguments.
* ravanan/workflow.scm (<scheduler-proc>):[cwl-or-propnet]: Rename to
script-or-propnet. [resource-requirement]: New field.
* ravanan/workflow.scm (scheduler-proc): Rename cwl-or-propnet
argument to script-or-propnet. Add resource-requirement argument.
(workflow->scheduler-proc, workflow-class->propnet): Add
manifest-file, channels, scratch, store and guix-daemon-socket
arguments.
(workflow->scheduler-proc): Build script. Capture ResourceRequirement.
(workflow-scheduler): Remove manifest-file, channels, scratch and
guix-daemon-socket arguments.
[schedule]: Run built script instead of CWL. Update call to
run-command-line-tool.
(run-workflow): Do not pass manifest-file, channels, scratch and
guix-daemon-socket arguments to workflow-scheduler. Pass
manifest-file, channels, scratch, store and guix-daemon-socket
arguments to workflow->scheduler-proc.
|
|
* ravanan/workflow.scm (workflow->scheduler-proc): Make scatter and
scatter-method arguments optional.
|
|
* ravanan/workflow.scm (<scheduler-proc>): Rename constructor to
-scheduler-proc.
(scheduler-proc): New function.
|
|
Build out propagator network eagerly descending into the lowest
subworkflows. This will come in handy later on to build the G-exp
scripts ahead of time before the workflow is run.
* ravanan/workflow.scm (<scheduler-proc>)[cwl]: Rename to
cwl-or-propnet.
[formal-inputs, formal-outputs]: New fields.
* ravanan/workflow.scm (workflow->scheduler-proc): New function.
(workflow-class->propnet): Build out propagator network eagerly.
(workflow-scheduler)[schedule]: Handle eagerly built-out propagator
network <schedule-proc> objects.
(run-workflow): Use workflow->scheduler-proc to convert workflow to
<scheduler-proc> object.
|
|
command-line-tool->propagator was packaging up both CommandLineTool
and Workflow class workflows into <scheduler-proc> objects. It was
named erroneously. We now merge it into step->propagator.
* ravanan/workflow.scm (command-line-tool->propagator): Merge into ...
(workflow-class->propnet)[step->propagator]: ... here.
|
|
We build the script using only the CommandLineTool workflow itself,
and not its inputs. We pass the inputs in through a WORKFLOW_INPUTS
environment variable. To maintain bullet-proof caching, we construct
ravanan store paths by hashing both the script and the inputs
together. This change paves the way to pre-building all scripts ahead
of time.
* ravanan/command-line-tool.scm (<command-line-binding>,
command-line-binding->args): Move to (ravanan work command-line-tool).
(run-command-line-tool): Do not pass inputs to
build-command-line-tool-script. Call step-store-files-directory
instead of script->store-files-directory, step-store-data-file instead
of script->store-data-file, step-store-stdout-file instead of
script->store-stdout-file and step-store-stderr-file instead of
script->store-stderr-file. Put inputs into job state objects. Pass
inputs to the script through the WORKFLOW_INPUTS environment variable.
(capture-command-line-tool-output): Accept inputs. Call
step-store-data-file instead of script->store-data-file and
step-store-files-directory instead of script->store-files-directory.
(copy-input-files-gexp): Delete function.
(build-command-line-tool-script): Do not accept inputs; get them from
the WORKFLOW_INPUTS environment variable.
[coerce-argument]: New function.
[run-command-gexp]: Build command inside the G-expression.
[gexp]{copy-input-files}: New function.
* ravanan/job-state.scm (<single-machine-job-state>)[inputs]: New
field.
* ravanan/job-state.scm (<slurm-job-state>)[inputs]: New field.
* ravanan/job-state.scm (job-state-inputs): New public function.
* ravanan/store.scm: Import (srfi srfi-71), (gcrypt hash) and (guix
build utils).
(sha1-hash-sexp): New function.
(script->store-files-directory, script->store-data-file,
script->store-stdout-file, script->store-stderr-file): Delete
functions.
(step-store-files-directory, step-store-data-file,
step-store-stdout-file, step-store-stderr-file): New public functions.
* ravanan/work/command-line-tool.scm: Import (srfi srfi-9 gnu)
and (ravanan work ui).
* ravanan/workflow.scm (&job-failure)[inputs]: New field.
* ravanan/workflow.scm (workflow-scheduler)[poll]: Put inputs into
&job-failure condition.
[capture-output]: Call step-store-stdout-file instead of
script->store-stdout-file and step-store-stderr-file instead of
script->store-stderr-file. Pass inputs to
capture-command-line-tool-output.
(run-workflow): Call step-store-stdout-file instead of
script->store-stdout-file and step-store-stderr-file instead of
script->store-stderr-file.
|
|
* ravanan/command-line-tool.scm (run-command-line-tool): Define
environment only once.
|
|
* ravanan/store.scm: Import (guix base16) and (guix base32).
(intern-file): Encode store paths using base32 instead of base16.
|
|
* ravanan/work/command-line-tool.scm (sha1-hash-bytes): New public
function.
(sha1-hash): Use sha1-hash-bytes.
|
|
* ravanan/workflow.scm (workflow-scheduler): Bind class variable only
in the else arm of the if condition.
|
|
User errors are caused by incorrect input from the user, rather than
by actual bugs in the code. user-error is named after the similarly
named elisp function.
* bin/ravanan: Import (ravanan work ui).
(%options, invalid-option, main): Use user-error instead of error.
|
|
* ravanan/propnet.scm (<propagator>, <propnet-state>): Add record
printers.
|
|
* ravanan/propnet.scm (propnet?, propagator?, scheduler?,
state+status?): Export.
|
|
* bin/ravanan (print-short-usage): New function.
(main): Print usage when arguments are not matched.
|
|
*
ravanan/command-line-tool.scm (build-command-line-tool-script):[cores]:
Re-indent.
|
|
* ravanan/command-line-tool.scm (copy-input-files-gexp): Make inputs
subdirectory only if it does not exist.
|
