aboutsummaryrefslogtreecommitdiff
AgeCommit message (Collapse)Author
28 hoursstore: Canonicalize inputs JSON before constructing store paths.HEADmainArun Isaac
* ravanan/store.scm (step-store-basename): Canonicalize inputs JSON. * tests/store.scm: New file.
46 hourscommand-line-tool: Pass down inputs directory to recursive calls.Arun Isaac
* ravanan/command-line-tool.scm (build-command-line-tool-script)[copy-input-files]: Pass down inputs directory to recursive calls.
46 hourscommand-line-tool: Do not auto-compile manifest files on inferior.Arun Isaac
The comment was there earlier, but the code was missing! * ravanan/command-line-tool.scm (manifest-file->search-path-sexps): Do not auto-compile manifest files on inferior.
46 hourscommand-line-tool: Do not canonicalize manifest file paths.Arun Isaac
Canonicalization is unnecessary, and causes trouble when the manifest file does not exist. * ravanan/command-line-tool.scm (manifest-file->search-path-sexps, manifest-file->profile-derivation): Do not canonicalize manifest file paths.
47 hourscommand-line-tool: Wrap vectors and alists when embedding in G-exps.Arun Isaac
* ravanan/command-line-tool.scm (build-command-line-tool-script)[vector->gexp, alist->gexp]: New functions. [run-command-gexp]: Use vector->gexp and alist->gexp.
48 hourscommand-line-tool: Present a monadic interface to building scripts.Arun Isaac
* ravanan/command-line-tool.scm (build-gexp-script) (build-command-line-tool-script): Present a monadic interface. * ravanan/workflow.scm (workflow->scheduler-proc): Run monadic value returned by build-command-line-tool-script through the store. (workflow->scheduler-proc, workflow-class->propnet): Update docstring about the guix-store argument.
2 dayscommand-line-tool: Return all search path S-exps as monadic values.Arun Isaac
In the interest of consistency, return search path S-expressions from both manifest-file->search-path-sexps and software-packages->search-path-sexps as monadic values. * ravanan/command-line-tool.scm (software-packages->search-path-sexps): Return monadic value. * ravanan/command-line-tool.scm (build-command-line-tool-script): Use monadic values from software-packages->search-path-sexps.
2 dayscommand-line-tool: Handle manifest monadically.Arun Isaac
The monadic interface allows more of the building to be done in one go when build-gexp-script is invoked. * ravanan/command-line-tool.scm (manifest-file->environment, software-packages->environment, manifest->environment): Delete functions. (inferior-meval-with-store, manifest-file->search-path-sexps, manifest-file->profile-derivation, software-packages->manifest, software-packages->search-path-sexps, software-packages->profile-derivation): New functions. (build-command-line-tool-script): Use manifest-file->search-paths, manifest-file->profile-derivation, software-packages->search-path-sexps and software-packages->profile-derivation instead of manifest-file->environment and software-packages->environment.
2 daysworkflow: Re-use single connection to the Guix daemon.Arun Isaac
Re-using the connection improves performance and is cleaner than passing around the Guix daemon socket specification. * ravanan/command-line-tool.scm (manifest-file->environment, software-packages->environment, manifest->environment, build-command-line-tool-script, build-gexp-script): Work with store connection instead of the Guix daemon socket. * ravanan/workflow.scm: Import (guix store). (workflow->scheduler-proc, workflow-class->propnet): Work with store connection instead of the Guix daemon socket. (build-workflow): Pass a store connection, not the Guix daemon socket, to workflow->scheduler-proc.
2 daysworkflow: Spawn an inferior only once.Arun Isaac
Spawning inferiors is expensive. Do it only once and re-use the connection. * ravanan/command-line-tool.scm (call-with-inferior): Move to (ravanan workflow). (manifest-file->environment, software-packages->environment, build-command-line-tool-script): Accept inferior instead of channels. * ravanan/workflow.scm: Import (guix inferior). (workflow->scheduler-proc, workflow-class->propnet): Accept inferior instead of channels. (build-workflow): New function. (run-workflow): Use build-workflow instead of workflow->scheduler-proc.
2 dayscommand-line-tool: Clarify disabling WorkReuse message.Arun Isaac
* ravanan/command-line-tool.scm (build-command-line-tool-script): Clarify it is "disabling WorkReuse" that is unnecessary.
2 daysworkflow: Build scripts ahead of time.Arun Isaac
* ravanan/command-line-tool.scm (build-command-line-tool-script, find-requirement): Export. (run-command-line-tool): Accept script as argument instead of cwl. Add resource-requirement argument. Remove manifest-file, channels, scratch and guix-daemon-socket arguments. * ravanan/workflow.scm (<scheduler-proc>):[cwl-or-propnet]: Rename to script-or-propnet. [resource-requirement]: New field. * ravanan/workflow.scm (scheduler-proc): Rename cwl-or-propnet argument to script-or-propnet. Add resource-requirement argument. (workflow->scheduler-proc, workflow-class->propnet): Add manifest-file, channels, scratch, store and guix-daemon-socket arguments. (workflow->scheduler-proc): Build script. Capture ResourceRequirement. (workflow-scheduler): Remove manifest-file, channels, scratch and guix-daemon-socket arguments. [schedule]: Run built script instead of CWL. Update call to run-command-line-tool. (run-workflow): Do not pass manifest-file, channels, scratch and guix-daemon-socket arguments to workflow-scheduler. Pass manifest-file, channels, scratch, store and guix-daemon-socket arguments to workflow->scheduler-proc.
2 daysworkflow: Make scatter arguments optional in workflow->scheduler-proc.Arun Isaac
* ravanan/workflow.scm (workflow->scheduler-proc): Make scatter and scatter-method arguments optional.
2 daysworkflow: Introduce optional fields in scheduler-proc constructor.Arun Isaac
* ravanan/workflow.scm (<scheduler-proc>): Rename constructor to -scheduler-proc. (scheduler-proc): New function.
2 daysworkflow: Build out propagator network eagerly.Arun Isaac
Build out propagator network eagerly descending into the lowest subworkflows. This will come in handy later on to build the G-exp scripts ahead of time before the workflow is run. * ravanan/workflow.scm (<scheduler-proc>)[cwl]: Rename to cwl-or-propnet. [formal-inputs, formal-outputs]: New fields. * ravanan/workflow.scm (workflow->scheduler-proc): New function. (workflow-class->propnet): Build out propagator network eagerly. (workflow-scheduler)[schedule]: Handle eagerly built-out propagator network <schedule-proc> objects. (run-workflow): Use workflow->scheduler-proc to convert workflow to <scheduler-proc> object.
2 daysworkflow: Eliminate command-line-tool->propagator.Arun Isaac
command-line-tool->propagator was packaging up both CommandLineTool and Workflow class workflows into <scheduler-proc> objects. It was named erroneously. We now merge it into step->propagator. * ravanan/workflow.scm (command-line-tool->propagator): Merge into ... (workflow-class->propnet)[step->propagator]: ... here.
2 dayscommand-line-tool: Build script without inputs.Arun Isaac
We build the script using only the CommandLineTool workflow itself, and not its inputs. We pass the inputs in through a WORKFLOW_INPUTS environment variable. To maintain bullet-proof caching, we construct ravanan store paths by hashing both the script and the inputs together. This change paves the way to pre-building all scripts ahead of time. * ravanan/command-line-tool.scm (<command-line-binding>, command-line-binding->args): Move to (ravanan work command-line-tool). (run-command-line-tool): Do not pass inputs to build-command-line-tool-script. Call step-store-files-directory instead of script->store-files-directory, step-store-data-file instead of script->store-data-file, step-store-stdout-file instead of script->store-stdout-file and step-store-stderr-file instead of script->store-stderr-file. Put inputs into job state objects. Pass inputs to the script through the WORKFLOW_INPUTS environment variable. (capture-command-line-tool-output): Accept inputs. Call step-store-data-file instead of script->store-data-file and step-store-files-directory instead of script->store-files-directory. (copy-input-files-gexp): Delete function. (build-command-line-tool-script): Do not accept inputs; get them from the WORKFLOW_INPUTS environment variable. [coerce-argument]: New function. [run-command-gexp]: Build command inside the G-expression. [gexp]{copy-input-files}: New function. * ravanan/job-state.scm (<single-machine-job-state>)[inputs]: New field. * ravanan/job-state.scm (<slurm-job-state>)[inputs]: New field. * ravanan/job-state.scm (job-state-inputs): New public function. * ravanan/store.scm: Import (srfi srfi-71), (gcrypt hash) and (guix build utils). (sha1-hash-sexp): New function. (script->store-files-directory, script->store-data-file, script->store-stdout-file, script->store-stderr-file): Delete functions. (step-store-files-directory, step-store-data-file, step-store-stdout-file, step-store-stderr-file): New public functions. * ravanan/work/command-line-tool.scm: Import (srfi srfi-9 gnu) and (ravanan work ui). * ravanan/workflow.scm (&job-failure)[inputs]: New field. * ravanan/workflow.scm (workflow-scheduler)[poll]: Put inputs into &job-failure condition. [capture-output]: Call step-store-stdout-file instead of script->store-stdout-file and step-store-stderr-file instead of script->store-stderr-file. Pass inputs to capture-command-line-tool-output. (run-workflow): Call step-store-stdout-file instead of script->store-stdout-file and step-store-stderr-file instead of script->store-stderr-file.
2 dayscommand-line-tool: Define environment only once.Arun Isaac
* ravanan/command-line-tool.scm (run-command-line-tool): Define environment only once.
2 daysstore: Encode store paths using base32 instead of base16.Arun Isaac
* ravanan/store.scm: Import (guix base16) and (guix base32). (intern-file): Encode store paths using base32 instead of base16.
2 dayswork/command-line-tool: Add sha1-hash-bytes.Arun Isaac
* ravanan/work/command-line-tool.scm (sha1-hash-bytes): New public function. (sha1-hash): Use sha1-hash-bytes.
2 daysworkflow: Bind class variable with reduced scope.Arun Isaac
* ravanan/workflow.scm (workflow-scheduler): Bind class variable only in the else arm of the if condition.
2 daysbin: Use user-error instead of errors to report user errors.Arun Isaac
User errors are caused by incorrect input from the user, rather than by actual bugs in the code. user-error is named after the similarly named elisp function. * bin/ravanan: Import (ravanan work ui). (%options, invalid-option, main): Use user-error instead of error.
2 dayspropnet: Add record printers.Arun Isaac
* ravanan/propnet.scm (<propagator>, <propnet-state>): Add record printers.
2 dayspropnet: Export predicates.Arun Isaac
* ravanan/propnet.scm (propnet?, propagator?, scheduler?, state+status?): Export.
11 daysbin: Print usage when arguments are not matched.Arun Isaac
* bin/ravanan (print-short-usage): New function. (main): Print usage when arguments are not matched.
2025-05-29command-line-tool: Re-indent.Arun Isaac
* ravanan/command-line-tool.scm (build-command-line-tool-script):[cores]: Re-indent.
2025-05-16command-line-tool: Make inputs subdirectory only if it does not exist.Arun Isaac
* ravanan/command-line-tool.scm (copy-input-files-gexp): Make inputs subdirectory only if it does not exist.
2025-05-16command-line-tool: Copy input files into their own subdirectories.Arun Isaac
* ravanan/command-line-tool.scm (copy-input-files-gexp): Copy input files into their own subdirectories.
2025-05-06bin: Import (ravanan work vectors) for vector-map.Arun Isaac
* bin/ravanan: Import (ravanan work vectors).
2025-04-23store: Intern files into a directory of their own.Arun Isaac
This way, user-supplied files and intermediate files generated by steps are stored the same way. Also, basename is preserved and there is no need to strip store hashes from filenames. * ravanan/command-line-tool.scm (copy-input-files-gexp): Use basename instead of store-item-name. * ravanan/store.scm (intern-file): Intern files into a directory of their own. (%store-hash-length, store-item-name): Delete variables. * ravanan/workflow.scm (resolve-inputs): Use basename instead of store-item-name.
2025-04-23command-line-tool: Strip store hash when copying out of the store.Arun Isaac
Without stripping the hashes, the primary and secondary files have different hash prefixes. This interferes with successful command execution. * ravanan/command-line-tool.scm (copy-input-files-gexp): Strip store hash when copying files out of the store into the temporary inputs directory. Update basename, nameroot and nameext accordingly.
2025-04-23store: Update basename, nameroot and nameext when interning files.Arun Isaac
* ravanan/store.scm (intern-file): Update basename, nameroot and nameext.
2025-04-23workflow: Use secondary-path.Arun Isaac
* ravanan/workflow.scm (resolve-inputs)[match-secondary-file-pattern]: Accept secondary-file. Call secondary-path. [check-secondary-files]: Pass secondary-file, instead of just the pattern, to match-secondary-file-pattern.
2025-04-23command-line-tool: Move secondary-path to work module.Arun Isaac
* ravanan/command-line-tool.scm (build-command-line-tool-script): Move to ... * ravanan/work/command-line-tool.scm (secondary-path): ... here.
2025-04-22command-line-tool: Pass secondary-file to secondary-path.Arun Isaac
* ravanan/command-line-tool.scm (build-command-line-tool-script): Pass secondary-file, not pattern, to secondary-path.
2025-04-22command-line-tool: Stage secondary files too.Arun Isaac
* ravanan/command-line-tool.scm (build-command-line-tool-script): Stage secondary files too.
2025-04-22reader: Use canonicalize-file-value.Arun Isaac
* ravanan/reader.scm (normalize-input): Use canonicalize-file-value.
2025-04-22work/command-line-tool: Comment on trusting the checksum.Arun Isaac
* ravanan/work/command-line-tool.scm (canonicalize-file-value): Comment on trusting the checksum.
2025-04-22work/command-line-tool: Canonicalize secondary files too.Arun Isaac
* ravanan/work/command-line-tool.scm (canonicalize-file-value): Canonicalize secondary files too.
2025-04-22work/command-line-tool: Handle absent path and location correctly.Arun Isaac
* ravanan/work/command-line-tool.scm (canonicalize-file-value): Handle absent path and location correctly.
2025-04-22reader: Move location->path to work module.Arun Isaac
* ravanan/reader.scm (location->path): Move to ... * ravanan/work/command-line-tool.scm (location->path): ... here.
2025-04-22command-line-tool: Move canonicalize-file-value to work module.Arun Isaac
* ravanan/command-line-tool.scm (build-command-line-tool-script): Move canonicalize-file-value to ... * ravanan/work/command-line-tool.scm (canonicalize-file-value): ... here.
2025-04-17command-line-tool: Abstract out handling of SecondaryFile pattern.Arun Isaac
* ravanan/command-line-tool.scm (build-command-line-tool-script)[secondary-path]: New function. [capture-secondary-file]: Use secondary-path.
2025-04-17command-line-tool: Abstract out file staging into its own function.Arun Isaac
* ravanan/command-line-tool.scm (build-command-line-tool-script)[stage-file]: New function. [stage-files]: Use stage-file.
2025-04-14store: Intern secondary files.Arun Isaac
* ravanan/store.scm: Import (srfi srfi-26) and (ravanan work vectors). (intern-file): Intern secondary files. (%store-hash-length): New variable. (store-item-name): New public function. * ravanan/workflow.scm (resolve-inputs)[match-secondary-file-pattern]: Strip store hashes before matching secondary file patterns.
2025-04-14dir-locals: Indent call-with-atomic-output-file correctly.Arun Isaac
* .dir-locals.el (scheme-mode): Indent call-with-atomic-output-file correctly.
2025-03-18tests: Fix typo in test.Arun Isaac
* tests/reader.scm ("Normalize array of File arrays type formal output"): Replace normalize-formal-input with normalize-formal-output.
2025-03-18tests: Tolerate only path or location in File type inputs.Arun Isaac
* tests/reader.scm: Import (ice-9 filesystem), (web uri), (ravanan work command-line-tool) and (ravanan work utils). (normalize-input): New variable. ("Normalize inputs with only location", "Normalize inputs with only path"): New tests.
2025-03-18reader: Canonicalize File inputs completely in reader.Arun Isaac
It is the reader's job to completely canonicalize File type inputs. This should not be done later when the workflow is executing. * ravanan/reader.scm: Import (web uri). (normalize-formal-input): Normalize default value. (location->path): New function. (normalize-input): Canonicalize File type input completely. * ravanan/workflow.scm (workflow-scheduler): Update comment about partially specified File objects. (location->path): Delete function. (resolve-inputs): Do not canonicalize File type inputs; they already are.
2025-03-18reader: Spin off input normalization into separate function.Arun Isaac
Separate functions are easier to test. * ravanan/reader.scm (read-inputs): Spin off input normalization into ... (normalize-input): ... new function.
generated by cgit v1.2.3 (git 2.49.0) at 2025-06-28 16:39:27 +0000