- Parse key, value pairs from the command line properly
- Improved safety of module load error detection
- Parse multi-value arguments from the command line properly
- Added check flag to workflow construction from YAML
- Improved docs for job submission
- Improved cleanup on exit
- Added JSON output of all interfaces for a given workflow
- Improved maize --list to show nodes sorted by tags
- Added key-based telemetry authentication
- Allowed typecheck to succeed for certain subtypes (e.g. int-float)
- Removed
max_cpusandmax_gpusparameters - Removed ability to use shift operators to connect nodes
- More robust telemetry handling
- Added telemetry and usage statistics
- Added timeout function to utilities
- Additional logging for extra argument parsing
- Retry job submissions on IO errors
- Fixed GPU not listing all processes correctly
- Fixed run_multi spawning too many jobs when running locally
- Added basic citation and reference system
- Various documentation fixes
- Added node tagging system
- Removed excessive debug spam for post-run directory cleanup
- Set env variables before module loading
- Changed PSI/J logging to debug
- Fixed subgraphs not providing correct serialized summaries
- Fixed serialized_summary not including parent classes (fixes #43)
serialized_summary()now provides correct type information (fixes #44)
- Fixed required parameters defined in config not being recognized
- Changed directory setup to use unique IDs instead of incrementing to avoid race conditions
- Fixed incorrect working directory being used in some cases
- Command batching option for
run_multi
- Improved GPU status querying
- Fixed excessive writing of temporary job output files
- Pinned Numpy to <2.0 for now
- Workflow submission improvements
- Added more detailed job logging
- Fix for excessive queue polling on workflow submission
- Some minor SonarQube fixes
- Workflow submission functionality
- Added
wait_for_allcommand to wait for multiple submitted workflows - Added
gpu_infoutility to determine if it's safe to launch a command on GPU
- Parameters with a default are now automatically considered optional
- Modified JobHandler to cancel jobs on interrupt (#36)
- Multiple async job submission attempts
- Ensure that iterables of paths are correctly handled
- Various typing fixes
parallelcan now run with only constant inputs
- All node superclasses are now searched for type information
- Fix for incorrect generic type updates
- Fixed inconsistencies between
flow.scratchandflow.config.scratch
- Added logger flushing on shutdown
- Slightly improved error message for duplicate nodes
- Added extra batch job logging
- Added version flag
- Unset workflow-level parameters no longer raise (fixes #22)
- Writing to yaml now encodes paths as strings
- Fixed sonarqube bugs
- Fixed off-by-one in job completion logging
- Added active flag to all nodes to enable instant shutdown
- Improved graph connection logging
- Added
MultiParameterhook functionality to allow complex parameter mappings
- As a result of the hook functionality to
MultiParameter, this class is now a double generic. If you use it in your code directly (unlikely), you will need to update the type signature.
- Fixed wrong order in shell variable expansion for config command spec
- Fixed
MergeListsshutting down prematurely when an input closes while others still have data
- Added
Choiceplumbing node - Added
ContentValidator
- Added type information to serialized output
- Fixed ordering for serialized output
- Casting for scratch path spec
- Added option to specify scratch at workflow level
- Install package-based config if available
- Added
IndexDistribute - Added
IntegerMap - Added
FileBuffer
- Improved typing for dict utility functions
- Improved visualization colors
- Improved port activity determination for Multiplex
- Cleaned up workflow serialization
- Fixed typechecking of files passed as strings
- Fixed paths passed to inputs as parameters in a separate JSON failing to be cast
- Allowed mapping of inputs in serialized workflows
- Node list output is now sorted alphabetically
- Added diagrams to node documentation
- Added multiple new plumbing nodes (
TimeDistribute,MergeLists,CopyEveryNIter) - Node preparation is now cached, avoiding multiple dependency lookups
FileParameterwill now attempt casting strings to paths- Allowed caching in
MultiInput
- Job queues will now be kept saturated
- Deprecated static
MultiPorts - Environment variables in interpreter specification are now expanded
- Split
TestRig.setup_runfor explicit use with variable outputs
- Fixed incorrect job submission counts
- Fixed typing issues in
TestRig - Added proper shutdown for
Multiplex
- Interpreter - script pairs can now be non-path commands
- Added option to use list of paths for FileParameters
- Updated guide + dev instructions
- Added package directory as a search path for global config
- Made class tags private
- Added send and receive hook support
- Added component tagging option
- Config dependencies are now converted to absolute paths
- Removed init files causing problems with contrib
- Refactored execution to use correct logging
- Expanded test coverage
- Fix for
_preparecalls with missing interpreter - Fix for premature channel flush when handling converging data streams
- Added queue option to
JobResourceConfig - Added option to explicitly prefer batch submission
- Warning when receiving multiple times from the same port without looping
- Added warning for single char custom batch attributes
- Job submission will now only submit n_jobs if larger than max_jobs
- Improved file validation, will now wait for files until timeout
- Changed handling of flags to explicit
--flag/--no-flag prepare()no longer requires a call to the parent method
- Fix for receive not recognising optional unconnected ports
- Fixed looped nodes not handling cached input files properly
- Fix for
Workflow.from_dictnot recognizing input setting - More robust batch job submissions
- Fixed occassional deadlocks under high logging loads
- Fixed
Returnnodes potentially freezing looped workflows
- Set parameters are logged at workflow start
- Added asynchronous command execution
- It is now possible to map free inputs on the workflow level
- Added checks for common graph construction problems
- Added support for CUDA MPS to run multiple processes on one GPU
- Set default batch polling interval to 120s
- Added functionality to skip node execution if all parameters are unset / optional
Voidcan now take any number of inputs- Dynamic workflow creation is now possible using
expose - Added
working_diroption torun_command - Improved workflow status reporting
- Custom job attributes are now formatted correctly based on batch system
- Status updates now show full path for nodes in subgraphs with duplicate names
- Fixed missing cleanup when using relative scratch directory
- Avoid error when specifying duplicate
loop=Trueparameter - Fixed
typechecknot handling dictionaries properly - Fixed
common_parentnot breaking after divergence - Fixed looped nodes not sending the correct file when dealing with identical names
- Temporary fix for paths not being parsed from YAML files
- Added command inputs to node
run_multimethod FileChannelcan now send and receive dicts of files
- Changed
exclusive_usefor batch submission toFalseby default - Changed required Python to 3.10 to avoid odd beartype behaviour
- Documentation cleanup
- Fixed missing documentation propagation for inherited nodes
- Fixed
MultiPortnot being mapped properly in subgraphs - Removed weird error code check
- Added missing
defaultproperty to MultiInput - Fixed misbehaving subgraph looping test
- Refactored looping system, only way to loop a node now is to use
loop=True - Removed dynamic interface building
- Allowed setting walltime in per-job config
- Lowered logging noise
- Looping is now inherited from subgraphs
- Interface mapping sets attribute by default
- Fixed error in building docs
- Fixed validation failures not showing command output
- Fixed tests to match parameterization requirement
- Fix for incorrect walltime parsing for batch submission
- Throw a proper error when mapping ports with existing names
- More verbose message for missing connection
- Added
--parameterscommandline option to override workflow parameters using a JSON file - Added timeout option to
run_command - Added simple multiple file I/O nodes
- Unparameterized generic nodes will now cause graph construction to fail
FileParameterwill now cast strings toPathobjects when setting- Maize is now a proper namespace package
- Fixed cascading generic nodes sometimes not passing files correctly
- Fixed overeager parameter checks for serialized workflows
- Fixed bug preventing
run_multifrom running withoutbatch_options
- Updated dependencies
- Added contribution guidelines
- Prepared for initial public release
- Added mode option to
MultiPort - Added
custom_attributesoption for batch jobs
- Better batch submission tests
- Batch submission now only requires
batch_options
- Fixed resource management issue
- Fixed file copy issues with the
LoadFilenode - Fixed off-by-one in finished job count
- Various improvements to
parallel
- Added pre-execution option for
run_command - Allow setting default parameters in global config
- Added interface / component serialization
Inputcan now also act as aParameterInputcan now cache its data- Overhauled
FileChannel, now supports lists of files
- Improved type checking for paths
- Fix for PSI/J execution freezing when using
fork - Resolved occasional node looping issues
SaveFilenow handles directories properly- Inherited nodes now set datatypes properly
- Better missing graphviz information
- Improved working directory cleanup
LoadFilenow never moves filesrun_commandnow parses whitespace / strings correctly- Missing config warnings
- Can now submit jobs to arbitrary resource manager systems (SLURM, PBS, etc)
run_commandnow acceptsstdincommand input (Thanks Marco)- Added tool to convert from functions to nodes
- Added experimental node parallelization macro
- Added utility nodes for batching data, with example workflow
- Added
Barriernode for conditional sending,Yesas an equivalent to the Unix command - Workflow visualization improvements (depth limit, node status)
- All execution is now performed with Exaworks PSI/J (new
psij-pythondependency) - Dynamic typechecking now uses
beartype - Parameters with no default value will cause an error if not set to optional
- Status summaries now show approximate number of items in channel
- Channel size can be set globally for the whole workflow
- Many fixes in dynamic typechecking
- More informative error messages
- Fixed a race condition during certain executions
- Fixed an issue where channel data would get flushed in long-running workflows
- Fixed issues relating to Graphviz
- Lowered chances of zombie / orphan processes
- Fixed issue where the config would not get read properly
Initial release.