Workdir caching WITHOUT output caching?

[reesekneeland]

Hello!

I am curious if there is a way to utilize the workdir caching functionality to cache the source code of a python operation (and execute in the same thread without a cluster) without caching the output of that python operation. I want to use this to make my code for my outermost execution layer robust to working code changes, while the inner operations of my code will have their own tasks configured to cache their outputs and utilize appropriate resources for each individual task. My outermost execution flow changes almost every time so it doesn't make much sense to cache this, and will just create clutter if left enabled.

When I try to implement it using the following configurations (with the "folder" operation specified in the WorkDir config but not in the outer "infra" config, I get the error below:

infra = {
        "version" : "1",
        "workdir" : {
            "folder": cache_path,
            "copied" : [filename, "alljoined"]
            },
        "folder": None, #cfg.paths.exca_cache_path,
        "cluster": None
 }

Produces the following error when executed.

Traceback (most recent call last):
  File "execute_exca.py", line 52, in <module>
    task = CachedTask(cfg=cfg, infra=infra)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "<env>/lib/python3.12/site-packages/pydantic/main.py", line 253, in __init__
    validated_self = self.__pydantic_validator__.validate_python(data, self_instance=self)
                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
pydantic_core._pydantic_core.ValidationError: 1 validation error for CachedTask
infra
  Value error, Workdir requires a folder [type=value_error, input_value={'version': '1', 'workdir...: None, 'cluster': None}, input_type=dict]
    For further information visit https://errors.pydantic.dev/2.11/v/value_error

[jrapin]

The fact that this was not allowed was not intended, I started to fix it with #100 however this raises a few questions (also related to #95

1. if there is no folder, then one cannot use slurm / local process
2. if we do not use slurm / local process, then the code will not be reloaded from the copy
   -> I'm afraid that people will not realize that the code that is loaded is not necessarily the copy, even though it's not supposed to load changes if we do not change process...

I'd tend to say that it's up to the user to decide what to do, but as a user i would find it hard to figure when the copy is useful or not, but I am not sure how to make it clear... any idea? what is your use case ? I expect one main task to trigger sub-computation on slurm as stated in #95?

[reesekneeland]

Yes my use case is the one stated in #95, I want my outer task to only facilitate the caching of the source code so it is uninterrupted by local changes while the inner tasks handle output caching and resource management.

My suggestion would be to add a new "mode" option to the TaskInfra class called something like "run" (instead of "cached'), which still accepts the folder argument (allowing it to be used for the WorkDir and cluster functionality), and just disables the output caching functionality. In this mode we won't be making the cached file structure with a hashed set of code operations, but rather just saving the workdir cache directly in the folder (if that function is enabled), and saving slurm logs in a subfolder of the slurm job id (and perhaps a PID or timestamp if running in a local subprocess).

Because of 2. above, we probably want to enforce cluster != None when using the workdir functionality, to avoid cases of confusion.

[reesekneeland]

It's also occurring to me that given 1. above, there is no way to even run processes in slurm without caching the output? Correct me if I am wrong, but if that is the case, the design I discussed above would also help enable this case.

[jrapin]

Because of 2. above, we probably want to enforce cluster != None when using the workdir functionality, to avoid cases of confusion.

even cluster debug would not reload stuff either, and cluster local can be confusing as it spawns a process (makes debugging more complicated for instance)... there are many special cases to handle so not sure what is actually best, I'm considering cluster=None can make sense as workdir does not only make a copy but also changes the current working directly (which does work with None) (+ in any case, with cluster = None the code you execute is necessarily the current one anyway)

It's also occurring to me that given 1. above, there is no way to even run processes in slurm without caching the output?

currently no, then again it's feasible (cf SubmitInfra in task.py)

My suggestion would be to add a new "mode" option to the TaskInfra class called something like "run"

could work, but probably not completely straightforward, I'll need to give more thoughts to it

[reesekneeland]

Hmmm I see what you mean about the workdir, I think running with cluster=local will be a requirement for the use case I am envisioning then, the separate process I don't think will be an issue as I understand it, as any resource allocating TaskInfra tasks or MapInfra tasks are nested inside of this with cluster=slurm, and so the external process shouldn't be duplicated.

Thanks for taking a look at this!

[reesekneeland]

It's also occurring to me that given 1. above, there is no way to even run processes in slurm without caching the output?

currently no, then again it's feasible (cf SubmitInfra in task.py)

As I get deeper into playing with exca, it is becoming clearer to me that this is a fundamental requirement for my use case in the way my architecture is currently designed, there are data processing operations in my pipeline that:

• have very high memory requirements, and so need to be executed in slurm on a compute node,
• return data that is hundreds of gigabytes in size, making it very impractical to cache,
• Take only a few seconds to run, meaning the value of caching them is small.

When you say this is feasible, can you provide more information on what part of the codebase needs to be extended to provide a "don't cache the output" flag to infra tasks? I am happy to work on this and make a PR but would need some pointers as to where to start.

I may need to just redesign my system spec around this, as my understanding of submitit is that the only way we can return a value from a slurm process to the user in python is by saving and loading it on the disk, even if we were to clean up this "cached" output, the file I/O operations would be pretty costly, and it would probably make more sense for me to not split these processes out into a separate slurm process.

[jrapin]

What is the value of having an infra on your function then?
If you need to compute it on slurm and recover it in the main process, you need caching in some way anyway ("submitit" that does the scheduling on slurm uses pickles, so you'd get GBs of pickles)

One option would be to compute it on slurm and use it on slurm, so you'd end up using the infra on some larger function