Skip to content

cwltool#MPIRequirement implementation needed for Singularity #2208

Description

@kinow

To use Singularity with MPI we normally use something like mpirun -np N singularity image.sif args

Then MPI takes care to communicate with the daemon and ask it to launch N processes using the Singularity container, for the image.sif with the right args, and passing the rank, MPI world size, etc.

That can be in hybrid or host/bind modes. The syntax of both is similar, but in the bind version we need to pass the --bind or --mount args to add the MPI installation from host inside the container (and have compatible glibc, other libs, maybe some MPI binaries too, etc).

CWLTool fails to run MPI + Singularity in either mode, on my laptop or on HPCs due to the isolation of environment variables and to how it isolates the temporary directory.

Each MPI implementation may use a different set of environment variables. Using --preserve-entire-environment doesn't seem to work with containers, and the command line used in Singularity includes --cleanenv and --contain even when that's used.

The MPI daemon/launcher process will use temporary directories to keep metadata, sockets for IPC, and other files. So when I launch mpirun in the host, what I think is happening is mpirun creates the set of files/variables needed for the MPI processes, mpirun -np N singularity starts separate processes for singularity, but because there's the mount for a different tmpdir, each process launched isn't properly set up for the current group/communicator/etc., and the program crashes.

To confirm that this is happening, I tried this diff against commit

commit dfc950ca3e4c725fea2807b2f71a4a87c87e3ce4 (HEAD -> main, upstream/main, upstream/HEAD)
Author: Michael R. Crusoe <michael.crusoe@gmail.com>
Date:   Tue Feb 24 17:58:52 2026 +0100

    ci: pin setuptools/pip version for "no leftovers" test
diff --git a/cwltool/singularity.py b/cwltool/singularity.py
index 41db3b69..2a5a04c8 100644
--- a/cwltool/singularity.py
+++ b/cwltool/singularity.py
@@ -593,9 +593,9 @@ class SingularityCommandLineJob(ContainerCommandLineJob):
             "singularity",
             "--quiet",
             "run" if (is_apptainer_1_1_or_newer() or is_version_3_10_or_newer()) else "exec",
-            "--contain",
+            # "--contain",
             "--ipc",
-            "--cleanenv",
+            # "--cleanenv",
         ]
         if is_apptainer_1_1_or_newer() or is_version_3_10_or_newer():
             runtime.append("--no-eval")
@@ -624,9 +624,9 @@ class SingularityCommandLineJob(ContainerCommandLineJob):
                 writable=True,
             )
 
-        self.append_volume(
-            runtime, os.path.realpath(self.tmpdir), self.CONTAINER_TMPDIR, writable=True
-        )
+        # self.append_volume(
+        #     runtime, os.path.realpath(self.tmpdir), self.CONTAINER_TMPDIR, writable=True
+        # )
 
         self.add_volumes(
             self.pathmapper,

By not isolating the env vars, and by not creating a separate temporary directory, MPI is able to correctly launch the application.

For the env vars, I think we can add something in the docs that tell users to use preserve-entire or to preserve some envs (like everything MPICH_*). But we need to ensure those are passed to the container with --env or some other way.

I don't know what to do about the temporary directory. I tried hard-coding in cwltool code to a directory I created /tmp/mpi, and launching CWLTool with TMPDIR=/tmp/mpi cwltool .... But that still failed. It might be a bit more difficult to find a fix for this one.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type
    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions