RuntimeError: Error building extension 'lib3dgrt_cc' on Ubuntu 24.04, CUDA : 11.8, gcc 11

[yunusstalha]

Environment is installed correctly. Running a training code generates the following output.
GPU is NVIDIA RTX A5000

[15:15:33] [INFO] Git hash: ab33fd4058d096f7e18061821680c031831c6769                                                                                                                                                                                                                      logger.py:67
           [INFO] Compiling native code..                                                                                                                                                                                                                                                 logger.py:67
─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────── Load Datasets ────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────
[15:15:39] [INFO] 📷 EXIF: No exposure data found in 1164 images                                                                                                                                                                                                                          logger.py:67
Load Dataset (train) ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 100% :: 🕒 Elapsed 0:00:00
Load Dataset (val) ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 100% :: 🕒 Elapsed 0:00:00
miniconda3/envs/3dgrut/lib/python3.11/site-packages/torch/utils/data/dataloader.py:557: UserWarning: This DataLoader will create 24 worker processes in total. Our suggested max number of worker in current system is 20, which is smaller than what this DataLoader is going to create. Please be aware that excessive worker creation might get DataLoader running slow or even freeze, lower the worker number to avoid potential slowness/freeze if necessary.
  warnings.warn(_create_warning_msg(
────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────── Initialize Model ──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────
Using .cache/torch_extensions/py311_cu118 as PyTorch extensions root...
Detected CUDA files, patching ldflags
Emitting ninja build file /.cache/torch_extensions/py311_cu118/lib3dgrt_cc/build.ninja...
Building extension module lib3dgrt_cc...
Allowing ninja to set a default number of workers... (overridable by setting the environment variable MAX_JOBS=N)
[1/3] /usr/bin/g++-11 -MMD -MF optixTracer.o.d -DTORCH_EXTENSION_NAME=lib3dgrt_cc -DTORCH_API_INCLUDE_EXTENSION_H -DPYBIND11_COMPILER_TYPE=\"_gcc\" -DPYBIND11_STDLIB=\"_libstdcpp\" -DPYBIND11_BUILD_ABI=\"_cxxabi1011\" -I/miniconda3/envs/3dgrut/targets/x86_64-linux/include -I/git/3dgrut/threedgrt_tracer/include -I/git/3dgrut/threedgrt_tracer/dependencies/optix-dev/include -isystem /miniconda3/envs/3dgrut/lib/python3.11/site-packages/torch/include -isystem /miniconda3/envs/3dgrut/lib/python3.11/site-packages/torch/include/torch/csrc/api/include -isystem /miniconda3/envs/3dgrut/lib/python3.11/site-packages/torch/include/TH -isystem /miniconda3/envs/3dgrut/lib/python3.11/site-packages/torch/include/THC -isystem /miniconda3/envs/3dgrut/include -isystem miniconda3/envs/3dgrut/include/python3.11 -D_GLIBCXX_USE_CXX11_ABI=0 -fPIC -std=c++17 -DNVDR_TORCH -c git/3dgrut/threedgrt_tracer/src/optixTracer.cpp -o optixTracer.o 
[2/3] /miniconda3/envs/3dgrut/bin/nvcc  -ccbin /usr/bin/gcc-11 -DTORCH_EXTENSION_NAME=lib3dgrt_cc -DTORCH_API_INCLUDE_EXTENSION_H -DPYBIND11_COMPILER_TYPE=\"_gcc\" -DPYBIND11_STDLIB=\"_libstdcpp\" -DPYBIND11_BUILD_ABI=\"_cxxabi1011\" -Iminiconda3/envs/3dgrut/targets/x86_64-linux/include -I/git/3dgrut/threedgrt_tracer/include -I/git/3dgrut/threedgrt_tracer/dependencies/optix-dev/include -isystem /miniconda3/envs/3dgrut/lib/python3.11/site-packages/torch/include -isystem /miniconda3/envs/3dgrut/lib/python3.11/site-packages/torch/include/torch/csrc/api/include -isystem/miniconda3/envs/3dgrut/lib/python3.11/site-packages/torch/include/TH -isystem /miniconda3/envs/3dgrut/lib/python3.11/site-packages/torch/include/THC -isystem /miniconda3/envs/3dgrut/include -isystem /miniconda3/envs/3dgrut/include/python3.11 -D_GLIBCXX_USE_CXX11_ABI=0 -D__CUDA_NO_HALF_OPERATORS__ -D__CUDA_NO_HALF_CONVERSIONS__ -D__CUDA_NO_BFLOAT16_CONVERSIONS__ -D__CUDA_NO_HALF2_OPERATORS__ --expt-relaxed-constexpr -gencode=arch=compute_70,code=sm_70 -gencode=arch=compute_75,code=sm_75 -gencode=arch=compute_80,code=sm_80 -gencode=arch=compute_86,code=sm_86 -gencode=arch=compute_90,code=compute_90 -gencode=arch=compute_90,code=sm_90 --compiler-options '-fPIC' -DNVDR_TORCH -std=c++17 --extended-lambda --expt-relaxed-constexpr -Xcompiler=-fno-strict-aliasing -c /git/3dgrut/threedgrt_tracer/src/particlePrimitives.cu -o particlePrimitives.cuda.o 
FAILED: [code=137] particlePrimitives.cuda.o 
/miniconda3/envs/3dgrut/bin/nvcc  -ccbin /usr/bin/gcc-11 -DTORCH_EXTENSION_NAME=lib3dgrt_cc -DTORCH_API_INCLUDE_EXTENSION_H -DPYBIND11_COMPILER_TYPE=\"_gcc\" -DPYBIND11_STDLIB=\"_libstdcpp\" -DPYBIND11_BUILD_ABI=\"_cxxabi1011\" -I/miniconda3/envs/3dgrut/targets/x86_64-linux/include -I/git/3dgrut/threedgrt_tracer/include -I/git/3dgrut/threedgrt_tracer/dependencies/optix-dev/include -isystem /miniconda3/envs/3dgrut/lib/python3.11/site-packages/torch/include -isystemminiconda3/envs/3dgrut/lib/python3.11/site-packages/torch/include/torch/csrc/api/include -isystem /miniconda3/envs/3dgrut/lib/python3.11/site-packages/torch/include/TH -isystem miniconda3/envs/3dgrut/lib/python3.11/site-packages/torch/include/THC -isystem miniconda3/envs/3dgrut/include -isystem /miniconda3/envs/3dgrut/include/python3.11 -D_GLIBCXX_USE_CXX11_ABI=0 -D__CUDA_NO_HALF_OPERATORS__ -D__CUDA_NO_HALF_CONVERSIONS__ -D__CUDA_NO_BFLOAT16_CONVERSIONS__ -D__CUDA_NO_HALF2_OPERATORS__ --expt-relaxed-constexpr -gencode=arch=compute_70,code=sm_70 -gencode=arch=compute_75,code=sm_75 -gencode=arch=compute_80,code=sm_80 -gencode=arch=compute_86,code=sm_86 -gencode=arch=compute_90,code=compute_90 -gencode=arch=compute_90,code=sm_90 --compiler-options '-fPIC' -DNVDR_TORCH -std=c++17 --extended-lambda --expt-relaxed-constexpr -Xcompiler=-fno-strict-aliasing -c /git/3dgrut/threedgrt_tracer/src/particlePrimitives.cu -o particlePrimitives.cuda.o 
Killed
ninja: build stopped: subcommand failed.
Error executing job with overrides: ['path=data/dark', 'out_dir=runs']
Traceback (most recent call last):
  File "/git/3dgrut/threedgrt_tracer/tracer.py", line 39, in load_3dgrt_plugin
    from . import lib3dgrt_cc as tdgrt  # type: ignore
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ImportError: cannot import name 'lib3dgrt_cc' from 'threedgrt_tracer' (/git/3dgrut/threedgrt_tracer/__init__.py)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/miniconda3/envs/3dgrut/lib/python3.11/site-packages/torch/utils/cpp_extension.py", line 2100, in _run_ninja_build
    subprocess.run(
  File "/miniconda3/envs/3dgrut/lib/python3.11/subprocess.py", line 571, in run
    raise CalledProcessError(retcode, process.args,
subprocess.CalledProcessError: Command '['ninja', '-v']' returned non-zero exit status 137.

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/git/3dgrut/train.py", line 50, in main
    trainer = Trainer3DGRUT(conf)
              ^^^^^^^^^^^^^^^^^^^
  File "/miniconda3/envs/3dgrut/lib/python3.11/contextlib.py", line 81, in inner
    return func(*args, **kwds)
           ^^^^^^^^^^^^^^^^^^^
  File "/git/3dgrut/threedgrut/trainer.py", line 129, in __init__
    self.init_model(conf, self.scene_extent)
  File "/git/3dgrut/threedgrut/trainer.py", line 189, in init_model
    self.model = MixtureOfGaussians(conf, scene_extent=scene_extent)
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/git/3dgrut/threedgrut/model/model.py", line 196, in __init__
    self.renderer = threedgrt_tracer.Tracer(conf)
                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/git/3dgrut/threedgrt_tracer/tracer.py", line 177, in __init__
    load_3dgrt_plugin(conf)
  File "/git/3dgrut/threedgrt_tracer/tracer.py", line 43, in load_3dgrt_plugin
    tdgrt = setup_3dgrt(conf)
            ^^^^^^^^^^^^^^^^^
  File "/git/3dgrut/threedgrt_tracer/setup_3dgrt.py", line 81, in setup_3dgrt
    return jit.load(
           ^^^^^^^^^
  File "/git/3dgrut/threedgrut/utils/jit.py", line 111, in load
    module = torch.utils.cpp_extension.load(
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/miniconda3/envs/3dgrut/lib/python3.11/site-packages/torch/utils/cpp_extension.py", line 1308, in load
    return _jit_compile(
           ^^^^^^^^^^^^^
  File "/miniconda3/envs/3dgrut/lib/python3.11/site-packages/torch/utils/cpp_extension.py", line 1710, in _jit_compile
    _write_ninja_file_and_build_library(
  File "/miniconda3/envs/3dgrut/lib/python3.11/site-packages/torch/utils/cpp_extension.py", line 1823, in _write_ninja_file_and_build_library
    _run_ninja_build(
  File "/miniconda3/envs/3dgrut/lib/python3.11/site-packages/torch/utils/cpp_extension.py", line 2116, in _run_ninja_build
    raise RuntimeError(message) from e
RuntimeError: Error building extension 'lib3dgrt_cc'

Set the environment variable HYDRA_FULL_ERROR=1 for a complete stack trace.

[yunusstalha]

export TORCH_CUDA_ARCH_LIST="8.6";
solves training problem for my case. However, now if I try to use gui I get the following error

[15:43:27] [INFO] 🤸 Initiating new 3dgrut training..                                                                                 logger.py:67
           [INFO] Found 277374 colmap points                                                                                          logger.py:67
[15:43:28] [INFO] 🔆 Using Adam optimizer                                                                                             logger.py:67
           [INFO] 🔆 Using Adam optimizer                                                                                             logger.py:67
           [INFO] 📊 Training logs & will be saved to: runs/flat1-1004_154328                                                         logger.py:67
Using /.cache/torch_extensions/py311_cu118 as PyTorch extensions root...
Detected CUDA files, patching ldflags
Emitting ninja build file /.cache/torch_extensions/py311_cu118/lib3dgrut_gui_cc/build.ninja...
Building extension module lib3dgrut_gui_cc...
Using envvar MAX_JOBS (1) as the number of workers...
[1/2] /usr/bin/g++-11 -MMD -MF bindings.o.d -DTORCH_EXTENSION_NAME=lib3dgrut_gui_cc -DTORCH_API_INCLUDE_EXTENSION_H -DPYBIND11_COMPILER_TYPE=\"_gcc\" -DPYBIND11_STDLIB=\"_libstdcpp\" -DPYBIND11_BUILD_ABI=\"_cxxabi1011\" -I/miniconda3/envs/3dgrut/targets/x86_64-linux/include -I/git/3dgrut/threedgrut/gui/include -isystem /miniconda3/envs/3dgrut/lib/python3.11/site-packages/torch/include -isystem miniconda3/envs/3dgrut/lib/python3.11/site-packages/torch/include/torch/csrc/api/include -isystem miniconda3/envs/3dgrut/lib/python3.11/site-packages/torch/include/TH -isystem /miniconda3/envs/3dgrut/lib/python3.11/site-packages/torch/include/THC -isystem /miniconda3/envs/3dgrut/include -isystem /miniconda3/envs/3dgrut/include/python3.11 -D_GLIBCXX_USE_CXX11_ABI=0 -fPIC -std=c++17 -DNVDR_TORCH -DUSE_CUGL_INTEROP=1 -c /git/3dgrut/threedgrut/gui/bindings.cpp -o bindings.o 
FAILED: [code=1] bindings.o 
/usr/bin/g++-11 -MMD -MF bindings.o.d -DTORCH_EXTENSION_NAME=lib3dgrut_gui_cc -DTORCH_API_INCLUDE_EXTENSION_H -DPYBIND11_COMPILER_TYPE=\"_gcc\" -DPYBIND11_STDLIB=\"_libstdcpp\" -DPYBIND11_BUILD_ABI=\"_cxxabi1011\" -I/miniconda3/envs/3dgrut/targets/x86_64-linux/include -Igit/3dgrut/threedgrut/gui/include -isystem /miniconda3/envs/3dgrut/lib/python3.11/site-packages/torch/include -isystem /miniconda3/envs/3dgrut/lib/python3.11/site-packages/torch/include/torch/csrc/api/include -isystem /miniconda3/envs/3dgrut/lib/python3.11/site-packages/torch/include/TH -isystem //miniconda3/envs/3dgrut/lib/python3.11/site-packages/torch/include/THC -isystem /miniconda3/envs/3dgrut/include -isystem /miniconda3/envs/3dgrut/include/python3.11 -D_GLIBCXX_USE_CXX11_ABI=0 -fPIC -std=c++17 -DNVDR_TORCH -DUSE_CUGL_INTEROP=1 -c /git/3dgrut/threedgrut/gui/bindings.cpp -o bindings.o 
In file included from /git/3dgrut/threedgrut/gui/include/gl_interop.h:22,
                 from /git/3dgrut/threedgrut/gui/bindings.cpp:31:
/miniconda3/envs/3dgrut/include/cuda_gl_interop.h:66:10: fatal error: GL/gl.h: No such file or directory
   66 | #include <GL/gl.h>
      |          ^~~~~~~~~
compilation terminated.
ninja: build stopped: subcommand failed.
Error executing job with overrides: ['path=data/flat1', 'out_dir=runs']
Traceback (most recent call last):
  File "/git/3dgrut/threedgrut/gui/ps_extension.py", line 60, in initialize_cugl_interop
    import cuda
ModuleNotFoundError: No module named 'cuda'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/git/3dgrut/threedgrut/gui/ps_extension.py", line 27, in load_3dgrut_gui_plugin
    from . import lib3dgrut_gui_cc as tdgui  # type: ignore
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ImportError: cannot import name 'lib3dgrut_gui_cc' from 'threedgrut.gui' (/git/3dgrut/threedgrut/gui/__init__.py)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/miniconda3/envs/3dgrut/lib/python3.11/site-packages/torch/utils/cpp_extension.py", line 2100, in _run_ninja_build
    subprocess.run(
  File "/miniconda3/envs/3dgrut/lib/python3.11/subprocess.py", line 571, in run
    raise CalledProcessError(retcode, process.args,
subprocess.CalledProcessError: Command '['ninja', '-v', '-j', '1']' returned non-zero exit status 1.

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/git/3dgrut/train.py", line 50, in main
    trainer = Trainer3DGRUT(conf)
              ^^^^^^^^^^^^^^^^^^^
  File "/miniconda3/envs/3dgrut/lib/python3.11/contextlib.py", line 81, in inner
    return func(*args, **kwds)
           ^^^^^^^^^^^^^^^^^^^
  File "/git/3dgrut/threedgrut/trainer.py", line 136, in __init__
    self.init_gui(conf, self.model, self.train_dataset, self.val_dataset, self.scene_bbox)
  File "/git/3dgrut/threedgrut/trainer.py", line 333, in init_gui
    gui = GUI(conf, model, train_dataset, val_dataset, scene_bbox)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/git/3dgrut/threedgrut/utils/gui.py", line 43, in __init__
    initialize_cugl_interop()
  File "/git/3dgrut/threedgrut/gui/ps_extension.py", line 66, in initialize_cugl_interop
    set_custom_cugl_bindings()  # polyscope is available without cupy / cuda-python, use custom bindings
    ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/git/3dgrut/threedgrut/gui/ps_extension.py", line 39, in set_custom_cugl_bindings
    load_3dgrut_gui_plugin()
  File "/git/3dgrut/threedgrut/gui/ps_extension.py", line 31, in load_3dgrut_gui_plugin
    tdgui = setup_gui()
            ^^^^^^^^^^^
  File "/git/3dgrut/threedgrut/gui/setup_gui.py", line 38, in setup_gui
    return jit.load(
           ^^^^^^^^^
  File "/git/3dgrut/threedgrut/utils/jit.py", line 111, in load
    module = torch.utils.cpp_extension.load(
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/miniconda3/envs/3dgrut/lib/python3.11/site-packages/torch/utils/cpp_extension.py", line 1308, in load
    return _jit_compile(
           ^^^^^^^^^^^^^
  File "/miniconda3/envs/3dgrut/lib/python3.11/site-packages/torch/utils/cpp_extension.py", line 1710, in _jit_compile
    _write_ninja_file_and_build_library(
  File "/miniconda3/envs/3dgrut/lib/python3.11/site-packages/torch/utils/cpp_extension.py", line 1823, in _write_ninja_file_and_build_library
    _run_ninja_build(
  File "/miniconda3/envs/3dgrut/lib/python3.11/site-packages/torch/utils/cpp_extension.py", line 2116, in _run_ninja_build
    raise RuntimeError(message) from e
RuntimeError: Error building extension 'lib3dgrut_gui_cc'

Set the environment variable HYDRA_FULL_ERROR=1 for a complete stack trace.

[wilsonCernWq]

There might be two separate issues in this thread:

1. Initial training failure (exit code 137 = OOM during nvcc)
The compiler was killed by the OS because building all CUDA architectures
simultaneously exhausted system memory. Fix:
export TORCH_CUDA_ARCH_LIST="8.6" # match your GPU's compute capability
Then re-run training. You can find your GPU's SM version with:
nvidia-smi --query-gpu=compute_cap --format=csv,noheader

2. GUI failure: GL/gl.h: No such file or directory
The OpenGL development headers are missing. Install them with:
sudo apt install libglu1-mesa-dev libgl-dev

Then retry launching the playground.