Very low GPU utilization (0%) during Chronos-2 LoRA fine-tuning with two GPUs

[Claire-bx]

Hello Chronos team,

I am fine-tuning Chronos-2 with LoRA and noticed that GPU utilization remains 0% throughout training when using two A100 80GB GPUs. I would like to confirm whether this behavior is expected or if I am missing something.

Model Loading

from chronos import BaseChronosPipeline, Chronos2Pipeline

pipeline: Chronos2Pipeline = BaseChronosPipeline.from_pretrained(
    "/home/local_model/chronos-2",
    device_map="cuda",
)

# Explicitly move model to GPU
pipeline.model.to("cuda")

Fine-tuning Code

finetuned_pipeline = pipeline.fit(
    inputs=train_inputs,
    validation_inputs=val_inputs,
    context_length=history_len,
    prediction_length=prediction_length,
    learning_rate=learning_rate,
    num_steps=num_steps,
    batch_size=batch_size,
    logging_steps=100,
    finetune_mode="lora",
    dataloader_num_workers=16,
    bf16=True,
    tf32=True,
    output_dir="/home/fineune_model",
    finetuned_ckpt_name="load",
    do_eval=True,
    eval_strategy="epoch",
    save_strategy="epoch",
    load_best_model_at_end=True,
    metric_for_best_model="eval_loss",
    greater_is_better=False,
    per_device_train_batch_size=batch_size,
    per_device_eval_batch_size=batch_size,
    save_total_limit=1,
    ddp_find_unused_parameters=False,
)

Observed GPU Behavior (nvidia-smi)

GPU Name	Memory-Usage	GPU-Util
A100 80GB (GPU 0)	~1.4 GB	0%
A100 80GB (GPU 1)	~1.4 GB	0%

GPU utilization stays at 0% during the entire training process

[shchur]

Which version of the package are you using? Older versions of the package might be falling back to DataParallel when running on a multi-GPU setup, which dramatically slows down fine-tuning.

You can either set CUDA_VISIBLE_DEVICES=1 before running the script, or upgrade the chronos-forecasting package to the latest version and run fine-tuning with model.fit(..., disable_data_parallel=True).

[abdulfatir]

@Claire-bx can you install nvitop and check if the utilization live? Does it not move at all? Also, how are you using the 2 GPUs? Are you using DDP? How did you set it up exactly?

[wwfcnu]

想问一下，怎么根据我的数据去估算微调需要的 num_steps