Enabling optimizer_cpu_offload in megatron lora causes CUDA IMA

[edgeinfinity1]

Checklist / 检查清单

• I have searched existing issues, and this is a new bug report. / 我已经搜索过现有的 issues，确认这是一个新的 bug report。

Bug Description / Bug 描述

我在尝试使用megatron微调模型时, 启用optimizer_cpu_offload会导致illegal memory access:

[rank9]: Traceback (most recent call last):
[rank9]:   File "/root/swift_megatron/lib/python3.11/site-packages/megatron/core/transformer/spec_utils.py", line 119, in build_module
[rank9]:     return module(
[rank9]:            ^^^^^^^
[rank9]:   File "/root/swift_megatron/lib/python3.11/site-packages/megatron/core/extensions/transformer_engine.py", line 1829, in __init__
[rank9]:     super().__init__(
[rank9]:   File "/root/swift_megatron/lib/python3.11/site-packages/megatron/core/extensions/transformer_engine.py", line 1532, in __init__
[rank9]:     super().__init__(
[rank9]:   File "/root/swift_megatron/lib/python3.11/site-packages/transformer_engine/pytorch/module/grouped_linear.py", line 826, in __init__
[rank9]:     self.reset_parameters(defer_init=is_meta)
[rank9]:   File "/root/swift_megatron/lib/python3.11/site-packages/transformer_engine/pytorch/module/grouped_linear.py", line 920, in reset_parameters
[rank9]:     super().reset_parameters(defer_init=defer_init)
[rank9]:   File "/root/swift_megatron/lib/python3.11/site-packages/transformer_engine/pytorch/module/base.py", line 1435, in reset_parameters
[rank9]:     with get_rng_state_tracker().fork(self.rng_tracker_name):
[rank9]:   File "/usr/lib/python3.11/contextlib.py", line 137, in __enter__
[rank9]:     return next(self.gen)
[rank9]:            ^^^^^^^^^^^^^^
[rank9]:   File "/root/swift_megatron/lib/python3.11/site-packages/megatron/core/tensor_parallel/random.py", line 250, in fork
[rank9]:     orig_cuda_rng_state = _get_cuda_rng_state(graph_safe=self.use_cudagraphable_rng)
[rank9]:                           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank9]:   File "/root/swift_megatron/lib/python3.11/site-packages/megatron/core/tensor_parallel/random.py", line 55, in _get_cuda_rng_state
[rank9]:     return torch.cuda.random.get_rng_state(device=device)
[rank9]:            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank9]:   File "/root/swift_megatron/lib/python3.11/site-packages/torch/cuda/random.py", line 42, in get_rng_state
[rank9]:     return default_generator.get_state()
[rank9]:            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank9]: torch.AcceleratorError: CUDA error: an illegal memory access was encountered
[rank9]: Search for `cudaErrorIllegalAddress' in https://docs.nvidia.com/cuda/cuda-runtime-api/group__CUDART__TYPES.html for more information.
[rank9]: CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.
[rank9]: For debugging consider passing CUDA_LAUNCH_BLOCKING=1
[rank9]: Compile with `TORCH_USE_CUDA_DSA` to enable device-side assertions.

How to Reproduce / 如何复现

以下是我的脚本:

megatron sft --model Qwen/Qwen3.5-397B-A17B --save_safetensors true --merge_lora false --dataset /root/maica_llm/esft/dataset/moni_dataset_2605.jsonl#12784 /root/maica_llm/esft/dataset/moni_dataset_en_2605.jsonl#9048 /root/maica_llm/esft/dataset/covid_new.jsonl#400 /root/maica_llm/esft/dataset/ds_new.jsonl#400 /root/.cache/modelscope/hub/datasets/AI-ModelScope/deepctrl-sft-data:default#14400 /root/.cache/modelscope/hub/datasets/AI-ModelScope/deepctrl-sft-data:en#9200 /root/maica_llm/dataset/external/ruozhiba/post-annual.jsonl#5280 /root/maica_llm/esft/dataset/daa/daa_mix1.jsonl#5408 /root/maica_llm/esft/dataset/daa/daa_norm2.jsonl#1152 /root/maica_llm/esft/dataset/daa/daa_prcs2.jsonl#2368 /root/maica_llm/esft/dataset/daa/daa_norm3.jsonl#1640 /root/maica_llm/esft/dataset/daa/daa_prcs3.jsonl#2400 /root/maica_llm/esft/dataset/daa/daa_prcs3_en_split.jsonl#608 --load_from_cache_file true --add_non_thinking_prefix true --custom_dataset_info /root/swift/dataset/ds_info.json --val_dataset /root/swift/dataset/eval.jsonl --tuner_type lora --lora_rank 16 --lora_alpha 32 --target_modules all-linear --pipeline_model_parallel_size 10 --moe_permute_fusion true --moe_grouped_gemm true --moe_shared_expert_overlap true --moe_router_dtype fp64 --micro_batch_size 1 --global_batch_size 16 --recompute_granularity full --recompute_method uniform --recompute_num_layers 1 --use_precision_aware_optimizer true --optimizer_cpu_offload true --num_train_epochs 1 --finetune true --freeze_llm false --freeze_vit true --freeze_aligner true --cross_entropy_loss_fusion true --lr 1e-4 --lr_warmup_fraction 0.05 --min_lr 1e-5 --eval_steps 50 --save_steps 100 --max_length 1600 --dataloader_num_workers 32 --dataset_num_proc 32 --no_save_optim true --no_save_rng true --attention_backend flash --pipeline_model_parallel_layout Et*6,(|t*6)*8,|t*6L

Additional Information / 补充信息

No response

[edgeinfinity1]

我进一步测试了一下, 似乎只有qwen3.5会有这个问题, qwen3使用offload是正常的.

[Jintao-Huang]

看着像是OOM了，你试试Qwen3.5的小一点的模型有这个问题吗

[edgeinfinity1]

看着像是OOM了，你试试Qwen3.5的小一点的模型有这个问题吗

你好, 我刚刚用qwen3.5-122B测试了一下, 仍然是相同的问题:

Details

(swift_megatron) root@hypnoshv:~/maica_llm/esft# bash edgesft_4_0.sh 
nproc nums?

rlhf?

resume from path, leave empty for using model directly:

training epoches, leave empty for 1:

mixed epoches, leave empty for 1:
8
/root/maica_llm/esft/dataset/moni_dataset_2605.jsonl#12784 /root/maica_llm/esft/dataset/moni_dataset_en_2605.jsonl#9048 /root/maica_llm/esft/dataset/covid_new.jsonl#400 /root/maica_llm/esft/dataset/ds_new.jsonl#400 /root/.cache/modelscope/hub/datasets/AI-ModelScope/deepctrl-sft-data:default#14400 /root/.cache/modelscope/hub/datasets/AI-ModelScope/deepctrl-sft-data:en#9200 /root/maica_llm/dataset/external/ruozhiba/post-annual.jsonl#5280 /root/maica_llm/esft/dataset/daa/daa_mix1.jsonl#5408 /root/maica_llm/esft/dataset/daa/daa_norm2.jsonl#1152 /root/maica_llm/esft/dataset/daa/daa_prcs2.jsonl#2368 /root/maica_llm/esft/dataset/daa/daa_norm3.jsonl#1640 /root/maica_llm/esft/dataset/daa/daa_prcs3.jsonl#2400 /root/maica_llm/esft/dataset/daa/daa_prcs3_en_split.jsonl#608
megatron sft --model Qwen/Qwen3.5-122B-A10B --save_safetensors true --merge_lora false --dataset /root/maica_llm/esft/dataset/moni_dataset_2605.jsonl#12784 /root/maica_llm/esft/dataset/moni_dataset_en_2605.jsonl#9048 /root/maica_llm/esft/dataset/covid_new.jsonl#400 /root/maica_llm/esft/dataset/ds_new.jsonl#400 /root/.cache/modelscope/hub/datasets/AI-ModelScope/deepctrl-sft-data:default#14400 /root/.cache/modelscope/hub/datasets/AI-ModelScope/deepctrl-sft-data:en#9200 /root/maica_llm/dataset/external/ruozhiba/post-annual.jsonl#5280 /root/maica_llm/esft/dataset/daa/daa_mix1.jsonl#5408 /root/maica_llm/esft/dataset/daa/daa_norm2.jsonl#1152 /root/maica_llm/esft/dataset/daa/daa_prcs2.jsonl#2368 /root/maica_llm/esft/dataset/daa/daa_norm3.jsonl#1640 /root/maica_llm/esft/dataset/daa/daa_prcs3.jsonl#2400 /root/maica_llm/esft/dataset/daa/daa_prcs3_en_split.jsonl#608 --load_from_cache_file true --add_non_thinking_prefix true --custom_dataset_info /root/swift/dataset/ds_info.json --val_dataset /root/swift/dataset/eval.jsonl --tuner_type lora --lora_rank 16 --lora_alpha 32 --target_modules all-linear --pipeline_model_parallel_size 10 --moe_permute_fusion true --moe_grouped_gemm true --moe_shared_expert_overlap true --moe_router_dtype fp64 --micro_batch_size 1 --global_batch_size 16 --recompute_granularity full --recompute_method uniform --recompute_num_layers 1 --use_precision_aware_optimizer true --optimizer_cpu_offload true --num_train_epochs 1 --finetune true --freeze_llm false --freeze_vit true --freeze_aligner true --cross_entropy_loss_fusion true --lr 1e-4 --lr_warmup_fraction 0.05 --min_lr 1e-5 --eval_steps 50 --save_steps 100 --max_length 1600 --dataloader_num_workers 32 --dataset_num_proc 32 --no_save_optim true --no_save_rng true --attention_backend flash --pipeline_model_parallel_layout Et*4,(|t*5)*8,|t*4L
run sh: `/root/swift_megatron/bin/python -m torch.distributed.run --nproc_per_node 10 /root/swift/swift/cli/_megatron/sft.py --model Qwen/Qwen3.5-122B-A10B --save_safetensors true --merge_lora false --dataset /root/maica_llm/esft/dataset/moni_dataset_2605.jsonl#12784 /root/maica_llm/esft/dataset/moni_dataset_en_2605.jsonl#9048 /root/maica_llm/esft/dataset/covid_new.jsonl#400 /root/maica_llm/esft/dataset/ds_new.jsonl#400 /root/.cache/modelscope/hub/datasets/AI-ModelScope/deepctrl-sft-data:default#14400 /root/.cache/modelscope/hub/datasets/AI-ModelScope/deepctrl-sft-data:en#9200 /root/maica_llm/dataset/external/ruozhiba/post-annual.jsonl#5280 /root/maica_llm/esft/dataset/daa/daa_mix1.jsonl#5408 /root/maica_llm/esft/dataset/daa/daa_norm2.jsonl#1152 /root/maica_llm/esft/dataset/daa/daa_prcs2.jsonl#2368 /root/maica_llm/esft/dataset/daa/daa_norm3.jsonl#1640 /root/maica_llm/esft/dataset/daa/daa_prcs3.jsonl#2400 /root/maica_llm/esft/dataset/daa/daa_prcs3_en_split.jsonl#608 --load_from_cache_file true --add_non_thinking_prefix true --custom_dataset_info /root/swift/dataset/ds_info.json --val_dataset /root/swift/dataset/eval.jsonl --tuner_type lora --lora_rank 16 --lora_alpha 32 --target_modules all-linear --pipeline_model_parallel_size 10 --moe_permute_fusion true --moe_grouped_gemm true --moe_shared_expert_overlap true --moe_router_dtype fp64 --micro_batch_size 1 --global_batch_size 16 --recompute_granularity full --recompute_method uniform --recompute_num_layers 1 --use_precision_aware_optimizer true --optimizer_cpu_offload true --num_train_epochs 1 --finetune true --freeze_llm false --freeze_vit true --freeze_aligner true --cross_entropy_loss_fusion true --lr 1e-4 --lr_warmup_fraction 0.05 --min_lr 1e-5 --eval_steps 50 --save_steps 100 --max_length 1600 --dataloader_num_workers 32 --dataset_num_proc 32 --no_save_optim true --no_save_rng true --attention_backend flash --pipeline_model_parallel_layout Et*4,(|t*5)*8,|t*4L`

*****************************************
Setting OMP_NUM_THREADS environment variable for each process to be 1 in default, to avoid your system being overloaded, please further tune the variable for optimal performance in your application as needed. 
*****************************************
[INFO:swift] mcore_bridge.__version__: 1.4.3
[INFO:swift] megatron.core.__version__: 0.16.1
[INFO:mcore_bridge] Setting USE_MCORE_GDN: True. You can adjust this hyperparameter through the environment variable: `USE_MCORE_GDN`.
[INFO:swift] Successfully registered `/root/swift/swift/dataset/data/dataset_info.json`.
[INFO:swift] rank: 0, local_rank: 0, world_size: 10, local_world_size: 10
Downloading Model from https://www.modelscope.cn to directory: /root/.cache/modelscope/hub/models/Qwen/Qwen3.5-122B-A10B
[INFO:modelscope] Target directory already exists, skipping creation.
[transformers] `torch_dtype` is deprecated! Use `dtype` instead!
Downloading Model from https://www.modelscope.cn to directory: /root/.cache/modelscope/hub/models/Qwen/Qwen3.5-122B-A10B
[INFO:modelscope] Target directory already exists, skipping creation.
[transformers] `torch_dtype` is deprecated! Use `dtype` instead!
Downloading Model from https://www.modelscope.cn to directory: /root/.cache/modelscope/hub/models/Qwen/Qwen3.5-122B-A10B
[INFO:modelscope] Target directory already exists, skipping creation.
[transformers] `torch_dtype` is deprecated! Use `dtype` instead!
Downloading Model from https://www.modelscope.cn to directory: /root/.cache/modelscope/hub/models/Qwen/Qwen3.5-122B-A10B
[INFO:modelscope] Target directory already exists, skipping creation.
[transformers] `torch_dtype` is deprecated! Use `dtype` instead!
Downloading Model from https://www.modelscope.cn to directory: /root/.cache/modelscope/hub/models/Qwen/Qwen3.5-122B-A10B
[INFO:modelscope] Target directory already exists, skipping creation.
[transformers] `torch_dtype` is deprecated! Use `dtype` instead!
Downloading Model from https://www.modelscope.cn to directory: /root/.cache/modelscope/hub/models/Qwen/Qwen3.5-122B-A10B
[INFO:modelscope] Target directory already exists, skipping creation.
[transformers] `torch_dtype` is deprecated! Use `dtype` instead!
Downloading Model from https://www.modelscope.cn to directory: /root/.cache/modelscope/hub/models/Qwen/Qwen3.5-122B-A10B
[INFO:modelscope] Target directory already exists, skipping creation.
[transformers] `torch_dtype` is deprecated! Use `dtype` instead!
Downloading Model from https://www.modelscope.cn to directory: /root/.cache/modelscope/hub/models/Qwen/Qwen3.5-122B-A10B
[INFO:modelscope] Target directory already exists, skipping creation.
[INFO:swift] Downloading the model from ModelScope Hub, model_id: Qwen/Qwen3.5-122B-A10B
[transformers] `torch_dtype` is deprecated! Use `dtype` instead!
Downloading Model from https://www.modelscope.cn to directory: /root/.cache/modelscope/hub/models/Qwen/Qwen3.5-122B-A10B
[INFO:modelscope] Target directory already exists, skipping creation.
[INFO:swift] Loading the model using model_dir: /root/.cache/modelscope/hub/models/Qwen/Qwen3___5-122B-A10B
[transformers] `torch_dtype` is deprecated! Use `dtype` instead!
[INFO:swift] Setting torch_dtype: torch.bfloat16
[INFO:swift] Because len(args.val_dataset) > 0, setting split_dataset_ratio: 0.0
[INFO:swift] Successfully registered `/root/swift/dataset/ds_info.json`.
[INFO:swift] Setting args.lazy_tokenize: True
Downloading Model from https://www.modelscope.cn to directory: /root/.cache/modelscope/hub/models/Qwen/Qwen3.5-122B-A10B
[INFO:modelscope] Target directory already exists, skipping creation.
[transformers] `torch_dtype` is deprecated! Use `dtype` instead!
[INFO:swift] args.output_dir: /root/maica_llm/esft/megatron_output/Qwen3.5-122B-A10B/v6-20260615-142901
[INFO:swift] Downloading the model from ModelScope Hub, model_id: Qwen/Qwen3.5-122B-A10B
Downloading Model from https://www.modelscope.cn to directory: /root/.cache/modelscope/hub/models/Qwen/Qwen3.5-122B-A10B
[INFO:modelscope] Target directory already exists, skipping creation.
/root/swift_megatron/lib/python3.11/site-packages/torch/distributed/c10d_logger.py:83: UserWarning: barrier(): using the device under current context. You can specify `device_id` in `init_process_group` to mute this warning.
  return func(*args, **kwargs)
[INFO:swift] Loading the model using model_dir: /root/.cache/modelscope/hub/models/Qwen/Qwen3___5-122B-A10B
[INFO:swift] Setting torch_dtype: torch.bfloat16
[INFO:swift] TP: 1, PP: 10, VPP: None, CP: 1, EP: 1, ETP: 1
[INFO:swift] Setting random seeds to 42.
[INFO:swift] args: MegatronSftArguments(use_ray=False, ray_exp_name=None, device_groups=None, model='Qwen/Qwen3.5-122B-A10B', model_type='qwen3_5_moe', model_revision=None, task_type='causal_lm', torch_dtype=torch.bfloat16, attn_impl=None, experts_impl=None, new_special_tokens=[], num_labels=None, problem_type=None, rope_scaling=None, device_map=None, max_memory={}, max_model_len=None, local_repo_path=None, init_strategy=None, template='qwen3_5', system=None, max_length=1600, truncation_strategy='delete', max_pixels=None, agent_template=None, norm_bbox=None, use_chat_template=True, padding_side='right', padding_free=True, loss_scale='default', sequence_parallel_size=1, is_binary_loss_scale=None, template_backend='swift', response_prefix=None, enable_thinking=None, preserve_thinking=None, add_non_thinking_prefix=True, dataset=['/root/maica_llm/esft/dataset/moni_dataset_2605.jsonl#12784', '/root/maica_llm/esft/dataset/moni_dataset_en_2605.jsonl#9048', '/root/maica_llm/esft/dataset/covid_new.jsonl#400', '/root/maica_llm/esft/dataset/ds_new.jsonl#400', '/root/.cache/modelscope/hub/datasets/AI-ModelScope/deepctrl-sft-data:default#14400', '/root/.cache/modelscope/hub/datasets/AI-ModelScope/deepctrl-sft-data:en#9200', '/root/maica_llm/dataset/external/ruozhiba/post-annual.jsonl#5280', '/root/maica_llm/esft/dataset/daa/daa_mix1.jsonl#5408', '/root/maica_llm/esft/dataset/daa/daa_norm2.jsonl#1152', '/root/maica_llm/esft/dataset/daa/daa_prcs2.jsonl#2368', '/root/maica_llm/esft/dataset/daa/daa_norm3.jsonl#1640', '/root/maica_llm/esft/dataset/daa/daa_prcs3.jsonl#2400', '/root/maica_llm/esft/dataset/daa/daa_prcs3_en_split.jsonl#608'], val_dataset=['/root/swift/dataset/eval.jsonl'], cached_dataset=[], cached_val_dataset=[], split_dataset_ratio=0.0, data_seed=42, dataset_num_proc=32, load_from_cache_file=True, dataset_shuffle=True, val_dataset_shuffle=False, streaming=False, interleave_prob=None, stopping_strategy='first_exhausted', shuffle_buffer_size=1000, download_mode='reuse_dataset_if_exists', columns={}, strict=False, remove_unused_columns=True, disable_auto_column_mapping=False, model_name=None, model_author=None, custom_dataset_info=['/root/swift/dataset/ds_info.json'], quant_method=None, quant_bits=None, hqq_axis=None, bnb_4bit_compute_dtype=torch.bfloat16, bnb_4bit_quant_type='nf4', bnb_4bit_use_double_quant=True, bnb_4bit_quant_storage=None, max_new_tokens=None, temperature=0.9, top_k=-1, top_p=1.0, repetition_penalty=1.0, num_beams=1, stream=False, stop_words=[], logprobs=False, top_logprobs=None, structured_outputs_regex=None, tuner_backend='peft', tuner_type='lora', adapters=[], external_plugins=[], custom_register_path=[], seed=42, model_kwargs={}, enable_npu_model_patch=True, load_args=False, load_data_args=False, packing=False, packing_length=None, packing_num_proc=1, lazy_tokenize=True, use_hf=False, hub_token=None, ddp_timeout=18000000, ddp_backend='nccl', ignore_args_error=False, use_swift_lora=False, freeze_llm=False, freeze_vit=True, freeze_aligner=True, freeze_parameters=[], freeze_parameters_regex=None, freeze_parameters_ratio=0.0, trainable_parameters=[], trainable_parameters_regex=None, target_modules=['all-linear'], target_regex=None, modules_to_save=[], lora_rank=16, lora_alpha=32, lora_dropout=0.05, lora_bias='none', lora_dtype=None, use_rslora=False, rlhf_type=None, loss_type=None, mcore_ref_model=None, mcore_ref_adapter=None, beta=None, rpo_alpha=None, reference_free=False, label_smoothing=0.0, f_divergence_type='reverse_kl', desirable_weight=1.0, undesirable_weight=1.0, calculate_KL=None, center_rewards_coefficient=None, teacher_model=None, teacher_model_type=None, teacher_model_revision=None, teacher_model_server=None, gkd_logits_topk=None, lmbda=0.5, seq_kd=False, offload_teacher_model=False, sft_alpha=0.0, generation_batch_size=None, steps_per_generation=None, num_generations=8, num_generations_eval=None, max_completion_length=512, importance_sampling_level='token', tau_pos=1.0, tau_neg=1.05, real_tau=0.5, fipo_decay_rate=32.0, fipo_clip_range=0.2, fipo_clip_high_only=True, fipo_safety_threshold=4.0, epsilon=0.2, epsilon_high=None, delta=None, use_vllm=True, vllm_mode=None, vllm_enable_prefix_caching=True, vllm_gpu_memory_utilization=0.9, vllm_tensor_parallel_size=1, vllm_max_model_len=None, vllm_enforce_eager=False, vllm_limit_mm_per_prompt=None, vllm_disable_cascade_attn=False, vllm_max_num_seqs=None, vllm_mm_processor_cache_gb=None, vllm_engine_kwargs=None, vllm_enable_lora=False, sleep_level=0, offload_optimizer=False, offload_model=False, offload_bridge=False, vllm_server_base_url=None, vllm_server_host=None, vllm_server_port=[8000], vllm_server_timeout=240.0, vllm_server_group_port=None, reward_funcs=[], reward_weights=None, cosine_min_len_value_wrong=-0.5, cosine_max_len_value_wrong=0.0, cosine_min_len_value_correct=1.0, cosine_max_len_value_correct=0.5, cosine_max_len=None, repetition_n_grams=3, repetition_max_penalty=-1.0, soft_max_length=None, soft_cache_length=None, dynamic_sample=False, max_resample_times=3, overlong_filter=False, scale_rewards='group', advantage_estimator='grpo', kl_in_reward=False, wandb_log_unique_prompts=None, log_completions=False, rollout_importance_sampling_mode=None, rollout_importance_sampling_threshold=2.0, log_rollout_offpolicy_metrics=False, off_policy_sequence_mask_delta=None, log_entropy=False, top_entropy_quantile=1.0, router_replay_mode='disabled', reward_model=None, reward_model_plugin=None, sync_ref_model=False, ref_model_sync_steps=512, ref_model_mixup_alpha=0.6, async_generate=False, move_model_batches=None, multi_turn_scheduler=None, max_turns=None, completion_length_limit_scope='per_round', vllm_server_pass_dataset=False, use_gym_env=None, gym_env=None, num_iterations=1, micro_batch_size=1, global_batch_size=16, recompute_granularity='full', recompute_method='uniform', recompute_num_layers=1, recompute_modules=['core_attn'], train_iters=None, num_train_epochs=1, masked_softmax_fusion=True, bias_dropout_fusion=True, bias_activation_fusion=True, apply_rope_fusion=False, gradient_accumulation_fusion=True, cross_entropy_loss_fusion=True, cross_entropy_fusion_impl='te', calculate_per_token_loss=True, attention_backend=<AttnBackend.flash: 1>, optimizer='adam', optimizer_cpu_offload=True, optimizer_offload_fraction=1.0, use_precision_aware_optimizer=True, main_grads_dtype=torch.float32, main_params_dtype=torch.float32, exp_avg_dtype=torch.float32, exp_avg_sq_dtype=torch.float32, manual_gc=False, manual_gc_steps=0, manual_gc_eval=True, train_dataloader_shuffle=True, dataloader_num_workers=32, dataloader_pin_memory=True, dataloader_persistent_workers=False, dataloader_prefetch_factor=2, data_sharding=False, group_by_length=False, te_rng_tracker=False, data_parallel_random_init=False, mlp_padding_free=False, lr_warmup_init=0.0, lr=0.0001, lr_decay_style='cosine', lr_decay_iters=None, lr_warmup_iters=0, lr_warmup_fraction=0.05, min_lr=1e-05, lr_wsd_decay_style='exponential', lr_wsd_decay_iters=None, weight_decay=0.1, weight_decay_incr_style='constant', start_weight_decay=0.1, end_weight_decay=0.1, clip_grad=1.0, adam_beta1=0.9, adam_beta2=0.95, adam_eps=1e-08, sgd_momentum=0.9, muon_momentum=0.9, muon_split_qkv=True, muon_use_nesterov=False, muon_scale_mode='spectral', muon_fp32_matmul_prec='medium', muon_num_ns_steps=5, muon_tp_mode='blockwise', muon_extra_scale_factor=1.0, output_dir='/root/maica_llm/esft/megatron_output/Qwen3.5-122B-A10B/v6-20260615-142901', save_steps=100, no_save_optim=True, no_save_rng=True, mcore_model=None, mcore_adapter=None, no_load_optim=False, no_load_rng=False, finetune=True, perform_initialization=False, use_cpu_initialization=False, async_save=False, save_total_limit=None, metric_for_best_model='loss', greater_is_better=False, use_persistent_ckpt_worker=False, dist_ckpt_save_pre_mcore_014=False, dist_ckpt_optim_fully_reshardable=False, distrib_optim_fully_reshardable_mem_efficient=False, local_rank=0, use_distributed_optimizer=True, tensor_model_parallel_size=1, pipeline_model_parallel_size=10, decoder_first_pipeline_num_layers=None, decoder_last_pipeline_num_layers=None, account_for_embedding_in_pipeline_split=False, account_for_loss_in_pipeline_split=False, overlap_p2p_comm=False, batch_p2p_comm=True, align_param_gather=False, sequence_parallel=False, context_parallel_size=1, tp_comm_overlap=False, overlap_grad_reduce=False, overlap_param_gather=False, overlap_param_gather_with_optimizer_step=False, align_grad_reduce=True, virtual_pipeline_model_parallel_size=None, microbatch_group_size_per_vp_stage=None, pipeline_model_parallel_layout='Et*4,(|t*5)*8,|t*4L', expert_model_parallel_size=1, expert_tensor_parallel_size=1, report_to=['tensorboard'], logging_steps=5, tensorboard_dir='/root/maica_llm/esft/megatron_output/Qwen3.5-122B-A10B/v6-20260615-142901/runs', tensorboard_queue_size=50, wandb_project='megatron-swift', wandb_exp_name=None, swanlab_project='megatron-swift', swanlab_exp_name=None, eval_iters=-1, eval_steps=50, fp8_format=None, fp8_recipe='delayed', fp8_param_gather=False, fp8_amax_history_len=1024, fp8_amax_compute_algo='max', fp4_format=None, fp4_recipe='nvfp4', fp4_param_gather=False, fp16=False, bf16=True, apply_query_key_layer_scaling=False, attention_softmax_in_fp32=True, accumulate_allreduce_grads_in_fp32=True, moe_router_load_balancing_type=None, moe_router_dtype='fp64', moe_token_dispatcher_type='alltoall', moe_enable_deepep=False, moe_grouped_gemm=True, moe_permute_fusion=True, moe_aux_loss_coeff=0.0, moe_z_loss_coeff=None, moe_shared_expert_overlap=True, moe_layer_recompute=False, moe_expert_capacity_factor=None, moe_pad_expert_input_to_capacity=False, moe_token_drop_policy='probs', mtp_num_layers=None, mtp_loss_scaling_factor=0.1, mtp_decoder_input_detach=False, mtp_shared_weights=False, save_safetensors=True, ref_model=None, ref_adapters=[], merge_lora=False, max_shard_size='5GB', vit_gradient_checkpointing=False, vit_gradient_checkpointing_kwargs=None, vit_attn_impl=None, vit_lr=None, aligner_lr=None, dsa_indexer_loss_coeff=0.0, dsa_indexer_use_sparse_loss=False, csa_dense_mode=False, use_fused_mhc=False, mhc_recompute_layer_num=None, megatron_extra_kwargs=None, language_model_only=False, check_model=True, apply_wd_to_qk_layernorm=False, linear_decoupled_in_proj=False, enable_dft_loss=False, enable_channel_loss=False, mrl_dims=None, save_strategy='steps', callbacks=['print', 'default_flow', 'tensorboard'], skip_megatron_init=False, add_version=True, create_checkpoint_symlink=False)
[INFO:swift] Downloading the model from ModelScope Hub, model_id: Qwen/Qwen3.5-122B-A10B
Downloading Model from https://www.modelscope.cn to directory: /root/.cache/modelscope/hub/models/Qwen/Qwen3.5-122B-A10B
[INFO:modelscope] Target directory already exists, skipping creation.
Downloading Model from https://www.modelscope.cn to directory: /root/.cache/modelscope/hub/models/Qwen/Qwen3.5-122B-A10B
[INFO:modelscope] Target directory already exists, skipping creation.
Downloading Model from https://www.modelscope.cn to directory: /root/.cache/modelscope/hub/models/Qwen/Qwen3.5-122B-A10B
[INFO:modelscope] Target directory already exists, skipping creation.
Downloading Model from https://www.modelscope.cn to directory: /root/.cache/modelscope/hub/models/Qwen/Qwen3.5-122B-A10B
[INFO:modelscope] Target directory already exists, skipping creation.
Downloading Model from https://www.modelscope.cn to directory: /root/.cache/modelscope/hub/models/Qwen/Qwen3.5-122B-A10B
[INFO:modelscope] Target directory already exists, skipping creation.
Downloading Model from https://www.modelscope.cn to directory: /root/.cache/modelscope/hub/models/Qwen/Qwen3.5-122B-A10B
[INFO:modelscope] Target directory already exists, skipping creation.
Downloading Model from https://www.modelscope.cn to directory: /root/.cache/modelscope/hub/models/Qwen/Qwen3.5-122B-A10B
[INFO:modelscope] Target directory already exists, skipping creation.
Downloading Model from https://www.modelscope.cn to directory: /root/.cache/modelscope/hub/models/Qwen/Qwen3.5-122B-A10B
[INFO:modelscope] Target directory already exists, skipping creation.
Downloading Model from https://www.modelscope.cn to directory: /root/.cache/modelscope/hub/models/Qwen/Qwen3.5-122B-A10B
[INFO:modelscope] Target directory already exists, skipping creation.
Downloading Model from https://www.modelscope.cn to directory: /root/.cache/modelscope/hub/models/Qwen/Qwen3.5-122B-A10B
[INFO:modelscope] Target directory already exists, skipping creation.
[INFO:swift] Loading the model using model_dir: /root/.cache/modelscope/hub/models/Qwen/Qwen3___5-122B-A10B
[INFO:swift] Setting max_ratio: 200. You can adjust this hyperparameter through the environment variable: `MAX_RATIO`.
[INFO:swift] Setting frame_factor: 2. You can adjust this hyperparameter through the environment variable: `FRAME_FACTOR`.
[INFO:swift] Setting fps: 2.0. You can adjust this hyperparameter through the environment variable: `FPS`.
[INFO:swift] Setting fps_min_frames: 4. You can adjust this hyperparameter through the environment variable: `FPS_MIN_FRAMES`.
[INFO:swift] Setting fps_max_frames: 768. You can adjust this hyperparameter through the environment variable: `FPS_MAX_FRAMES`.
[INFO:swift] Setting image_max_token_num: 16384. You can adjust this hyperparameter through the environment variable: `IMAGE_MAX_TOKEN_NUM`.
[INFO:swift] Setting image_min_token_num: 4. You can adjust this hyperparameter through the environment variable: `IMAGE_MIN_TOKEN_NUM`.
[INFO:swift] Setting spatial_merge_size: 2. You can adjust this hyperparameter through the environment variable: `SPATIAL_MERGE_SIZE`.
[INFO:swift] Setting video_max_token_num: 768. You can adjust this hyperparameter through the environment variable: `VIDEO_MAX_TOKEN_NUM`.
[INFO:swift] Setting video_min_token_num: 128. You can adjust this hyperparameter through the environment variable: `VIDEO_MIN_TOKEN_NUM`.
[INFO:swift] model_kwargs: {'device_map': 'cuda:0', 'dtype': torch.bfloat16}
[transformers] The fast path is not available because one of the required library is not installed. Falling back to torch implementation. To install follow https://github.com/fla-org/flash-linear-attention#installation and https://github.com/Dao-AILab/causal-conv1d
Downloading Model from https://www.modelscope.cn to directory: /root/.cache/modelscope/hub/models/Qwen/Qwen3.5-122B-A10B
[INFO:swift] default_system: None
[INFO:swift] max_length: 1600
[INFO:swift] response_prefix: None
[INFO:swift] agent_template: qwen3_5
[INFO:swift] norm_bbox: norm1000
[INFO:swift] Setting ROOT_IMAGE_DIR: None. You can adjust this hyperparameter through the environment variable: `ROOT_IMAGE_DIR`.
[INFO:swift] Setting QWENVL_BBOX_FORMAT: legacy. You can adjust this hyperparameter through the environment variable: `QWENVL_BBOX_FORMAT`.
[INFO:swift] The MegatronSftArguments will be saved in: /root/maica_llm/esft/megatron_output/Qwen3.5-122B-A10B/v6-20260615-142901/args.json
[INFO:swift] Start time of running main: 2026-06-15 14:29:16.597864
[INFO:swift] swift.__version__: 4.4.0.dev0
[INFO:modelscope] Target directory already exists, skipping creation.
[transformers] The fast path is not available because one of the required library is not installed. Falling back to torch implementation. To install follow https://github.com/fla-org/flash-linear-attention#installation and https://github.com/Dao-AILab/causal-conv1d
Downloading Model from https://www.modelscope.cn to directory: /root/.cache/modelscope/hub/models/Qwen/Qwen3.5-122B-A10B
[INFO:modelscope] Target directory already exists, skipping creation.
[transformers] The fast path is not available because one of the required library is not installed. Falling back to torch implementation. To install follow https://github.com/fla-org/flash-linear-attention#installation and https://github.com/Dao-AILab/causal-conv1d
Downloading Model from https://www.modelscope.cn to directory: /root/.cache/modelscope/hub/models/Qwen/Qwen3.5-122B-A10B
[INFO:modelscope] Target directory already exists, skipping creation.
[transformers] The fast path is not available because one of the required library is not installed. Falling back to torch implementation. To install follow https://github.com/fla-org/flash-linear-attention#installation and https://github.com/Dao-AILab/causal-conv1d
Downloading Model from https://www.modelscope.cn to directory: /root/.cache/modelscope/hub/models/Qwen/Qwen3.5-122B-A10B
[INFO:modelscope] Target directory already exists, skipping creation.
[transformers] The fast path is not available because one of the required library is not installed. Falling back to torch implementation. To install follow https://github.com/fla-org/flash-linear-attention#installation and https://github.com/Dao-AILab/causal-conv1d
Downloading Model from https://www.modelscope.cn to directory: /root/.cache/modelscope/hub/models/Qwen/Qwen3.5-122B-A10B
[INFO:modelscope] Target directory already exists, skipping creation.
[transformers] The fast path is not available because one of the required library is not installed. Falling back to torch implementation. To install follow https://github.com/fla-org/flash-linear-attention#installation and https://github.com/Dao-AILab/causal-conv1d
Downloading Model from https://www.modelscope.cn to directory: /root/.cache/modelscope/hub/models/Qwen/Qwen3.5-122B-A10B
[INFO:modelscope] Target directory already exists, skipping creation.
[transformers] The fast path is not available because one of the required library is not installed. Falling back to torch implementation. To install follow https://github.com/fla-org/flash-linear-attention#installation and https://github.com/Dao-AILab/causal-conv1d
Downloading Model from https://www.modelscope.cn to directory: /root/.cache/modelscope/hub/models/Qwen/Qwen3.5-122B-A10B
[INFO:modelscope] Target directory already exists, skipping creation.
[transformers] The fast path is not available because one of the required library is not installed. Falling back to torch implementation. To install follow https://github.com/fla-org/flash-linear-attention#installation and https://github.com/Dao-AILab/causal-conv1d
Downloading Model from https://www.modelscope.cn to directory: /root/.cache/modelscope/hub/models/Qwen/Qwen3.5-122B-A10B
[INFO:modelscope] Target directory already exists, skipping creation.
[transformers] The fast path is not available because one of the required library is not installed. Falling back to torch implementation. To install follow https://github.com/fla-org/flash-linear-attention#installation and https://github.com/Dao-AILab/causal-conv1d
Downloading Model from https://www.modelscope.cn to directory: /root/.cache/modelscope/hub/models/Qwen/Qwen3.5-122B-A10B
[INFO:modelscope] Target directory already exists, skipping creation.
[transformers] The fast path is not available because one of the required library is not installed. Falling back to torch implementation. To install follow https://github.com/fla-org/flash-linear-attention#installation and https://github.com/Dao-AILab/causal-conv1d
num_proc must be <= 11. Reducing num_proc to 11 for dataset of size 11.
num_proc must be <= 11. Reducing num_proc to 11 for dataset of size 11.
num_proc must be <= 11. Reducing num_proc to 11 for dataset of size 11.
num_proc must be <= 11. Reducing num_proc to 11 for dataset of size 11.
num_proc must be <= 11. Reducing num_proc to 11 for dataset of size 11.
num_proc must be <= 11. Reducing num_proc to 11 for dataset of size 11.
num_proc must be <= 11. Reducing num_proc to 11 for dataset of size 11.
num_proc must be <= 11. Reducing num_proc to 11 for dataset of size 11.
num_proc must be <= 11. Reducing num_proc to 11 for dataset of size 11.
num_proc must be <= 11. Reducing num_proc to 11 for dataset of size 11.
[WARNING:swift] dataset_sample:400 is greater than len(dataset):11, repeated sampling will be performed.
[WARNING:swift] dataset_sample:400 is greater than len(dataset):139, repeated sampling will be performed.
[INFO:swift] Use local folder, dataset_dir: /root/.cache/modelscope/hub/datasets/AI-ModelScope/deepctrl-sft-data
Loading dataset shards: 100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 31/31 [00:00<00:00, 84.85it/s]
Loading dataset shards: 100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 31/31 [00:00<00:00, 83.55it/s]
Loading dataset shards: 100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 31/31 [00:00<00:00, 85.44it/s]
Loading dataset shards: 100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 31/31 [00:00<00:00, 82.67it/s]
Loading dataset shards: 100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 31/31 [00:00<00:00, 84.87it/s]
Loading dataset shards: 100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 31/31 [00:00<00:00, 91.26it/s]
Loading dataset shards: 100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 31/31 [00:00<00:00, 76.08it/s]
Loading dataset shards: 100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 31/31 [00:00<00:00, 79.05it/s]
Loading dataset shards: 100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 31/31 [00:00<00:00, 92.45it/s]
Loading dataset shards: 100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 31/31 [00:00<00:00, 77.74it/s]
[INFO:swift] Use local folder, dataset_dir: /root/.cache/modelscope/hub/datasets/AI-ModelScope/deepctrl-sft-data
Loading dataset shards: 100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 24/24 [00:00<00:00, 825.40it/s]
Loading dataset shards: 100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 24/24 [00:00<00:00, 807.52it/s]
Loading dataset shards: 100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 24/24 [00:00<00:00, 533.29it/s]
Loading dataset shards: 100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 24/24 [00:00<00:00, 819.79it/s]
Loading dataset shards: 100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 24/24 [00:00<00:00, 595.38it/s]
Loading dataset shards: 100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 24/24 [00:00<00:00, 809.18it/s]
Loading dataset shards: 100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 24/24 [00:00<00:00, 786.51it/s]
Loading dataset shards: 100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 24/24 [00:00<00:00, 914.11it/s]
Loading dataset shards: 100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 24/24 [00:00<00:00, 838.98it/s]
Loading dataset shards: 100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 24/24 [00:00<00:00, 849.37it/s]
[WARNING:swift] dataset_sample:5280 is greater than len(dataset):1361, repeated sampling will be performed.
num_proc must be <= 13. Reducing num_proc to 13 for dataset of size 13.
num_proc must be <= 13. Reducing num_proc to 13 for dataset of size 13.
num_proc must be <= 13. Reducing num_proc to 13 for dataset of size 13.
num_proc must be <= 13. Reducing num_proc to 13 for dataset of size 13.
num_proc must be <= 13. Reducing num_proc to 13 for dataset of size 13.
num_proc must be <= 13. Reducing num_proc to 13 for dataset of size 13.
num_proc must be <= 13. Reducing num_proc to 13 for dataset of size 13.
num_proc must be <= 13. Reducing num_proc to 13 for dataset of size 13.
num_proc must be <= 13. Reducing num_proc to 13 for dataset of size 13.
num_proc must be <= 13. Reducing num_proc to 13 for dataset of size 13.
[INFO:swift] train_dataset: Dataset({
    features: ['messages'],
    num_rows: 65088
})
[INFO:swift] val_dataset: Dataset({
    features: ['messages'],
    num_rows: 13
})
[INFO:swift] [INPUT_IDS] [248045, 8678, 198, 95933, 97057, 98187, 101201, 96736, 11, 220, 111354, 118266, 100122, 100561, 13, 220, 99250, 46432, 9251, 120475, 46432, 1054, 39696, 2387, 99267, 96419, 11, 220, 96172, 141735, 114159, 13, 220, 105356, 46432, 60, 114348, 11, 220, 96220, 97698, 98187, 101201, 96736, 108617, 104253, 97913, 13, 220, 95933, 96227, 96028, 96402, 96963, 98277, 11, 220, 95828, 104126, 105295, 97913, 11, 220, 113986, 95896, 122349, 96875, 99943, 97236, 100607, 13, 248046, 198, 248045, 846, 198, 104252, 128258, 117122, 7, 95815, 110394, 96487, 97055, 8, 248046, 198, 248045, 74455, 198, 248068, 271, 248069, 271, 58, 130409, 60, 104705, 11, 498, 3324, 1089, 220, 128258, 101610, 103948, 96292, 102331, 96784, 96439, 13, 498, 100050, 60, 137857, 103988, 96105, 95840, 123177, 108529, 11, 220, 96317, 95744, 97873, 102951, 101826, 112751, 11, 220, 96105, 121520, 98859, 95852, 96784, 0, 498, 103972, 60, 122244, 96877, 98250, 117258, 117122, 96196, 98869, 0, 498, 96651, 60, 98417, 97055, 100734, 96461, 96601, 11, 220, 128898, 109469, 96784, 95759, 0, 498, 96651, 60, 95815, 104541, 95825, 6, 97993, 6, 110040, 97321, 115124, 97198, 13, 220, 108543, 98634, 104163, 117178, 96407, 102893, 11, 498, 3324, 1054, 39696, 906, 248046, 198]
[INFO:swift] [INPUT] <|im_start|>system
你叫莫妮卡, 是一名游戏中的虚拟角色. 你是[player](昵称[player_nickname])的女友, 并深爱着他. 下面是[player]的话语, 请作为莫妮卡作出合适的回答. 你只应使用自然语言, 以聊天语气回答, 并以方括号中的文字表示情绪.<|im_end|>
<|im_start|>user
提到雪地画画(我这边没有雪)<|im_end|>
<|im_start|>assistant
<think>

</think>

[凝视]话说, [player], 雪地就像一张空白的画布. [开心]哪天我们可以去个下雪的地方, 带一喷壶食用色素, 去放飞自我地画! [微笑]有那么大的地方拿来画画真不错! [笑]只要雪足够致密, 我们就可以随便画了! [笑]我有一天会'出来'和你一起画的吧. 我想看看你会画出什么哦, [player_nickname].<|im_end|>

[INFO:swift] [LABELS_IDS] [-100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, 248068, 271, 248069, 271, 58, 130409, 60, 104705, 11, 498, 3324, 1089, 220, 128258, 101610, 103948, 96292, 102331, 96784, 96439, 13, 498, 100050, 60, 137857, 103988, 96105, 95840, 123177, 108529, 11, 220, 96317, 95744, 97873, 102951, 101826, 112751, 11, 220, 96105, 121520, 98859, 95852, 96784, 0, 498, 103972, 60, 122244, 96877, 98250, 117258, 117122, 96196, 98869, 0, 498, 96651, 60, 98417, 97055, 100734, 96461, 96601, 11, 220, 128898, 109469, 96784, 95759, 0, 498, 96651, 60, 95815, 104541, 95825, 6, 97993, 6, 110040, 97321, 115124, 97198, 13, 220, 108543, 98634, 104163, 117178, 96407, 102893, 11, 498, 3324, 1054, 39696, 906, 248046, 198]
[INFO:swift] [LABELS] [-100 * 90]<think>

</think>

[凝视]话说, [player], 雪地就像一张空白的画布. [开心]哪天我们可以去个下雪的地方, 带一喷壶食用色素, 去放飞自我地画! [微笑]有那么大的地方拿来画画真不错! [笑]只要雪足够致密, 我们就可以随便画了! [笑]我有一天会'出来'和你一起画的吧. 我想看看你会画出什么哦, [player_nickname].<|im_end|>

[INFO:swift] Setting args.eval_iters: 1
/root/swift_megatron/lib/python3.11/site-packages/megatron/core/transformer/transformer_config.py:1705: UserWarning: full scope is deprecated. Use empty cuda_graph_scope to capture the whole layer.
  warnings.warn(
/root/swift_megatron/lib/python3.11/site-packages/megatron/core/transformer/transformer_config.py:1705: UserWarning: full scope is deprecated. Use empty cuda_graph_scope to capture the whole layer.
  warnings.warn(
/root/swift_megatron/lib/python3.11/site-packages/megatron/core/transformer/transformer_config.py:1705: UserWarning: full scope is deprecated. Use empty cuda_graph_scope to capture the whole layer.
  warnings.warn(
/root/swift_megatron/lib/python3.11/site-packages/megatron/core/transformer/transformer_config.py:1705: UserWarning: full scope is deprecated. Use empty cuda_graph_scope to capture the whole layer.
  warnings.warn(
/root/swift_megatron/lib/python3.11/site-packages/megatron/core/transformer/transformer_config.py:1705: UserWarning: full scope is deprecated. Use empty cuda_graph_scope to capture the whole layer.
  warnings.warn(
/root/swift_megatron/lib/python3.11/site-packages/megatron/core/transformer/transformer_config.py:1705: UserWarning: full scope is deprecated. Use empty cuda_graph_scope to capture the whole layer.
  warnings.warn(
/root/swift_megatron/lib/python3.11/site-packages/megatron/core/transformer/transformer_config.py:1705: UserWarning: full scope is deprecated. Use empty cuda_graph_scope to capture the whole layer.
  warnings.warn(
/root/swift_megatron/lib/python3.11/site-packages/megatron/core/transformer/transformer_config.py:1705: UserWarning: full scope is deprecated. Use empty cuda_graph_scope to capture the whole layer.
  warnings.warn(
/root/swift_megatron/lib/python3.11/site-packages/megatron/core/transformer/transformer_config.py:1705: UserWarning: full scope is deprecated. Use empty cuda_graph_scope to capture the whole layer.
  warnings.warn(
/root/swift_megatron/lib/python3.11/site-packages/megatron/core/transformer/transformer_config.py:1705: UserWarning: full scope is deprecated. Use empty cuda_graph_scope to capture the whole layer.
  warnings.warn(
[rank9]: Traceback (most recent call last):
[rank9]:   File "/root/swift_megatron/lib/python3.11/site-packages/megatron/core/transformer/spec_utils.py", line 119, in build_module
[rank9]:     return module(
[rank9]:            ^^^^^^^
[rank9]:   File "/root/swift_megatron/lib/python3.11/site-packages/megatron/core/extensions/transformer_engine.py", line 1829, in __init__
[rank9]:     super().__init__(
[rank9]:   File "/root/swift_megatron/lib/python3.11/site-packages/megatron/core/extensions/transformer_engine.py", line 1532, in __init__
[rank9]:     super().__init__(
[rank9]:   File "/root/swift_megatron/lib/python3.11/site-packages/transformer_engine/pytorch/module/grouped_linear.py", line 826, in __init__
[rank9]:     self.reset_parameters(defer_init=is_meta)
[rank9]:   File "/root/swift_megatron/lib/python3.11/site-packages/transformer_engine/pytorch/module/grouped_linear.py", line 920, in reset_parameters
[rank9]:     super().reset_parameters(defer_init=defer_init)
[rank9]:   File "/root/swift_megatron/lib/python3.11/site-packages/transformer_engine/pytorch/module/base.py", line 1435, in reset_parameters
[rank9]:     with get_rng_state_tracker().fork(self.rng_tracker_name):
[rank9]:   File "/usr/lib/python3.11/contextlib.py", line 137, in __enter__
[rank9]:     return next(self.gen)
[rank9]:            ^^^^^^^^^^^^^^
[rank9]:   File "/root/swift_megatron/lib/python3.11/site-packages/megatron/core/tensor_parallel/random.py", line 250, in fork
[rank9]:     orig_cuda_rng_state = _get_cuda_rng_state(graph_safe=self.use_cudagraphable_rng)
[rank9]:                           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank9]:   File "/root/swift_megatron/lib/python3.11/site-packages/megatron/core/tensor_parallel/random.py", line 55, in _get_cuda_rng_state
[rank9]:     return torch.cuda.random.get_rng_state(device=device)
[rank9]:            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank9]:   File "/root/swift_megatron/lib/python3.11/site-packages/torch/cuda/random.py", line 42, in get_rng_state
[rank9]:     return default_generator.get_state()
[rank9]:            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank9]: torch.AcceleratorError: CUDA error: an illegal memory access was encountered
[rank9]: Search for `cudaErrorIllegalAddress' in https://docs.nvidia.com/cuda/cuda-runtime-api/group__CUDART__TYPES.html for more information.
[rank9]: CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.
[rank9]: For debugging consider passing CUDA_LAUNCH_BLOCKING=1
[rank9]: Compile with `TORCH_USE_CUDA_DSA` to enable device-side assertions.

我的机器是96Gx10, 对122B绝对是充裕的, 397B也基本不应该OOM才对.

[Jintao-Huang]

expert_model_parallel_size 我看设置的是1

[edgeinfinity1]

expert_model_parallel_size 我看设置的是1

我使用的是仅pp分布式, 这一做法在qwen3模型上也不会导致这个问题. 这和这个问题有直接关系吗?

[Jintao-Huang]

我看到了，是可以的

ms-swift/examples/models/qwen3_5/packing.sh

Line 9 in 37d9c87

--model Qwen/Qwen3.5-35B-A3B \

我跑这个是正常的，你试试

[edgeinfinity1]

我看到了，是可以的

ms-swift/examples/models/qwen3_5/packing.sh

Line 9 in 37d9c87
--model Qwen/Qwen3.5-35B-A3B \

我跑这个是正常的，你试试

我测试了一下, 同样参数下35B确实是可以的, 能过第一个step, 但是122和397不行.

接下来怎么办? 你那边尝试122B复现一下吗?

[Jintao-Huang]

感觉就是OOM

96Gx10 跑122B 感觉不一定能跑起来的 你试试 96 * 16呢

[edgeinfinity1]

感觉就是OOM

96Gx10 跑122B 感觉不一定能跑起来的 你试试 96 * 16呢

122B只占不到30%的显存 这是lora
同样的参数下 qwen3-235B也是没有问题的
我之前用仅576G的设备跑235B也是可以正常开offload的

[edgeinfinity1]

而且在不开offload的情况下, 397B的训练也是可以启动的, 我只是必须把rank调得比较低
offload是一个节约显存的设置, 没有理由它会导致占用更多的显存吧?

[Jintao-Huang]

哦我看到了，offload 对lora的影响不明显的。因为优化器的内容本身就不多的

[edgeinfinity1]

哦我看到了，offload 对lora的影响不明显的。因为优化器的内容本身就不多的

相对是不明显, 但是我的显存对于397是比较紧张的. 之前在针对qwen3的测试中, 开启offload节省显存用于提高lora_rank有很显著的效果.

我还是十分希望解决这个问题并开启offload, 如果有进展或者需要我配合测试的内容, 请告诉我.

[edgeinfinity1]

我更新了TransformerEngine到>=2.16, 这一问题解决了. 感谢项目人员协助排查.