Qwen3.5moe

[vvvdwbvvv]

Summary

Adding Qwen3.5_MOE model support
https://huggingface.co/docs/transformers/en/model_doc/qwen3_5_moe
Related issue: #1110

Note

There's undeterministic problem on all qwen's moe model series, not sure what it is. Maybe we should re-examine all the MoE model to check the convergence test
error log:

num_tokens_per_expert = torch.histc(histc_input, bins=self.num_experts, min=0, max=self.num_experts - 1)
E       RuntimeError: _histc_cuda does not have a deterministic implementation, but you set 'torch.use_deterministic_algorithms(True)'. You can turn off determinism just for this operation, or you can use the 'warn_only=True' option, if that's acceptable for your application. You can also file an issue at https://github.com/pytorch/pytorch/issues to help us prioritize adding deterministic support for this operation.

Testing Done

• Hardware Type:
• [x ] run make test to ensure correctness
• [x ] run make checkstyle to ensure code style
• [] run make test-convergence to ensure convergence

[albertvillanova]

I think for qwen3_5_moe model, mm_token_type_ids param should also be added to lce_forward, as already done for other VL models:

• #1120

See code in transformers:

• https://github.com/huggingface/transformers/blob/7235d44257e7e4765317d475622e2085a09c9e3b/src/transformers/models/qwen3_5_moe/modeling_qwen3_5_moe.py#L1840-L1851

CC: @kashif

[albertvillanova]

Suggested change

	mm_token_type_ids: Optional[torch.IntTensor] = None,
	cache_position: Optional[torch.LongTensor] = None,

[albertvillanova]

Suggested change

	mm_token_type_ids=mm_token_type_ids,
	cache_position=cache_position,

[vvvdwbvvv]

@albertvillanova Thank you for reviewing, I'll fix that.

[vvvdwbvvv]

closed for it is done at #1109