Fix mm server

[tastelikefeet]

PR type

• Bug Fix
• New Feature
• Document Updates
• More Models or Datasets Support

PR information

Write the detail information belongs to this PR.

Experiment results

Paste your experiment result here(if needed).

[gemini-code-assist]

Summary of Changes

Hello, I'm Gemini Code Assist¹! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!

This pull request primarily addresses issues related to the multi-modal server setup and introduces a comprehensive example for multi-modal LoRA training. It refines server configurations for specific models, enhances debugging capabilities in data loading, and improves the flexibility and robustness of model and input handling within the Twinkle framework. The changes aim to stabilize the server and provide clearer guidance for multi-modal fine-tuning workflows.

Highlights

• Server Configuration Update: The server_config_4b.yaml file was updated to disable Megatron for the Qwen3.5-4B model and explicitly specify Qwen3_5ForConditionalGeneration as the model class, streamlining the model serving setup.
• New Multi-Modal Training Example: A new Python script, multi_modal.py, was added as an example for fine-tuning multi-modal models (specifically Qwen3.5-4B) using LoRA within the Twinkle client-server architecture. This script demonstrates dataset preparation, model configuration, and the training loop.
• Improved Error Handling in DataLoader: The retry_sampler.py now includes traceback.print_exc() within its exception handling, providing more detailed debugging information when data loading retries fail.
• Flexible Model Class Loading: The multi_lora_transformers.py file was modified to dynamically load model classes from the transformers library using getattr, allowing for more flexible model instantiation based on string names.
• Multi-Modal Token Type Support: The InputProcessor in base.py was updated to include mm_token_type_ids in its padding configuration, enhancing support for multi-modal inputs.
• Robust Input Handling in Server: The forward_backward method in twinkle_handlers.py now explicitly converts list-based inputs to torch.tensor objects before passing them to the model, ensuring correct data types for model processing.

🧠 New Feature in Public Preview: You can now enable Memory to help Gemini Code Assist learn from your team's feedback. This makes future code reviews more consistent and personalized to your project's style. Click here to enable Memory in your admin console.

Using Gemini Code Assist

The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.

Invoking Gemini

You can request assistance from Gemini at any point by creating a comment using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands on the current page.

Feature	Command	Description
Code Review	/gemini review	Performs a code review for the current pull request in its current state.
Pull Request Summary	/gemini summary	Provides a summary of the current pull request in its current state.
Comment	@gemini-code-assist	Responds in comments when explicitly tagged, both in pull request comments and review comments.
Help	/gemini help	Displays a list of available commands.

Customization

To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.

Limitations & Feedback

Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for GitHub and other Google products, sign up here.

Footnotes

1. Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution. ↩

[gemini-code-assist]

Code Review

The pull request modifies the megatron server configuration to disable megatron and specify the model class. It also introduces a new example script for fine-tuning a language model using LoRA. Additionally, the pull request includes changes to the retry sampler to print the traceback on exceptions, adds transformers import and a check for model_cls type in multi_lora_transformers, adds mm_token_type_ids to InputProcessor, and converts lists to tensors in twinkle_handlers. The reviewer pointed out that the os module was not imported in multi_modal.py and suggested adding import os at the top of the file. The reviewer also suggested a more robust way to convert lists to tensors in twinkle_handlers.py by checking if the list contains numbers before attempting conversion.

[gemini-code-assist]

This logic for converting lists to tensors is not robust. It assumes any value that is a list should be converted to a torch.tensor. This could fail if a field contains a list of non-numeric types, such as a list of strings. A safer approach is to check if the list contains numbers before attempting conversion.

Suggested change

	if isinstance(inputs[key], list):
	inputs[key] = torch.tensor(inputs[key])
	if isinstance(inputs[key], list) and inputs[key] and isinstance(inputs[key][0], (int, float)):
	inputs[key] = torch.tensor(inputs[key])