CosyVoice3.0 TRT LLM

[ChristianPalaArtificialy]

Is adding TRT support to version 3.0 on the roadmap? What is missing is the config.json file in the HF repository.

[aluminumbox]

the trtllm support is from nvidia @yuekaizhang hope they have some time to do it

[yuekaizhang]

Is adding TRT support to version 3.0 on the roadmap? What is missing is the config.json file in the HF repository.

Yeah, I would like start to support cosyvoice3 in a month.

[JohnHerry]

How about the flow-matching TRT fp16 support in cosyvoice3?

[BruceLee569]

Is adding TRT support to version 3.0 on the roadmap? What is missing is the config.json file in the HF repository.

Yeah, I would like start to support cosyvoice3 in a month.

Hope it can support the local environment. My machine does not support deploying docker images. 🫡

[Brakanier]

Hi, you can try trt flow + trt llm in my repo:
https://github.com/Brakanier/FastCosyVoice

[JohnHerry]

Hi, you can try trt flow + trt llm in my repo: https://github.com/Brakanier/FastCosyVoice

Thank you for the help. I had learned you pipeline of the kernel functions export_flow_decoder_estimator_onnx and convert_onn_to_trt
in the export_flow_decoder_estimator_onnx function ,there are two opitonal arguments optimize and fp16, but in your project both of them are set as default False, in this case ,this function has no difference with the export_to_onnx.py in the original CosyVoice project.
in the convert_onnx_to_trt function, you keep the SOTMAX layer precision as Fp32, but I am confused by this code:

try:
    output_tensor.dtype = tensor_dtype
except Exception:
    output_tensor.dtype = trt.DataType.FLOAT

will a value set cause exception?

If your pipeline is the same as my torch-> onnx fp32 -> trt fp16, I had tried that, it will fail for inference value overflow [Nan generated], and if trt bf16, will generate ill speech with noise.

[Brakanier]

Hi, you can try trt flow + trt llm in my repo: https://github.com/Brakanier/FastCosyVoice

Thank you for the help. I had learned you pipeline of the kernel functions export_flow_decoder_estimator_onnx and convert_onn_to_trt in the export_flow_decoder_estimator_onnx function ,there are two opitonal arguments optimize and fp16, but in your project both of them are set as default False, in this case ,this function has no difference with the export_to_onnx.py in the original CosyVoice project. in the convert_onnx_to_trt function, you keep the SOTMAX layer precision as Fp32, but I am confused by this code:

try:
    output_tensor.dtype = tensor_dtype
except Exception:
    output_tensor.dtype = trt.DataType.FLOAT

will a value set cause exception?

If your pipeline is the same as my torch-> onnx fp32 -> trt fp16, I had tried that, it will fail for inference value overflow [Nan generated], and if trt bf16, will generate ill speech with noise.

1. You just need run FastCosyVoice instance with needed params (fp16, trt, trt llm), no need start convert script manually, all should auto conver to trt format. Try to run run_fast.py script with example.
2. Yep, got error with fp16 for SOTMAX. The strange code for allowing use fp32, as I remember (it was fixed with opus 4.5)

[JohnHerry]

I can't run the run_fast.py directly because our project is somewhat different with opensource cosyvoice, and now I am working on only the flow-matching part, which is ,also, somewhat different. but we had aligned parameters on the inference API. so what is valuable for me is the process you convert the pytorch checkpoint into tensorrt fp16, should we set the fp16=True and optimize=True to call export_flow_decoder_estimator_onnx ?
we had tested the onnx inference and the result is ok, so the problem arised from the process of onnxfp32 -> trt fp16; and I noticed that your tensorrt version is 10.11+, while mine is V9.3. I will try a newer version.

[Brakanier]

should we set the fp16=True and optimize=True to call export_flow_decoder_estimator_onnx ?

Yep, and you will get 2 artifacts, fp16 and optimized, but the optimized version did not show any improvements in my tests, I am using only fp16.
But flow matching is very similar to original repo, trt and fp16 for llm part - bring more speed up.

[zmy1116]

I did TRTLLM conversion for both llm part and the flow part.

The qwen-0.5 module can be converted following the basically same script from repo, but need to update tokenizer, as @Brakanier did at https://github.com/Brakanier/FastCosyVoice/blob/main/scripts/convert_cosyvoice3_to_hf.py

For the flow part:

• once llm part is optimized.. the computation bottleneck is actually on the flow part
• torch -> onnx -> trt fp32 doesnt really accelerate much (at least on L4, T4 GPU there is no difference comparing with torch model)
• torch -> onnx -> trt fp16 accelerate signifcantly, but over 10 flow matching steps the discrepancy is so much larger that the final generated audio is very noisy.

Therefore, I did the the flow estimator part also in TRTLLM.

• rewrite the estimator model with TRTLLM API
• weights mapping
• trtllm build

On L4 I see 5x speed improvement on flow part with trtllm. For ~ 1000 length melspec generation (including reference part):

• torch model takes ~1.2 sec
• trtllm model takes ~0.2 sec

I would like to contribute. @yuekaizhang would you mind reviewing my conversion code for the flow part once I share my code? While I can confirm it works... I sort of hacked everything together and I dont think I follow best practices, and I have so many questions...

Thanks

[yuekaizhang]

@zmy1116 Great! You are very welcome to contribute PR. If there's anything I can help with, feel free to @ me directly.

[ukemamaster]

@zmy1116 any updates on this?

[zmy1116]

@zmy1116 any updates on this?

#1796

[ukemamaster]

@zmy1116
#1796 thanks for this. Do you plan to support streamig?