Question of .pt inference and eval

[zhangxuchen0806]

Great work!

If I want to perform SFT or RL training after the LLAVA-OV-1.5 Stage 2, should I use the script you provided to convert the hf weights to .pt format?

And then do I need to use this .pt format of the weights for training?

So how can I perform inference/evaluation on the trained .pt weights? Or is it only converted to HF format weights before doing inference and evaluation?

I'm not familiar with the Megatron training framework. Anyway, thanks.