For finetuning, add an alternative to LlamaFactory

[zhangfaen]

For finetuning, add an alternative to LlamaFactory

[zhangfaen]

https://github.com/zhangfaen/finetune-Qwen2-VL has 50+ stars and many repost in twitter.
I think it is a good thing to let more people to know and use QwenLM/Qwen2-VL.
Would you please merge this PR? many thanks

[zhangfaen]

Thank you deeksha! Hope this PR will be merged to main branch soon.

[CarlHuangNuc]

Cound you have a plan to add continue pretraining support ?

[zhangfaen]

Cound you have a plan to add continue pretraining support ?

I think Finetune itself is kind of continue pretraining.

[CarlHuangNuc]

Cound you have a plan to add continue pretraining support ?

I think Finetune itself is kind of continue pretraining.

yes, you are right. I want to add some new task in stage2 (multi_task pretraining) and then SFT on stage 3. whether your PR can more easy for our project. The biggest obstacle is that we can not obtain the model output from stage 1 from Qwen2-VL team which just release stage3 model(Qwen2-VL-7B-instruct).

[zhangfaen]

Below is from Qwen2-vl tech report:

Following Qwen-VL (Bai et al., 2023b), we adopt a three-stage training methodology. In the first stage, we
focus exclusively on training the Vision Transformer (ViT) component, utilizing a vast corpus of image-text
pairs to enhance semantic understanding within the Large Language Model (LLM). In the second stage, we
unfreeze all parameters and train with a wider range of data for more comprehensive learning. In the final
stage, we lock the ViT parameters and perform exclusive fine-tuning of the LLM using instructional datasets.

It seems the stage2 is just to optimize all parameters. For your purpose, I think it should be OK that you use https://github.com/zhangfaen/finetune-Qwen2-VL and prepare your data.

In case you want to do stage3, you can just add a few lines in my code script, for example:

model.vision_model.requires_grad_(False) # 冻结这module中的参数， 不更新... 
model.language_model.requires_grad_(False)  # 冻结这module中的参数， 不更新... 

logger.info(f"total params for Lora training: {sum(p.numel() for p in model.parameters())}")
logger.info(f"total trainable params for Lora training: {sum(p.numel() for p in model.parameters() if p.requires_grad)}")

[CarlHuangNuc]

thanks,,good idea.

[CarlHuangNuc]

Why Qwen-VL release Qwen-VL + Qwen-VL-Char, but Qwen2-VL only release Qwen2-VL-Instruct , do not release Qwen2-VL which maybe belong to pretrain model.

[ahmedheakl]

I agree with @CarlHuangNuc.

Can you guys please provide your pretraining code? @deekshaaneja