Add multi-architecture support for ColPali (SmolVLM, Qwen, PaliGemma) #322
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Updated ColpaliEmbeddingModel to support dynamic model loading based on environment variable. Added support for ColIdefics3 architecture and adjusted batch sizes accordingly.
Enhance ColpaliEmbeddingModel with dynamic model loading
Added support for COLPALI model configuration.
Updated PDF DPI setting for ColPali processing.
Updated COLPALI_MODEL_NAME to use ColSmolVLM 256/500M architecture.
Added guide comment for colpali model setup.
Added support for PaliGemma model.
Updated comments for ColPali model selection and clarified default model name.
Collaborator
|
Thanks @husaynirfan1, could you pls restore the metrics collection, seems to be removed. Additionally, for non-secrets, it's preferred to have it be morphik.toml driven. Currently, you're reading model from .env. Could we make that toml riven pls? |
Changes from .env loading method to morphik.toml global config.
Added options for ColPali model selection in configuration.
Removed comments and default model name from .env.example
Updated ColPali model selection options with descriptions.
Contributor
Author
The final commit had corrected the processors for ColPali model, added metrics collection and utilize
|
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This PR refactors colpali_embedding_model.py to support dynamic loading of different ColPali architectures. Previously, the model class was hardcoded to ColQwen2_5, which prevented the use of lightweight models like SmolVLM or the standard PaliGemma checkpoints.
The system now detects the architecture type based on the COLPALI_MODEL_NAME environment variable and instantiates the correct class from colpali_engine.
Key Changes
Dynamic Imports: Added ColIdefics3 (for SmolVLM) and ColPali (for PaliGemma) to imports.
Model Selector: Implemented logic in init to route model loading:
Contains "smol" or "idefics" → Loads ColIdefics3 (with optimized batch sizes).
Contains "qwen" → Loads ColQwen2_5.
Default/Else → Loads ColPali (Standard v1.2/v1.3 behavior).
Environment Config: Now utilizes os.getenv("COLPALI_MODEL_NAME") to allow hot-swapping models without code changes.
How to Test
Update .env with a specific model:
Start the service.
Verify logs show: INFO: Detected SmolVLM/Idefics3 architecture.
Performance Impact
SmolVLM (256M/500M): significantly faster ingestion due to increased batch size (up to 32x) and smaller parameter count. (and give options to low-end users.)
Standard/Qwen: No change in performance; maintains previous batch settings.