feat: add greedy packing, MiniCPM packing support, and dataset progress tracking#7904

Open

Lollipop wants to merge 16 commits intomodelscope:release/3.12from

Lollipop:feat/greedy-packing-3.12

Commits on Jan 26, 2026

feat(minicpm): add packing support for MiniCPMV models
liuxiaoming
committed
feat(train): add dataset progress tracking to TensorBoard
liuxiaoming
committed
fix(minicpm): correct packing data format for MiniCPMV models
liuxiaoming
committed
refactor(train): use collator wrapper for dataset progress tracking
liuxiaoming
committed
fix(train): ensure all ranks participate in gather for dataset progress
liuxiaoming
committed
refactor(train): update dataset progress logging to show epoch count
liuxiaoming
committed
fix(dataset): improve streaming packing safety
liuxiaoming
committed
refactor(dataset): implement independent packing processor for subprocesses
liuxiaoming
committed
refactor(dataset): remove dataset progress tracking from Template class
liuxiaoming
committed
feat: add greedy_packing and tokens/s statistics
liuxiaoming
committed
feat(greedy_packing): add batch_size support and fix statistics
liuxiaoming
committed
fix(greedy_packing): use _batch_sources for dataset progress tracking
liuxiaoming
committed
fix(greedy_packing): collect sources and lengths before packing_collator
liuxiaoming
committed
feat(dataset): enhance progress tracking with original dataset sizes
liuxiaoming
committed
refactor: remove unnecessary spawn mode changes in IterablePackingDataset
liuxiaoming
committed
refactor(collator): improve ProgressTrackingCollator per code review
liuxiaoming
committed