Draft: Data loading optimizations#44

Closed

PatrickRMiles wants to merge 29 commits intoLBANN:mainfrom

PatrickRMiles:miles30/dataset_format_v2_loader_perf

Commits on Apr 2, 2026

apply optimizer every batch, not every epoch; unscale gradients before clipping
Patrick Miles
committed
trainer tweaks
Patrick Miles
committed
apply optimizer every batch, not every epoch; unscale gradients before clipping
Patrick Miles
committed
extract warmup to separate method; switch to warming up set number of batches (user configurable)
Patrick Miles
committed
whitespace; num_workers revert
Patrick Miles
committed
ruff
Patrick Miles
committed
make dataloader num_workers user-configurable
Patrick Miles
committed
make dataloader num_workers user-configurable
Patrick Miles
committed
shift dataloader preprocessing work into dataset generation for speedup, maintaining support for old datasets
Patrick Miles
committed
data_loading.py: restore .contiguous() and dtype cast calls, but change order to avoid redundant copies
Patrick Miles
committed
ruff
Patrick Miles
committed
Apply optimizer every batch, not every epoch; unscale gradients before clipping (#40)

PatrickRMiles
and
Patrick Miles
authored
Warmup changes: only warm a few batches; extract to separate method in trainer class (#43)

authored
apply optimizer every batch, not every epoch; unscale gradients before clipping
Patrick Miles
committed
trainer tweaks
Patrick Miles
committed
apply optimizer every batch, not every epoch; unscale gradients before clipping
Patrick Miles
committed
extract warmup to separate method; switch to warming up set number of batches (user configurable)
Patrick Miles
committed
whitespace; num_workers revert
Patrick Miles
committed
make parallelstrategy, spatial_mesh, ddp_placements attrs of trainer; other small tweaks
Patrick Miles
committed
make dataloader num_workers user-configurable
Patrick Miles
committed
make dataloader num_workers user-configurable
Patrick Miles
committed
shift dataloader preprocessing work into dataset generation for speedup, maintaining support for old datasets
Patrick Miles
committed
data_loading.py: restore .contiguous() and dtype cast calls, but change order to avoid redundant copies
Patrick Miles
committed
ruff
Patrick Miles
committed
remove acciodental adds
Patrick Miles
committed
Merge branch 'miles30/dataset_format_v2_loader_perf' of github.com:PatrickRMiles/ScaFFold into miles30/dataset_format_v2_loader_perf
Patrick Miles
committed
remove accidental adds
Patrick Miles
committed
remove accidental adds
Patrick Miles
committed
remove accidental adds
Patrick Miles
committed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Draft: Data loading optimizations#44

Draft: Data loading optimizations#44
PatrickRMiles wants to merge 29 commits intoLBANN:mainfrom
PatrickRMiles:miles30/dataset_format_v2_loader_perf

Commits on Apr 2, 2026

apply optimizer every batch, not every epoch; unscale gradients before clipping

trainer tweaks

apply optimizer every batch, not every epoch; unscale gradients before clipping

extract warmup to separate method; switch to warming up set number of batches (user configurable)

whitespace; num_workers revert

ruff

make dataloader num_workers user-configurable

make dataloader num_workers user-configurable

shift dataloader preprocessing work into dataset generation for speedup, maintaining support for old datasets

data_loading.py: restore .contiguous() and dtype cast calls, but change order to avoid redundant copies

ruff

Apply optimizer every batch, not every epoch; unscale gradients before clipping (#40)

Warmup changes: only warm a few batches; extract to separate method in trainer class (#43)

apply optimizer every batch, not every epoch; unscale gradients before clipping

trainer tweaks

apply optimizer every batch, not every epoch; unscale gradients before clipping

extract warmup to separate method; switch to warming up set number of batches (user configurable)

whitespace; num_workers revert

make parallelstrategy, spatial_mesh, ddp_placements attrs of trainer; other small tweaks

make dataloader num_workers user-configurable

make dataloader num_workers user-configurable

shift dataloader preprocessing work into dataset generation for speedup, maintaining support for old datasets

data_loading.py: restore .contiguous() and dtype cast calls, but change order to avoid redundant copies

ruff

remove acciodental adds

Merge branch 'miles30/dataset_format_v2_loader_perf' of github.com:PatrickRMiles/ScaFFold into miles30/dataset_format_v2_loader_perf

remove accidental adds

remove accidental adds

remove accidental adds