Skip to content

Data loading optimizations#46

Merged
PatrickRMiles merged 9 commits intoLBANN:mainfrom
PatrickRMiles:miles30/dataset_format_v2_loader_perf_new
Apr 3, 2026
Merged

Data loading optimizations#46
PatrickRMiles merged 9 commits intoLBANN:mainfrom
PatrickRMiles:miles30/dataset_format_v2_loader_perf_new

Conversation

@PatrickRMiles
Copy link
Copy Markdown
Collaborator

  • shift dataloader preprocessing work into dataset generation for speedup, maintaining
    support for old datasets
  • restore .contiguous() and dtype cast calls, but change order to avoid
    redundant copies
  • Make dataloader num_workers user-configurable

@michaelmckinsey1 michaelmckinsey1 self-requested a review April 3, 2026 15:50
@PatrickRMiles PatrickRMiles merged commit 63b598a into LBANN:main Apr 3, 2026
1 check passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants