Skip to content

Extend lazy MFD dask assembly to stream/hand modules and study the convergence phase #2885

@brendancol

Description

@brendancol

Follow-up to #2865, which made the dask output assembly lazy in flow_accumulation_mfd, flow_length_mfd, and watershed_mfd.

Remaining eager paths

The same eager-assembly pattern (.compute() per tile, then da.from_array / da.block) lives in other MFD dask paths that #2865 left untouched to keep its diff scoped and reduce conflicts with parallel hydro work:

  • xrspatial/hydro/stream_link_mfd.py, _stream_link_mfd_dask
  • xrspatial/hydro/stream_order_mfd.py
  • xrspatial/hydro/hand_mfd.py

These can take the same da.map_blocks treatment. They have extra secondary inputs (accumulation, stream rasters) that need the same chunk-alignment handling used for pour points in watershed_mfd.

Convergence phase

The iterative boundary-convergence sweep that runs before assembly still reads tile data eagerly. That is inherent to the algorithm: converging cross-tile boundaries needs real values across sweeps. Making it lazy would need a different approach (for example a fixed number of map_overlap rounds, or a delayed reduction), which is a separate design question worth its own investigation. Filing it here so it is tracked rather than lost.

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or requestoomOut-of-memory risk with large datasets

    Type

    No type
    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions