PERF: SegmentFeature Filters, IdentifySample, FillBadData OoC Optimization

[joeykleingers]

Summary

• Optimizes 5 Group D filters (ScalarSegmentFeatures, EBSDSegmentFeatures, CAxisSegmentFeatures, FillBadData, IdentifySample) for out-of-core (ZarrStore) performance
• Adds chunk-sequential Connected Component Labeling (CCL) with Union-Find to SegmentFeatures base class, replacing DFS flood-fill for OOC paths
• Adds rolling slice buffer pre-loading to eliminate per-element OOC overhead during CCL neighbor comparisons
• Runtime dispatch selects BFS/DFS for in-core and CCL for OOC based on storage type
• Splits FillBadData and IdentifySample into separate BFS/CCL algorithm files
• Adds UnionFind.hpp utility class with path-halving compression and union-by-rank
• Restructures all 5 filter test suites with tier-based exemplar testing pattern

Algorithm Decision

Problem

The original DFS flood-fill in SegmentFeatures uses random data access — each stack pop reads an arbitrary voxel then checks 6 scattered neighbors. For in-core DataStore this is fast, but for ZarrStore-backed arrays every random access can trigger a chunk load/evict cycle, causing 3,000x–8,000x slowdowns on 200^3 datasets.

Solution: Chunk-Sequential CCL with Union-Find (Segment Features)

A multi-phase scanline algorithm that processes the grid in strict Z-Y-X order:

Phase 1 — Forward Labeling: Iterate every voxel in scanline order. For each valid voxel, check only backward neighbors (-X, -Y, -Z for face connectivity, or all 13 backward neighbors for FaceEdgeVertex). Assign provisional labels and union equivalences in a Union-Find structure. A 2-slice rolling buffer (current + previous Z-slice) keeps all neighbor reads in RAM.

Phase 1b — Periodic Boundary Merge (when isPeriodic=true): The forward pass cannot detect periodic connections because the wrapped neighbor has a higher linear index and hasn't been processed yet. This phase reads back provisional labels from the data store and unites labels of similar voxels on opposite boundary faces. Face connectivity checks the 3 axis-aligned face pairs; FaceEdgeVertex checks all 26-neighbor pairs that wrap across any boundary.

Phase 2 — Resolution and Relabeling: Flatten the Union-Find, then two sequential passes remap provisional labels to final feature IDs (assigned in seed-discovery order for compatibility with the original DFS numbering).

Rolling Slice Buffer Pre-Loading

Even after converting to CCL, the areNeighborsSimilar() virtual method still reads input arrays (quaternions, phases, mask, scalar data) through the OOC operator[] path. Each access costs ~50–100 ns due to virtual dispatch → mutex lock → N-D position calculation → chunk cache search → element access → mutex unlock. For EBSD with face connectivity, each voxel triggers ~36 OOC reads (8 quat + 2 phase + 2 validity per backward neighbor × 3 neighbors), totaling 300M+ OOC accesses for a 200^3 dataset.

The prepareForSlice() virtual hook in executeCCL() allows subclasses to pre-load input data for the current Z-slice into local std::vector buffers at the start of each slice iteration. All neighbor comparisons then read from these buffers with zero OOC overhead. The buffers use a rolling 2-slot scheme (slot = iz % 2) so only current + previous Z-slice are in memory. Phase 1b disables buffering via prepareForSlice(-1) since it accesses arbitrary Z-slices, but Phase 1b is O(dim²) so the overhead is negligible.

Memory cost for a 2000³ EBSD dataset: 128 MB (quats) + 32 MB (phases) + 8 MB (mask) = 168 MB for the rolling buffers — acceptable for a workstation processing a 128 GB quats array.

Why This Algorithm

1. Eliminates random access entirely — every read is from the rolling buffer (RAM) or a sequential chunk load
2. Memory scales with slice size, not volume — O(dimX × dimY) for the buffer + O(num_features) for Union-Find
3. Preserves feature numbering order — Phase 2 remaps to match original DFS seed-discovery order
4. Shared base class — executeCCL() in SegmentFeatures serves all 3 subclasses via isValidVoxel() and areNeighborsSimilar() virtual methods

Dispatch

Each filter checks IsOutOfCore(*featureIdsArray) || ForceOocAlgorithm():

• True → executeCCL() (chunk-sequential CCL)
• False → execute() (original DFS flood-fill)

FillBadData Algorithm

Split into FillBadDataBFS.cpp (in-core) and FillBadDataCCL.cpp (OOC) with dispatch in the main algorithm class:

1. CCL with negative labels for bad-data regions
2. Flatten Union-Find and accumulate region sizes
3. Classify regions by size (small → fill, large → preserve as voids)
4. Iterative morphological dilation to fill marked regions

IdentifySample Algorithm

Split into IdentifySampleBFS.cpp (in-core) and IdentifySampleCCL.cpp (OOC) with shared logic in IdentifySampleCommon.hpp:

1. CCL to label all connected components in the mask array
2. Identify the largest component by voxel count (this is the "sample")
3. Set all non-sample mask voxels to false
4. Optional hole-filling pass using iterative dilation within the sample bounding box
5. Slice-by-slice mode processes each 2D slice independently on the specified plane (XY/XZ/YZ)

DispatchAlgorithm<IdentifySampleBFS, IdentifySampleCCL> selects the path at runtime based on mask array residency.

Bug Fixes

• Restore periodic boundary condition neighbor wrapping in SegmentFeatures — the m_IsPeriodic flag was set but never read. Add periodic wrapping to both DFS (getFaceNeighbors/getAllNeighbors) and CCL (Phase 1b boundary merge) paths.
• Fix phantom feature from hardcoded seed=0 in DFS execute() — use getSeed() to skip masked voxels, preventing inflated feature counts and ghost Active entries.

Test Restructuring

• Add SegmentFeaturesTestUtils.hpp with shared builder and verification functions
• Restructure all 5 filter test suites to the tier-based exemplar pattern:
  • Tier 1 (200^3): exemplar comparison with GENERATE(false, true) for both algorithm paths
  • Tier 2 (15^3): exemplar comparison, every parameter combo with GENERATE(false, true) for both algorithm paths
  • Error, Randomize, High Tolerance, and FaceEdgeVertex connectivity tests
• Publish 3 exemplar archives to Data_Archive: segment_features_exemplars.tar.gz, fill_bad_data_exemplars.tar.gz, identify_sample_exemplars.tar.gz
• All exemplar datasets were programmatically validated (feature counts, array shapes, Active array integrity, mask consistency) and manually validated in DREAM3D-NX
• Orientation test data uses clean Z-layer quaternion scheme (0°/30°/60° X-axis rotations in EBSDlib vector-scalar order) with a merge pair override for visual verification

OOC Performance (200^3 dataset, arm64 Release, 1 Z-slice per chunk)

Filter	OOC Before	OOC After	Speedup
ScalarSegmentFeatures	708.3s	~2.8s	253x
EBSDSegmentFeatures	972.6s	~8.3s	117x
CAxisSegmentFeatures	824.1s	~5.9s	140x
FillBadData	8.6s	~5.6s	1.5x
IdentifySample	825.0s	~3.9s	212x

All timings measure the filter's execute() method only, excluding test setup (data generation, exemplar loading, result comparison). This reflects the actual cost a user would see when running the filter in a pipeline.

FillBadData baseline was already partially OOC-friendly, explaining the modest improvement.

Test Plan

• 29/29 Group D tests pass (in-core build)
• 29/29 Group D tests pass (OOC build)
• All Tier 1 and Tier 2 tests run both in-core and OOC algorithm paths via GENERATE(false, true)
• FaceEdgeVertex (26-neighbor) connectivity tested for all 3 segment features filters
• Periodic BC tested in both DFS and CCL paths (Scalar + EBSD)
• Exemplar datasets programmatically and manually validated in DREAM3D-NX