2026-04-09 Engineering Team

[joecastiglione]

Agenda

Items with Progress

• We apparently have a problem with non-deterministic sort order on ProtoPop persons, which has become evident on BOTH of the next two items.

  • Sort order problem appears in 1 of 2 tests of SANDAG-Example#41 here

• Utility not aligned #1022 - possible cause identified and fix implemented in #1027, pending review from PSRC

• Remove Three Zone Code and Examples #1056 and SANDAG-Example#41

• Lower default threshold for skipped failed choices to 0.1% #1063 - Awaiting note in changelog then merge

• Explicit Error Terms #1064

• PopulationSim intermediate geographies camsys#1

Items without Progress

• Park and Ride #1001 - review pending

• Speed Up Nearest Zone Calculation in Disaggregate Accessibilities #1031 - review pending

• Adding ignore list to skim load for unused skims while running with sharrow #1036 - review pending

• Implement deletion of temporary variables in summarize.py #1062 - pending docs/testing

• Model component recompilation when running MP with sharrow.

• Comprehensive testing plan.

• A bunch of other contributed or smaller PRs

New Contributed PRs

• Poisson sampling methodology for EET #1065 - Draft proposed new algorithm for sampling

[jpn--]

Meeting Overview

Agenda

• Non-deterministic sort order issue in proto population persons table
• PSRC full-scale two-zone model CI failures related to NumPy/sparse integer overflow bug
• Removal of three-zone system from ActivitySim (David Hensle's PR)
• Explicit error term (EET) pull request status update (Jan Zill)
• PopulationSim progress update

Participants

• Joe Castiglione (@joecastiglione) — Zephyr Foundation
• Jeff Newman (@jpn--) — Driftless
• Sijia Wang (@i-am-sijia) — WSP
• Jan Zill (@janzill) — Outer Loop
• Joe Flood (@JoeJimFlood) — CS

Duration

Approximately 50 minutes

---

Executive Summary

Jeff Newman identified and resolved two related CI test reliability issues: a non-deterministic sort order in the proto population persons table, and a NumPy/sparse integer overflow bug causing failures in PSRC's full-scale two-zone model on Windows. The root fix for both involves upgrading NumPy to version 2, which standardizes 64-bit integer defaults across platforms and eliminates a longstanding class of Windows-specific bugs. Jan Zill provided an update on the explicit error term (EET) pull request, which is nearly ready for review and merge, pending addition of a zone-ID-based ordering fix for sampled choices. A PopulationSim config-reading issue remains blocked pending Nick Fournier's return.

---

Meeting Notes

Non-Deterministic Sort Order in Proto Population

Jeff Newman described a non-deterministic issue discovered in the proto population persons table. The problem surfaced when David Hensle's pull request removing an unused setting from the SANDAG model caused the CI tests to run twice on identical code — one pass, one failure — because the person row ordering differed between runs.

The root cause appears to be that the existing sorting code sorted on a single key (person ID) while implicitly relying on the table already being partially sorted by household. When that implicit assumption fails, the row order is non-deterministic, causing disaggregate accessibility to use different random number seeds (which are keyed to index row number), leading to approximately 1% variance in log sum outputs. While small, this is enough to cause CI regression test failures.

The fix is to make all sorting explicit by adding compound sort keys (e.g., sort the persons table by household ID and person ID, and the tours table by multiple keys), so the sort outcome is fully determined without relying on any prior ordering. Jeff noted this only needs to be done in one place — the proto population generation — since the rest of ActivitySim operates on an exogenously provided population with stable IDs.

Joe Castiglione asked whether the fix introduces any real cost. Jeff confirmed the overhead is negligible and that going forward all sorts should be made explicit rather than relying on stable-sort assumptions.

NumPy Integer Overflow Bug (PSRC Two-Zone Model)

Jeff Newman also described a related bug causing PSRC's full-scale two-zone model to fail in Sharrow mode when running under uv on Windows. The failure arises from a specific combination of conditions:

• Running on Windows (which defaults to 32-bit integers)
• NumPy version < 2
• The sparse library at certain versions
• Sharrow enabled
• A zone matrix large enough that the number of cells exceeds the 32-bit integer limit (~42,000 zones × 42,000 zones)

Under these conditions, sparse matrix indices overflow, causing values in a portion of the walk table to be misaligned.

The solution Jeff implemented is to upgrade NumPy to version 2, which standardizes on 64-bit integers across all platforms. This required also upgrading Pandera (pinned at v18, which is incompatible with NumPy 2) and a few other dependencies. Jeff noted that existing workaround code (conditionals guarding against int32 overflow) can remain in place harmlessly — it will simply never trigger — and can be cleaned up later.

Sijia Wang clarified that the overflow issue is specific to Sharrow mode (not legacy/non-Sharrow). Jeff confirmed that PSRC's model runs fine with Sharrow disabled; the bug only surfaces when all the trigger conditions are met simultaneously. Stefan Coe is expected to test the fix on his Windows machine when he returns.

Three-Zone Code Removal (David Hensle)

David Hensle (not present) has completed and pushed the pull request removing the three-zone system code from ActivitySim. Jeff Newman noted this needs to be reviewed and merged.

Explicit Error Term (EET) Pull Request — Jan Zill

Jan Zill gave an update on the EET pull request. The PR includes documentation and unit tests for the explicit error term functionality. During pre-submission testing, Jan discovered an edge case in the existing trip scheduling model related to how trips get rescheduled on subsequent iterations in small-sample (20-household) test runs, which could produce divergent paths depending on multi- vs. single-processing. This was addressed with a random seed fix, documented in the PR.

One remaining item is a fix for alternative order sampling: currently, random numbers for sampled destination choices are indexed by sample index rather than zone ID, making them unstable. A fix based on MATS's implementation is in progress and could be added to the current PR or submitted separately. Jan indicated this could be completed within ~12 hours.

The PR has substantial changes and will require agency review. Jeff noted David Hensle's availability is limited in the near term. Jan confirmed the PR is otherwise ready to merge and will complete the ordering fix before tagging reviewers.

Sijia Wang asked whether the probabilistic trip scheduling changes were tested against the logit-based trip scheduling model (DTPS) used by ARC and others. Jan confirmed that the underlying choice mechanisms are unit tested, and that models using DTPS go through the same underlying ActivitySim Core path, which is covered.

Testing Infrastructure Discussion

The group discussed CI testing constraints. GitHub provides free CI runners for open-source repositories, but they are significantly slower than local machines (~6–10× slower per Jan). Current longer-running tests take 60–90 minutes on CI. Jeff noted the GitHub per-job cap may be around 3 hours (exact limit unclear), and that adding more tests should be done by breaking out additional jobs rather than extending existing ones.

Jeff expressed a preference for moving toward smaller, faster unit tests that each run in seconds, reserving large integration/regression tests for pre-release validation rather than every push. A more robust system for running those larger tests on non-GitHub infrastructure was noted as worth exploring, though it has been discussed without resolution multiple times in the past.

PopulationSim Config Issue — Joe Flood

Joe Flood reported that the PopulationSim issue — where the tool reads config files from two different directories, causing setting conflicts and crashes — remains unresolved. Nick Fournier, who owns this work, is out this week. The group agreed to wait for Nick's return before deciding next steps.

---

Action Items

• Jeff Newman: Finalize and merge the explicit sort fix in the proto population code to resolve non-deterministic ordering.
• Jeff Newman: Tag reviewers for David Hensle's three-zone removal PR and merge after review.
• Stefan Coe (upon return): Test the NumPy 2 upgrade fix on a Windows machine to confirm the PSRC two-zone/Sharrow failure is resolved.
• Jan Zill: Add the zone-ID-based ordering fix for sampled choices to the EET pull request, then tag reviewers for agency review.
• Jeff Newman: Tag agency reviewers (including RSG) for the EET pull request once Jan finalizes it.
• Joe Flood: Follow up with Nick Fournier on the PopulationSim config directory issue when Nick returns.
• Jeff Newman / team: Explore options for running larger-scale integration tests outside the GitHub CI free tier, as a longer-term testing infrastructure improvement.

[janzill]

@jpn-- looks like tagging reviewers for the EET PR (ActivitySim/activitysim#1064) is on your and my todo list :) The work is done but I cannot add reviewers, could you do this please?