2026-03-24 Australia Team

[joecastiglione]

Agenda

• Admin
  • Meeting Structure & Cadence
  • Meeting Time Zone Accommodation Poll Results
    • Friday 9a EADT / 8a EAST?
  • Phase 11C Schedule Review
• EET Tasks
  • Implementation and Improvement (Outer Loop)
  • Updated Random Number Generation: RNG Alternatives Testing (RSG)
  • Meeting Recording
  • Meeting Notes
  • Victoria EET Research update?
• Engineering Updates
  • Skip-on-fail default configs #1023
  • Testing
    • Testing Plan
    • Draft component test for the park-and-ride scenario #1001

Notes

Admin Items

Meeting Time

The group is working to establish a regularly scheduled meeting slot compatible with Australian time zones. Based on a poll, a Thursday U.S. / Friday Australia slot around 9:00–9:30 AM AEST appears to be the leading option, though not universally available. Joe will resend the poll link; Shaun offered to forward it to TfNSW staff. The goal is to hold at least every other engineering meeting at this Australia-friendly time, resulting in roughly three meetings per month.

Phase 11 Review Schedule & Update Cadence

Joe shared a task schedule showing when formal presentations are planned. Gaps exist where some consultants have no updates scheduled for weeks at a time. The approach agreed upon previously — and reaffirmed here — is to carry standing agenda items for each task so consultants can give informal check-ins, even if just to confirm no issues. This avoids surprises later in the schedule.

---

EET Briefing — Jan Zill (Outer Loop)

Jan repeated the presentation given earlier in the day for the benefit of Australian participants who could not attend the morning session. Key points:

Background

Explicit Error Terms (EET) is an alternative to Monte Carlo simulation for modeling individual choices in ActivitySim. A proof-of-concept was implemented in 2022, evaluated as useful in Phase 10, and is now being made production-ready.

Tests & Documentation

EET touches nearly every choice model in ActivitySim and must be thoroughly tested. The plan covers unit tests for core models, parity tests against Monte Carlo, and integration tests. A merge request with tests and documentation is targeted for end of next week.

Runtime Problem & Fix

Phase 10 testing revealed major slowdowns, especially in disaggregate accessibility (~12x slower) and location choice models (3–6x). The root cause: EET requires a random error term per alternative per chooser, and ActivitySim's current implementation re-seeds the random number generator for every chooser on every sample pass. Because location choice uses importance sampling (typically 30–50 passes), the number of random draws that must be generated and discarded grows quadratically — potentially hundreds of billions for large models.

A vectorized fix was implemented: drawing all random numbers at once in a single NumPy call rather than in a Python for loop. For a profiling test case, this reduced runtime from 281 seconds to 3.4 seconds. Compared to the Phase 10 baseline, the disaggregate accessibility slowdown factor dropped from ~242x to ~11x, with location choice models now running 1.2–1.6x slower than Monte Carlo (down from 2.9–6.2x). A merge request incorporating this fix is nearly ready. A memory trade-off exists — storing all random numbers at once requires a larger array — but ActivitySim's chunking mechanism should be able to manage this.

Paths to Further Runtime Improvement

Two strategies are under investigation:

1. Faster random draws — RSG/WSP/Driftless are evaluating alternative random number generators (RNGs) that may reduce per-draw cost while meeting ActivitySim's reproducibility requirements.
2. Fewer random draws — Outer Loop is exploring single-pass sampling approaches (Poisson sampling, best-N, or a mixture) that would replace repeated importance sampling with replacement. These would reduce draws proportionally to the number of sample passes. Jan plans to present findings in ~3 weeks.

Inconsistency Investigation

Two known sources of noise in scenario comparisons are being addressed:

• Sample indexing: error terms are currently keyed to sample position rather than zone ID, causing changes when the sample set changes. Fix: index by zone ID.
• Shadow pricing: random number offsets are not reset between shadow pricing iterations, causing both runtime penalties and potential instability. Fix: reset offsets per iteration.

---

Victoria / VLC Update

Paris (VLC's client representative) shared that VLC independently encountered the same severe runtime issues when testing EET on the Victorian Integrated Model (VINM2), seeing slowdowns of up to 50x in some modules — likely amplified by Victoria's sample size of 50 (vs. the consortium's typical 30) and shadow pricing behavior. Rather than simply benchmarking the prototype, VLC redirected scope to implement fixes directly. Three changes were made:

1. Poisson sampling: Instead of asking "should this zone be in the sample?" 50 separate times, Poisson sampling asks once — "what is the probability this zone appears in a sample of 50?" — generating one random number per alternative rather than repeating the process. This dramatically reduces draws.
2. Zone-indexed error terms: Fixed the inconsistency where error terms were tied to sample array position rather than zone ID, matching what Jan's team independently identified.
3. Shadow pricing randomness reset: Fixed the RNG offset accumulation between shadow pricing iterations to improve convergence stability.

These changes brought runtime from ~50x slower than Monte Carlo down to roughly 10–20% slower. Behavioral results look promising: demand matrices are more stable, and illogical choices are reduced. Cost-benefit analysis testing is underway and showing positive early results. VLC's development work is complete; they are now in internal testing and expect to be ready to share the implementation soon.

VLC intends to present their full results to the consortium and is also being encouraged to submit a paper to the Australasian Transport Research Forum (ATRF) in November. The group noted that Poisson sampling also appears applicable to Monte Carlo (not just EET), potentially reducing runtime for the standard approach as well.

---

Action Items

Owner	Action
Joe	Resend meeting time poll; send to Shaun for TfNSW distribution
Shaun	Forward meeting poll to TfNSW team
Jan Zill	Submit EET tests + documentation merge request by end of next week
Jan Zill	Present runtime reduction (sampling methodology) comparison in ~3 weeks
Jan Zill	Coordinate with VLC/Paris on Poisson sampling implementation details
VLC / Paris & Kate	Complete internal testing; prepare to share implementation with consortium
Kate Johns	Schedule VLC presentation to consortium (Wednesday meeting)
VLC	Submit paper to ATRF (November)