2025-12-11 Engineering Team

[joecastiglione]

Agenda

Review of Status

Summary of Progress this week

• CDAP estimation bug fixed in ActivitySim#1026
• Trace IDs fix ActivitySim#1021 reviewed and merged
• Skip failed households PR #1023 substantially done, to review

Discussion

• Do we want to merge the EET branch?

[jpn--]

ActivitySim Engineering Team Meeting Notes

Meeting Overview

Agenda

• Status review of recent development work
• CDAP estimation bug fix discussion
• Skip failed households implementation approach
• UV status update across agencies
• Explicit error terms (ET) branch merge discussion
• Upcoming holiday meeting schedule

Participants

• Joe Castiglione (@joecastiglione) - Zephyr Foundation
• Jeff Newman (@jpn--) - Driftless
• David Hensle (@dhensle) - RSG
• Bo Wen (@bwentl) - TransLink

Duration

37 minutes

Executive Summary

Jeff Newman reported fixing a CDAP estimation bug involving interaction terms for three, four, and five-person households where parameter assignments were being overwritten due to identical logic keys. The team discussed implementation approaches for the skip failed households feature, with debate over component-level versus global settings. UV status updates indicated several agencies are still in progress, with acknowledgment that holiday schedules may slow completion. The group also discussed whether to merge the explicit error terms branch into main, with consensus to wait until after Phase 11C work is completed.

Meeting Notes

CDAP Estimation Bug Fix

Jeff Newman explained a bug in the CDAP estimation code where interaction terms for three, four, and five-person households were incorrectly assigned. The issue occurred because households of these sizes shared identical underlying logic for an interaction term (when everyone in the household had the same pattern day). During the loop process, because the logic was the same, the dictionary key was identical and got overwritten, resulting in all three household sizes using the five-person parameter instead of unique parameters for each size.

Jeff fixed the issue and opened a pull request, though noted the last commit didn't initially push. David Hensle asked about implications for previously estimated models. Jeff clarified that the bug only affects estimation and specifically the Larch code implementation. He noted that RSG likely caught this issue in their work since Joel originally flagged it. The bug affects approximately one out of 300 parameters in the CDAP model, impacting only three and four-person households while two and five-person households remain unaffected. Jeff emphasized that anyone estimating CDAP should dig deep into the details since it involves five models estimated simultaneously with overlapping parameters and complex interaction effects.

Jeff also reported that Sijia Wang completed the trace pull request successfully, which has been merged into main. David Hensle volunteered to review the CDAP fix.

Skip Failed Households Implementation

The team discussed the implementation approach for the skip failed households feature. Jeff explained that the original task called for a global option to skip failed households everywhere, but the implementation provides component-level options instead. David noted this was discussed at an engineering meeting Jeff missed, explaining that only specific models like trip destination and mode choice are expected to fail in edge cases. Having component-level settings would help identify unexpected errors in other models rather than silently skipping them with a global setting.

Joe Castiglione expressed confusion about the conclusion, noting that Sijia's email suggested reverting to a global setting. Jeff clarified that no final decision has been made and he will review the code Sijia wrote over the next two weeks before she returns from holiday break. The goal is to allow a small number of household failures in large-scale models (three or four out of a million households) without affecting statistical validity, while warning or failing if failure rates become significant (such as 20% of households).

The team discussed whether thresholds should be configurable, with Jeff confirming they will be with default options. David noted the current implementation appears to have a global setting and asked about implications for estimation mode. Jeff argued the logic should be the same across estimation and application modes - users shouldn't be able to ignore large amounts of missing data, but single corrupt records shouldn't stop the entire process. Joe shared a link to the shared drive where executed task orders are available for transparency on consultant scopes.

UV Status Update

Joe raised the issue of UV status across agencies, noting from engineering meeting notes that many agencies are at various stages of implementation. He mentioned ARC, SEMCOG, Metropolitan Council, CMAP, and Victoria are all waiting for information or consultant input. Jeff confirmed this was the main topic of the previous week's engineering meeting and acknowledged that December timing poses challenges with Sijia away for two weeks and many others taking holiday time. He suggested not rushing people before vacations and noted that among all Phase 11B tasks, UV implementation dragging into the new year would not be a disaster. Joe agreed while encouraging agencies to complete any outstanding items on their end when possible.

Explicit Error Terms Branch Merge Discussion

Jeff raised the question of whether to merge the explicit error terms branch into main. He noted the branch has been separate for seven months since technical implementation finished, with subsequent analysis work on performance and outcomes. While ET performance remains a concern, Jeff argued that if the consortium wants to continue developing and maintaining ET functionality, it should be merged to avoid leaving it behind. He warned that the longer they wait, the more difficult merging becomes as the main branch continues to evolve (currently 135 commits behind).

Bo Wen indicated TransLink is testing the random number generator for ET and doing additional testing. David suggested the branch might actually be on Jan's fork rather than the main repository and offered to confirm the correct location. Joe expressed concern about merging before resolving methodological questions, noting that Phase 11C includes output analysis work to investigate unexpected changes. David agreed it would be better to pull in one complete ET implementation rather than risk losing track of changes across partial merges.

Jeff acknowledged good arguments exist either way but emphasized the need to either prepare for merging or decide never to merge during Phase 11C to avoid compounding merge difficulties. Joe explained that Phase 11C ET work includes output analysis from both methodological and performance perspectives, plus random number generator work, but likely won't result in a fully settled solution. He suggested revisiting the merge question after Phase 11C when more information is available. David noted this merge appears less risky than past major merges like the state object, which touched nearly every file.

Meeting Schedule

Joe announced an all-hands meeting scheduled for Tuesday, December 23rd, and asked about attendance given holiday schedules. He indicated he would cancel the engineering meeting on Christmas Day (December 25th) and noted the following week's meeting falls on New Year's Day (January 1st), which will also be cancelled.

Action Items

• Jeff Newman: Push final commit for CDAP bug fix pull request and mark for review
• Jeff Newman: Review skip failed households code implementation over the next two weeks
• David Hensle: Review CDAP estimation bug fix pull request
• David Hensle: Confirm correct ET branch location (possibly on Jan's fork)
• Bo Wen: Tag David and Sumit on ET next steps outstanding issue for tracking
• Agencies: Complete any outstanding UV implementation items when possible during holiday period
• Joe Castiglione: Cancel engineering meetings on December 25th and January 1st