High memory usage for the YZ-Sim

[HaiwangYu]

ICARUS reported high High memory usage due to the YZ-Sim. Likely related to the duplicated mapping in memory.

[HaiwangYu]

GH repo for this issue:
https://github.com/HaiwangYu/wcdev-icarus

First round resolved the long initialization issue. See #471
But the memory is still at 12GB level.
Need further investigation.

[HaiwangYu]

Valgrind profiling from Tracy Usher

/exp/icarus/app/users/usher/wirecelltest/massif.out.929

2 wide peaks at initialization and 4 narrow peak per each TPC

[HaiwangYu]

Massif analysis: 2 + 4 peaks in wcls-multitpc-sim-drift-simchannel-yzsim-refactored.jsonnet

Context

Profile: /exp/icarus/app/users/yuhw/wcdev-icarus/massif.out.929
(symlink → /exp/icarus/app/users/usher/wirecelltest/massif.out.929)

Command run under valgrind:

lar --trace -n 1 -c standard_mc_all_detsim_icarus.fcl \
    -s prod_muplus_0-1GeV_isotropic_icarus_*_gen_*-G4.root \
    -o detsim.root

driving Wire-Cell via wcls-multitpc-sim-drift-simchannel-yzsim-refactored.jsonnet on the current
branch yzmap-service2.

Goal: explain the 2 wide early humps and the 4 narrow late spikes in the massif curve,
and rank concrete ways to reduce them. No code changes are proposed for this turn —
this is a read-only analysis + a ranked menu of fixes.

1. Where the peaks are on the timeline

ms_print shows 153 snapshots. Mapping snapshots → ASCII features:

ASCII feature	Snap range	Time	Heap (GB)	Dominant allocator	What is running
Wide peak A	1 → 16	0.06 – 1.10 s	up to 5.85	Json::Value::CZString Rb_tree (libjsoncpp)	LArSoft + WCT startup, config parse
Dip	17	1.15 s	3.13	(intermediate tree freed)	—
Wide peak B	18 → 23	1.21 – 1.54 s	up to 6.22	same — JSON tree _M_copy	configurables receive their config
Valley	24 → 29	1.58 – 1.83 s	2.17–2.18	(configurator drops its copy)	components built, config released
Plateau	30 → 110	1.88 – 6.85 s	4.2–4.8	(steady event ramp)	depo loading + drift + diffusion
Narrow peak 1	112	6.92 s	5.32	jsoncpp tree (2.19 GB) + SimpleTrace	TPC 0 chain
Narrow peak 2	117	7.09 s	5.57	same shape	TPC 1 chain
Narrow peak 3	123	7.31 s	6.88	SimpleTrace from Digitizer (2.48 GB) + AddNoise (0.87 GB)	TPC 2 chain
Narrow peak 4 (global)	138	7.50 s	7.79	SimpleTrace from Digitizer (3.46 GB) + AddNoise (0.87 GB)	TPC 3 chain
Drop	139 → 150	7.54 – 7.92 s	3.16 → 4.98 → 3.23	—	output written, frames released

Useful-heap @ peak ≈ 7.79 GB, extra-heap ≈ 0.43 GB → ≈ 8.2 GB RSS-equivalent.

2. Why the 2 wide early peaks exist

Massif attributes essentially all of snapshots 1–23 to one allocation kind:

Json::Value::CZString-keyed std::_Rb_tree nodes (libjsoncpp.so.1.9.5)
   ← _M_copy<…>  ← _M_copy<…>  ← _M_copy<…>  …            (10+ levels deep)

At snap 23: 6.55 GB out of 6.68 GB is Rb_tree<…CZString,…Value>::_M_copy. Two
features feed this:

(a) the configuration blob is enormous

wcls-multitpc-sim-drift-simchannel-yzsim-refactored.jsonnet realizes a graph with

• 360 per-(anode, plane) DepoSetDrifter instances (line 195–206)
• 360 DepoSetFilterYZ + 360 Scaler (lines 438–449)
• 360 wclsDepoFluxWriter SimChannel sinks
• 4 FrameSummerYZ (mult=90 each, lines 429–436)
• 4 per-TPC actpipes: reframer → IncoherentAddNoise → CoherentAddNoise → Digitizer → wclsFrameSaver (line 454)

So the jsonnet itself emits ≈ 1500 component configurations, several with bulky
sub-trees (PlaneImpactResponse/field responses, ColdElec, noise models, the two
YZ maps). The recent commits 7c18f62 (extract YZMap to shared service) and
505f50b ("Bulk loads of large configs") already chipped at this, but the dominant
cost remains.

(b) Wire-Cell deep-copies the whole config tree several times per component

Verified by reading the code:

• util/inc/WireCellUtil/Configuration.h:142
  Configuration branch(Configuration cfg, const std::string& dotpath); ← pass-by-value, full Rb_tree deep copy every call.
• apps/src/Main.cxx has six for (auto c : m_cfgmgr.all()) / for (auto c : m_apps) loops at lines 312, 325, 331, 350, 375, 408 — each iteration deep-copies a full top-level config entry.
• apps/src/Main.cxx:364-365
  Configuration cfg = cfgobj->default_configuration(); cfg = update(cfg, c["data"]);
  produces two full copies (default + merged) per component, even when the default is empty.
• util/src/ConfigManager.cxx:29
  for (auto one : more) { … } — copy-on-iterate during bulk load.

Json::Value's object backing is shims::map<CZString, Json::Value> → std::map → std::_Rb_tree, which matches exactly the massif stack. Recursive _M_copy frames in the trace are the Rb_tree clone of a JSON object containing JSON objects containing JSON objects (≥ 10 levels of field_response / pir / noise_models etc.) — every top-level deep copy doubles a multi-MB sub-tree.

Result: ~1500 components × several full-tree copies × a deeply nested per-component sub-tree
→ the 5–6 GB humps. Peak B is higher than peak A because by then every component has been
constructed and both the originals (held by ConfigManager) and the per-component copies
exist briefly before the originals are destroyed.

3. Why the 4 narrow late peaks exist

Each peak corresponds to one ICARUS TPC running through

reframer[n] → noises[n] → coh_noises[n] → digitizers[n] → wcls_output.sim_digits[n]

(line 454, n ∈ 0..3).

The Pgrapher's DFP scheduler runs these chains effectively serially over the 4 TPCs, so memory
spikes once per TPC. Within each spike the offenders, taken directly from the snapshot-138
heap tree, are:

Digitizer (gen/src/Digitizer.cxx lines 130–152) — 3.46 GB at peak:

ITrace::vector adctraces(nrows);                           // one slot per channel
for (size_t irow = 0; irow < nrows; ++irow) {
    …
    ITrace::ChargeSequence adcwave(ncols);                 // NEW float vector, ~4096 floats
    for (size_t icol = 0; icol < ncols; ++icol) { … }
    adctraces[irow] = make_shared<SimpleTrace>(ch, tbinmm.first, adcwave);  // COPY into SimpleTrace
}

Both adcwave (line 144) and m_charge inside the new SimpleTrace (SimpleTrace.cxx:6 does m_charge(charge), copy-constructed) coexist for the duration of one channel iteration, and then the full channel-set of new traces (≈ frame_size) lives until the chain's output frame is consumed by wclsFrameSaver. Crucially, the upstream voltage frame is still held by the pgraph queue while this is running.

CoherentAddNoise (gen/src/AddNoise.cxx lines 161–201) — 0.87 GB at peak:

for (const auto& intrace : *inframe->traces()) {
    …
    auto charge = intrace->charge();                       // COPY #1: full float vector
    …
    Waveform::increase(charge, gwlu[grpid]);               // in-place add of noise
    auto trace = make_shared<SimpleTrace>(chid, intrace->tbin(), charge);  // COPY #2
    outtraces.push_back(trace);
}

Per channel: two float-vector copies of the same waveform. intrace (and the whole input
frame) is not released before the new SimpleTrace is constructed.

Why the peak grows from 5.32 → 7.79 GB across the four TPCs: the per-TPC output frames
are held by their wclsFrameSaver until the art Event is finalized, so frame N's ADC
data is still resident when frame N+1 is being produced. This is the staircase between
peaks 1 → 4 visible in the ASCII (each "plateau" between spikes climbs).

Note on the jsoncpp residue at the narrow peaks: ~2 GB of jsoncpp Rb_tree memory is
still alive during the digitization spikes (snap 112 shows 2.19 GB of jsoncpp). So the
"config bloat" of section 2 is paying a tax during event processing too — not just
startup. Fixing section 2 would lower the floor that the narrow peaks rise off of.

4. Ranked reduction suggestions

Each item lists: what to change, where, rough peak-memory impact, invasiveness.

Group A — shrink the 2 wide startup peaks (peaks A + B)

#	Change	File:line	Est. impact	Risk
A1	Make branch() take const Configuration& and stop returning by value (return a view: keep a Json::Path-style accessor, or return Configuration by NRVO but with the input taken by const ref).	util/inc/WireCellUtil/Configuration.h:142, util/src/Configuration.cxx:8	Removes one full deep-copy per get<T>(cfg, "data.…") call site (≈ dozens during init). 0.5–1 GB.	Low — pure refactor; ABI signature change so rebuild all users.
A2	Switch the six for (auto c : …) loops in apps/src/Main.cxx (lines 312, 325, 331, 350, 375, 408) to for (const auto& c : …).	as listed	This is the biggest single win — each loop currently clones every top-level config entry. ≈ 1–2 GB.	Trivial. Read-only iteration only — already what the code wants.
A3	In apps/src/Main.cxx:364-365 avoid the default-then-merge double copy when the default is empty (default_configuration() returns Json::nullValue).	same	≈ 0.3–0.7 GB.	Low.
A4	In util/src/ConfigManager.cxx:29 change for (auto one : more) → for (const auto& one : more).	same	Small (~0.1 GB) but free.	Trivial.
A5	Continue the YZMap-style externalization for the remaining bulk arrays still inlined into per-component configs: PIR / field-response coefficients, ColdElec response, noise spectra. Each large table should be loaded once by a named service and component configs should reference it by name. (This is the same pattern as commit 7c18f62a; current branch is mid-migration.)	jsonnet/cfg files (e.g. cfg/pgrapher/experiment/icarus/simparams.jsonnet, cfg/pgrapher/experiment/icarus/sim.jsonnet)	The remaining inlined arrays appear to drive the ≈ 2 GB jsoncpp baseline still resident during event processing — externalizing the largest one or two could cut that in half.	Medium — needs a service per data type, but precedent exists.
A6	Drop the --trace flag from the run command when not debugging. --trace in art instruments every call and inflates allocations modestly.	n/a, run-time flag	Small but free.	None.

Expected combined impact of A1+A2+A3+A4 (low-risk, no API changes for users): peaks A and B drop from ~6 GB → ~2 GB, and the steady-state jsoncpp residue during event processing falls correspondingly.

Group B — shrink the 4 narrow event-processing peaks

#	Change	File:line	Est. impact	Risk
B1	Add a SimpleTrace constructor SimpleTrace(int, int, ChargeSequence&&) that moves the float vector in. Then use std::move(charge) at AddNoise.cxx:198 and std::move(adcwave) at Digitizer.cxx:151.	aux/inc/WireCellAux/SimpleTrace.h:22, aux/src/SimpleTrace.cxx:5	Removes 1 full vector copy per channel per stage. With ~5632 ch × ~4096 ticks × 4 B × 4 TPCs ≈ 0.36 GB per stage, applied 2× (Digitizer + AddNoise) ≈ 0.7 GB off peak 138.	Low — additive overload, existing callers unchanged.
B2	At AddNoise.cxx:165 the initial auto charge = intrace->charge(); is a copy because charge() returns const ChargeSequence&. If the input trace will not be reused downstream (it won't, in this pipeline), we could (a) take the input frame by IFrame::pointer& and clear traces from it as we consume, or (b) emit outframe traces in-place over inframe. Option (a) is the smallest change — release inframe before constructing outframe.	AddNoise.cxx:201	~0.7 GB at the narrow-peak transient (overlap of input + output frames).	Medium — need to confirm no other graph node holds the same inframe.
B3	Switch the ADC payload type from float to int16_t once the digitization is done (it is already quantized at Digitizer.cxx:147). Either: (i) introduce SimpleTraceInt16 / a templated trace, or (ii) cast at the wclsFrameSaver boundary.	gen/src/Digitizer.cxx:144-151 and downstream sink.	Halves every post-digitizer trace from 4 B/tick to 2 B/tick. For 4 TPCs × 5632 ch × 4096 ticks held simultaneously ≈ 1.4 GB off peak.	High — interface change touching ITrace, downstream consumers, and the wcls↔larsoft RawDigit conversion. Probably the biggest single win for narrow peaks but the most invasive.
B4	Make wcls_output.sim_digits[n] flush its accumulated RawDigit collection to the art event eagerly (after the per-TPC chain completes) rather than holding all four until process() returns.	larwirecell wclsFrameSaver — out of this repo but worth flagging	The staircase between the 4 narrow peaks (5.32 → 7.79 GB) shows ≈ 2.5 GB of cross-TPC retention. Eliminating it would cap all four peaks near the level of peak 1.	Medium — depends on art's Producer semantics; may already be the only allowed shape.
B5	Reduce concurrency in pgraph for this subgraph (force the 4 actpipes to truly serialize, not just appear to). The current "narrow peak" shape already suggests serialization, but if any overlap remains, eliminating it would shave further.	jsonnet: g.pipeline(...) at line 454 — investigate scheduling policy in Pgrapher	Bounded by how serial it already is. Diagnostic value > savings.	Low.
B6	In Digitizer.cxx:122-129 (just above the shown lines), the dense Eigen arr matrix (nrows × ncols floats ≈ ~92 MB per TPC) is constructed alongside the new traces. Quantize directly into adctraces[irow]'s buffer without materializing arr.	as listed	~0.4 GB at peak.	Medium — changes the Eigen-style "load-then-loop" pattern.

Expected combined impact of B1+B2 (low-risk, additive): peak 138 drops from 7.79 → ~6.4 GB.
Add B3 → ~5.0 GB. Add B4 → flatten the staircase so peak ≈ first-TPC peak (~5.3 GB → ~4 GB
once Group A also lands).

Cross-cutting

• Profile after each fix, in the same configuration (same fcl, same jsonnet, same input).
  valgrind --tool=massif --pages-as-heap=no --time-unit=ms --detailed-freq=1 lar -n 1 -c ….
• Rough order of merit / invasiveness: A2 ≫ A1 ≈ A3 > B1 > B2 > A5 ≈ A4 > B3 > B4 > B6 > B5.

5. Verification recipe

1. Re-run massif against the same fcl + jsonnet (in-sl7 wrapper for any rebuild):
   tools/in-sl7.sh ./wcb -p --notests build install
2. Sanity-check the event content hasn't changed by diffing the produced detsim.root
RawDigit collection against a reference (existing ICARUS regression scripts under
   cfg/pgrapher/experiment/icarus/ or your local notebook).
3. For Group A changes, expect: snapshot 23 useful-heap to drop from ~6.5 GB to ~2 GB and
   the post-startup plateau to drop accordingly.
4. For Group B changes, expect: peak snapshot (138) useful-heap to drop proportionally and
   the staircase between the 4 narrow peaks to flatten.

6. Key files referenced

Code (read-only in this analysis):

• apps/src/Main.cxx — six for (auto …) copies (lines 312, 325, 331, 350, 375, 408); default_configuration() merge (364-365)
• util/inc/WireCellUtil/Configuration.h — branch() declaration line 142
• util/src/Configuration.cxx — branch() definition line 8
• util/src/ConfigManager.cxx — extend() loop line 29 (recent commit 505f50b improved adjacent code)
• gen/src/Digitizer.cxx — trace allocation lines 144–151
• gen/src/AddNoise.cxx — CoherentAddNoise::operator() lines 161–201
• aux/inc/WireCellAux/SimpleTrace.h — copy-only constructor line 22
• aux/src/SimpleTrace.cxx — m_charge(charge) line 6

Config (read-only in this analysis):

• cfg/pgrapher/experiment/icarus/wcls-multitpc-sim-drift-simchannel-yzsim-refactored.jsonnet — 360-way fanout (line 206), YZMap (lines 220–265), per-TPC actpipes (line 454)
• cfg/pgrapher/experiment/icarus/funcs.jsonnet — fandrifter topology (lines 494–693)
• cfg/pgrapher/experiment/icarus/params.jsonnet — nticks=4096 (line 84–86)

No edits made. Ready for your direction on which items from Group A or Group B to act on next.

[HaiwangYu]

I did two additional optimizations, the peak is now 6GB.
Summary: https://github.com/HaiwangYu/wcdev-icarus/blob/main/PER-TPC-SUMMARY.md
1, delete configuration once c++ components are configured (C5)
2, make 4 WireCellToolkit for each TPC, so that at the end of each one the raw::digits can be offloaded immediately after once TPC processing is finished (per_TPC modules).
Comparing no-fix, up to C5 and per_TPC modules in the plot

The artROOT is here:

/exp/icarus/app/users/yuhw/wcdev-icarus/run-20260516-per-tpc-top/detsim.root