Sharrow Memory Usage

[jpn--]

Would like to add as an agenda item the following memory profile produced by TransLink running their full sample with sharrow on. TransLink developed a script to view the total system memory of the machine. Shows similar behavior as reported in #850

cc @bwentl

Originally posted by @dhensle in #77

[joecastiglione]

Some earlier tests: ActivitySim Data Types Task - Progress Update 09-26-2023.pptx

[bwentl]

@jpn-- I did more memory profiling with multiprocessing + sharrow, and I think the issue is related to sharrow compile on windows. Take our CDAP model for example, on 8mp, it took 20 minutes and ~16GB of memory to compile it in multiprocessing mode. And with 4mp, it took ~10 minutes and ~8GB of memory to compile it in multiprocessing mode.

I am curious if this is a problem on Linux or MacOS.

Here are the tests I did:

• 1_noshcache - 8 multiprocessing with no sharrow cache.
• 2a_shcache_8mp - 8 mp with sharrow cache precompiled from 1_noshcache.
• 2b_shcache_4mp - 4 mp with sharrow cache precompiled. Showed little to no difference compared to 2a_shcache_8mp.
• 3_newskims_4mp - 4 mp with sharrow cache precompiled, with new smaller set of skims for testing our pnr model.
• 3b_noshcache_newskims_4mp - 4 mp with no sharrow cache, smaller set of skims for pnr testing.