Skip to content

Default settings results in large memory overhead while using multiple threads #92

@DanielOlson

Description

@DanielOlson

Example:

/usr/local/ultra-1.2.1/ultra --json -o foo.json foo.fa 
----------------------------
Maximum repeat period: 100
Number of model states: 1970
Total sequence window size: 100200
DP matrix cells: 197394000
Threads: 1
Sequence window queue length: 1024
Total size: 0.83 GB (892180800 bytes)

/usr/local/ultra-1.2.1/ultra --json -o foo.json foo.fa --threads 24
----------------------------
Maximum repeat period: 100
Number of model states: 1970
Total sequence window size: 100200
DP matrix cells: 197394000
Threads: 24
Sequence window queue length: 1024
Total size: 17.74 GB (19052428800 bytes)

0.83GB seems reasonable for a single threaded run, but perhaps the memory allocation model should reduce window size when using multiple threads.

Additionally, it may be beneficial to have a --use_memory command; this would be much more intuitive than manually adjusting --win_size.

note The sizes above may appear to scale incorrectly; 0.83GB*24 does not equal 17.74 GB. The reason for this is because the 0.83 GB includes additional overhead that multiple threads do not have to pay (the actual total per-thread memory cost in this case will be 197394000 * 4 bytes * 24). The memory print statement should perhaps be improved to reflect this.

Metadata

Metadata

Assignees

No one assigned

    Type

    No type
    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions