Memory Profiler causes extreme memory usage during profiling.

[envygeeks]

Using MemoryProfiler.start on one of our jobs, and then waiting to dump MemoryProfile.stop&.pretty_print took a job that uses 2gb of memory to 76gb of memory. Is there a way we can prevent this kind of memory usage? Even running memory profiler for a few short seconds causes it to jump to exponential levels.

[SamSaffron]

its is very very tricky, collection all this data comes at a cost and we have no way of simply streaming this data to disk and then running analysis on data stored on disk.

Is there any way you can simply collect less info?

[envygeeks]

Kinda we did, it was a bit of a mess, but memory profiling acting crazy prompted me to look closer at other stuff with a manual audit, which helped me notice an allocation leak in one of the libraries we use, where the library was allocating millions of objects over and over again.

[jgmontoya]

I'm having the same issue, to the point where I can't really measure the job because I'm running out of memory... How can I reduce the amount of info collected? I'm already trying to evaluate a single class (using trace: [TargetClass])

[GMolini]

This is happening to me when Im trying to measure the amount of memory consumed while reading from a stream
Ive got a zip file that has a CSV in it. I was trying to see how much memory would be used if I processed the file as a stream

report = MemoryProfiler.report do

  file_path =  "/tmp/myzip"
  zip_file = Zip::File.open(file_path)
  zip_file.entries.each do |entry|
   
    Rails.logger.info "Count lines"
    total_lines = 0
    entry.get_input_stream.each do |line|
      total_lines += 1
    end
  end
end

If I execute that code without the memory profiler, using htop I can see that the memory usage barely changes. But If I put the block inside the MemoryProfiler.report do statement, the memory usage absolutely explodes (I stopped the console after it had risen from 6GB of used RAM to 50GB)

[kadru]

I have a similar issue, I'm trying to profile a sidekiq job trough a middleware with memory profiler (sidekiq is running only with 1 thread). When I execute the code without the profiler the memory use is 2GB aprox, with the profiler it reaches up to 28GB until is killed by the OOM. The background job is using CSV to read a cvs file line per line to generate another csv file.

[segiddins]

https://github.com/ruby/ruby/blob/0a630fa461a7260235842e482f682deca30172d6/ext/objspace/object_tracing.c#L445C40-L445C40 looks like it allocates a new string each time we get the allocation source file, which definitely could be an fstring (my guess is its coming from the iseq, which already should have an associated fstring for the file name?)

profile with rbspy is telling me that the CPU time is being spent in allocation_sourcefile, allocation_generation, __id__, []=, and maybe some class's new ?

[segiddins]

I think another thing that might help would be an option to skip retained and only look at allocated, then we can skip the __id__ call and associated hash?