bug: Memory Exhaustion on Large Files

[akramcodez]

Description

The read-data-file tool reads entire files into memory synchronously and parses them immediately, causing excessive memory usage.

// packages/core/src/tools/read-data-file.ts
const content = await fs.promises.readFile(filePath, 'utf-8'); 
const data = JSON.parse(content); 

Impact

• Immediate process crash (OOM) for files larger than ~500MB
• Inability to analyze large files (logs, datasets, etc.)
• Instability of the entire agent session

Potential Fix

Use stream-based processing with fs.createReadStream and a streaming JSON parser or line-by-line reader (e.g., readline).

[sahilkhedkar]

working on it...

[vyagh]

I submitted a PR for this, @sahilkhedkar is still working on their solution, happy to coordinate or close mine if theirs is preferred