Skip to content

⚡ perf: optimize migrate-mdx.js with async I/O #22

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
sunnylqm wants to merge 1 commit into
base: master
Choose a base branch
from
+38 −42
Open

⚡ perf: optimize migrate-mdx.js with async I/O #22

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
80 changes: 38 additions & 42 deletions site/migrate-mdx.js
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,44 +1,40 @@
const fs = require('fs');
const path = require('path');

function walk(dir) {
let results = [];
const list = fs.readdirSync(dir);
list.forEach(file => {
const fullPath = path.join(dir, file);
const stat = fs.statSync(fullPath);
if (stat && stat.isDirectory()) results = results.concat(walk(fullPath));
else results.push(fullPath);
});
return results;
import fs from 'node:fs/promises';
import path from 'node:path';

async function main() {
const dirents = await fs.readdir('./pages', { recursive: true, withFileTypes: true });
const files = dirents
.filter(dirent => !dirent.isDirectory() && (dirent.name.endsWith('.mdx') || dirent.name.endsWith('.md')))
.map(dirent => path.join(dirent.parentPath || dirent.path, dirent.name));

await Promise.all(files.map(async file => {
let content = await fs.readFile(file, 'utf8');

// replace <Callout type="warning"> ... </Callout> with :::warning ... :::
content = content.replace(/<Callout[^>]*type="([^"]+)"[^>]*>([\s\S]*?)<\/Callout>/g, (_, type, body) => {
return `:::${type}\n${body.trim()}\n:::`;
});
content = content.replace(/<Callout>([\s\S]*?)<\/Callout>/g, (_, body) => {
return `:::info\n${body.trim()}\n:::`;
});

// Replace imports
content = content.replace(/import\s*\{([^}]*)\}\s*from\s*"nextra\/components";?/g, (match, imports) => {
const list = imports.split(',').map(i => i.trim()).filter(i => i !== 'Callout' && i !== '');
if (list.length === 0) return '';
if (list.includes('Tabs') && !list.includes('Tab')) {
list.push('Tab');
}
return `import { ${list.join(', ')} } from "rspress/theme";`;
});

// Replace <Tabs.Tab> with <Tab>
content = content.replace(/Tabs\.Tab/g, 'Tab');

await fs.writeFile(file, content, 'utf8');
}));
Comment on lines +10 to +35
Copy link

@coderabbitai coderabbitai bot Mar 20, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🟠 Major

Bound parallel I/O to avoid EMFILE/resource spikes at scale.

Running all file reads/writes in one Promise.all is unbounded; large trees can overwhelm file descriptors and memory. Use a fixed concurrency worker pool.

Suggested fix 
 async function main() {
   const dirents = await fs.readdir('./pages', { recursive: true, withFileTypes: true });
   const files = dirents
     .filter(dirent => !dirent.isDirectory() && (dirent.name.endsWith('.mdx') || dirent.name.endsWith('.md')))
     .map(dirent => path.join(dirent.parentPath || dirent.path, dirent.name));
 
-  await Promise.all(files.map(async file => {
+  const CONCURRENCY = 32;
+  let idx = 0;
+
+  async function worker() {
+    while (idx < files.length) {
+      const file = files[idx++];
+      await processFile(file);
+    }
+  }
+
+  async function processFile(file) {
     let content = await fs.readFile(file, 'utf8');
 
     // replace <Callout type="warning"> ... </Callout> with :::warning ... :::
     content = content.replace(/<Callout[^>]*type="([^"]+)"[^>]*>([\s\S]*?)<\/Callout>/g, (_, type, body) => {
       return `:::${type}\n${body.trim()}\n:::`;
@@
     // Replace <Tabs.Tab> with <Tab>
     content = content.replace(/Tabs\.Tab/g, 'Tab');
 
     await fs.writeFile(file, content, 'utf8');
-  }));
+  }
+
+  await Promise.all(Array.from({ length: Math.min(CONCURRENCY, files.length) }, () => worker()));
🤖 Prompt for AI Agents 
Verify each finding against the current code and only fix it if needed.

In `@site/migrate-mdx.js` around lines 10 - 35, The current code runs all file I/O
in an unbounded Promise.all (files.map(...)) using fs.readFile and fs.writeFile
which can spike descriptors; change to a bounded concurrency worker pool (e.g.,
use p-limit or an async queue) and replace the Promise.all(files.map(...))
pattern with a limited runner that processes N files at a time (configurable,
e.g., 5-20), invoking the same async handler that reads, transforms
(Callout/Tabs import logic, Tabs.Tab replacement) and writes each file; ensure
errors are propagated and awaited for all tasks before exit.


console.log('Done migrating MDX');
}

const files = walk('./pages').filter(f => f.endsWith('.mdx') || f.endsWith('.md'));

files.forEach(file => {
let content = fs.readFileSync(file, 'utf8');

// replace <Callout type="warning"> ... </Callout> with :::warning ... :::
content = content.replace(/<Callout[^>]*type="([^"]+)"[^>]*>([\s\S]*?)<\/Callout>/g, (_, type, body) => {
return `:::${type}\n${body.trim()}\n:::`;
});
content = content.replace(/<Callout>([\s\S]*?)<\/Callout>/g, (_, body) => {
return `:::info\n${body.trim()}\n:::`;
});

// Replace imports
content = content.replace(/import\s*\{([^}]*)\}\s*from\s*"nextra\/components";?/g, (match, imports) => {
const list = imports.split(',').map(i => i.trim()).filter(i => i !== 'Callout' && i !== '');
if (list.length === 0) return '';
if (list.includes('Tabs') && !list.includes('Tab')) {
list.push('Tab');
}
return `import { ${list.join(', ')} } from "rspress/theme";`;
});

// Replace <Tabs.Tab> with <Tab>
content = content.replace(/Tabs\.Tab/g, 'Tab');

fs.writeFileSync(file, content, 'utf8');
});
console.log('Done migrating MDX');
main().catch(console.error);
Copy link

@coderabbitai coderabbitai bot Mar 20, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🟠 Major

Return non-zero exit status on migration failure.

main().catch(console.error) logs the error but can still exit successfully. Set process.exitCode = 1 so CI/scripts fail correctly on partial/failed migrations.

Suggested fix 
-main().catch(console.error);
+main().catch(err => {
+  console.error(err);
+  process.exitCode = 1;
+});
📝 Committable suggestion

‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.

Suggested change
main().catch(console.error);
main().catch(err => {
console.error(err);
process.exitCode = 1;
});
🤖 Prompt for AI Agents 
Verify each finding against the current code and only fix it if needed.

In `@site/migrate-mdx.js` at line 40, The script currently swallows migration
failures by only logging errors via main().catch(console.error), which can still
yield a zero exit status; update the catch handler to set process.exitCode = 1
(or call process.exit(1)) after logging the error so CI/scripts detect
failures—modify the invocation around main() and its catch (the main() call) to
log the error and then set process.exitCode = 1 to indicate a non-zero exit on
failure.