Skip to content

Parallel builds #36

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
lightninglu10 merged 9 commits into from
Nov 29, 2025
Merged

Parallel builds #36

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
9 commits
Select commit Hold shift + click to select a range
96c1fde
parallel builds
lightninglu10 Nov 25, 2025
f2f99cb
default 2
lightninglu10 Nov 25, 2025
b954a1f
updating webpack-plugin
lightninglu10 Nov 25, 2025
340cc0d
updating version
lightninglu10 Nov 26, 2025
10b636b
build in parallel
lightninglu10 Nov 26, 2025
e37a03e
fix release
lightninglu10 Nov 26, 2025
95dcd13
better tsconfig reading
lightninglu10 Nov 26, 2025
102ea47
package.json update
lightninglu10 Nov 27, 2025
95254a4
fix pr comment issue
lightninglu10 Nov 29, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
79 changes: 70 additions & 9 deletions .github/workflows/release.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -95,10 +95,64 @@ jobs:
echo "changed=true" >> $GITHUB_OUTPUT
fi

build-and-publish:
# Build each WASM band in parallel using matrix strategy
build-wasm:
needs: check-version
if: needs.check-version.outputs.version_changed == 'true'
runs-on: ubuntu-latest
strategy:
fail-fast: false
matrix:
band: [v0_82_87, v26, v42, v48]
steps:
- name: Checkout
uses: actions/checkout@v4

- name: Setup Node
uses: actions/setup-node@v4
with:
node-version: ${{ env.NODE_VERSION }}

- name: Install pnpm
uses: pnpm/action-setup@v4
with:
version: 10.22.0

- name: Setup Rust (stable + WASI)
uses: dtolnay/rust-toolchain@stable
with:
targets: wasm32-wasip1

- name: Cache Cargo registry
uses: actions/cache@v4
with:
path: |
~/.cargo/registry
~/.cargo/git
key: ${{ runner.os }}-cargo-${{ matrix.band }}-${{ hashFiles('codepress-swc-plugin/src/**') }}
restore-keys: |
${{ runner.os }}-cargo-${{ matrix.band }}-
${{ runner.os }}-cargo-

- name: Install deps
run: pnpm install --frozen-lockfile

- name: Build WASM band ${{ matrix.band }}
run: node scripts/build-swc.mjs
env:
BAND: ${{ matrix.band }}

- name: Upload WASM artifact
uses: actions/upload-artifact@v4
with:
name: wasm-${{ matrix.band }}
path: swc/*.wasm
retention-days: 1

build-and-publish:
needs: [check-version, build-wasm]
if: needs.check-version.outputs.version_changed == 'true'
runs-on: ubuntu-latest
permissions:
contents: write
packages: write
Expand All @@ -118,16 +172,26 @@ jobs:
with:
version: 10.22.0

- name: Setup Rust (stable + WASI)
uses: dtolnay/rust-toolchain@stable
with:
targets: wasm32-wasip1

- name: Install deps
run: pnpm install --frozen-lockfile
env:
NODE_AUTH_TOKEN: ${{ secrets.NPM_TOKEN }}

- name: Download all WASM artifacts
uses: actions/download-artifact@v4
with:
pattern: wasm-*
path: swc-artifacts
merge-multiple: false

- name: Combine WASM artifacts
run: |
mkdir -p swc
for dir in swc-artifacts/wasm-*; do
cp "$dir"/*.wasm swc/ 2>/dev/null || true
done
ls -la swc/

- name: Update package version for release
env:
RELEASE_VERSION: ${{ needs.check-version.outputs.release_version }}
Expand All @@ -144,9 +208,6 @@ jobs:
- name: Build JS
run: pnpm run build

- name: Build Rust WASM plugin
run: pnpm run build:rust

- name: Run tests
run: pnpm test

Expand Down
4 changes: 2 additions & 2 deletions package.json
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
{
"name": "@codepress/codepress-engine",
"version": "0.7.9",
"version": "0.7.10",
"packageManager": "pnpm@10.22.0",
"description": "CodePress engine - Babel and SWC plug-ins",
"main": "./dist/index.js",
Expand Down Expand Up @@ -100,6 +100,7 @@
"@babel/preset-typescript": "^7.26.0",
"@eslint/eslintrc": "^3.3.1",
"@eslint/js": "^9.37.0",
"@swc/core": "^1.15.3",
"@swc/wasm": "^1",
"@types/babel__core": "^7.20.5",
"@types/jest": "^30.0.0",
Expand All @@ -124,7 +125,6 @@
},
"dependencies": {
"@fastify/cors": "^11.0.1",
"@swc/core": "^1.15.3",
"fastify": "^5.3.3",
"node-fetch": "^2.6.7",
"prettier": "^3.1.0",
Expand Down
6 changes: 3 additions & 3 deletions pnpm-lock.yaml
View file
Open in desktop

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

84 changes: 81 additions & 3 deletions scripts/build-swc.mjs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -63,13 +63,35 @@ const BANDS = [
// BAND=v42 node scripts/build-swc.mjs
// BANDS=v26,v42 node scripts/build-swc.mjs
// node scripts/build-swc.mjs v42 v26
const args = process.argv.slice(2).filter(Boolean);
const cliArgs = process.argv.slice(2).filter(Boolean);
// Filter out CLI flags and their values so they don't get misinterpreted as band IDs
// Flags that take values: -n, --next, -b, --band, -t, --target, -p, --parallel
const flagsWithValues = new Set([
"-n",
"--next",
"-b",
"--band",
"-t",
"--target",
"-p",
"--parallel",
]);
const bandArgsFromCli = [];
for (let i = 0; i < cliArgs.length; i++) {
const arg = cliArgs[i];
if (arg.startsWith("-")) {
// Skip flags; if it takes a value, skip the next arg too
if (flagsWithValues.has(arg)) i++;
continue;
}
bandArgsFromCli.push(arg);
}
const bandEnvRaw = process.env.BAND || process.env.BANDS || "";
const bandIdsFromEnv = bandEnvRaw
.split(",")
.map((s) => s.trim())
.filter(Boolean);
const requestedIds = [...bandIdsFromEnv, ...args];
const requestedIds = [...bandIdsFromEnv, ...bandArgsFromCli];
const validIds = new Set(BANDS.map((b) => b.id));
let BANDS_TO_BUILD = BANDS;
if (requestedIds.length) {
Expand Down Expand Up @@ -206,6 +228,7 @@ function parseArgs(argv) {
next: undefined,
band: undefined,
target: undefined,
parallel: 2, // number of concurrent builds, 0 = unlimited
listBands: false,
help: false,
};
Expand All @@ -224,6 +247,24 @@ function parseArgs(argv) {
case "-t":
args.target = argv[++i];
break;
case "--parallel":
case "-p": {
const raw = argv[++i];
const n = Number(raw);
if (
raw == null ||
!Number.isFinite(n) ||
!Number.isInteger(n) ||
n < 0
) {
console.error(
`[codepress] --parallel expects a non-negative integer (0 = unlimited); received: "${raw ?? ""}"`
);
process.exit(1);
}
args.parallel = n;
break;
}
case "--list-bands":
args.listBands = true;
break;
Expand All @@ -239,6 +280,32 @@ function parseArgs(argv) {
return args;
}

async function runWithConcurrency(tasks, concurrency) {
if (concurrency <= 0) {
// Unlimited parallelism
return Promise.all(tasks.map((fn) => fn()));
}
const results = [];
const executing = new Set();
for (const task of tasks) {
const p = Promise.resolve()
.then(task)
.finally(() => {
executing.delete(p);
});
results.push(p);
executing.add(p);
if (executing.size >= concurrency) {
try {
await Promise.race(executing);
} catch {
// Swallow here; final Promise.all(results) will reject with the first error.
}
}
}
return Promise.all(results);
}

function parseSemver(input) {
if (!input) return null;
const m = String(input)
Expand Down Expand Up @@ -275,11 +342,14 @@ function usage() {
` -n, --next <version> Build band matching Next.js version (e.g. 15.4.0)\n` +
` -b, --band <id> Build specific band id (one of: ${BANDS.map((b) => b.id).join(", ")})\n` +
` -t, --target <t> Build target(s): wasip1 | wasi | all | comma-list (default: all)\n` +
` -p, --parallel <n> Concurrency limit (default: 2, 0 = unlimited)\n` +
` --list-bands Print available band ids and exit\n` +
` -h, --help Show this help\n\n` +
`Examples:\n` +
` node scripts/build-swc.mjs --next 15.4.0\n` +
` node scripts/build-swc.mjs --band v26 --target wasip1\n` +
` node scripts/build-swc.mjs --parallel 2 # Build 2 bands at a time\n` +
Copy link

@github-actions github-actions bot Nov 25, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔴 REQUIRED: runWithConcurrency lacks rejection handling for in-flight tasks. If any task rejects while throttling, await Promise.race(executing) throws and the function returns early, leaving other executing promises without rejection handlers. This can produce unhandled promise rejections and leave spawned cargo processes running after the script exits (critical stability issue).

  Evidence: p is created with task().then(...delete...) only; no .catch/.finally on rejection. The loop awaits Promise.race(executing) without try/catch. On first rejection, execution unwinds before reaching the final Promise.all(results), so other rejecting promises may be unhandled.

  Required: ensure each task always has a rejection handler and that the pool drains correctly. Use .finally to delete from the set, and catch the race to allow the loop to continue attaching handlers; the eventual Promise.all will still surface the error.
Suggested change
` node scripts/build-swc.mjs --parallel 2 # Build 2 bands at a time\n` +
async function runWithConcurrency(tasks, concurrency) {
if (concurrency &lt;= 0) {
return Promise.all(tasks.map((fn) =&gt; fn()));
}
const results = [];
const executing = new Set();
for (const task of tasks) {
const p = Promise.resolve()
.then(task)
.finally(() =&gt; {
executing.delete(p);
});
results.push(p);
executing.add(p);
if (executing.size &gt;= concurrency) {
try {
await Promise.race(executing);
} catch {
// Swallow here; final Promise.all(results) will reject with the first error.
}
}
}
return Promise.all(results);
}

` node scripts/build-swc.mjs --parallel 0 # Build all bands in parallel\n` +
` npm run build:rust -- --next 15.4.0\n`
);
}
Expand Down Expand Up @@ -336,7 +406,15 @@ function selectTargets(targetArg) {
}

const targets = selectTargets(args.target);
const concurrency = args.parallel;

for (const band of bands) await buildOneBand(band, targets);
console.log(
`[codepress] Building ${bands.length} band(s)` +
(concurrency > 0
? ` (max ${concurrency} concurrent)`
: " (unlimited parallelism)")
);
const tasks = bands.map((band) => () => buildOneBand(band, targets));
await runWithConcurrency(tasks, concurrency);
console.log("Finished SWC bands built.");
})();
Loading