Encountered SIGSEGV coming frequently from my nodes after coming up to most recent SHA

[toddtarsi]

Environment

• Platform: AWS ECS Fargate, linux variant v1.4.0
• Docker Version: I am unable to answer this
• Image Tag:

This image has the issue:
public.ecr.aws/docker/library/node@sha256:16858294071a56ffd4cce9f17b57136cc39e41507b40e245b4f8e906f7a19463

This is the previous, good image in AWS ECR:
public.ecr.aws/docker/library/node@sha256:6178e78b972f79c335df281f4b7674a2d85071aae2af020ffa39f0a770265435

Expected Behavior

My containers should not exit with SIGSEGV.

Current Behavior

Segfault style errors coming in at a rate of about once a minute. Sometimes spiking and leading to a full cluster outage. It appears to be uncorrelated to CPU or memory, as increasing limits either did nothing or exacerbated the crashes.

Possible Solution

Stay on this image for now:

public.ecr.aws/docker/library/node@sha256:6178e78b972f79c335df281f4b7674a2d85071aae2af020ffa39f0a770265435

Steps to Reproduce

I wish I had more here. I'm running a decent amount of traffic, but not too much on my ECS boxes. I tried doubling memory and CPU, no effect. We ran a scan over cloudwatch logs and it was on the 26th, when the new version landed, so we moved back versions here using the old build logs, and that got us out of this problem. Production was really catching on fire a lot though.

I do a lot with building in the container environment and then using COPY at the docker boundary more aggressively than I should (copying modules, Prisma with specified binary targets, etc). So, there's a chance I have some native module build issue. That's my main suspect. However, that hasn't been an issue until this increment.

Thank you for maintaining such important code and if I can provide anything else, please let me know. Thanks!

Just in case it might help, here is a list of my native files. Maybe there is specifically something in my ecosystem causing an issue that you know of:

find node_modules -type f -name "*.node" 2>/dev/null | grep -v "obj\.target"

---

node_modules/lightningcss-darwin-arm64/lightningcss.darwin-arm64.node
node_modules/@tailwindcss/oxide-darwin-arm64/tailwindcss-oxide.darwin-arm64.node
node_modules/@rollup/rollup-darwin-arm64/rollup.darwin-arm64.node
node_modules/prisma/libquery_engine-rhel-openssl-3.0.x.so.node
node_modules/prisma/libquery_engine-darwin-arm64.dylib.node
node_modules/.prisma/client/libquery_engine-rhel-openssl-3.0.x.so.node
node_modules/.prisma/client/libquery_engine-darwin-arm64.dylib.node
node_modules/@prisma/engines/libquery_engine-darwin-arm64.dylib.node
node_modules/@parcel/watcher-darwin-arm64/watcher.node
node_modules/@unrs/resolver-binding-darwin-arm64/resolver.darwin-arm64.node
node_modules/fsevents/fsevents.node

[MikeMcC399]

@toddtarsi

public.ecr.aws/docker/library/node@sha256:6178e78b972f79c335df281f4b7674a2d85071aae2af020ffa39f0a770265435
uses Node.js 20.19.5 with tag node:20.19.5-alpine3.22

public.ecr.aws/docker/library/node@sha256:16858294071a56ffd4cce9f17b57136cc39e41507b40e245b4f8e906f7a19463
uses Node.js 20.19.6 with tag node:20.19.6-alpine3.22

You can check the changes in Node.js 20.19.6 in the announcement

In your situation I would check the version of prisma you are using and report your issue on their issue list https://github.com/prisma/prisma/issues.