JVM Crash in cfb_decrypt with AWS Graviton 2 (Java Corretto 21)

[alexwatson]

Repeatable JVM crash using Java corretto 21.0.9.11.1 when decrypting a PGP file with Bouncy Castle LTS v2.73.5 (and also v2.73.10).
JVM crash occurs on AWS server using Elastic Beanstalk Tomcat 11 / Corretto 21 running on 64 bit Amazon Linux 2023/5.9.3.
The same code works perfectly on MacBook Pro with Java corretto 21.0.8.

# A fatal error has been detected by the Java Runtime Environment:
#
#  SIGILL (0x4) at pc=0x0000ffffa012adc8, pid=64494, tid=65024
#
# JRE version: OpenJDK Runtime Environment Corretto-21.0.9.11.1 (21.0.9+11) (build 21.0.9+11-LTS)
# Java VM: OpenJDK 64-Bit Server VM Corretto-21.0.9.11.1 (21.0.9+11-LTS, mixed mode, sharing, tiered, compressed oops, compressed class ptrs, serial gc, linux-aarch64)
# Problematic frame:
# C  [libbc-lts-neon-le.so+0xedc8]  cfb_decrypt+0x104
#
# Core dump will be written. Default location: Core dumps may be processed with "/usr/lib/systemd/systemd-coredump %P %u %g %s %t %c %h" (or dumping to /usr/share/tomcat11/core.64494)
#
# If you would like to submit a bug report, please visit:
#   https://github.com/corretto/corretto-21/issues/
# The crash happened outside the Java Virtual Machine in native code.
# See problematic frame for where to report the bug.

Maven dependencies (with versions 2.73.5 and 2.73.10):

<dependency>
	<groupId>org.bouncycastle</groupId>
	<artifactId>bcprov-lts8on</artifactId>
	<version>${bouncy-castle-version}</version>
</dependency>
<dependency>
	<groupId>org.bouncycastle</groupId>
	<artifactId>bcpkix-lts8on</artifactId>
	<version>${bouncy-castle-version}</version>
</dependency>
<dependency>
	<groupId>org.bouncycastle</groupId>
	<artifactId>bcpg-lts8on</artifactId>
	<version>${bouncy-castle-version}</version>
</dependency>

Host: AArch64, 1 cores, 1G, Amazon Linux release 2023.10.20260120 (Amazon Linux)
Time: Wed Feb 11 12:49:36 2026 AEDT elapsed time: 2337.653122 seconds (0d 0h 38m 57s)

---------------  T H R E A D  ---------------

Current thread (0x0000ffff9f5beb00):  JavaThread "au.com.vikingc.webmonitortools.jobs.JobManagerExecutorThread" daemon [_thread_in_native, id=65024, stack(0x0000ffff65c0e000,0x0000ffff65e0c000) (2040K)]

Stack: [0x0000ffff65c0e000,0x0000ffff65e0c000],  sp=0x0000ffff65e09520,  free space=2029k
Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native code)
C  [libbc-lts-neon-le.so+0xedc8]  cfb_decrypt+0x104
C  [libbc-lts-neon-le.so+0xf1c8]  Java_org_bouncycastle_crypto_engines_AESNativeCFB_processBytes+0x198
j  org.bouncycastle.crypto.engines.AESNativeCFB.processBytes(J[BII[BI)I+0
j  org.bouncycastle.crypto.engines.AESNativeCFB.processBytes([BII[BI)I+31
j  org.bouncycastle.crypto.engines.AESNativeCFB.processBlocks([BII[BI)I+30
j  org.bouncycastle.crypto.DefaultBufferedBlockCipher.processBytes([BII[BI)I+209
j  org.bouncycastle.jcajce.provider.symmetric.util.BaseBlockCipher$BufferedGenericBlockCipher.processBytes([BII[BI)I+11
j  org.bouncycastle.jcajce.provider.symmetric.util.BaseBlockCipher.engineDoFinal([BII)[B+26
j  javax.crypto.Cipher.doFinal([BII)[B+46 java.base@21.0.9
j  org.bouncycastle.openpgp.operator.jcajce.JcePBESecretKeyDecryptorBuilder$1.recoverKeyData(I[B[B[BII)[B+61
j  org.bouncycastle.openpgp.PGPSecretKey.extractKeyData(Lorg/bouncycastle/openpgp/operator/PBESecretKeyDecryptor;)[B+147
j  org.bouncycastle.openpgp.PGPSecretKey.extractPrivateKey(Lorg/bouncycastle/openpgp/operator/PBESecretKeyDecryptor;)Lorg/bouncycastle/openpgp/PGPPrivateKey;+19
j  au.com.vikingc.universalfilesystem.encryption.PGPEncryptionProvider.decrypt(Ljava/io/InputStream;)Ljava/io/InputStream;+193

[mwcw]

Hi,

SIGILL usually means there is an instruction the machine does not support.

Are you able to share the output of "cat /proc/cpuinfo" or the machine type code eg "T4g" please.

Thanks,

MW

[xnox]

Is it Graviton2? and it works fine on Graviton3?

[alexwatson]

The server is a c6g.medium (Graviton 2).
I will try a c7g.medium (Graviton 3) to see if that works.

[alexwatson]

Tried on c7g.medium (Graviton 3) server, and the decryption worked without JDK error.
The issue seems to be Graviton 2 specific.

Is this a known issue? Is there a fix coming soon?

[c6g.medium server]
> cat /proc/cpuinfo
processor	: 0
BogoMIPS	: 243.75
Features	: fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm lrcpc dcpop asimddp
CPU implementer	: 0x41
CPU architecture: 8
CPU variant	: 0x3
CPU part	: 0xd0c
CPU revision	: 1

[c7g.medium server]
> cat /proc/cpuinfo
processor	: 0
BogoMIPS	: 2100.00
Features	: fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm paca pacg dcpodp svei8mm svebf16 i8mm bf16 dgh rng
CPU implementer	: 0x41
CPU architecture: 8
CPU variant	: 0x1
CPU part	: 0xd40
CPU revision	: 1

[mwcw]

Hi,

Tried on c7g.medium (Graviton 3) server, and the decryption worked without JDK error.
The issue seems to be Graviton 2 specific.

Is this a known issue? Is there a fix coming soon?

Not a known issue, but an unexpected one, and I am guessing we will need to include a specific native interface compiled to match the Graviton 2 CPU Features.

The fix will require a new release, which I don't have a timeframe for.

In the meantime if you need to use a Graviton 2 you can run it with:

-Dorg.bouncycastle.native.cpu_variant=java

But you will not have any native accelerations, it will only be pure java.

Thanks for taking the time collect the information about the machines, very much appreciated.

MW

[xnox]

Separately, there are many issues known with hw acceleration on Graviton2 w.r.t. sve instructions. This has been observed in the past in other open source projects.

A middle ground would be to detect Graviton2 CPU and disable SVE, or set a CPU acceleration level without SVE.

Note, in general price performance of Graviton3 is often better (cheaper for same or better performance), is there a reason to still use Graviton2?

[alexwatson]

Thanks @xnox - agree Graviton 3 is better value for money, but Graviton 2 is slightly cheaper in my region (6%) - and current servers are not performance critical. We also have reserved instances for Graviton 2 server - so there could be costs for switching.

Ultimately we view Java as interpreted JVM and therefore CPU/OS agnostic. Historically we've had a high level of confidence migrating Java web applications between CPU instances - so my 2c feedback would be to for Bouncy Castle to prefer robustness and perhaps allow opt-in for hardware acceleration rather than defaulting to hardware acceleration that can be buggy/unportable. This could be done via choosing between pure Java and native accelerated libraries (so that a developer choosing which library to use becomes aware of this distinction).

Until we had the fatal JDK error, I was unaware that BC used native libraries and that using BC was affecting hardware portability. I'm now aware of the system property (thanks to @mwcw's comment), but I want to help anyone else who has the same experience as me (ie. how can we improve documentation / choices).

For my benefit, what is the performance gain from using hardware acceleration? Are we talking 5-10% or 200-300% gain?

I'm also aware that Graviton 4 and Graviton 5 are available, so I assume any hardware acceleration problems could be expected across each new chipset?

Thanks again for all of your help and guidance.

[olivergillespie]

I reproduced it and the bad instruction is eor3 which is from the SHA3 extensions, which Graviton 2 does not support.

sha3 is marked as supported here even if it's not.

BouncyCastle APIs (LTS edition) v2.73.7
Native Build Date: 2024-11-07T12:43:41
Native Status: READY
Native Variant: neon-le
Native Features: AES/CBC AES/CCM AES/CFB AES/CTR AES/ECB AES/GCM MULACC SHA2 SHA224 SHA256


CPU Features and Variant availability.
--------------------------------------------------------------------------------
Variant   CPU features + or -:                              Supported
--------------------------------------------------------------------------------
neon-le   +aes +sha256 -sha512 -sha3 +neon                  Variant Supported

It selects neon-le which enables sha3 extensions even though it knows they're not supported. I think the correct graviton 2 set would be -march=armv8.2-a+crypto.

[xnox]

Thanks @xnox - agree Graviton 3 is better value for money, but Graviton 2 is slightly cheaper in my region (6%) - and current servers are not performance critical. We also have reserved instances for Graviton 2 server - so there could be costs for switching.

Ultimately we view Java as interpreted JVM and therefore CPU/OS agnostic.

False, all java implementations have native code by default, or via jit, or via native code implementation. Each new version adds more.

This is definitely a bug in Graviton2 advertised features, which is in contradiction as to what is supported - hence correct runtime detection leads to failures not observed on Apple, Ampere, Google, Microsoft, Qualcomm and Mediatek chips.

Historically we've had a high level of confidence migrating Java web applications between CPU instances - so my 2c feedback would be to for Bouncy Castle to prefer robustness and perhaps allow opt-in for hardware acceleration rather than defaulting to hardware acceleration that can be buggy/unportable. This could be done via choosing between pure Java and native accelerated libraries (so that a developer choosing which library to use becomes aware of this distinction).

Until we had the fatal JDK error, I was unaware that BC used native libraries and that using BC was affecting hardware portability. I'm now aware of the system property (thanks to @mwcw's comment), but I want to help anyone else who has the same experience as me (ie. how can we improve documentation / choices).

For my benefit, what is the performance gain from using hardware acceleration? Are we talking 5-10% or 200-300% gain?

For AES it is often 10x and more, so 1,000% maybe? So yeah, AES without hw acceleration is typically toast.

Hence most OSes require AES HW.

Acceleration for SHA3 on arm chips pre-SVE2 can be hit or miss; hence in OpenSSL they have per-chip allow list adding chips only after manual benchmark confirms gains.

I'm also aware that Graviton 4 and Graviton 5 are available, so I assume any hardware acceleration problems could be expected across each new chipset?

Thanks again for all of your help and guidance.

[olivergillespie]

This is definitely a bug in Graviton2 advertised features

Isn't the problem here where -march=armv8.4-a+crypto+sha2+sha3 is set for all neon cpus, but that's not the right setting for graviton 2?

I know the explicit use of those instructions from BC code is gated by runtime checks, but GCC is also free to sprinkle its own un-gated sha3 instructions elsewhere, like the eor3 it adds in cfb_decrypt -> decrypt_blocks. Not saying it's the right approach, but this patch fixes the issue for me.

[xnox]

This is definitely a bug in Graviton2 advertised features

Isn't the problem here where -march=armv8.4-a+crypto+sha2+sha3 is set for all neon cpus, but that's not the right setting for graviton 2?

Yes, This is way too high of march for single build. It is ok if one does multiple builds and ships multiple copies of libraries in for example different subfolders that glibc hwcaps can resolve and load separately.

If one relies on runtime checks and guarded codepaths, the march must be lower; with all other things opt-in.

Cause yes gravito2 does not have sha3 support.

or for example, java code should test first if there is support for all of armv8.4-a+crypto+sha2+sha3 before loading that library

I know the explicit use of those instructions from BC code is gated by runtime checks, but GCC is also free to sprinkle its own un-gated sha3 instructions elsewhere, like the eor3 it adds in cfb_decrypt -> decrypt_blocks. Not saying it's the right approach, but this patch fixes the issue for me.

yeah.

[alexwatson]

Any updates on this ticket? Is there a patch / version that I can test? Any ETA for a proper release?
Sorry if I should be looking elsewhere for this info - please guide me to the right place.