bouncycastle failures related to TLS 1.3 Hybrid Key Exchange - DER input not an octet string

[cushon]

I am seeing a regression that affects bouncycastle when testing with the latest JDK EA builds that include the changes for JEP 527: Post-Quantum Hybrid Key Exchange for TLS 1.3.

In JDK versions that include those changes, the following test fails.

import java.net.HttpURLConnection;
import java.net.URL;
import java.security.Security;
import java.time.Duration;
import org.bouncycastle.jce.provider.BouncyCastleProvider;

public final class DownloadTest {

  private static final Duration TIMEOUT = Duration.ofSeconds(1);

  public static void main(String[] args) throws Exception {

    if (Security.getProvider(BouncyCastleProvider.PROVIDER_NAME) == null) {
      Security.insertProviderAt(new BouncyCastleProvider(), 1);
    }

    String url = "https://storage.googleapis.com/diagnostic_service_storage/google_wifi_512dp.png";
    HttpURLConnection connection = (HttpURLConnection) new URL(url).openConnection();
    connection.setReadTimeout((int) TIMEOUT.toMillis());
    connection.setConnectTimeout((int) TIMEOUT.toMillis());
    connection.setRequestMethod("GET");
    connection.connect();
  }
}

$ java -Djdk.tls.namedGroups=X25519MLKEM768 -cp bcprov-jdk18on-1.83.jar DownloadTest.java
...
Exception in thread "main" javax.net.ssl.SSLException: (bad_record_mac) mac check in GCM failed
	at java.base/sun.security.ssl.Alert.createSSLException(Alert.java:132)
	at java.base/sun.security.ssl.TransportContext.fatal(TransportContext.java:375)
	at java.base/sun.security.ssl.TransportContext.fatal(TransportContext.java:318)
	at java.base/sun.security.ssl.TransportContext.fatal(TransportContext.java:313)
	at java.base/sun.security.ssl.SSLTransport.decode(SSLTransport.java:123)
	at java.base/sun.security.ssl.SSLSocketImpl.decode(SSLSocketImpl.java:1506)
	at java.base/sun.security.ssl.SSLSocketImpl.readHandshakeRecord(SSLSocketImpl.java:1421)
	at java.base/sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:455)
	at java.base/sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:426)
	at java.base/sun.net.www.protocol.https.HttpsClient.afterConnect(HttpsClient.java:490)
	at java.base/sun.net.www.protocol.https.AbstractDelegateHttpsURLConnection.connect(AbstractDelegateHttpsURLConnection.java:187)
	at java.base/sun.net.www.protocol.https.HttpsURLConnectionImpl.connect(HttpsURLConnectionImpl.java:141)
	at DownloadTest.main(DownloadTest.java:22)
Caused by: javax.crypto.AEADBadTagException: mac check in GCM failed
	at java.base/jdk.internal.reflect.DirectConstructorHandleAccessor.newInstance(DirectConstructorHandleAccessor.java:62)
	at java.base/java.lang.reflect.Constructor.newInstanceWithCaller(Constructor.java:499)
	at java.base/java.lang.reflect.Constructor.newInstance(Constructor.java:483)
	at org.bouncycastle.jcajce.provider.symmetric.util.BaseBlockCipher$AEADGenericBlockCipher.doFinal(Unknown Source)
	at org.bouncycastle.jcajce.provider.symmetric.util.BaseBlockCipher.engineDoFinal(Unknown Source)
	at java.base/javax.crypto.CipherSpi.bufferCrypt(CipherSpi.java:788)
	at java.base/javax.crypto.CipherSpi.engineDoFinal(CipherSpi.java:723)
	at java.base/javax.crypto.Cipher.doFinal(Cipher.java:2539)
	at java.base/sun.security.ssl.SSLCipher$T13GcmReadCipherGenerator$GcmReadCipher.decrypt(SSLCipher.java:1909)
	at java.base/sun.security.ssl.SSLSocketInputRecord.decodeInputRecord(SSLSocketInputRecord.java:264)
	at java.base/sun.security.ssl.SSLSocketInputRecord.decode(SSLSocketInputRecord.java:181)
	at java.base/sun.security.ssl.SSLTransport.decode(SSLTransport.java:111)
	at java.base/sun.security.ssl.SSLSocketImpl.decode(SSLSocketImpl.java:1506)
	at java.base/sun.security.ssl.SSLSocketImpl.readHandshakeRecord(SSLSocketImpl.java:1421)
	at java.base/sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:455)
	at java.base/sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:426)
	at java.base/sun.net.www.protocol.https.HttpsClient.afterConnect(HttpsClient.java:490)
	at java.base/sun.net.www.protocol.https.AbstractDelegateHttpsURLConnection.connect(AbstractDelegateHttpsURLConnection.java:187)
	at java.base/sun.net.www.protocol.https.HttpsURLConnectionImpl.connect(HttpsURLConnectionImpl.java:141)
	at DownloadTest.main(DownloadTest.java:22)

I was initially investigating this problem on a different machine, where the same repro fails with a different error, the DER input not an octet string crash below:

javax.net.ssl.SSLHandshakeException: Could not generate secret
        at java.base/sun.security.ssl.KAKeyDerivation.t13DeriveKey(KAKeyDerivation.java:228)
        at java.base/sun.security.ssl.KAKeyDerivation.deriveKey(KAKeyDerivation.java:87)
        at java.base/sun.security.ssl.ServerHello$T13ServerHelloConsumer.consume(ServerHello.java:1347)
        at java.base/sun.security.ssl.ServerHello$ServerHelloConsumer.onServerHello(ServerHello.java:1061)
        at java.base/sun.security.ssl.ServerHello$ServerHelloConsumer.consume(ServerHello.java:941)
        at java.base/sun.security.ssl.SSLHandshake.consume(SSLHandshake.java:421)
        at java.base/sun.security.ssl.HandshakeContext.dispatch(HandshakeContext.java:477)
        at java.base/sun.security.ssl.HandshakeContext.dispatch(HandshakeContext.java:448)
        at java.base/sun.security.ssl.TransportContext.dispatch(TransportContext.java:198)
        at java.base/sun.security.ssl.SSLTransport.decode(SSLTransport.java:172)
        at java.base/sun.security.ssl.SSLSocketImpl.decode(SSLSocketImpl.java:1506)
        at java.base/sun.security.ssl.SSLSocketImpl.readHandshakeRecord(SSLSocketImpl.java:1421)
        at java.base/sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:455)
        at java.base/sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:426)
        at java.base/sun.net.www.protocol.https.HttpsClient.afterConnect(HttpsClient.java:490)
        at java.base/sun.net.www.protocol.https.AbstractDelegateHttpsURLConnection.connect(AbstractDelegateHttpsURLConnection.java:187)
        at java.base/sun.net.www.protocol.https.HttpsURLConnectionImpl.connect(HttpsURLConnectionImpl.java:141)
        at DownloadTest.download(DownloadTest.java:26)
        ... 21 trimmed
Caused by: java.security.InvalidKeyException: Cannot parse input
        at java.base/sun.security.pkcs.NamedPKCS8Key.<init>(NamedPKCS8Key.java:95)
        at java.base/sun.security.provider.NamedKeyFactory.fromPKCS8(NamedKeyFactory.java:160)
        at java.base/sun.security.provider.NamedKeyFactory.engineTranslateKey(NamedKeyFactory.java:272)
        at java.base/sun.security.provider.NamedKEM.engineNewDecapsulator(NamedKEM.java:96)
        at java.base/javax.crypto.KEM$DelayedKEM.newDecapsulator(KEM.java:508)
        at java.base/javax.crypto.KEM.newDecapsulator(KEM.java:770)
        at java.base/javax.crypto.KEM.newDecapsulator(KEM.java:751)
        at java.base/sun.security.ssl.Hybrid$KEMImpl.engineNewDecapsulator(Hybrid.java:292)
        at java.base/javax.crypto.KEM.newDecapsulator(KEM.java:771)
        at java.base/javax.crypto.KEM.newDecapsulator(KEM.java:751)
        at java.base/sun.security.ssl.KAKeyDerivation.t13DeriveKey(KAKeyDerivation.java:214)
        ... 39 more
Caused by: java.io.IOException: DER input not an octet string
        at java.base/sun.security.util.DerInputStream.getOctetString(DerInputStream.java:155)
        at java.base/sun.security.pkcs.NamedPKCS8Key.<init>(NamedPKCS8Key.java:93)
        ... 49 more

[xnox]

@cushon is it required to use Java JSSE? Have you considered to use bouncycastle jsse with bouncycastle? See https://downloads.bouncycastle.org/fips-java/docs/BC-FJA-(D)TLSUserGuide-2.0.19.pdf ?

It has been the case that Java JSSE often does not work with bouncycastle; and it is recommended to use bouncycastle jsse instead.

[cushon]

[@cushon](https://github.com/cushon) is it required to use Java JSSE? Have you considered to use bouncycastle jsse with bouncycastle? See https://downloads.bouncycastle.org/fips-java/docs/BC-FJA-(D)TLSUserGuide-2.0.19.pdf ?

It has been the case that Java JSSE often does not work with bouncycastle; and it is recommended to use bouncycastle jsse instead.

Thanks! It's helpful to know this isn't a recommended configuration.

The example above was minimized from a failing test I saw testing with the latest JDK EA builds, I don't have full context on what the requirements are for that code, I'll look into bouncycastle jsse as an alternative.

[peterdettman]

I was able to reproduce the first error using the most recent JDK 27 EA and BC 1.83. Some notes from my debugging to help bracket the issue (all tests based on the given example and URL):

1. SunJSSE, no BC, X25519 OK.
2. SunJSSE, no BC, X25519MLKEM768: OK.
3. SunJSSE, BC, X25519 OK.
4. SunJSSE, BC, X25519MLKEM768: failed (with AEADBadTagException stack trace from above).
5. BCJSSE, BC, X25519 OK.
6. BCJSSE, BC, X25519MLKEM768 OK.

In particular, (3) implies that both X25519 and GCM from BC are working fine within SunJSSE. Therefore the bad_record_mac in (4) is likely not a failure of GCM itself, but rather the client somehow derives incorrect key material from the MLKEM768 part of the hybrid. I considered both sending of an incorrect ML-KEM-768 public key and/or an incorrect secret output from decapsulation, however initial debugging attempts did not reveal any glaring problems with either of these. I will have to look in more detail to attribute the error to either BC or SunJSSE.

It's worth noting that SunJSSE is using BC's ML-KEM implementation via the relatively new javax.crypto.KEM API, whereas BCJSSE is using BC's KeyGenerator-based implementation with custom specs (because the KEM API wasn't available when we first added this support). This difference seems like the most productive place to investigate.

The second error is similar to the key encoding issue for ML-DSA noted here: #2186 i.e. the JDK doesn't support the PKCS#8 private key encoding that BC is using . It;s likely that the configuration is not actually the same as the first case, because it seems that a BC private key is being checked for use with a JDK KEM instance, which suggests either the BC provider isn't highest priority or an earlier version of BC is in use (where we don't implement the KEM API). In any case I believe (Open)JDK support for the full set of possible private key encodings is under active development, so I expect this aspect of the issue will resolve itself in due course.

[xnox]

If you look at https://datatracker.ietf.org/doc/html/draft-ietf-tls-hybrid-design-16 you will notice how concatenated shared secret happens in the middle of the key schedule.

Nothing validates if it is correct or not; and things progress; until they eventually blow up.

This might be silly, but the largest mistakes around hybrid KEMs are to do with the order of shared secret concatenation; and one would not spot this if one is testing the same implementation against itself.

https://www.ietf.org/archive/id/draft-kwiatkowski-tls-ecdhe-mlkem-02.html#name-shared-secret

The ECDHE leads over MLKEM; which leads over X25519. This order of concatenation is chosen, such that algorithm that is most likely to have been FIPS validated the soonest is the earliest in the exchange. It does mean that X25519MLKEM768 and SecP256r1MLKEM768 use different order of secrets.

Out of curiosity can you test SecP256r1MLKEM768 ? and see if that also fails or passes? Because if results for SecP256r1MLKEM768 are different to results for X25519MLKEM768 i would bet a cup of coffee the concatenation of the secrets is wrong way around for one of these hybrids.

[peterdettman]

@xnox My testing was against the googleapi server URL above, and then https://pq.cloudflareresearch.com, so cases (2) and (6) should rule out any basic implementation errors. In any case I could see in the debugger that the ML-KEM secret is in first (left) position for X25519MLKEM.

One more datapoint for the failing case (4) is that the BC ML-KEM implementation is not failing decapsulation (with "implicit rejection" mechanism), so it seems the error must lie narrowly within the SunJSSE usage and/or BC implementation of the javax.crypto.KEM API.

[peterdettman]

After all that, the issue turned out to be a known bug (already fixed) in our new HKDF implementation: #2209 . It is triggered specifically for the hybrids since they (try to) take advantage of passing multiple IKMs to HKDF.

There was a secondary issue where we didn't support HKDFParameterSpec.Expand, but this was not fatal since the JDK could failover to the next HKDF implementation. I've now added that support also.

[xnox]

So it is / was concatenation bug 😀 I am going out for a takeout coffee!