Skip to content

rsa-oaep-support #821

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
NESYOR wants to merge 1 commit into
base: main
Choose a base branch
from
Open

rsa-oaep-support #821

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
390 changes: 251 additions & 139 deletions src/lib/SoftHSM.cpp
View file
Open in desktop

Large diffs are not rendered by default.

2 changes: 1 addition & 1 deletion src/lib/SoftHSM.h
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -507,7 +507,7 @@ class SoftHSM
ByteString &keydata
);

CK_RV MechParamCheckRSAPKCSOAEP(CK_MECHANISM_PTR pMechanism);
CK_RV MechParamCheckRSAPKCSOAEP(CK_MECHANISM_PTR pMechanism, RSA_PKCS_OAEP_PARAMS* outParams = NULL);
Copy link

@coderabbitai coderabbitai bot Sep 28, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🔴 Critical

Restore missing declaration for RSA_PKCS_OAEP_PARAMS

SoftHSM.h now uses RSA_PKCS_OAEP_PARAMS in the public signature, but the header neither includes the definition nor forward declares the struct. Compiling a TU that includes SoftHSM.h before any other OAEP headers currently fails with “RSA_PKCS_OAEP_PARAMS has not been declared.” Please add a declaration (or the defining include) in this header so it remains self-contained.

+#include "AsymmetricAlgorithm.h"
📝 Committable suggestion

‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.

Suggested change
CK_RV MechParamCheckRSAPKCSOAEP(CK_MECHANISM_PTR pMechanism, RSA_PKCS_OAEP_PARAMS* outParams = NULL);
#include "AsymmetricAlgorithm.h"
CK_RV MechParamCheckRSAPKCSOAEP(CK_MECHANISM_PTR pMechanism, RSA_PKCS_OAEP_PARAMS* outParams = NULL);
🤖 Prompt for AI Agents 
In src/lib/SoftHSM.h around line 510 the signature MechParamCheckRSAPKCSOAEP
references RSA_PKCS_OAEP_PARAMS but the header neither forward-declares nor
includes its definition; fix by either adding a forward declaration (e.g., a
typedef struct RSA_PKCS_OAEP_PARAMS RSA_PKCS_OAEP_PARAMS;) near the top of
SoftHSM.h or by including the header that defines RSA_PKCS_OAEP_PARAMS (for
example the appropriate pkcs11 header used in this project), and ensure the
include is guarded and placed before the function prototype so the file becomes
self-contained.

CK_RV MechParamCheckRSAAESKEYWRAP(CK_MECHANISM_PTR pMechanism);

bool isMechanismPermitted(OSObject* key, CK_MECHANISM_PTR pMechanism);
Expand Down
16 changes: 8 additions & 8 deletions src/lib/crypto/AsymmetricAlgorithm.cpp
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -156,20 +156,20 @@ bool AsymmetricAlgorithm::isWrappingMech(AsymMech::Type padding)
}

// Wrap/Unwrap keys
bool AsymmetricAlgorithm::wrapKey(PublicKey* publicKey, const ByteString& data, ByteString& encryptedData, const AsymMech::Type padding)
bool AsymmetricAlgorithm::wrapKey(PublicKey* publicKey, const ByteString& data, ByteString& encryptedData, const AsymMech::Type padding, const void* param /* = NULL */, const size_t paramLen /* = 0 */)
{
if (!isWrappingMech(padding))
return false;
if (!isWrappingMech(padding))
return false;

return encrypt(publicKey, data, encryptedData, padding);
return encrypt(publicKey, data, encryptedData, padding, param, paramLen);
}

bool AsymmetricAlgorithm::unwrapKey(PrivateKey* privateKey, const ByteString& encryptedData, ByteString& data, const AsymMech::Type padding)
bool AsymmetricAlgorithm::unwrapKey(PrivateKey* privateKey, const ByteString& encryptedData, ByteString& data, const AsymMech::Type padding, const void* param /* = NULL */, const size_t paramLen /* = 0 */)
{
if (!isWrappingMech(padding))
return false;
if (!isWrappingMech(padding))
return false;

return decrypt(privateKey, encryptedData, data, padding);
return decrypt(privateKey, encryptedData, data, padding, param, paramLen);
}


Expand Down
25 changes: 16 additions & 9 deletions src/lib/crypto/AsymmetricAlgorithm.h
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -111,9 +111,16 @@ struct AsymRSAMGF

struct RSA_PKCS_PSS_PARAMS
{
HashAlgo::Type hashAlg;
AsymRSAMGF::Type mgf;
size_t sLen;
HashAlgo::Type hashAlg;
AsymRSAMGF::Type mgf;
size_t sLen;
};

struct RSA_PKCS_OAEP_PARAMS
{
HashAlgo::Type hashAlg;
AsymRSAMGF::Type mgf;
size_t hashLen;
};
Comment on lines +119 to 124
Copy link

@coderabbitai coderabbitai bot Sep 28, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🟠 Major

OAEP label not represented; risks silent label ignore

PKCS#11 CK_RSA_PKCS_OAEP_PARAMS includes label fields; the current struct lacks them. Either add and plumb label support or ensure upstream rejects non-empty labels to avoid non-compliance and silent behavior differences.

 struct RSA_PKCS_OAEP_PARAMS
 {
         HashAlgo::Type hashAlg;
         AsymRSAMGF::Type mgf;
         size_t hashLen;
+        // Optional OAEP label (may be NULL when labelLen == 0)
+        const unsigned char* label;
+        size_t labelLen;
 };

If adding later, document that labelLen must be 0 for now and enforce rejection in backends. Based on learnings.

📝 Committable suggestion

‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.

Suggested change
struct RSA_PKCS_OAEP_PARAMS
{
HashAlgo::Type hashAlg;
AsymRSAMGF::Type mgf;
size_t hashLen;
};
struct RSA_PKCS_OAEP_PARAMS
{
HashAlgo::Type hashAlg;
AsymRSAMGF::Type mgf;
size_t hashLen;
// Optional OAEP label (may be NULL when labelLen == 0)
const unsigned char* label;
size_t labelLen;
};


class AsymmetricAlgorithm
Expand All @@ -138,14 +145,14 @@ class AsymmetricAlgorithm
virtual bool verifyFinal(const ByteString& signature);

// Encryption functions
virtual bool encrypt(PublicKey* publicKey, const ByteString& data, ByteString& encryptedData, const AsymMech::Type padding) = 0;
virtual bool encrypt(PublicKey* publicKey, const ByteString& data, ByteString& encryptedData, const AsymMech::Type padding, const void* param = NULL, const size_t paramLen = 0) = 0;

// Decryption functions
virtual bool decrypt(PrivateKey* privateKey, const ByteString& encryptedData, ByteString& data, const AsymMech::Type padding) = 0;
// Decryption functions
virtual bool decrypt(PrivateKey* privateKey, const ByteString& encryptedData, ByteString& data, const AsymMech::Type padding, const void* param = NULL, const size_t paramLen = 0) = 0;

// Wrap/Unwrap keys
bool wrapKey(PublicKey* publicKey, const ByteString& data, ByteString& encryptedData, const AsymMech::Type padding);
bool unwrapKey(PrivateKey* privateKey, const ByteString& encryptedData, ByteString& data, const AsymMech::Type padding);
// Wrap/Unwrap keys
bool wrapKey(PublicKey* publicKey, const ByteString& data, ByteString& encryptedData, const AsymMech::Type padding, const void* param = NULL, const size_t paramLen = 0);
bool unwrapKey(PrivateKey* privateKey, const ByteString& encryptedData, ByteString& data, const AsymMech::Type padding, const void* param = NULL, const size_t paramLen = 0);

// Key factory
virtual bool generateKeyPair(AsymmetricKeyPair** ppKeyPair, AsymmetricParameters* parameters, RNG* rng = NULL) = 0;
Expand Down
4 changes: 2 additions & 2 deletions src/lib/crypto/BotanDH.cpp
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -93,7 +93,7 @@ bool BotanDH::verifyFinal(const ByteString& /*signature*/)

// Encryption functions
bool BotanDH::encrypt(PublicKey* /*publicKey*/, const ByteString& /*data*/,
ByteString& /*encryptedData*/, const AsymMech::Type /*padding*/)
ByteString& /*encryptedData*/, const AsymMech::Type /*padding*/, const void* /*param*/ /* = NULL */, const size_t /*paramLen*/ /* = 0 */)
{
ERROR_MSG("DH does not support encryption");

Expand All @@ -102,7 +102,7 @@ bool BotanDH::encrypt(PublicKey* /*publicKey*/, const ByteString& /*data*/,

// Decryption functions
bool BotanDH::decrypt(PrivateKey* /*privateKey*/, const ByteString& /*encryptedData*/,
ByteString& /*data*/, const AsymMech::Type /*padding*/)
ByteString& /*data*/, const AsymMech::Type /*padding*/, const void* /*param*/ /* = NULL */, const size_t /*paramLen*/ /* = 0 */)
{
ERROR_MSG("DH does not support decryption");

Expand Down
4 changes: 2 additions & 2 deletions src/lib/crypto/BotanDH.h
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -54,10 +54,10 @@ class BotanDH : public AsymmetricAlgorithm
virtual bool verifyFinal(const ByteString& signature);

// Encryption functions
virtual bool encrypt(PublicKey* publicKey, const ByteString& data, ByteString& encryptedData, const AsymMech::Type padding);
virtual bool encrypt(PublicKey* publicKey, const ByteString& data, ByteString& encryptedData, const AsymMech::Type padding, const void* param = NULL, const size_t paramLen = 0);

// Decryption functions
virtual bool decrypt(PrivateKey* privateKey, const ByteString& encryptedData, ByteString& data, const AsymMech::Type padding);
virtual bool decrypt(PrivateKey* privateKey, const ByteString& encryptedData, ByteString& data, const AsymMech::Type padding, const void* param = NULL, const size_t paramLen = 0);

// Key factory
virtual bool generateKeyPair(AsymmetricKeyPair** ppKeyPair, AsymmetricParameters* parameters, RNG* rng = NULL);
Expand Down
4 changes: 2 additions & 2 deletions src/lib/crypto/BotanDSA.cpp
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -485,7 +485,7 @@ bool BotanDSA::verifyFinal(const ByteString& signature)

// Encryption functions
bool BotanDSA::encrypt(PublicKey* /*publicKey*/, const ByteString& /*data*/,
ByteString& /*encryptedData*/, const AsymMech::Type /*padding*/)
ByteString& /*encryptedData*/, const AsymMech::Type /*padding*/, const void* /*param*/ /* = NULL */, const size_t /*paramLen*/ /* = 0 */)
{
ERROR_MSG("DSA does not support encryption");

Expand All @@ -494,7 +494,7 @@ bool BotanDSA::encrypt(PublicKey* /*publicKey*/, const ByteString& /*data*/,

// Decryption functions
bool BotanDSA::decrypt(PrivateKey* /*privateKey*/, const ByteString& /*encryptedData*/,
ByteString& /*data*/, const AsymMech::Type /*padding*/)
ByteString& /*data*/, const AsymMech::Type /*padding*/, const void* /*param*/ /* = NULL */, const size_t /*paramLen*/ /* = 0 */)
{
ERROR_MSG("DSA does not support decryption");

Expand Down
4 changes: 2 additions & 2 deletions src/lib/crypto/BotanDSA.h
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -59,10 +59,10 @@ class BotanDSA : public AsymmetricAlgorithm
virtual bool verifyFinal(const ByteString& signature);

// Encryption functions
virtual bool encrypt(PublicKey* publicKey, const ByteString& data, ByteString& encryptedData, const AsymMech::Type padding);
virtual bool encrypt(PublicKey* publicKey, const ByteString& data, ByteString& encryptedData, const AsymMech::Type padding, const void* param = NULL, const size_t paramLen = 0);

// Decryption functions
virtual bool decrypt(PrivateKey* privateKey, const ByteString& encryptedData, ByteString& data, const AsymMech::Type padding);
virtual bool decrypt(PrivateKey* privateKey, const ByteString& encryptedData, ByteString& data, const AsymMech::Type padding, const void* param = NULL, const size_t paramLen = 0);

// Key factory
virtual bool generateKeyPair(AsymmetricKeyPair** ppKeyPair, AsymmetricParameters* parameters, RNG* rng = NULL);
Expand Down
4 changes: 2 additions & 2 deletions src/lib/crypto/BotanECDH.cpp
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -94,7 +94,7 @@ bool BotanECDH::verifyFinal(const ByteString& /*signature*/)

// Encryption functions
bool BotanECDH::encrypt(PublicKey* /*publicKey*/, const ByteString& /*data*/,
ByteString& /*encryptedData*/, const AsymMech::Type /*padding*/)
ByteString& /*encryptedData*/, const AsymMech::Type /*padding*/, const void* /*param*/ /* = NULL */, const size_t /*paramLen*/ /* = 0 */)
{
ERROR_MSG("ECDH does not support encryption");

Expand All @@ -103,7 +103,7 @@ bool BotanECDH::encrypt(PublicKey* /*publicKey*/, const ByteString& /*data*/,

// Decryption functions
bool BotanECDH::decrypt(PrivateKey* /*privateKey*/, const ByteString& /*encryptedData*/,
ByteString& /*data*/, const AsymMech::Type /*padding*/)
ByteString& /*data*/, const AsymMech::Type /*padding*/, const void* /*param*/ /* = NULL */, const size_t /*paramLen*/ /* = 0 */)
{
ERROR_MSG("ECDH does not support decryption");

Expand Down
4 changes: 2 additions & 2 deletions src/lib/crypto/BotanECDH.h
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -54,10 +54,10 @@ class BotanECDH : public AsymmetricAlgorithm
virtual bool verifyFinal(const ByteString& signature);

// Encryption functions
virtual bool encrypt(PublicKey* publicKey, const ByteString& data, ByteString& encryptedData, const AsymMech::Type padding);
virtual bool encrypt(PublicKey* publicKey, const ByteString& data, ByteString& encryptedData, const AsymMech::Type padding, const void* param = NULL, const size_t paramLen = 0);

// Decryption functions
virtual bool decrypt(PrivateKey* privateKey, const ByteString& encryptedData, ByteString& data, const AsymMech::Type padding);
virtual bool decrypt(PrivateKey* privateKey, const ByteString& encryptedData, ByteString& data, const AsymMech::Type padding, const void* param = NULL, const size_t paramLen = 0);

// Key factory
virtual bool generateKeyPair(AsymmetricKeyPair** ppKeyPair, AsymmetricParameters* parameters, RNG* rng = NULL);
Expand Down
4 changes: 2 additions & 2 deletions src/lib/crypto/BotanECDSA.cpp
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -321,7 +321,7 @@ bool BotanECDSA::verifyFinal(const ByteString& /*signature*/)

// Encryption functions
bool BotanECDSA::encrypt(PublicKey* /*publicKey*/, const ByteString& /*data*/,
ByteString& /*encryptedData*/, const AsymMech::Type /*padding*/)
ByteString& /*encryptedData*/, const AsymMech::Type /*padding*/, const void* /*param*/ /* = NULL */, const size_t /*paramLen*/ /* = 0 */)
{
ERROR_MSG("ECDSA does not support encryption");

Expand All @@ -330,7 +330,7 @@ bool BotanECDSA::encrypt(PublicKey* /*publicKey*/, const ByteString& /*data*/,

// Decryption functions
bool BotanECDSA::decrypt(PrivateKey* /*privateKey*/, const ByteString& /*encryptedData*/,
ByteString& /*data*/, const AsymMech::Type /*padding*/)
ByteString& /*data*/, const AsymMech::Type /*padding*/, const void* /*param*/ /* = NULL */, const size_t /*paramLen*/ /* = 0 */)
{
ERROR_MSG("ECDSA does not support decryption");

Expand Down
4 changes: 2 additions & 2 deletions src/lib/crypto/BotanECDSA.h
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -59,10 +59,10 @@ class BotanECDSA : public AsymmetricAlgorithm
virtual bool verifyFinal(const ByteString& signature);

// Encryption functions
virtual bool encrypt(PublicKey* publicKey, const ByteString& data, ByteString& encryptedData, const AsymMech::Type padding);
virtual bool encrypt(PublicKey* publicKey, const ByteString& data, ByteString& encryptedData, const AsymMech::Type padding, const void* param = NULL, const size_t paramLen = 0);

// Decryption functions
virtual bool decrypt(PrivateKey* privateKey, const ByteString& encryptedData, ByteString& data, const AsymMech::Type padding);
virtual bool decrypt(PrivateKey* privateKey, const ByteString& encryptedData, ByteString& data, const AsymMech::Type padding, const void* param = NULL, const size_t paramLen = 0);

// Key factory
virtual bool generateKeyPair(AsymmetricKeyPair** ppKeyPair, AsymmetricParameters* parameters, RNG* rng = NULL);
Expand Down
4 changes: 2 additions & 2 deletions src/lib/crypto/BotanEDDSA.cpp
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -256,7 +256,7 @@ bool BotanEDDSA::verifyFinal(const ByteString& /*signature*/)

// Encryption functions
bool BotanEDDSA::encrypt(PublicKey* /*publicKey*/, const ByteString& /*data*/,
ByteString& /*encryptedData*/, const AsymMech::Type /*padding*/)
ByteString& /*encryptedData*/, const AsymMech::Type /*padding*/, const void* /*param*/ /* = NULL */, const size_t /*paramLen*/ /* = 0 */)
{
ERROR_MSG("EDDSA does not support encryption");

Expand All @@ -265,7 +265,7 @@ bool BotanEDDSA::encrypt(PublicKey* /*publicKey*/, const ByteString& /*data*/,

// Decryption functions
bool BotanEDDSA::decrypt(PrivateKey* /*privateKey*/, const ByteString& /*encryptedData*/,
ByteString& /*data*/, const AsymMech::Type /*padding*/)
ByteString& /*data*/, const AsymMech::Type /*padding*/, const void* /*param*/ /* = NULL */, const size_t /*paramLen*/ /* = 0 */)
{
ERROR_MSG("EDDSA does not support decryption");

Expand Down
4 changes: 2 additions & 2 deletions src/lib/crypto/BotanEDDSA.h
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -59,10 +59,10 @@ class BotanEDDSA : public AsymmetricAlgorithm
virtual bool verifyFinal(const ByteString& signature);

// Encryption functions
virtual bool encrypt(PublicKey* publicKey, const ByteString& data, ByteString& encryptedData, const AsymMech::Type padding);
virtual bool encrypt(PublicKey* publicKey, const ByteString& data, ByteString& encryptedData, const AsymMech::Type padding, const void* param = NULL, const size_t paramLen = 0);

// Decryption functions
virtual bool decrypt(PrivateKey* privateKey, const ByteString& encryptedData, ByteString& data, const AsymMech::Type padding);
virtual bool decrypt(PrivateKey* privateKey, const ByteString& encryptedData, ByteString& data, const AsymMech::Type padding, const void* param = NULL, const size_t paramLen = 0);

// Key factory
virtual bool generateKeyPair(AsymmetricKeyPair** ppKeyPair, AsymmetricParameters* parameters, RNG* rng = NULL);
Expand Down
4 changes: 2 additions & 2 deletions src/lib/crypto/BotanGOST.cpp
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -326,7 +326,7 @@ bool BotanGOST::verifyFinal(const ByteString& signature)

// Encryption functions
bool BotanGOST::encrypt(PublicKey* /*publicKey*/, const ByteString& /*data*/,
ByteString& /*encryptedData*/, const AsymMech::Type /*padding*/)
ByteString& /*encryptedData*/, const AsymMech::Type /*padding*/, const void* /*param*/ /* = NULL */, const size_t /*paramLen*/ /* = 0 */)
{
ERROR_MSG("GOST does not support encryption");

Expand All @@ -335,7 +335,7 @@ bool BotanGOST::encrypt(PublicKey* /*publicKey*/, const ByteString& /*data*/,

// Decryption functions
bool BotanGOST::decrypt(PrivateKey* /*privateKey*/, const ByteString& /*encryptedData*/,
ByteString& /*data*/, const AsymMech::Type /*padding*/)
ByteString& /*data*/, const AsymMech::Type /*padding*/, const void* /*param*/ /* = NULL */, const size_t /*paramLen*/ /* = 0 */)
{
ERROR_MSG("GOST does not support decryption");

Expand Down
4 changes: 2 additions & 2 deletions src/lib/crypto/BotanGOST.h
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -57,10 +57,10 @@ class BotanGOST : public AsymmetricAlgorithm
virtual bool verifyFinal(const ByteString& signature);

// Encryption functions
virtual bool encrypt(PublicKey* publicKey, const ByteString& data, ByteString& encryptedData, const AsymMech::Type padding);
virtual bool encrypt(PublicKey* publicKey, const ByteString& data, ByteString& encryptedData, const AsymMech::Type padding, const void* param = NULL, const size_t paramLen = 0);

// Decryption functions
virtual bool decrypt(PrivateKey* privateKey, const ByteString& encryptedData, ByteString& data, const AsymMech::Type padding);
virtual bool decrypt(PrivateKey* privateKey, const ByteString& encryptedData, ByteString& data, const AsymMech::Type padding, const void* param = NULL, const size_t paramLen = 0);

// Key factory
virtual bool generateKeyPair(AsymmetricKeyPair** ppKeyPair, AsymmetricParameters* parameters, RNG* rng = NULL);
Expand Down
Loading