openssl operations involving pcks11 and softHSM result in segfault on exit #729
Description
recently this bug was reported to openssl:
openssl/openssl#22508
Analysis of the problem revealed what is something of an intractable problem.
To summarize the issue:
- openssl on initalization, loads the pkcs11 library
- When configured appropriately for this use case, the pkcs11 library loads the SoftHSM library
- SoftHSM on load creates several global class instances (a SoftHSM object, SecurityManager object, etc)
- Because SoftHSM is written in C++, the C11 standard mandates that those objects be deleted on exit, so as an implementation detail, the compiler implicitly emits calls to __cxa_atexit, to call the destructor method for each object instance
- on exit, becuase posix mandates that atexit handlers be run in the reverse order of their registration, the softHSM library deletes the global object instances prior to any previously registered handlers
- because openssl preforms cleanup in some cases using an atexit handler, it (openssl) makes calls through the pkcs11 engine, which in turn attempts to reference data/code in the softHSM library that has already been deleted via (5), leading to a segfault.
There are a few potential workarounds for this, but it seems to me that the most correct fix would be for softHSM to not delete that data until all references to it were freed from using libraries. While this is more arguably a shortcoming in the C++ standard, I think the best fix would be for softHSM to modify the implementation of the PCKS api in main.cpp such that it can detect when its global object constructors have been called (via a global variable), and return an appropriate status code without attempting to access the now-deleted object data