SkipPublicKey and memory exhaustion

When the configuration option "SkipPublicKey" (config->use_pubkey) is set, the public keys generated by C_GenerateKeyPair are temporarily stored by the HSM for the session duration thanks to the CKA_TOKEN CK_ATTRIBUTE set to CK_FALSE.

The code of hsm_generate_rsa_key (libhsm.c) therefore drops the CK_OBJECT_HANDLE value returned by the C_GenerateKeyPair call, actually storing 0 as the value of the field public_key of the returned hsm_key_t object.

When a key is destroyed, in general because it is declared dead, the private key CK_OBJECT_HANDLE is retrieved from the database, and a hsm_key_t object is created within the function hsm_key_new_privkey_object_handle. If the SkipPublicKey configuration option is set, the public key CK_OBJECT_HANDLE is set to 0.

When a session is kept opened long enough, and several key rotations are performed within the same session, memory of small HSMs such as smartcards can be exhausted. This actually happened on my setup: HSM returning CKR_GENERAL_ERROR because of memory/storage exhaustion caused by unwanted public keys without associated private keys filling the HSM, even with the SkipPublicKey option set.

This report is therefore referencing to the following issues:

• Setting CKA_Token to CK_FALSE is not sufficient to ensure that HSM storage capability (be it long term or memory-short-term storage) is not exceeded: the actual public key object should be destroyed immediately after being generated by the C_GenerateKeyPair call to ensure that no object survives that is not desired.

• If the SkipPublicKey configuration option is changed during operations, I think the public key that have been actually generated (and desired at that time) are not destroyed when the associated private key is.

I think proper fix would destroy the undesired public key right after generation and key destruction would always try to retrieve the public key handle and set it to 0 if not found, regardless of the current SkipPublicKey option value.

Note: OpenDNSSEC2.X versions may be affected but the issue might also be larger than just hsm_generate_rsa_key as other algorithms are handled.