At least since revision 903, the function hsm_list_keys_session_internal, contained in libhsm.c, is prone to a denial of service (segfault).
realloc is used to allocate an ever-extending table of CK_OBJECT_HANDLE, based on the results of C_FindObjects calls.
No more than max_object_count (100) object handles can be fetched at once. However as long as objects are found (0 != objectCount), the non-null objectCount is added to the total_count counter. This total_count is then used in the realloc call, where it is multiplied by the sizeof CK_OBJECT_HANDLE.
The result of the realloc is not checked and the while(objectCount > 0) is unbound. This means that an adversarial HSM could return non-null objectCount until memory is filled and realloc returns NULL.
A few lines later, this would result in an attempt to dereference a NULL pointer and write there, thus causing a denial of service (segfault).
On top of that denial of service, total_count being a CK_ULONG, the multiplication "total_count * sizeof(CK_OBJECT_HANDLE)" could result in an integer overflow.
I am not sure whether this integer overflow resulting in a very low amount of reallocated memory could result in an exploitable heap buffer overflow. The answer might be dependent on the libc realloc implementation.
I think a check should be performed, in any case.
Please find in attachment a patch for this issue.
OPENDNSSEC-539 Memory corruption (improper realloc call) in hsm_list_keys_session_internal (libhsm.c)