in hsm_get_key_rdata, the public key exponent for a DNSKEY is written with a 1- or 3-byte length. The choice between the formats is wrong,
public_exponent_len <= 256
and should be
public_exponent_len <= 255
according to RFC 3110, section 2:
For interoperability, the exponent and modulus are each limited to
4096 bits in length. The public key exponent is a variable length
unsigned integer. Its length in octets is represented as one octet
if it is in the range of 1 to 255 and by a zero octet followed by a
two octet unsigned length if it is longer than 255 bytes.
A trivial libhsm.c patch follows.
I doubt anyone runs into trouble with this one; public key exponents are generally chosen to be 65537, which fits in a length of only 3 bytes. Still, it's a coded fault.