If you have a miss-match between the token label in conf.xml and the actual token label, then libhsm cannot access the token. The Signer Engine will thus close down.
However, there are two log messages which indicates that the threads are not handled correctly. Maybe because the threads have not been started yet.
Jun 22 01:39:18 ubuntu ods-signerd: [hsm] hsm_get_slot_id(): could not find token with the name opendnssec
Jun 22 01:39:18 ubuntu ods-signerd: [engine] setup failed: HSM error
Jun 22 01:39:18 ubuntu ods-signerd: [engine] signer shutdown
Jun 22 01:39:18 ubuntu ods-signerd: ../../../signer/src/daemon/dnshandler.c at 231 could not pthread_kill(dnshandler->thread_id, 1): No such process
Jun 22 01:39:18 ubuntu ods-signerd: ../../../signer/src/daemon/engine.c at 246 could not pthread_join(engine->dnshandler->thread_id, NULL): Invalid argument