Could not enter critical section during program termination in AsyncCallback

[petervdonovan]

This describes an error that appeared in CI on macOS (here) in a PR that probably wouldn't introduce any concurrency bugs (the PR, lf-lang/reactor-c#342, is instead mostly just changing the build system and directory structure).

AsyncCallback just flaked on me with a failed termination sequence:

    DEBUG: Scheduler: Advancing tag.
    DEBUG: Scheduler: Reached stop tag.
    DEBUG: Worker 0: Stop requested. Exiting.
    LOG: ---- All worker threads exited successfully.
    LOG: ---- Terminating environment 0, normal termination: 1
    LOG: --------- Start time step at tag (2000000000, 0).
    ---- Elapsed logical time (in nsec): 2,000,000,000
    ---- Elapsed physical time (in nsec): 2,027,747,000
    DEBUG: Freeing token from _lf_token_recycling_bin: 0x600001dc8030
    DEBUG: Trace buffer 0 has 201 records.
    DEBUG: Trace buffer 1 has 0 records.
    FATAL ERROR: Could not enter critical section

This is probably an error thrown from C11 mtx_lock. I assume the problem is that it tried to acquire the same lock twice and the lock was not configured for recursive locking.

[erlingrj]

I will try to reproduce this locally. Note to self is that our LF_ASSERT macros should also print out the file and line number where the assertion fails.

[erlingrj]

With this PR: lf-lang/reactor-c#376 it will be easier to debug such failures since it will also print the exact place where the assertion failed.