fix for elusive crashing issue

[gisforgirard]

I kinda jumped the gun and posted a pull request the other day without testing it, but I've been running with this change all day and like 50,000 calls, everything appears to be functioning perfectly fine... I think it does seem to be ending calls sooner, but not like any earlier than they should be, just tends to seem like less of a silence gap between calls within a single recording.

Does anyone know why it would continue processing call audio if terminate_call is true? This seems to have solved the crashing issue I was having, and looking back at a previous issue that I thought was solved, I noticed in the debug info I posted back then, it was crashing while terminate_call was true, so it appears this could have been the actual issue at hand and I didn't notice because for whatever reason this crash happens randomly a bunch of times in a row and then goes away completely, but this time I actually had a chance to track it down and figure it out while it was happening, compile a bunch of versions with extra debug info, and narrow it down to pretty much this exact spot where it was having issues.

[tadscottsmith]

I believe the idea was to make sure all of the audio samples got put on to the output_queue before the termination was sent. There are 100s (thousands?) of instances running and I would think this would be much more prominent if it were an issue. What is pointing you in your examples to this? Have you tried building a debug version and getting the stack trace?

[gisforgirard]

yeah this didnt solve it cause it still crashed with terminate_call = false, thats why i made it a draft cause i figured it would probably end up not working lol

(gdb) bt full
#0  0x00007fde7c034235 in gr::op25_repeater::p25_frame_assembler_impl::general_work (this=0x55e4597ef370, noutput_items=15552, ninput_items=..., input_items=..., output_items=...) at /mnt/synology/some-raidz2/gisforgirard/delldesktop_rollingbackup/home/gisforgirard/dev/sdr-radio/trunk-recorder/lib/op25_repeater/lib/p25_frame_assembler_impl.cc:229
        i = 35279
        src_id = 34312
        grp_id = 4002
        out = 0x7fde665ddc62
        in = 0x7fde665fab3c "\002\002\002\003"
        terminate_call = false
        p2_ptt_src_id = -1
        p2_ptt_grp_id = -1
        amt_produce = 36000
#1  0x00007fde7a68e2ec in gr::block_executor::run_one_iteration (this=this@entry=0x7fda62ec46b0) at /home/gisforgirard/dev/sdr-radio/gnuradio/gnuradio-runtime/lib/block_executor.cc:500
        i = <optimized out>
        n = <optimized out>
        msg = <optimized out>
        try_again = <optimized out>
        noutput_items = 15552
        max_items_avail = <optimized out>
        max_noutput_items = <optimized out>
        new_alignment = 0
        alignment_state = <optimized out>
        m = 0x55e4597fdef8
        d = <optimized out>
#2  0x00007fde7a6e21c7 in gr::tpb_thread_body::tpb_thread_body (this=0x7fda62ec46b0, block=..., start_sync=..., max_noutput_items=<optimized out>) at /home/gisforgirard/dev/sdr-radio/gnuradio/gnuradio-runtime/lib/tpb_thread_body.cc:110
        s = <optimized out>
        msg = {<std::__shared_ptr<pmt::pmt_base, (__gnu_cxx::_Lock_policy)2>> = {<std::__shared_ptr_access<pmt::pmt_base, (__gnu_cxx::_Lock_policy)2, false, false>> = {<No data fields>}, _M_ptr = <optimized out>, _M_refcount = {_M_pi = 0x0}}, <No data fields>}
        max_nmsgs = 8192
        config_file = {static npos = 18446744073709551615, _M_dataplus = {<std::allocator<char>> = {<__gnu_cxx::new_allocator<char>> = {<No data fields>}, <No data fields>}, _M_p = 0x7fda62ec4460 ""}, _M_string_length = 0, {_M_local_buf = '\000' <repeats 15 times>, _M_allocated_capacity = 0}}
        log_level = {static npos = 18446744073709551615, _M_dataplus = {<std::allocator<char>> = {<__gnu_cxx::new_allocator<char>> = {<No data fields>}, <No data fields>}, _M_p = 0x7fda62ec4480 "debug"}, _M_string_length = 5, {_M_local_buf = "debug\000\000\000\000\000\000\000\000\000\000", _M_allocated_capacity = 444351014244}}
        d = <optimized out>
        p = <optimized out>
        log_file = {static npos = 18446744073709551615, _M_dataplus = {<std::allocator<char>> = {<__gnu_cxx::new_allocator<char>> = {<No data fields>}, <No data fields>}, _M_p = 0x7fda62ec44a0 "stdout"}, _M_string_length = 6, {_M_local_buf = "stdout\000\000\000\000\000\000\000\000\000", _M_allocated_capacity = 128047728850035}}
        LOG = 0x7fde5c001940
        d = <optimized out>
        s = <optimized out>
        msg = <optimized out>
        p = <optimized out>
        max_nmsgs = <optimized out>
        config_file = <optimized out>
        log_level = <optimized out>
        log_file = <optimized out>
        LOG = <optimized out>
        i = <optimized out>
        __for_range = <optimized out>
        __for_begin = <optimized out>
        __for_end = <optimized out>
        guard = <optimized out>
        timeout = <optimized out>
        guard = <optimized out>
#3  0x00007fde7a6d1114 in gr::tpb_container::operator() (this=0x55e45a2f8980) at /home/gisforgirard/dev/sdr-radio/gnuradio/gnuradio-runtime/lib/scheduler_tpb.cc:38
        body = {d_exec = {d_block = {<std::__shared_ptr<gr::block, (__gnu_cxx::_Lock_policy)2>> = {<std::__shared_ptr_access<gr::block, (__gnu_cxx::_Lock_policy)2, false, false>> = {<No data fields>}, _M_ptr = 0x55e4597fdef8, _M_refcount = {_M_pi = 0x55e4597e3d90}}, <No data fields>}, d_logger = 0x7fde5c000ed0, d_debug_logger = 0x7fde5c0012c0, d_ninput_items_required = {<std::_Vector_base<int, std::allocator<int> >> = {
                _M_impl = {<std::allocator<int>> = {<__gnu_cxx::new_allocator<int>> = {<No data fields>}, <No data fields>}, <std::_Vector_base<int, std::allocator<int> >::_Vector_impl_data> = {_M_start = 0x7fdd2c001c80, _M_finish = 0x7fdd2c001c84, _M_end_of_storage = 0x7fdd2c001c84}, <No data fields>}}, <No data fields>}, d_ninput_items = {<std::_Vector_base<int, std::allocator<int> >> = {
                _M_impl = {<std::allocator<int>> = {<__gnu_cxx::new_allocator<int>> = {<No data fields>}, <No data fields>}, <std::_Vector_base<int, std::allocator<int> >::_Vector_impl_data> = {_M_start = 0x7fdd2c001f30, _M_finish = 0x7fdd2c001f34, _M_end_of_storage = 0x7fdd2c001f34}, <No data fields>}}, <No data fields>}, d_input_items = {<std::_Vector_base<void const*, std::allocator<void const*> >> = {
                _M_impl = {<std::allocator<void const*>> = {<__gnu_cxx::new_allocator<void const*>> = {<No data fields>}, <No data fields>}, <std::_Vector_base<void const*, std::allocator<void const*> >::_Vector_impl_data> = {_M_start = 0x7fdd2c001f50, _M_finish = 0x7fdd2c001f58, _M_end_of_storage = 0x7fdd2c001f58}, <No data fields>}}, <No data fields>}, d_input_done = {<std::_Bvector_base<std::allocator<bool> >> = {
                _M_impl = {<std::allocator<unsigned long>> = {<__gnu_cxx::new_allocator<unsigned long>> = {<No data fields>}, <No data fields>}, <std::_Bvector_base<std::allocator<bool> >::_Bvector_impl_data> = {_M_start = {<std::_Bit_iterator_base> = {<std::iterator<std::random_access_iterator_tag, bool, long, bool*, bool&>> = {<No data fields>}, _M_p = 0x7fdd2c001f70, _M_offset = 0}, <No data fields>}, 
                    _M_finish = {<std::_Bit_iterator_base> = {<std::iterator<std::random_access_iterator_tag, bool, long, bool*, bool&>> = {<No data fields>}, _M_p = 0x7fdd2c001f70, _M_offset = 1}, <No data fields>}, _M_end_of_storage = 0x7fdd2c001f78}, <No data fields>}}, <No data fields>}, d_output_items = {<std::_Vector_base<void*, std::allocator<void*> >> = {
                _M_impl = {<std::allocator<void*>> = {<__gnu_cxx::new_allocator<void*>> = {<No data fields>}, <No data fields>}, <std::_Vector_base<void*, std::allocator<void*> >::_Vector_impl_data> = {_M_start = 0x7fdd2c001f90, _M_finish = 0x7fdd2c001f98, _M_end_of_storage = 0x7fdd2c001f98}, <No data fields>}}, <No data fields>}, d_start_nitems_read = {<std::_Vector_base<unsigned long, std::allocator<unsigned long> >> = {
                _M_impl = {<std::allocator<unsigned long>> = {<__gnu_cxx::new_allocator<unsigned long>> = {<No data fields>}, <No data fields>}, <std::_Vector_base<unsigned long, std::allocator<unsigned long> >::_Vector_impl_data> = {_M_start = 0x7fdd2c001fb0, _M_finish = 0x7fdd2c001fb8, _M_end_of_storage = 0x7fdd2c001fb8}, <No data fields>}}, <No data fields>}, d_returned_tags = {<std::_Vector_base<gr::tag_t, std::allocator<gr::tag_t> >> = {
                _M_impl = {<std::allocator<gr::tag_t>> = {<__gnu_cxx::new_allocator<gr::tag_t>> = {<No data fields>}, <No data fields>}, <std::_Vector_base<gr::tag_t, std::allocator<gr::tag_t> >::_Vector_impl_data> = {_M_start = 0x7fdd2c002650, _M_finish = 0x7fdd2c002650, _M_end_of_storage = 0x7fdd2c0028d0}, <No data fields>}}, <No data fields>}, d_max_noutput_items = 32768, d_use_pc = false}}
        body = <optimized out>
#4  gr::thread::thread_body_wrapper<gr::tpb_container>::operator() (this=0x55e45a2f8980) at /home/gisforgirard/dev/sdr-radio/gnuradio/gnuradio-runtime/lib/../include/gnuradio/thread/thread_body_wrapper.h:50
No locals.
#5  0x00007fde7a4530cb in ?? () from /lib/x86_64-linux-gnu/libboost_thread.so.1.74.0
No symbol table info available.
#6  0x00007fde79f33b43 in ?? () from /lib/x86_64-linux-gnu/libc.so.6
No symbol table info available.
#7  0x00007fde79fc5a00 in ?? () from /lib/x86_64-linux-gnu/libc.so.6
No symbol table info available.

[gisforgirard]

I'm pretty certain it has something to do with the fact that I'm using networked storage mounted as a local folder, which, whenever there is some latency with the connection, it basically locks up the entire computer waiting for it to respond, and so the buffer fills up, way more than it's supposed to, and once it starts responding again it tries to process too much at one time and crashes, it's driving me crazy and I'm too dumb to figure this out but I'm going to keep trying regardless...

[tadscottsmith]

Is your 'catpureDir' on the networked storage? Can you let trunk-recorder create the files locally and then have the uploadScript or a separate process move them to networked storage? You probably wouldn't need much local storage.

[gisforgirard]

this was stupid and didn't actually work

[gisforgirard]

Is your 'catpureDir' on the networked storage? Can you let trunk-recorder create the files locally and then have the uploadScript or a separate process move them to networked storage? You probably wouldn't need much local storage.

that's a good idea, but I actually think I finally tracked it down and figured it out, I'm just trying to decide if it's actually fixed before opening a third pull request for no reason lol