TiFlash crash in the failpoint test

[lilinghai]

Bug Report

Please answer these questions before submitting your issue. Thanks!

1. Minimal reproduce step (Required)

[2023/03/04 21:04:27.253 +08:00] [ERROR] [BaseDaemon.cpp:376] [########################################] [source=BaseDaemon] [thread_id=2117]
[2023/03/04 21:04:27.253 +08:00] [ERROR] [BaseDaemon.cpp:377] ["(from thread 828) Received signal Segmentation fault(11)."] [source=BaseDaemon] [thread_id=2117]
[2023/03/04 21:04:27.253 +08:00] [ERROR] [BaseDaemon.cpp:405] ["Address: NULL pointer."] [source=BaseDaemon] [thread_id=2117]
[2023/03/04 21:04:27.253 +08:00] [ERROR] [BaseDaemon.cpp:413] ["Access: read."] [source=BaseDaemon] [thread_id=2117]
[2023/03/04 21:04:27.253 +08:00] [ERROR] [BaseDaemon.cpp:425] ["Unknown si_code."] [source=BaseDaemon] [thread_id=2117]
[2023/03/04 21:04:27.280 +08:00] [WARN] [ExchangeReceiver.cpp:922] ["connection end. meet error: true, err msg: Exchange receiver meet error : push packets fail, current alive connections: 1"] [source="MPP<query:<query_ts:1677935053525118130, local_query_id:49059, server_id:1318176, start_ts:439860606649499649>,task_id:13> ExchangeReceiver_75 tunnel10+13"] [thread_id=1068]
[2023/03/04 21:04:27.286 +08:00] [WARN] [ExchangeReceiver.cpp:922] ["connection end. meet error: true, err msg: Exchange receiver meet error : push packets fail, current alive connections: 0"] [source="MPP<query:<query_ts:1677935053525118130, local_query_id:49059, server_id:1318176, start_ts:439860606649499649>,task_id:13> ExchangeReceiver_75 tunnel11+13"] [thread_id=1068]
[2023/03/04 21:04:27.337 +08:00] [ERROR] [BaseDaemon.cpp:569] ["\n       0xa91b6ed\tfaultSignalHandler(int, siginfo_t*, void*) [tiflash+177321709]\n                \tlibs/libdaemon/src/BaseDaemon.cpp:220\n  0x7f03b298bd90\t<unknown symbol> [libc.so.6+347536]\n       0x3f01f79\tlong std::__1::__cxx_atomic_fetch_sub<long>(std::__1::__cxx_atomic_base_impl<long>*, long, std::__1::memory_order) [tiflash+66068345]\n                \t/usr/local/bin/../include/c++/v1/atomic:1082\n       0x3f01ce3\tstd::__1::__atomic_base<long, true>::fetch_sub(long, std::__1::memory_order) [tiflash+66067683]\n                \t/usr/local/bin/../include/c++/v1/atomic:1736\n       0x3f0e687\tMemoryTracker::free(long) [tiflash+66119303]\n                \tdbms/src/Common/MemoryTracker.cpp:176\n       0x3f0e6f1\tMemoryTracker::free(long) [tiflash+66119409]\n                \tdbms/src/Common/MemoryTracker.cpp:191\n       0xb667089\tDB::MemTrackerWrapper::free(unsigned long) [tiflash+191262857]\n                \tdbms/src/Flash/Mpp/TrackedMppDataPacket.h:81\n       0xb666fe9\tDB::MemTrackerWrapper::freeAll() [tiflash+191262697]\n                \tdbms/src/Flash/Mpp/TrackedMppDataPacket.h:104\n       0x3e2ebf5\tDB::MemTrackerWrapper::~MemTrackerWrapper() [tiflash+65203189]\n                \tdbms/src/Flash/Mpp/TrackedMppDataPacket.h:99\n       0x3e37cde\tDB::TrackedMppDataPacket::~TrackedMppDataPacket() [tiflash+65240286]\n                \tdbms/src/Flash/Mpp/TrackedMppDataPacket.h:111\n       0x3e37c9b\tvoid std::__1::destroy_at<DB::TrackedMppDataPacket>(DB::TrackedMppDataPacket*) [tiflash+65240219]\n                \t/usr/local/bin/../include/c++/v1/__memory/construct_at.h:50\n       0x3e37c39\tvoid std::__1::allocator_traits<std::__1::allocator<DB::TrackedMppDataPacket> >::destroy<DB::TrackedMppDataPacket, void, void>(std::__1::allocator<DB::TrackedMppDataPacket>&, DB::TrackedMppDataPacket*) [tiflash+65240121]\n                \t/usr/local/bin/../include/c++/v1/__memory/allocator_traits.h:317\n       0x3e37b5d\tstd::__1::__shared_ptr_emplace<DB::TrackedMppDataPacket, std::__1::allocator<DB::TrackedMppDataPacket> >::__on_zero_shared() [tiflash+65239901]\n                \t/usr/local/bin/../include/c++/v1/__memory/shared_ptr.h:313\n       0x3eff58c\tstd::__1::__shared_count::__release_shared() [tiflash+66057612]\n                \t/usr/local/bin/../include/c++/v1/__memory/shared_ptr.h:177\n       0x3eff546\tstd::__1::__shared_weak_count::__release_shared() [tiflash+66057542]\n                \t/usr/local/bin/../include/c++/v1/__memory/shared_ptr.h:219\n       0xb6632b0\tstd::__1::shared_ptr<DB::TrackedMppDataPacket>::~shared_ptr() [tiflash+191247024]\n                \t/usr/local/bin/../include/c++/v1/__memory/shared_ptr.h:959\n       0xb8399c1\tvoid DB::broadcastOrPassThroughWrite<true, DB::MPPTunnelSetWriterBase::broadcastWrite(std::__1::vector<DB::Block, std::__1::allocator<DB::Block> >&, DB::MPPDataPacketVersion, DB::CompressionMethod)::$_4, DB::MPPTunnelSetWriterBase::broadcastWrite(std::__1::vector<DB::Block, std::__1::allocator<DB::Block> >&, DB::MPPDataPacketVersion, DB::CompressionMethod)::$_5>(unsigned long, unsigned long, std::__1::vector<DB::Block, std::__1::allocator<DB::Block> >&, DB::MPPDataPacketVersion, DB::CompressionMethod, DB::MPPTunnelSetWriterBase::broadcastWrite(std::__1::vector<DB::Block, std::__1::allocator<DB::Block> >&, DB::MPPDataPacketVersion, DB::CompressionMethod)::$_4&&, DB::MPPTunnelSetWriterBase::broadcastWrite(std::__1::vector<DB::Block, std::__1::allocator<DB::Block> >&, DB::MPPDataPacketVersion, DB::CompressionMethod)::$_5&&) [tiflash+193173953]\n                \tdbms/src/Flash/Mpp/MPPTunnelSetWriter.cpp:288\n       0xb839862\tDB::MPPTunnelSetWriterBase::broadcastWrite(std::__1::vector<DB::Block, std::__1::allocator<DB::Block> >&, DB::MPPDataPacketVersion, DB::CompressionMethod) [tiflash+193173602]\n                \tdbms/src/Flash/Mpp/MPPTunnelSetWriter.cpp:294\n       0x3e5c377\tDB::BroadcastOrPassThroughWriter<std::__1::shared_ptr<DB::SyncMPPTunnelSetWriter> >::writeBlocks() [tiflash+65389431]\n                \tdbms/src/Flash/Mpp/BroadcastOrPassThroughWriter.cpp:109\n       0x3e5c168\tDB::BroadcastOrPassThroughWriter<std::__1::shared_ptr<DB::SyncMPPTunnelSetWriter> >::write(DB::Block const&) [tiflash+65388904]\n                \tdbms/src/Flash/Mpp/BroadcastOrPassThroughWriter.cpp:92\n       0xb781068\tDB::ExchangeSenderBlockInputStream::readImpl() [tiflash+192417896]\n                \tdbms/src/DataStreams/ExchangeSenderBlockInputStream.cpp:44\n       0xaa23650\tDB::IProfilingBlockInputStream::read(DB::PODArray<unsigned char, 4096ul, Allocator<false>, 15ul, 16ul>*&, bool) [tiflash+178402896]\n                \tdbms/src/DataStreams/IProfilingBlockInputStream.cpp:75\n       0xaa234e7\tDB::IProfilingBlockInputStream::read() [tiflash+178402535]\n                \tdbms/src/DataStreams/IProfilingBlockInputStream.cpp:43\n       0xb8660ff\tDB::DataStreamExecutor::execute(DB::ResultHandler&&) [tiflash+193356031]\n                \tdbms/src/Flash/Executor/DataStreamExecutor.cpp:44\n       0xb86c37a\tDB::QueryExecutor::execute() [tiflash+193381242]\n                \tdbms/src/Flash/Executor/QueryExecutor.cpp:22\n       0xb7e3fc3\tDB::MPPTask::runImpl() [tiflash+192823235]\n                \tdbms/src/Flash/Mpp/MPPTask.cpp:403\n       0xb7e8c0d\tDB::MPPTask::run()::$_13::operator()() const [tiflash+192842765]\n                \tdbms/src/Flash/Mpp/MPPTask.cpp:152\n       0xb7e8bd5\tdecltype(static_cast<DB::MPPTask::run()::$_13&>(fp)()) std::__1::__invoke<DB::MPPTask::run()::$_13&>(DB::MPPTask::run()::$_13&) [tiflash+192842709]\n                \t/usr/local/bin/../include/c++/v1/type_traits:3918\n       0xb7e8b8d\tvoid std::__1::__invoke_void_return_wrapper<void, true>::__call<DB::MPPTask::run()::$_13&>(DB::MPPTask::run()::$_13&) [tiflash+192842637]\n                \t/usr/local/bin/../include/c++/v1/__functional/invoke.h:61\n       0xb7e8b5d\tstd::__1::__function::__alloc_func<DB::MPPTask::run()::$_13, std::__1::allocator<DB::MPPTask::run()::$_13>, void ()>::operator()() [tiflash+192842589]\n                \t/usr/local/bin/../include/c++/v1/__functional/function.h:171\n       0xb7e7c99\tstd::__1::__function::__func<DB::MPPTask::run()::$_13, std::__1::allocator<DB::MPPTask::run()::$_13>, void ()>::operator()() [tiflash+192838809]\n                \t/usr/local/bin/../include/c++/v1/__functional/function.h:345\n       0x3fe2766\tstd::__1::__function::__value_func<void ()>::operator()() const [tiflash+66987878]\n                \t/usr/local/bin/../include/c++/v1/__functional/function.h:498\n       0x3fdfab5\tstd::__1::function<void ()>::operator()() const [tiflash+66976437]\n                \t/usr/local/bin/../include/c++/v1/__functional/function.h:1175\n       0x3fe5d45\tdecltype(static_cast<std::__1::function<void ()>>(fp)()) std::__1::__invoke_constexpr<std::__1::function<void ()> >(std::__1::function<void ()>&&) [tiflash+67001669]\n                \t/usr/local/bin/../include/c++/v1/type_traits:3924\n       0x3fe5d21\tdecltype(auto) std::__1::__apply_tuple_impl<std::__1::function<void ()>, std::__1::tuple<> >(std::__1::function<void ()>&&, std::__1::tuple<>&&, std::__1::__tuple_indices<>) [tiflash+67001633]\n                \t/usr/local/bin/../include/c++/v1/tuple:1536\n       0x3fe5cf1\tdecltype(auto) std::__1::apply<std::__1::function<void ()>, std::__1::tuple<> >(std::__1::function<void ()>&&, std::__1::tuple<>&&) [tiflash+67001585]\n                \t/usr/local/bin/../include/c++/v1/tuple:1545\n       0x3fe5c8c\tauto DB::wrapInvocable<std::__1::function<void ()> >(bool, std::__1::function<void ()>&&)::'lambda'()::operator()() [tiflash+67001484]\n                \tdbms/src/Common/wrapInvocable.h:36\n       0x3fe6959\tDB::ExecutableTask<auto DB::wrapInvocable<std::__1::function<void ()> >(bool, std::__1::function<void ()>&&)::'lambda'()>::execute() [tiflash+67004761]\n                \tdbms/src/Common/ExecutableTask.h:52\n       0x3f016dd\tDB::DynamicThreadPool::executeTask(std::__1::unique_ptr<DB::IExecutableTask, std::__1::default_delete<DB::IExecutableTask> >&) [tiflash+66066141]\n                \tdbms/src/Common/DynamicThreadPool.cpp:101\n       0x3f00eff\tDB::DynamicThreadPool::fixedWork(unsigned long) [tiflash+66064127]\n                \tdbms/src/Common/DynamicThreadPool.cpp:115\n       0x3f09a1c\tdecltype(*(static_cast<DB::DynamicThreadPool*>(fp0)).*fp(static_cast<unsigned long&>(fp1))) std::__1::__invoke<void (DB::DynamicThreadPool::* const&)(unsigned long), DB::DynamicThreadPool*, unsigned long&, void>(void (DB::DynamicThreadPool::* const&)(unsigned long), DB::DynamicThreadPool*&&, unsigned long&) [tiflash+66099740]\n                \t/usr/local/bin/../include/c++/v1/type_traits:3859\n       0x3f09926\tstd::__1::invoke_result<void (DB::DynamicThreadPool::* const&)(unsigned long), DB::DynamicThreadPool*, unsigned long&>::type std::__1::invoke<void (DB::DynamicThreadPool::* const&)(unsigned long), DB::DynamicThreadPool*, unsigned long&>(void (DB::DynamicThreadPool::* const&)(unsigned long), DB::DynamicThreadPool*&&, unsigned long&) [tiflash+66099494]\n                \t/usr/local/bin/../include/c++/v1/__functional/invoke.h:93\n       0x3f09815\tauto std::__1::thread DB::ThreadFactory::newThread<void (DB::DynamicThreadPool::*)(unsigned long), DB::DynamicThreadPool*, unsigned long&>(bool, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, void (DB::DynamicThreadPool::*&&)(unsigned long), DB::DynamicThreadPool*&&, unsigned long&)::'lambda'(auto&&...)::operator()<DB::DynamicThreadPool*, unsigned long>(auto&&...) const [tiflash+66099221]\n                \tdbms/src/Common/ThreadFactory.h:47\n       0x3f09645\tdecltype(static_cast<void (DB::DynamicThreadPool::*>(fp)(static_cast<DB::DynamicThreadPool*>(fp0), static_cast<unsigned long&>(fp0))) std::__1::__invoke<std::__1::thread DB::ThreadFactory::newThread<void (DB::DynamicThreadPool::*)(unsigned long), DB::DynamicThreadPool*, unsigned long&>(bool, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, void (DB::DynamicThreadPool::*&&)(unsigned long), DB::DynamicThreadPool*&&, unsigned long&)::'lambda'(auto&&...), DB::DynamicThreadPool*, unsigned long>(void (DB::DynamicThreadPool::*&&)(unsigned long), DB::DynamicThreadPool*&&, unsigned long&) [tiflash+66098757]\n                \t/usr/local/bin/../include/c++/v1/type_traits:3918\n       0x3f09596\tvoid std::__1::__thread_execute<std::__1::unique_ptr<std::__1::__thread_struct, std::__1::default_delete<std::__1::__thread_struct> >, std::__1::thread DB::ThreadFactory::newThread<void (DB::DynamicThreadPool::*)(unsigned long), DB::DynamicThreadPool*, unsigned long&>(bool, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, void (DB::DynamicThreadPool::*&&)(unsigned long), DB::DynamicThreadPool*&&, unsigned long&)::'lambda'(auto&&...), DB::DynamicThreadPool*, unsigned long, 2ul, 3ul>(std::__1::tuple<void (DB::DynamicThreadPool::*)(unsigned long), std::__1::thread DB::ThreadFactory::newThread<void (DB::DynamicThreadPool::*)(unsigned long), DB::DynamicThreadPool*, unsigned long&>(bool, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, void (DB::DynamicThreadPool::*&&)(unsigned long), DB::DynamicThreadPool*&&, unsigned long&)::'lambda'(auto&&...), DB::DynamicThreadPool*, unsigned long>&, std::__1::__tuple_indices<2ul, 3ul>) [tiflash+66098582]\n                \t/usr/local/bin/../include/c++/v1/thread:280\n       0x3f08e9b\tvoid* std::__1::__thread_proxy<std::__1::tuple<std::__1::unique_ptr<std::__1::__thread_struct, std::__1::default_delete<std::__1::__thread_struct> >, std::__1::thread DB::ThreadFactory::newThread<void (DB::DynamicThreadPool::*)(unsigned long), DB::DynamicThreadPool*, unsigned long&>(bool, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, void (DB::DynamicThreadPool::*&&)(unsigned long), DB::DynamicThreadPool*&&, unsigned long&)::'lambda'(auto&&...), DB::DynamicThreadPool*, unsigned long> >(void*) [tiflash+66096795]\n                \t/usr/local/bin/../include/c++/v1/thread:291\n  0x7f03b29d6802\tstart_thread [libc.so.6+653314]"] [source=BaseDaemon] [thread_id=2117]

2. What did you expect to see? (Required)

3. What did you see instead (Required)

4. What is your TiFlash version? (Required)

[2023/03/04 12:29:48.350 +08:00] [INFO] [client.go:508] ["Cluster version information"] [type=tikv] [version=6.7.0-alpha] [git_hash=d74fd1325280999b367424ee332c49e11bbf80b0]
[2023/03/04 12:29:48.350 +08:00] [INFO] [client.go:508] ["Cluster version information"] [type=tidb] [version=6.7.0-alpha] [git_hash=7a8cace3bf353e14978961fa2798e0175334b718]
[2023/03/04 12:29:48.350 +08:00] [INFO] [client.go:508] ["Cluster version information"] [type=pd] [version=6.7.0-alpha] [git_hash=c40e319f50822678cda71ae62ee2fd70a9cac010]
[2023/03/04 12:29:48.350 +08:00] [INFO] [client.go:508] ["Cluster version information"] [type=tiflash] [version=6.7.0-alpha] [git_hash=f6a59f6680be1c3d9538d2c01d3fc6e1d34dec19]