Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

PageFile::Writer::closeFd raise Bad file descriptor error #5687

Closed
lilinghai opened this issue Aug 24, 2022 · 2 comments
Closed

PageFile::Writer::closeFd raise Bad file descriptor error #5687

lilinghai opened this issue Aug 24, 2022 · 2 comments
Labels
affects-6.1 This bug affects the 6.1.x(LTS) versions. component/storage severity/moderate type/bug The issue is confirmed as a bug.

Comments

@lilinghai
Copy link

lilinghai commented Aug 24, 2022 •
edited by JaySon-Huang
Loading

Bug Report

Please answer these questions before submitting your issue. Thanks!

1. Minimal reproduce step (Required)

[2022/08/24 08:13:36.428 +08:00] [ERROR] [MPPTask.cpp:324] ["MPPTask:MPP<query:435499829991833636,task:9> task running meets error: DB::Exception: Query was cancelled Stack Trace : 
       0x1d28013    StackTrace::StackTrace() [tiflash+30572563]
                    dbms/src/Common/StackTrace.cpp:23
       0x1d25616    DB::Exception::Exception(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, int) [tiflash+30561814]
                    dbms/src/Common/Exception.h:41
       0x1df061e    DB::SharedQueryBlockInputStream::waitThread() [tiflash+31393310]
                    dbms/src/DataStreams/SharedQueryBlockInputStream.h:169
       0x1def484    DB::SharedQueryBlockInputStream::readSuffix() [tiflash+31388804]
                    dbms/src/DataStreams/SharedQueryBlockInputStream.h:80
       0x668b0b9    DB::IProfilingBlockInputStream::readSuffix() [tiflash+107524281]
                    dbms/src/DataStreams/IProfilingBlockInputStream.cpp:135
       0x668b0b9    DB::IProfilingBlockInputStream::readSuffix() [tiflash+107524281]
                    dbms/src/DataStreams/IProfilingBlockInputStream.cpp:135
       0x7b8c8ba    DB::UnionBlockInputStream<(DB::StreamUnionMode)0, true>::readSuffix() [tiflash+129550522]
                    dbms/src/DataStreams/UnionBlockInputStream.h:256
       0x7bd05f1    DB::MPPTask::runImpl() [tiflash+129828337]
                    dbms/src/Flash/Mpp/MPPTask.cpp:310
       0x1db233c    auto DB::wrapInvocable<std::__1::function<void ()> >(bool, std::__1::function<void ()>&&)::'lambda'()::operator()() [tiflash+31138620]
                    dbms/src/Common/wrapInvocable.h:36
       0x1da1a78    DB::DynamicThreadPool::executeTask(std::__1::unique_ptr<DB::IExecutableTask, std::__1::default_delete<DB::IExecutableTask> >&) [tiflash+31070840]
                    dbms/src/Common/DynamicThreadPool.cpp:101
       0x1da1956    DB::DynamicThreadPool::dynamicWork(std::__1::unique_ptr<DB::IExecutableTask, std::__1::default_delete<DB::IExecutableTask> >) [tiflash+31070550]
                    dbms/src/Common/DynamicThreadPool.cpp:142
       0x1da35cd    auto std::__1::thread DB::ThreadFactory::newThread<void (DB::DynamicThreadPool::*)(std::__1::unique_ptr<DB::IExecutableTask, std::__1::default_delete<DB::IExecutableTask> >), DB::DynamicThreadPool*, std::__1::unique_ptr<DB::IExecutableTask, std::__1::default_delete<DB::IExecutableTask> > >(bool, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, void (DB::DynamicThreadPool::*&&)(std::__1::unique_ptr<DB::IExecutableTask, std::__1::default_delete<DB::IExecutableTask> >), DB::DynamicThreadPool*&&, std::__1::unique_ptr<DB::IExecutableTask, std::__1::default_delete<DB::IExecutableTask> >&&)::'lambda'(auto&&...)::operator()<DB::DynamicThreadPool*, std::__1::unique_ptr<DB::IExecutableTask, std::__1::default_delete<DB::IExecutableTask> > >(auto&&...) const [tiflash+31077837]
                    dbms/src/Common/ThreadFactory.h:47
       0x1da33a9    void* std::__1::__thread_proxy<std::__1::tuple<std::__1::unique_ptr<std::__1::__thread_struct, std::__1::default_delete<std::__1::__thread_struct> >, std::__1::thread DB::ThreadFactory::newThread<void (DB::DynamicThreadPool::*)(std::__1::unique_ptr<DB::IExecutableTask, std::__1::default_delete<DB::IExecutableTask> >), DB::DynamicThreadPool*, std::__1::unique_ptr<DB::IExecutableTask, std::__1::default_delete<DB::IExecutableTask> > >(bool, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, void (DB::DynamicThreadPool::*&&)(std::__1::unique_ptr<DB::IExecutableTask, std::__1::default_delete<DB::IExecutableTask> >), DB::DynamicThreadPool*&&, std::__1::unique_ptr<DB::IExecutableTask, std::__1::default_delete<DB::IExecutableTask> >&&)::'lambda'(auto&&...), DB::DynamicThreadPool*, std::__1::unique_ptr<DB::IExecutableTask, std::__1::default_delete<DB::IExecutableTask> > > >(void*) [tiflash+31077289]
                    /usr/local/bin/../include/c++/v1/thread:291
  0x7f55f3b03ea5    start_thread [libpthread.so.0+32421]
  0x7f55f361696d    __clone [libc.so.6+1042797]"] [thread_id=236]
[2022/08/24 08:13:36.428 +08:00] [ERROR] [Exception.cpp:80] ["MPPTask:MPP<query:435499829991833636,task:9> Failed to write error DB::Exception: Query was cancelled to tunnel: tunnel9+-1: Code: 0, e.displayText() = DB::Exception: write to tunnel which is already closed,grpc writes failed., e.what() = DB::Exception, Stack trace:


       0x1d28013    StackTrace::StackTrace() [tiflash+30572563]
                    dbms/src/Common/StackTrace.cpp:23
       0x1d25616    DB::Exception::Exception(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, int) [tiflash+30561814]
                    dbms/src/Common/Exception.h:41
       0x7be0f96    DB::MPPTunnelBase<DB::PacketWriter>::write(mpp::MPPDataPacket const&, bool) [tiflash+129896342]
                    dbms/src/Flash/Mpp/MPPTunnel.cpp:161
       0x7bd2106    DB::MPPTask::writeErrToAllTunnels(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&) [tiflash+129835270]
                    dbms/src/Flash/Mpp/MPPTask.cpp:378
       0x7bd0808    DB::MPPTask::runImpl() [tiflash+129828872]
                    dbms/src/Flash/Mpp/MPPTask.cpp:357
       0x1db233c    auto DB::wrapInvocable<std::__1::function<void ()> >(bool, std::__1::function<void ()>&&)::'lambda'()::operator()() [tiflash+31138620]
                    dbms/src/Common/wrapInvocable.h:36
       0x1da1a78    DB::DynamicThreadPool::executeTask(std::__1::unique_ptr<DB::IExecutableTask, std::__1::default_delete<DB::IExecutableTask> >&) [tiflash+31070840]
                    dbms/src/Common/DynamicThreadPool.cpp:101
       0x1da1956    DB::DynamicThreadPool::dynamicWork(std::__1::unique_ptr<DB::IExecutableTask, std::__1::default_delete<DB::IExecutableTask> >) [tiflash+31070550]
                    dbms/src/Common/DynamicThreadPool.cpp:142
       0x1da35cd    auto std::__1::thread DB::ThreadFactory::newThread<void (DB::DynamicThreadPool::*)(std::__1::unique_ptr<DB::IExecutableTask, std::__1::default_delete<DB::IExecutableTask> >), DB::DynamicThreadPool*, std::__1::unique_ptr<DB::IExecutableTask, std::__1::default_delete<DB::IExecutableTask> > >(bool, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, void (DB::DynamicThreadPool::*&&)(std::__1::unique_ptr<DB::IExecutableTask, std::__1::default_delete<DB::IExecutableTask> >), DB::DynamicThreadPool*&&, std::__1::unique_ptr<DB::IExecutableTask, std::__1::default_delete<DB::IExecutableTask> >&&)::'lambda'(auto&&...)::operator()<DB::DynamicThreadPool*, std::__1::unique_ptr<DB::IExecutableTask, std::__1::default_delete<DB::IExecutableTask> > >(auto&&...) const [tiflash+31077837]
                    dbms/src/Common/ThreadFactory.h:47
       0x1da33a9    void* std::__1::__thread_proxy<std::__1::tuple<std::__1::unique_ptr<std::__1::__thread_struct, std::__1::default_delete<std::__1::__thread_struct> >, std::__1::thread DB::ThreadFactory::newThread<void (DB::DynamicThreadPool::*)(std::__1::unique_ptr<DB::IExecutableTask, std::__1::default_delete<DB::IExecutableTask> >), DB::DynamicThreadPool*, std::__1::unique_ptr<DB::IExecutableTask, std::__1::default_delete<DB::IExecutableTask> > >(bool, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, void (DB::DynamicThreadPool::*&&)(std::__1::unique_ptr<DB::IExecutableTask, std::__1::default_delete<DB::IExecutableTask> >), DB::DynamicThreadPool*&&, std::__1::unique_ptr<DB::IExecutableTask, std::__1::default_delete<DB::IExecutableTask> >&&)::'lambda'(auto&&...), DB::DynamicThreadPool*, std::__1::unique_ptr<DB::IExecutableTask, std::__1::default_delete<DB::IExecutableTask> > > >(void*) [tiflash+31077289]
                    /usr/local/bin/../include/c++/v1/thread:291
  0x7f55f3b03ea5    start_thread [libpthread.so.0+32421]
  0x7f55f361696d    __clone [libc.so.6+1042797]"] [thread_id=236]
[2022/08/24 08:13:36.428 +08:00] [ERROR] [MPPTaskManager.cpp:164] ["TaskManager:The task MPP<query:435499829991833636,task:9> cannot be found and fail to unregister"] [thread_id=236]
[2022/08/24 08:13:36.428 +08:00] [WARN] [MPPTask.cpp:365] ["MPPTask:MPP<query:435499829991833636,task:9> finish task which was cancelled before"] [thread_id=236]
[2022/08/24 08:13:36.442 +08:00] [ERROR] [Exception.cpp:80] ["SharedQuery:MPP<query:435499829991833636,task:9> virtual DB::SharedQueryBlockInputStream::~SharedQueryBlockInputStream(): Code: 0, e.displayText() = DB::Exception: Query was cancelled, e.what() = DB::Exception, Stack trace:


       0x1d28013    StackTrace::StackTrace() [tiflash+30572563]
                    dbms/src/Common/StackTrace.cpp:23
       0x1d25616    DB::Exception::Exception(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, int) [tiflash+30561814]
                    dbms/src/Common/Exception.h:41
       0x1df061e    DB::SharedQueryBlockInputStream::waitThread() [tiflash+31393310]
                    dbms/src/DataStreams/SharedQueryBlockInputStream.h:169
       0x1def484    DB::SharedQueryBlockInputStream::readSuffix() [tiflash+31388804]
                    dbms/src/DataStreams/SharedQueryBlockInputStream.h:80
       0x1def5b1    DB::SharedQueryBlockInputStream::~SharedQueryBlockInputStream() [tiflash+31389105]
                    dbms/src/DataStreams/SharedQueryBlockInputStream.h:48
       0x1dc9b2e    DB::IBlockInputStream::~IBlockInputStream() [tiflash+31234862]
                    dbms/src/DataStreams/IBlockInputStream.h:111
       0x1dc9b2e    DB::IBlockInputStream::~IBlockInputStream() [tiflash+31234862]
                    dbms/src/DataStreams/IBlockInputStream.h:111
       0x1dc9b2e    DB::IBlockInputStream::~IBlockInputStream() [tiflash+31234862]
                    dbms/src/DataStreams/IBlockInputStream.h:111
       0x738d46c    DB::UnionBlockInputStream<(DB::StreamUnionMode)0, false>::~UnionBlockInputStream() [tiflash+121164908]
                    dbms/src/DataStreams/UnionBlockInputStream.h:136
       0x1dc9b2e    DB::IBlockInputStream::~IBlockInputStream() [tiflash+31234862]
                    dbms/src/DataStreams/IBlockInputStream.h:111
       0x1dc9b2e    DB::IBlockInputStream::~IBlockInputStream() [tiflash+31234862]
                    dbms/src/DataStreams/IBlockInputStream.h:111
       0x1dc9b2e    DB::IBlockInputStream::~IBlockInputStream() [tiflash+31234862]
                    dbms/src/DataStreams/IBlockInputStream.h:111
       0x1dc9b2e    DB::IBlockInputStream::~IBlockInputStream() [tiflash+31234862]
                    dbms/src/DataStreams/IBlockInputStream.h:111
       0x1dc9b2e    DB::IBlockInputStream::~IBlockInputStream() [tiflash+31234862]
                    dbms/src/DataStreams/IBlockInputStream.h:111
       0x1def65d    DB::SharedQueryBlockInputStream::~SharedQueryBlockInputStream() [tiflash+31389277]
                    dbms/src/DataStreams/SharedQueryBlockInputStream.h:54
       0x1dc9b2e    DB::IBlockInputStream::~IBlockInputStream() [tiflash+31234862]
                    dbms/src/DataStreams/IBlockInputStream.h:111
       0x1dc9b2e    DB::IBlockInputStream::~IBlockInputStream() [tiflash+31234862]
                    dbms/src/DataStreams/IBlockInputStream.h:111
       0x1dc9b2e    DB::IBlockInputStream::~IBlockInputStream() [tiflash+31234862]
                    dbms/src/DataStreams/IBlockInputStream.h:111
       0x7b8caac    DB::UnionBlockInputStream<(DB::StreamUnionMode)0, true>::~UnionBlockInputStream() [tiflash+129551020]
                    dbms/src/DataStreams/UnionBlockInputStream.h:136
       0x7615050    DB::ProcessListElement::releaseQueryStreams() [tiflash+123818064]
                    dbms/src/Interpreters/ProcessList.cpp:276
       0x7614934    DB::ProcessListEntry::~ProcessListEntry() [tiflash+123816244]
                    dbms/src/Interpreters/ProcessList.cpp:192
       0x1d70a05    DB::BlockIO::~BlockIO() [tiflash+30870021]
                    dbms/src/DataStreams/BlockIO.h:78
       0x727486f    DB::DAGContext::~DAGContext() [tiflash+120014959]
                    dbms/src/Flash/Coprocessor/DAGContext.h:113
       0x7bcdd68    DB::MPPTask::~MPPTask() [tiflash+129817960]
                    dbms/src/Flash/Mpp/MPPTask.cpp:78
       0x7bcd597    std::__1::__shared_ptr_pointer<DB::MPPTask*, std::__1::shared_ptr<DB::MPPTask>::__shared_ptr_default_delete<DB::MPPTask, DB::MPPTask>, std::__1::allocator<DB::MPPTask> >::__on_zero_shared() [tiflash+129815959]
                    /usr/local/bin/../include/c++/v1/__memory/shared_ptr.h:267
       0x7bd2ff1    std::__1::__function::__func<DB::MPPTask::run()::$_18, std::__1::allocator<DB::MPPTask::run()::$_18>, void ()>::destroy() [tiflash+129839089]
                    /usr/local/bin/../include/c++/v1/__functional/function.h:327
       0x1db282b    DB::ExecutableTask<auto DB::wrapInvocable<std::__1::function<void ()> >(bool, std::__1::function<void ()>&&)::'lambda'()>::~ExecutableTask() [tiflash+31139883]
                    dbms/src/Common/ExecutableTask.h:46
       0x1da1a8d    DB::DynamicThreadPool::executeTask(std::__1::unique_ptr<DB::IExecutableTask, std::__1::default_delete<DB::IExecutableTask> >&) [tiflash+31070861]
                    dbms/src/Common/DynamicThreadPool.cpp:102
       0x1da1956    DB::DynamicThreadPool::dynamicWork(std::__1::unique_ptr<DB::IExecutableTask, std::__1::default_delete<DB::IExecutableTask> >) [tiflash+31070550]
                    dbms/src/Common/DynamicThreadPool.cpp:142
       0x1da35cd    auto std::__1::thread DB::ThreadFactory::newThread<void (DB::DynamicThreadPool::*)(std::__1::unique_ptr<DB::IExecutableTask, std::__1::default_delete<DB::IExecutableTask> >), DB::DynamicThreadPool*, std::__1::unique_ptr<DB::IExecutableTask, std::__1::default_delete<DB::IExecutableTask> > >(bool, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, void (DB::DynamicThreadPool::*&&)(std::__1::unique_ptr<DB::IExecutableTask, std::__1::default_delete<DB::IExecutableTask> >), DB::DynamicThreadPool*&&, std::__1::unique_ptr<DB::IExecutableTask, std::__1::default_delete<DB::IExecutableTask> >&&)::'lambda'(auto&&...)::operator()<DB::DynamicThreadPool*, std::__1::unique_ptr<DB::IExecutableTask, std::__1::default_delete<DB::IExecutableTask> > >(auto&&...) const [tiflash+31077837]
                    dbms/src/Common/ThreadFactory.h:47
       0x1da33a9    void* std::__1::__thread_proxy<std::__1::tuple<std::__1::unique_ptr<std::__1::__thread_struct, std::__1::default_delete<std::__1::__thread_struct> >, std::__1::thread DB::ThreadFactory::newThread<void (DB::DynamicThreadPool::*)(std::__1::unique_ptr<DB::IExecutableTask, std::__1::default_delete<DB::IExecutableTask> >), DB::DynamicThreadPool*, std::__1::unique_ptr<DB::IExecutableTask, std::__1::default_delete<DB::IExecutableTask> > >(bool, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, void (DB::DynamicThreadPool::*&&)(std::__1::unique_ptr<DB::IExecutableTask, std::__1::default_delete<DB::IExecutableTask> >), DB::DynamicThreadPool*&&, std::__1::unique_ptr<DB::IExecutableTask, std::__1::default_delete<DB::IExecutableTask> >&&)::'lambda'(auto&&...), DB::DynamicThreadPool*, std::__1::unique_ptr<DB::IExecutableTask, std::__1::default_delete<DB::IExecutableTask> > > >(void*) [tiflash+31077289]
                    /usr/local/bin/../include/c++/v1/thread:291
  0x7f55f3b03ea5    start_thread [libpthread.so.0+32421]"] [thread_id=236]
[2022/08/24 08:15:25.964 +08:00] [WARN] [MPPTaskManager.cpp:82] ["TaskManager:Begin cancel query: 435499858696601684"] [thread_id=917]
[2022/08/24 08:15:25.964 +08:00] [WARN] [MPPTaskManager.cpp:93] ["TaskManager:Remaining task in query 435499858696601684 are: MPP<query:435499858696601684,task:9> "] [thread_id=917]
[2022/08/24 08:15:25.964 +08:00] [WARN] [MPPTask.cpp:391] ["MPPTask:MPP<query:435499858696601684,task:9> Begin cancel task: MPP<query:435499858696601684,task:9>"] [thread_id=1357]
[2022/08/24 08:15:25.965 +08:00] [WARN] [MPPTask.cpp:413] ["MPPTask:MPP<query:435499858696601684,task:9> Finish cancel task from running"] [thread_id=1357]
[2022/08/24 08:15:25.965 +08:00] [WARN] [MPPTaskManager.cpp:111] ["TaskManager:Finish cancel query: 435499858696601684"] [thread_id=917]
[2022/08/24 08:15:25.965 +08:00] [ERROR] [MPPTask.cpp:324] ["MPPTask:MPP<query:435499858696601684,task:9> task running meets error: DB::Exception: Query was cancelled Stack Trace : 
       0x1d28013    StackTrace::StackTrace() [tiflash+30572563]
                    dbms/src/Common/StackTrace.cpp:23
       0x1d25616    DB::Exception::Exception(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, int) [tiflash+30561814]
                    dbms/src/Common/Exception.h:41
       0x1df061e    DB::SharedQueryBlockInputStream::waitThread() [tiflash+31393310]
                    dbms/src/DataStreams/SharedQueryBlockInputStream.h:169
       0x1def484    DB::SharedQueryBlockInputStream::readSuffix() [tiflash+31388804]
                    dbms/src/DataStreams/SharedQueryBlockInputStream.h:80
       0x668b0b9    DB::IProfilingBlockInputStream::readSuffix() [tiflash+107524281]
                    dbms/src/DataStreams/IProfilingBlockInputStream.cpp:135
       0x668b0b9    DB::IProfilingBlockInputStream::readSuffix() [tiflash+107524281]
                    dbms/src/DataStreams/IProfilingBlockInputStream.cpp:135
       0x7b8c8ba    DB::UnionBlockInputStream<(DB::StreamUnionMode)0, true>::readSuffix() [tiflash+129550522]
                    dbms/src/DataStreams/UnionBlockInputStream.h:256
       0x7bd05f1    DB::MPPTask::runImpl() [tiflash+129828337]
                    dbms/src/Flash/Mpp/MPPTask.cpp:310
       0x1db233c    auto DB::wrapInvocable<std::__1::function<void ()> >(bool, std::__1::function<void ()>&&)::'lambda'()::operator()() [tiflash+31138620]
                    dbms/src/Common/wrapInvocable.h:36
       0x1da1a78    DB::DynamicThreadPool::executeTask(std::__1::unique_ptr<DB::IExecutableTask, std::__1::default_delete<DB::IExecutableTask> >&) [tiflash+31070840]
                    dbms/src/Common/DynamicThreadPool.cpp:101
       0x1da1956    DB::DynamicThreadPool::dynamicWork(std::__1::unique_ptr<DB::IExecutableTask, std::__1::default_delete<DB::IExecutableTask> >) [tiflash+31070550]
                    dbms/src/Common/DynamicThreadPool.cpp:142
       0x1da35cd    auto std::__1::thread DB::ThreadFactory::newThread<void (DB::DynamicThreadPool::*)(std::__1::unique_ptr<DB::IExecutableTask, std::__1::default_delete<DB::IExecutableTask> >), DB::DynamicThreadPool*, std::__1::unique_ptr<DB::IExecutableTask, std::__1::default_delete<DB::IExecutableTask> > >(bool, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, void (DB::DynamicThreadPool::*&&)(std::__1::unique_ptr<DB::IExecutableTask, std::__1::default_delete<DB::IExecutableTask> >), DB::DynamicThreadPool*&&, std::__1::unique_ptr<DB::IExecutableTask, std::__1::default_delete<DB::IExecutableTask> >&&)::'lambda'(auto&&...)::operator()<DB::DynamicThreadPool*, std::__1::unique_ptr<DB::IExecutableTask, std::__1::default_delete<DB::IExecutableTask> > >(auto&&...) const [tiflash+31077837]
                    dbms/src/Common/ThreadFactory.h:47
       0x1da33a9    void* std::__1::__thread_proxy<std::__1::tuple<std::__1::unique_ptr<std::__1::__thread_struct, std::__1::default_delete<std::__1::__thread_struct> >, std::__1::thread DB::ThreadFactory::newThread<void (DB::DynamicThreadPool::*)(std::__1::unique_ptr<DB::IExecutableTask, std::__1::default_delete<DB::IExecutableTask> >), DB::DynamicThreadPool*, std::__1::unique_ptr<DB::IExecutableTask, std::__1::default_delete<DB::IExecutableTask> > >(bool, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, void (DB::DynamicThreadPool::*&&)(std::__1::unique_ptr<DB::IExecutableTask, std::__1::default_delete<DB::IExecutableTask> >), DB::DynamicThreadPool*&&, std::__1::unique_ptr<DB::IExecutableTask, std::__1::default_delete<DB::IExecutableTask> >&&)::'lambda'(auto&&...), DB::DynamicThreadPool*, std::__1::unique_ptr<DB::IExecutableTask, std::__1::default_delete<DB::IExecutableTask> > > >(void*) [tiflash+31077289]
                    /usr/local/bin/../include/c++/v1/thread:291
  0x7f55f3b03ea5    start_thread [libpthread.so.0+32421]
  0x7f55f361696d    __clone [libc.so.6+1042797]"] [thread_id=1112]
[2022/08/24 08:15:25.966 +08:00] [ERROR] [Exception.cpp:80] ["MPPTask:MPP<query:435499858696601684,task:9> Failed to write error DB::Exception: Query was cancelled to tunnel: tunnel9+-1: Code: 0, e.displayText() = DB::Exception: write to tunnel which is already closed,grpc writes failed., e.what() = DB::Exception, Stack trace:


       0x1d28013    StackTrace::StackTrace() [tiflash+30572563]
                    dbms/src/Common/StackTrace.cpp:23
       0x1d25616    DB::Exception::Exception(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, int) [tiflash+30561814]
                    dbms/src/Common/Exception.h:41
       0x7be0f96    DB::MPPTunnelBase<DB::PacketWriter>::write(mpp::MPPDataPacket const&, bool) [tiflash+129896342]
                    dbms/src/Flash/Mpp/MPPTunnel.cpp:161
       0x7bd2106    DB::MPPTask::writeErrToAllTunnels(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&) [tiflash+129835270]
                    dbms/src/Flash/Mpp/MPPTask.cpp:378
       0x7bd0808    DB::MPPTask::runImpl() [tiflash+129828872]
                    dbms/src/Flash/Mpp/MPPTask.cpp:357
       0x1db233c    auto DB::wrapInvocable<std::__1::function<void ()> >(bool, std::__1::function<void ()>&&)::'lambda'()::operator()() [tiflash+31138620]
                    dbms/src/Common/wrapInvocable.h:36
       0x1da1a78    DB::DynamicThreadPool::executeTask(std::__1::unique_ptr<DB::IExecutableTask, std::__1::default_delete<DB::IExecutableTask> >&) [tiflash+31070840]
                    dbms/src/Common/DynamicThreadPool.cpp:101
       0x1da1956    DB::DynamicThreadPool::dynamicWork(std::__1::unique_ptr<DB::IExecutableTask, std::__1::default_delete<DB::IExecutableTask> >) [tiflash+31070550]
                    dbms/src/Common/DynamicThreadPool.cpp:142
       0x1da35cd    auto std::__1::thread DB::ThreadFactory::newThread<void (DB::DynamicThreadPool::*)(std::__1::unique_ptr<DB::IExecutableTask, std::__1::default_delete<DB::IExecutableTask> >), DB::DynamicThreadPool*, std::__1::unique_ptr<DB::IExecutableTask, std::__1::default_delete<DB::IExecutableTask> > >(bool, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, void (DB::DynamicThreadPool::*&&)(std::__1::unique_ptr<DB::IExecutableTask, std::__1::default_delete<DB::IExecutableTask> >), DB::DynamicThreadPool*&&, std::__1::unique_ptr<DB::IExecutableTask, std::__1::default_delete<DB::IExecutableTask> >&&)::'lambda'(auto&&...)::operator()<DB::DynamicThreadPool*, std::__1::unique_ptr<DB::IExecutableTask, std::__1::default_delete<DB::IExecutableTask> > >(auto&&...) const [tiflash+31077837]
                    dbms/src/Common/ThreadFactory.h:47
       0x1da33a9    void* std::__1::__thread_proxy<std::__1::tuple<std::__1::unique_ptr<std::__1::__thread_struct, std::__1::default_delete<std::__1::__thread_struct> >, std::__1::thread DB::ThreadFactory::newThread<void (DB::DynamicThreadPool::*)(std::__1::unique_ptr<DB::IExecutableTask, std::__1::default_delete<DB::IExecutableTask> >), DB::DynamicThreadPool*, std::__1::unique_ptr<DB::IExecutableTask, std::__1::default_delete<DB::IExecutableTask> > >(bool, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, void (DB::DynamicThreadPool::*&&)(std::__1::unique_ptr<DB::IExecutableTask, std::__1::default_delete<DB::IExecutableTask> >), DB::DynamicThreadPool*&&, std::__1::unique_ptr<DB::IExecutableTask, std::__1::default_delete<DB::IExecutableTask> >&&)::'lambda'(auto&&...), DB::DynamicThreadPool*, std::__1::unique_ptr<DB::IExecutableTask, std::__1::default_delete<DB::IExecutableTask> > > >(void*) [tiflash+31077289]
                    /usr/local/bin/../include/c++/v1/thread:291
  0x7f55f3b03ea5    start_thread [libpthread.so.0+32421]
  0x7f55f361696d    __clone [libc.so.6+1042797]"] [thread_id=1112]
[2022/08/24 08:15:25.966 +08:00] [ERROR] [MPPTaskManager.cpp:164] ["TaskManager:The task MPP<query:435499858696601684,task:9> cannot be found and fail to unregister"] [thread_id=1112]
[2022/08/24 08:15:25.966 +08:00] [WARN] [MPPTask.cpp:365] ["MPPTask:MPP<query:435499858696601684,task:9> finish task which was cancelled before"] [thread_id=1112]
[2022/08/24 08:15:25.973 +08:00] [ERROR] [Exception.cpp:80] ["SharedQuery:MPP<query:435499858696601684,task:9> virtual DB::SharedQueryBlockInputStream::~SharedQueryBlockInputStream(): Code: 0, e.displayText() = DB::Exception: Query was cancelled, e.what() = DB::Exception, Stack trace:


       0x1d28013    StackTrace::StackTrace() [tiflash+30572563]
                    dbms/src/Common/StackTrace.cpp:23
       0x1d25616    DB::Exception::Exception(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, int) [tiflash+30561814]
                    dbms/src/Common/Exception.h:41
       0x1df061e    DB::SharedQueryBlockInputStream::waitThread() [tiflash+31393310]
                    dbms/src/DataStreams/SharedQueryBlockInputStream.h:169
       0x1def484    DB::SharedQueryBlockInputStream::readSuffix() [tiflash+31388804]
                    dbms/src/DataStreams/SharedQueryBlockInputStream.h:80
       0x1def5b1    DB::SharedQueryBlockInputStream::~SharedQueryBlockInputStream() [tiflash+31389105]
                    dbms/src/DataStreams/SharedQueryBlockInputStream.h:48
       0x1dc9b2e    DB::IBlockInputStream::~IBlockInputStream() [tiflash+31234862]
                    dbms/src/DataStreams/IBlockInputStream.h:111
       0x1dc9b2e    DB::IBlockInputStream::~IBlockInputStream() [tiflash+31234862]
                    dbms/src/DataStreams/IBlockInputStream.h:111
       0x1dc9b2e    DB::IBlockInputStream::~IBlockInputStream() [tiflash+31234862]
                    dbms/src/DataStreams/IBlockInputStream.h:111
       0x738d46c    DB::UnionBlockInputStream<(DB::StreamUnionMode)0, false>::~UnionBlockInputStream() [tiflash+121164908]
                    dbms/src/DataStreams/UnionBlockInputStream.h:136
       0x1dc9b2e    DB::IBlockInputStream::~IBlockInputStream() [tiflash+31234862]
                    dbms/src/DataStreams/IBlockInputStream.h:111
       0x1dc9b2e    DB::IBlockInputStream::~IBlockInputStream() [tiflash+31234862]
                    dbms/src/DataStreams/IBlockInputStream.h:111
       0x1dc9b2e    DB::IBlockInputStream::~IBlockInputStream() [tiflash+31234862]
                    dbms/src/DataStreams/IBlockInputStream.h:111
       0x1dc9b2e    DB::IBlockInputStream::~IBlockInputStream() [tiflash+31234862]
                    dbms/src/DataStreams/IBlockInputStream.h:111
       0x1dc9b2e    DB::IBlockInputStream::~IBlockInputStream() [tiflash+31234862]
                    dbms/src/DataStreams/IBlockInputStream.h:111
       0x1def65d    DB::SharedQueryBlockInputStream::~SharedQueryBlockInputStream() [tiflash+31389277]
                    dbms/src/DataStreams/SharedQueryBlockInputStream.h:54
       0x1dc9b2e    DB::IBlockInputStream::~IBlockInputStream() [tiflash+31234862]
                    dbms/src/DataStreams/IBlockInputStream.h:111
       0x1dc9b2e    DB::IBlockInputStream::~IBlockInputStream() [tiflash+31234862]
                    dbms/src/DataStreams/IBlockInputStream.h:111
       0x1dc9b2e    DB::IBlockInputStream::~IBlockInputStream() [tiflash+31234862]
                    dbms/src/DataStreams/IBlockInputStream.h:111
       0x7b8caac    DB::UnionBlockInputStream<(DB::StreamUnionMode)0, true>::~UnionBlockInputStream() [tiflash+129551020]
                    dbms/src/DataStreams/UnionBlockInputStream.h:136
       0x7615050    DB::ProcessListElement::releaseQueryStreams() [tiflash+123818064]
                    dbms/src/Interpreters/ProcessList.cpp:276
       0x7614934    DB::ProcessListEntry::~ProcessListEntry() [tiflash+123816244]
                    dbms/src/Interpreters/ProcessList.cpp:192
       0x1d70a05    DB::BlockIO::~BlockIO() [tiflash+30870021]
                    dbms/src/DataStreams/BlockIO.h:78
       0x727486f    DB::DAGContext::~DAGContext() [tiflash+120014959]
                    dbms/src/Flash/Coprocessor/DAGContext.h:113
       0x7bcdd68    DB::MPPTask::~MPPTask() [tiflash+129817960]
                    dbms/src/Flash/Mpp/MPPTask.cpp:78
       0x7bcd597    std::__1::__shared_ptr_pointer<DB::MPPTask*, std::__1::shared_ptr<DB::MPPTask>::__shared_ptr_default_delete<DB::MPPTask, DB::MPPTask>, std::__1::allocator<DB::MPPTask> >::__on_zero_shared() [tiflash+129815959]
                    /usr/local/bin/../include/c++/v1/__memory/shared_ptr.h:267
       0x7bd2ff1    std::__1::__function::__func<DB::MPPTask::run()::$_18, std::__1::allocator<DB::MPPTask::run()::$_18>, void ()>::destroy() [tiflash+129839089]
                    /usr/local/bin/../include/c++/v1/__functional/function.h:327
       0x1db282b    DB::ExecutableTask<auto DB::wrapInvocable<std::__1::function<void ()> >(bool, std::__1::function<void ()>&&)::'lambda'()>::~ExecutableTask() [tiflash+31139883]
                    dbms/src/Common/ExecutableTask.h:46
       0x1da1a8d    DB::DynamicThreadPool::executeTask(std::__1::unique_ptr<DB::IExecutableTask, std::__1::default_delete<DB::IExecutableTask> >&) [tiflash+31070861]
                    dbms/src/Common/DynamicThreadPool.cpp:102
       0x1da1956    DB::DynamicThreadPool::dynamicWork(std::__1::unique_ptr<DB::IExecutableTask, std::__1::default_delete<DB::IExecutableTask> >) [tiflash+31070550]
                    dbms/src/Common/DynamicThreadPool.cpp:142
       0x1da35cd    auto std::__1::thread DB::ThreadFactory::newThread<void (DB::DynamicThreadPool::*)(std::__1::unique_ptr<DB::IExecutableTask, std::__1::default_delete<DB::IExecutableTask> >), DB::DynamicThreadPool*, std::__1::unique_ptr<DB::IExecutableTask, std::__1::default_delete<DB::IExecutableTask> > >(bool, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, void (DB::DynamicThreadPool::*&&)(std::__1::unique_ptr<DB::IExecutableTask, std::__1::default_delete<DB::IExecutableTask> >), DB::DynamicThreadPool*&&, std::__1::unique_ptr<DB::IExecutableTask, std::__1::default_delete<DB::IExecutableTask> >&&)::'lambda'(auto&&...)::operator()<DB::DynamicThreadPool*, std::__1::unique_ptr<DB::IExecutableTask, std::__1::default_delete<DB::IExecutableTask> > >(auto&&...) const [tiflash+31077837]
                    dbms/src/Common/ThreadFactory.h:47
       0x1da33a9    void* std::__1::__thread_proxy<std::__1::tuple<std::__1::unique_ptr<std::__1::__thread_struct, std::__1::default_delete<std::__1::__thread_struct> >, std::__1::thread DB::ThreadFactory::newThread<void (DB::DynamicThreadPool::*)(std::__1::unique_ptr<DB::IExecutableTask, std::__1::default_delete<DB::IExecutableTask> >), DB::DynamicThreadPool*, std::__1::unique_ptr<DB::IExecutableTask, std::__1::default_delete<DB::IExecutableTask> > >(bool, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, void (DB::DynamicThreadPool::*&&)(std::__1::unique_ptr<DB::IExecutableTask, std::__1::default_delete<DB::IExecutableTask> >), DB::DynamicThreadPool*&&, std::__1::unique_ptr<DB::IExecutableTask, std::__1::default_delete<DB::IExecutableTask> >&&)::'lambda'(auto&&...), DB::DynamicThreadPool*, std::__1::unique_ptr<DB::IExecutableTask, std::__1::default_delete<DB::IExecutableTask> > > >(void*) [tiflash+31077289]
                    /usr/local/bin/../include/c++/v1/thread:291
  0x7f55f3b03ea5    start_thread [libpthread.so.0+32421]"] [thread_id=1112]
[2022/08/24 08:16:43.997 +08:00] [WARN] [<unknown>] ["pingcap.tikv:region {27949,125,43} find error: peer is not leader for region 27949, leader may Some(id: 27952 store_id: 6)"] [thread_id=16]
[2022/08/24 08:18:05.319 +08:00] [ERROR] [WriteBatches.h:69] ["WriteBatches:!!!=========================Modifications in removed_log haven't persisted=========================!!! Stack trace: 
       0x1d28013    StackTrace::StackTrace() [tiflash+30572563]
                    dbms/src/Common/StackTrace.cpp:23
       0x77aca8f    DB::DM::WriteBatches::~WriteBatches()::'lambda'(DB::WriteBatch const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&)::operator()(DB::WriteBatch const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&) const [tiflash+125487759]
                    dbms/src/Storages/DeltaMerge/WriteBatches.h:65
       0x77a8ec1    DB::DM::WriteBatches::~WriteBatches() [tiflash+125472449]
                    dbms/src/Storages/DeltaMerge/WriteBatches.h:75
       0x78aed6a    DB::DM::DeltaValueSpace::compact(DB::DM::DMContext&) [tiflash+126545258]
                    dbms/src/Storages/DeltaMerge/Delta/DeltaValueSpace.cpp:272
       0x77d625a    DB::DM::Segment::compactDelta(DB::DM::DMContext&) [tiflash+125657690]
                    dbms/src/Storages/DeltaMerge/Segment.cpp:1365
       0x779bf34    DB::DM::DeltaMergeStore::handleBackgroundTask(bool) [tiflash+125419316]
                    dbms/src/Storages/DeltaMerge/DeltaMergeStore.cpp:1640
       0x76d3181    DB::BackgroundProcessingPool::threadFunction() [tiflash+124596609]
                    dbms/src/Storages/BackgroundProcessingPool.cpp:233
       0x76d3b01    void* std::__1::__thread_proxy<std::__1::tuple<std::__1::unique_ptr<std::__1::__thread_struct, std::__1::default_delete<std::__1::__thread_struct> >, DB::BackgroundProcessingPool::BackgroundProcessingPool(int)::$_2> >(void*) [tiflash+124599041]
                    /usr/local/bin/../include/c++/v1/thread:291
  0x7f55f3b03ea5    start_thread [libpthread.so.0+32421]
  0x7f55f361696d    __clone [libc.so.6+1042797]"] [thread_id=17]
[2022/08/24 08:18:05.319 +08:00] [ERROR] [DeltaMergeStore.cpp:1666] ["DeltaMergeStore:db_69.t_708 Task Compact on Segment [1] failed. Error msg: Cannot write to file /data0/db/data/t_708/log/page_3_0/meta,[truncate_res = -1],[errno_after_truncate = Invalid argument],[bytes_written=0,to_write=42,offset = 12269], errno: 29, strerror: Illegal seek"] [thread_id=17]
[2022/08/24 08:18:05.329 +08:00] [ERROR] [Exception.cpp:85] ["void DB::BackgroundProcessingPool::threadFunction():Code: 75, e.displayText() = DB::Exception: Cannot write to file /data0/db/data/t_708/log/page_3_0/meta,[truncate_res = -1],[errno_after_truncate = Invalid argument],[bytes_written=0,to_write=42,offset = 12269], errno: 29, strerror: Illegal seek, e.what() = DB::Exception, Stack trace:


       0x1d28013    StackTrace::StackTrace() [tiflash+30572563]
                    dbms/src/Common/StackTrace.cpp:23
       0x1d25616    DB::Exception::Exception(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, int) [tiflash+30561814]
                    dbms/src/Common/Exception.h:41
       0x1d30ffa    DB::throwFromErrno(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, int, int) [tiflash+30609402]
                    dbms/src/Common/Exception.cpp:59
       0x79d7d09    void DB::PageUtil::writeFile<std::__1::shared_ptr<DB::WritableFile> >(std::__1::shared_ptr<DB::WritableFile>&, unsigned long, char*, unsigned long, std::__1::shared_ptr<DB::WriteLimiter> const&, bool, bool, bool) [tiflash+127761673]
                    dbms/src/Storages/Page/PageUtil.h:209
       0x79f8c90    DB::PS::V2::PageFile::Writer::write(DB::WriteBatch&, DB::PS::V2::PageEntriesEdit&, std::__1::shared_ptr<DB::WriteLimiter> const&, bool) [tiflash+127896720]
                    dbms/src/Storages/Page/V2/PageFile.cpp:808
       0x7a0c12e    DB::PS::V2::PageStorage::writeImpl(DB::WriteBatch&&, std::__1::shared_ptr<DB::WriteLimiter> const&) [tiflash+127975726]
                    dbms/src/Storages/Page/V2/PageStorage.cpp:534
       0x7a90408    DB::PageWriter::write(DB::WriteBatch&&, std::__1::shared_ptr<DB::WriteLimiter>) const [tiflash+128517128]
                    dbms/src/Storages/Page/PageStorage.cpp:460
       0x77ac0a6    DB::DM::WriteBatches::writeRemoves() [tiflash+125485222]
                    dbms/src/Storages/DeltaMerge/WriteBatches.h:208
       0x78aea91    DB::DM::DeltaValueSpace::compact(DB::DM::DMContext&) [tiflash+126544529]
                    dbms/src/Storages/DeltaMerge/Delta/DeltaValueSpace.cpp:269
       0x77d625a    DB::DM::Segment::compactDelta(DB::DM::DMContext&) [tiflash+125657690]
                    dbms/src/Storages/DeltaMerge/Segment.cpp:1365
       0x779bf34    DB::DM::DeltaMergeStore::handleBackgroundTask(bool) [tiflash+125419316]
                    dbms/src/Storages/DeltaMerge/DeltaMergeStore.cpp:1640
       0x76d3181    DB::BackgroundProcessingPool::threadFunction() [tiflash+124596609]
                    dbms/src/Storages/BackgroundProcessingPool.cpp:233
       0x76d3b01    void* std::__1::__thread_proxy<std::__1::tuple<std::__1::unique_ptr<std::__1::__thread_struct, std::__1::default_delete<std::__1::__thread_struct> >, DB::BackgroundProcessingPool::BackgroundProcessingPool(int)::$_2> >(void*) [tiflash+124599041]
                    /usr/local/bin/../include/c++/v1/thread:291
  0x7f55f3b03ea5    start_thread [libpthread.so.0+32421]
  0x7f55f361696d    __clone [libc.so.6+1042797]"] [thread_id=17]
[2022/08/24 08:18:34.647 +08:00] [ERROR] [BaseDaemon.cpp:369] ["BaseDaemon:(from thread 10) Terminate called after throwing an instance of DB::ErrnoException
Code: 77, e.displayText() = DB::Exception: Cannot close file /data0/db/data/t_841/meta/page_1_0/page, errno: 9, strerror: Bad file descriptor, e.what() = DB::Exception
Stack trace:


       0x1d28013    StackTrace::StackTrace() [tiflash+30572563]
                    dbms/src/Common/StackTrace.cpp:23
       0x1ed2bbf    terminate_handler() [tiflash+32320447]
                    libs/libdaemon/src/BaseDaemon.cpp:634
  0x7f55f7967a13    std::__terminate(void (*)()) [libc++abi.so.1+236051]
  0x7f55f79679b8    std::terminate() [libc++abi.so.1+235960]
       0x1d11bfb    __clang_call_terminate [tiflash+30481403]
       0x79f8e7c    ext::ScopeGuard<DB::PS::V2::PageFile::Writer::closeFd()::$_19>::~ScopeGuard() [tiflash+127897212]
                    libs/libcommon/include/ext/scope_guard.h:33
       0x79f8e31    DB::PS::V2::PageFile::Writer::tryCloseIdleFd(std::__1::chrono::duration<long long, std::__1::ratio<1l, 1l> > const&) [tiflash+127897137]
                    db"] [thread_id=1358]
[2022/08/24 08:18:34.647 +08:00] [ERROR] [BaseDaemon.cpp:377] [BaseDaemon:########################################] [thread_id=1358]
[2022/08/24 08:18:34.647 +08:00] [ERROR] [BaseDaemon.cpp:378] ["BaseDaemon:(from thread 10) Received signal Aborted(6)."] [thread_id=1358]

2. What did you expect to see? (Required)

3. What did you see instead (Required)

4. What is your TiFlash version? (Required)

6.1.1
@lilinghai lilinghai added the type/bug The issue is confirmed as a bug. label Aug 24, 2022
@JaySon-Huang
Copy link
Contributor

JaySon-Huang commented Aug 24, 2022 •
edited
Loading

The reason is similar to #5285. ::close return -1 with errno "Bad file descriptor error"

@mornyx mornyx mentioned this issue Aug 25, 2022
Open
@jebter jebter added the affects-6.1 This bug affects the 6.1.x(LTS) versions. label Aug 27, 2022
@flowbehappy
Copy link
Contributor

Duplicate with #5285

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
affects-6.1 This bug affects the 6.1.x(LTS) versions. component/storage severity/moderate type/bug The issue is confirmed as a bug.
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
5 participants
@flowbehappy @jebter @JaySon-Huang @lilinghai @ti-chi-bot