[dev.icinga.com #6724] TLS Connections still unstable in 2.0.1 #1800

icinga-migration · 2014-07-15T05:16:23Z

This issue has been migrated from Redmine: https://dev.icinga.com/issues/6724

Created by dhanke on 2014-07-15 05:16:23 +00:00

Assignee: gbeutner
Status: Resolved (closed on 2014-08-05 08:25:57 +00:00)
Target Version: 2.0.2
Last Update: 2014-08-05 15:59:39 +00:00 (in Redmine)

Icinga Version: 2.0.1

[2014-07-11 09:48:41 +0200] critical/TlsStream: SSL_read() failed with code 336195711, "error:1409F07F:SSL routines:SSL3_WRITE_PENDING:bad write retry"
[2014-07-11 09:48:41 +0200] warning/ApiClient: Error while reading JSON-RPC message for identity 'xxxxxxxx': /usr/src/packages/BUILD/icinga2-2.0.1/lib/base/tlsstream.cpp(175): Throw in function virtual size_t icinga::TlsStream::Read(void*, size_t)
Dynamic exception type: boost::exception_detail::clone_implicinga::openssl_error
std::exception::what: std::exception
[boost::errinfo_api_function_*] = SSL_read
[icinga::StackTrace*] =
(0) libbase.so: void boost::exception_detail::throw_exception_icinga::openssl_error(icinga::openssl_error const&, char const**, char const**, int) (+0x1ea) [0x7f9052f2558a]
(1) libbase.so: icinga::TlsStream::Read(void*, unsigned long) (+0x2d0) [0x7f9052f23a40]
(2) libbase.so: icinga::NetString::ReadStringFromStream(boost::shared_ptricinga::Stream const&, icinga::String*) (+0x4b) [0x7f9052ee23eb]
(3) libremote.so: icinga::JsonRpc::ReadMessage(boost::shared_ptricinga::Stream const&) (+0x3e) [0x7f904f7658ce]
(4) libremote.so: icinga::ApiClient::ProcessMessage() (+0x39) [0x7f904f7206a9]
(5) libremote.so: icinga::ApiClient::MessageThreadProc() (+0x68) [0x7f904f721788]
(6) libboost_thread.so.1.54.0: (+0xd5e3) [0x7f9053a5e5e3]
(7) libpthread.so.0: (+0x7806) [0x7f9051133806]
(8) libc.so.6: clone (+0x6d) [0x7f905162db1d]

Changesets

2014-08-01 13:23:07 +00:00 by gbeutner 3f647bb

Fix OpenSSL errors during (re-)negotiation

fixes #6724

2014-08-04 07:50:47 +00:00 by gbeutner 6455c51

Fix a potential deadlock in ApiClient::Disconnect

refs #6724

2014-08-04 10:48:46 +00:00 by gbeutner 3c8e84e

Fix an invalid pointer access in ApiClient::DisconnectSync

refs #6724

2014-08-04 11:35:12 +00:00 by gbeutner 9c99826

Properly close TLS connections

refs #6724

2014-08-04 12:59:38 +00:00 by (unknown) 8d0fab9

Fix non-existing endpoint on ApiListener error

refs #6724

2014-08-04 15:23:41 +00:00 by (unknown) 7b8dc1d

Fix non-existing endpoint on ApiListener error

refs #6724

2014-08-05 06:41:52 +00:00 by gbeutner da41725

Partially revert 3f647bb7797b3e71405c59eb280a4be74305c6b2

refs #6724

2014-08-05 07:10:59 +00:00 by gbeutner 883170a

Avoid duplicate API connections

refs #6724

2014-08-05 07:35:28 +00:00 by gbeutner f3d5584

Add a missing lock in TlsStream::Close

fixes #6621
refs #6724

2014-08-05 09:30:06 +00:00 by (unknown) 87e63ed

Use a local buffer for ERR_error_string()

refs #6724
refs #6682

Relations:

relates #6724
duplicates #6724

The text was updated successfully, but these errors were encountered:

icinga-migration · 2014-08-01T11:33:56Z

Updated by gbeutner on 2014-08-01 11:33:56 +00:00

I'm also (still) seeing the same issue on RHEL 5:

[2014-08-01 13:33:20 +0200] critical/TlsStream: SSL_do_handshake() failed with code 336195711, "error:1409F07F:SSL routines:SSL3_WRITE_PENDING:bad write retry"
Context:
        (0) Handling new API client connection

[2014-08-01 13:33:20 +0200] critical/ApiListener: Client TLS handshake failed.
Context:
        (0) Handling new API client connection

[2014-08-01 13:33:25 +0200] critical/TlsStream: SSL_do_handshake() failed with code 336195711, "error:1409F07F:SSL routines:SSL3_WRITE_PENDING:bad write retry"
Context:
        (0) Handling new API client connection

[2014-08-01 13:33:25 +0200] critical/ApiListener: Client TLS handshake failed.
Context:
        (0) Handling new API client connection

(gdb) thread apply all bt

Thread 14 (Thread 0xb5bb8b90 (LWP 22740)):

#0  0xb7795424 in __kernel_vsyscall ()
#1  0xb6f7ecc5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/libpthread.so.0
#2  0xb7070e0d in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/libc.so.6
#3  0xb75820b6 in boost::condition_variable::wait(boost::unique_lock&) () from /usr/lib/icinga2/libbase.so
#4  0xb75ac976 in icinga::WorkQueue::WorkerThreadProc() () from /usr/lib/icinga2/libbase.so
#5  0xb75ad820 in boost::detail::thread_data, boost::_bi::list1 > > >::run() () from /usr/lib/icinga2/libbase.so
#6  0xb77803f0 in thread_proxy () from /usr/lib/libboost_thread-mt.so.5
#7  0xb6f7a912 in start_thread () from /lib/libpthread.so.0
#8  0xb706474e in clone () from /lib/libc.so.6

Thread 13 (Thread 0xb5b77b90 (LWP 22752)):

#0  0xb7795424 in __kernel_vsyscall ()
#1  0xb6f7ecc5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/libpthread.so.0
#2  0xb7070e0d in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/libc.so.6
#3  0xb75820b6 in boost::condition_variable::wait(boost::unique_lock&) () from /usr/lib/icinga2/libbase.so
#4  0xb75ac976 in icinga::WorkQueue::WorkerThreadProc() () from /usr/lib/icinga2/libbase.so
#5  0xb75ad820 in boost::detail::thread_data, boost::_bi::list1 > > >::run() () from /usr/lib/icinga2/libbase.so
#6  0xb77803f0 in thread_proxy () from /usr/lib/libboost_thread-mt.so.5
#7  0xb6f7a912 in start_thread () from /lib/libpthread.so.0
#8  0xb706474e in clone () from /lib/libc.so.6

Thread 12 (Thread 0xb59ffb90 (LWP 22756)):

#0  0xb7795424 in __kernel_vsyscall ()
#1  0xb6f7eff2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib/libpthread.so.0
#2  0xb7070e54 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib/libc.so.6
#3  0xb7786796 in boost::condition_variable::timed_wait(boost::unique_lock&, boost::posix_time::ptime const&) () from /usr/lib/libboost_thread-mt.so.5
#4  0xb5ce764b in icinga::CheckerComponent::CheckThreadProc() () from /usr/lib/icinga2/libchecker.so
#5  0xb5ceeb40 in boost::detail::thread_data, boost::_bi::list1 > > >::run() () from /usr/lib/icinga2/libchecker.so
#6  0xb77803f0 in thread_proxy () from /usr/lib/libboost_thread-mt.so.5
#7  0xb6f7a912 in start_thread () from /lib/libpthread.so.0
#8  0xb706474e in clone () from /lib/libc.so.6

Thread 11 (Thread 0xb59beb90 (LWP 22825)):

#0  0xb7795424 in __kernel_vsyscall ()
#1  0xb70653e8 in accept () from /lib/libc.so.6
#2  0xb7568135 in icinga::Socket::Accept() () from /usr/lib/icinga2/libbase.so
#3  0xb5dc2573 in icinga::ApiListener::ListenerThreadProc(boost::shared_ptr const&) () from /usr/lib/icinga2/libremote.so
#4  0xb5dd3847 in boost::detail::thread_data const&>, boost::_bi::list2, boost::_bi::value > > > >::run() () from /usr/lib/icinga2/libremote.so
#5  0xb77803f0 in thread_proxy () from /usr/lib/libboost_thread-mt.so.5
#6  0xb6f7a912 in start_thread () from /lib/libpthread.so.0
#7  0xb706474e in clone () from /lib/libc.so.6

--~~Type to continue, or q to quit--~~
Thread 10 (Thread 0xb5c7bb90 (LWP 22826)):

#0  0xb7795424 in __kernel_vsyscall ()
#1  0xb6f7eff2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib/libpthread.so.0
#2  0xb7070e54 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib/libc.so.6
#3  0xb7585912 in icinga::Timer::TimerThreadProc() () from /usr/lib/icinga2/libbase.so
#4  0xb758602f in boost::detail::thread_data::run() () from /usr/lib/icinga2/libbase.so
#5  0xb77803f0 in thread_proxy () from /usr/lib/libboost_thread-mt.so.5
#6  0xb6f7a912 in start_thread () from /lib/libpthread.so.0
#7  0xb706474e in clone () from /lib/libc.so.6

Thread 9 (Thread 0xb5c3ab90 (LWP 22833)):

#0  0xb7795424 in __kernel_vsyscall ()
#1  0xb6f7ecc5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/libpthread.so.0
#2  0xb7070e0d in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/libc.so.6
#3  0xb75820b6 in boost::condition_variable::wait(boost::unique_lock&) () from /usr/lib/icinga2/libbase.so
#4  0xb757bbbf in icinga::ThreadPool::WorkerThread::ThreadProc(icinga::ThreadPool::Queue&) () from /usr/lib/icinga2/libbase.so
#5  0xb757f902 in boost::detail::thread_data, boost::_bi::list2, boost::reference_wrapper > > >::run() () from /usr/lib/icinga2/libbase.so
#6  0xb77803f0 in thread_proxy () from /usr/lib/libboost_thread-mt.so.5
#7  0xb6f7a912 in start_thread () from /lib/libpthread.so.0
#8  0xb706474e in clone () from /lib/libc.so.6

Thread 8 (Thread 0xb5bf9b90 (LWP 22834)):

#0  0xb7795424 in __kernel_vsyscall ()
#1  0xb6f7ecc5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/libpthread.so.0
#2  0xb7070e0d in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/libc.so.6
#3  0xb75820b6 in boost::condition_variable::wait(boost::unique_lock&) () from /usr/lib/icinga2/libbase.so
#4  0xb757bbbf in icinga::ThreadPool::WorkerThread::ThreadProc(icinga::ThreadPool::Queue&) () from /usr/lib/icinga2/libbase.so
#5  0xb757f902 in boost::detail::thread_data, boost::_bi::list2, boost::reference_wrapper > > >::run() () from /usr/lib/icinga2/libbase.so
#6  0xb77803f0 in thread_proxy () from /usr/lib/libboost_thread-mt.so.5
#7  0xb6f7a912 in start_thread () from /lib/libpthread.so.0
#8  0xb706474e in clone () from /lib/libc.so.6

Thread 7 (Thread 0xb5cbcb90 (LWP 22835)):

#0  0xb7795424 in __kernel_vsyscall ()
#1  0xb6f7ecc5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/libpthread.so.0
#2  0xb7070e0d in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/libc.so.6
#3  0xb75820b6 in boost::condition_variable::wait(boost::unique_lock&) () from /usr/lib/icinga2/libbase.so
#4  0xb757bbbf in icinga::ThreadPool::WorkerThread::ThreadProc(icinga::ThreadPool::Queue&) () from /usr/lib/icinga2/libbase.so
#5  0xb757f902 in boost::detail::thread_data, boost::_bi::list2, boost::reference_wrapper > > >::run() () from /usr/lib/icinga2/libbase.so
#6  0xb77803f0 in thread_proxy () from /usr/lib/libboost_thread-mt.so.5
#7  0xb6f7a912 in start_thread () from /lib/libpthread.so.0
#8  0xb706474e in clone () from /lib/libc.so.6

Thread 6 (Thread 0xb597db90 (LWP 22836)):

#0  0xb7795424 in __kernel_vsyscall ()

--~~Type to continue, or q to quit--~~

#1  0xb6f7ecc5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/libpthread.so.0
#2  0xb7070e0d in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/libc.so.6
#3  0xb75820b6 in boost::condition_variable::wait(boost::unique_lock&) () from /usr/lib/icinga2/libbase.so
#4  0xb757bbbf in icinga::ThreadPool::WorkerThread::ThreadProc(icinga::ThreadPool::Queue&) () from /usr/lib/icinga2/libbase.so
#5  0xb757f902 in boost::detail::thread_data, boost::_bi::list2, boost::reference_wrapper > > >::run() () from /usr/lib/icinga2/libbase.so
#6  0xb77803f0 in thread_proxy () from /usr/lib/libboost_thread-mt.so.5
#7  0xb6f7a912 in start_thread () from /lib/libpthread.so.0
#8  0xb706474e in clone () from /lib/libc.so.6

Thread 5 (Thread 0xb593cb90 (LWP 22837)):

#0  0xb7795424 in __kernel_vsyscall ()
#1  0xb6f7eff2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib/libpthread.so.0
#2  0xb7070e54 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib/libc.so.6
#3  0xb757e46c in icinga::ThreadPool::ManagerThreadProc() () from /usr/lib/icinga2/libbase.so
#4  0xb757f940 in boost::detail::thread_data, boost::_bi::list1 > > >::run() () from /usr/lib/icinga2/libbase.so
#5  0xb77803f0 in thread_proxy () from /usr/lib/libboost_thread-mt.so.5
#6  0xb6f7a912 in start_thread () from /lib/libpthread.so.0
#7  0xb706474e in clone () from /lib/libc.so.6

Thread 4 (Thread 0xb58bab90 (LWP 26303)):

#0  0xb6f7df8e in pthread_mutex_unlock () from /lib/libpthread.so.0
#1  0xb7070fa6 in pthread_mutex_unlock () from /lib/libc.so.6
#2  0xb758e784 in ?? () from /usr/lib/icinga2/libbase.so
#3  0x096319dc in ?? ()
#4  0x00000000 in ?? ()

Thread 3 (Thread 0xb5879b90 (LWP 8669)):

#0  0xb7795424 in __kernel_vsyscall ()
#1  0xb705a963 in poll () from /lib/libc.so.6
#2  0xb7540cf3 in icinga::Process::IOThreadProc(int) () from /usr/lib/icinga2/libbase.so
#3  0xb75443a8 in boost::detail::thread_data > > >::run() () from /usr/lib/icinga2/libbase.so
#4  0xb77803f0 in thread_proxy () from /usr/lib/libboost_thread-mt.so.5
#5  0xb6f7a912 in start_thread () from /lib/libpthread.so.0
#6  0xb706474e in clone () from /lib/libc.so.6

Thread 2 (Thread 0xb56ffb90 (LWP 8670)):

#0  0xb7795424 in __kernel_vsyscall ()
#1  0xb705a963 in poll () from /lib/libc.so.6
#2  0xb7540cf3 in icinga::Process::IOThreadProc(int) () from /usr/lib/icinga2/libbase.so
#3  0xb75443a8 in boost::detail::thread_data > > >::run() () from /usr/lib/icinga2/libbase.so
#4  0xb77803f0 in thread_proxy () from /usr/lib/libboost_thread-mt.so.5
#5  0xb6f7a912 in start_thread () from /lib/libpthread.so.0
#6  0xb706474e in clone () from /lib/libc.so.6

Thread 1 (Thread 0xb61d16e0 (LWP 22714)):

#0  0xb7795424 in __kernel_vsyscall ()

--~~Type to continue, or q to quit--~~

#1  0xb7022a16 in nanosleep () from /lib/libc.so.6
#2  0xb705daac in usleep () from /lib/libc.so.6
#3  0xb75932b0 in icinga::Utility::Sleep(double) () from /usr/lib/icinga2/libbase.so
#4  0xb7506927 in icinga::Application::RunEventLoop() () from /usr/lib/icinga2/libbase.so
#5  0xb5fea1f1 in icinga::IcingaApplication::Main() () from /usr/lib/icinga2/libicinga.so
#6  0xb7505202 in icinga::Application::Run() () from /usr/lib/icinga2/libbase.so
#7  0x08060381 in Main() ()
#8  0x08060838 in main ()

(gdb)

icinga-migration · 2014-08-01T11:46:47Z

Updated by gbeutner on 2014-08-01 11:46:47 +00:00

Category set to Cluster
Assigned to set to gbeutner
Target Version set to 2.0.2

icinga-migration · 2014-08-01T12:52:00Z

Updated by gbeutner on 2014-08-01 12:52:00 +00:00

Duplicated set to 6830

icinga-migration · 2014-08-01T13:25:05Z

Updated by gbeutner on 2014-08-01 13:25:05 +00:00

Status changed from New to Resolved
Done % changed from 0 to 100

Applied in changeset 3f647bb.

icinga-migration · 2014-08-01T13:33:05Z

Updated by gbeutner on 2014-08-01 13:33:05 +00:00

Estimated Hours set to 4

icinga-migration · 2014-08-01T13:35:26Z

Updated by gbeutner on 2014-08-01 13:35:26 +00:00

I'm not entirely sure if that patch really fixes it. Please retest this.

icinga-migration · 2014-08-01T17:34:20Z

Updated by TechIsCool on 2014-08-01 17:34:20 +00:00

Here is the info you were looking for. version: v2.0.1-60-gb02a1ff)

[2014-08-01 10:28:33 -0700] information/ApiListener: New client connection for identity 'abydos'
[2014-08-01 10:28:33 -0700] warning/ApiListener: Ignoring config update. 'api' does not accept config.
[2014-08-01 10:28:33 -0700] critical/TlsStream: SSL_write() failed with code 0, "error:00000000:lib(0):func(0):reason(0)"
[2014-08-01 10:28:33 -0700] warning/ApiClient: Error while sending JSON-RPC message for identity 'abydos'
[2014-08-01 10:28:33 -0700] warning/ApiClient: API client disconnected for identity 'abydos'
[2014-08-01 10:28:33 -0700] critical/TlsStream: SSL_write() failed with code 0, "error:140D00CF:SSL routines:SSL_write:protocol is shutdown"
[2014-08-01 10:28:33 -0700] warning/ApiClient: Error while sending JSON-RPC message for identity 'abydos'
[2014-08-01 10:28:33 -0700] warning/ApiClient: API client disconnected for identity 'abydos'
[2014-08-01 10:28:35 -0700] critical/TlsStream: SSL_read() failed with code 0, "error:00000000:lib(0):func(0):reason(0)"
[2014-08-01 10:28:35 -0700] warning/ApiClient: Error while reading JSON-RPC message for identity 'abydos': /home/administrator/downloads/2.0.1.2/icinga2/lib/base/tlsstream.cpp(167): Throw in function virtual size_t icinga::TlsStream::Read(void*, size_t)
Dynamic exception type: boost::exception_detail::clone_impl
std::exception::what: std::exception
[icinga::StackTrace*] =
        (0) libbase.so: void boost::throw_exception(icinga::openssl_error const&) (+0xbe) [0x7f66df359dbe]
        (1) libbase.so: void boost::exception_detail::throw_exception_(icinga::openssl_error const&, char const*, char const*, int) (+0x40) [0x7f66df359e80]
        (2) libbase.so: icinga::TlsStream::Read(void*, unsigned long) (+0x230) [0x7f66df358930]
        (3) libbase.so: icinga::NetString::ReadStringFromStream(boost::shared_ptr const&, icinga::String*) (+0x4b) [0x7f66df32203b]
        (4) libremote.so: icinga::JsonRpc::ReadMessage(boost::shared_ptr const&) (+0x3e) [0x7f66d9e70e5e]
        (5) libremote.so: icinga::ApiClient::ProcessMessage() (+0x3b) [0x7f66d9e3a2cb]
        (6) libremote.so: icinga::ApiClient::MessageThreadProc() (+0x58) [0x7f66d9e3b168]
        (7) libboost_thread.so.1.54.0:  (+0xba4a) [0x7f66de292a4a]
        (8) libpthread.so.0:  (+0x8182) [0x7f66df60a182]
        (9) libc.so.6: clone (+0x6d) [0x7f66de79c30d]


[icinga::ContextTrace*] =

0, "error:00000000:lib(0):func(0):reason(0)"[boost::errinfo_api_function_*] = SSL_read

icinga-migration · 2014-08-03T19:05:46Z

Updated by TechIsCool on 2014-08-03 19:05:46 +00:00

So even though it is still throwing that error replication now seems to be working correctly. I read through the modified docs that where updated after I got the configuration set up and removed the folder /etc/icinga2/zone.d/config-ha-master from my slave that I had created. This started the replication functioning correctly.

icinga-migration · 2014-08-04T06:50:52Z

Updated by gbeutner on 2014-08-04 06:50:52 +00:00

Status changed from Resolved to Assigned

FWIW it's still not working properly for my own cluster.

icinga-migration · 2014-08-04T07:04:09Z

Updated by gbeutner on 2014-08-04 07:04:09 +00:00

The latest problem I'm seeing with TLS connections is this:

(gdb) thr 22
[Switching to thread 22 (Thread 0x7f99d2526700 (LWP 8362))]

#0  __lll_lock_wait () at ../nptl/sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:132

132 ../nptl/sysdeps/unix/sysv/linux/x86_64/lowlevellock.S: No such file or directory.
(gdb) bt

#0  __lll_lock_wait () at ../nptl/sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:132
#1  0x00007f99d8476065 in _L_lock_858 () from /lib/x86_64-linux-gnu/libpthread.so.0
#2  0x00007f99d8475eba in __pthread_mutex_lock (mutex=0x7f99b03fc020) at pthread_mutex_lock.c:61
#3  0x00007f99d7fb74e2 in icinga::WorkQueue::Join(bool) () from /usr/lib/x86_64-linux-gnu/icinga2/libbase.so
#4  0x00007f99d7fb7673 in icinga::WorkQueue::~WorkQueue() () from /usr/lib/x86_64-linux-gnu/icinga2/libbase.so
#5  0x00007f99d30fc39f in icinga::ApiClient::~ApiClient() () from /usr/lib/x86_64-linux-gnu/icinga2/libremote.so
#6  0x00007f99d3114ffa in boost::detail::sp_counted_impl_pd >::dispose() () from /usr/lib/x86_64-linux-gnu/icinga2/libremote.so
#7  0x00007f99d30fc2f9 in boost::detail::shared_count::~shared_count() () from /usr/lib/x86_64-linux-gnu/icinga2/libremote.so
#8  0x00007f99d30fc62a in boost::detail::function::functor_manager const&>, boost::_bi::list2 >, boost::_bi::value > > > >::manage(boost::detail::function::function_buffer const&, boost::detail::function::function_buffer&, boost::detail::function::functor_manager_operation_type) () from /usr/lib/x86_64-linux-gnu/icinga2/libremote.so
#9  0x00007f99d7fbd84f in std::deque >::~deque() () from /usr/lib/x86_64-linux-gnu/icinga2/libbase.so
#10 0x00007f99d7fb851b in icinga::WorkQueue::WorkerThreadProc() () from /usr/lib/x86_64-linux-gnu/icinga2/libbase.so
#11 0x00007f99d6ef1ce9 in thread_proxy () from /usr/lib/libboost_thread.so.1.46.1
#12 0x00007f99d8473e9a in start_thread (arg=0x7f99d2526700) at pthread_create.c:308
#13 0x00007f99d73f63fd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:112
#14 0x0000000000000000 in ?? ()

(gdb) up

#1  0x00007f99d8476065 in _L_lock_858 () from /lib/x86_64-linux-gnu/libpthread.so.0

(gdb) up

#2  0x00007f99d8475eba in __pthread_mutex_lock (mutex=0x7f99b03fc020) at pthread_mutex_lock.c:61

61 pthread_mutex_lock.c: No such file or directory.
(gdb) p *mutex
$2 = {_data = {lock = 2,count = 0,owner = 8362,nusers = 1,kind = 0,spins = 0,list = {prev = 0x0,_next = 0x0}},
_size = "\002\000\000\000\000\000\000\000\252 \000\000\001", '\000' <repeats 26 times>,_align = 2}
(gdb)

Apparently the WQ thread tried to recursively lock a mutex which it already held.

icinga-migration · 2014-08-05T07:36:28Z

Updated by gbeutner on 2014-08-05 07:36:28 +00:00

Relates set to 6621

icinga-migration · 2014-08-05T08:25:57Z

Updated by gbeutner on 2014-08-05 08:25:57 +00:00

Status changed from Assigned to Resolved

Please retest again. My own cluster seems to be working properly again.

icinga-migration · 2014-08-05T13:29:55Z

Updated by ccesario on 2014-08-05 13:29:55 +00:00

No more error here too.

After upgrade to version v2.0.1-85-g53d3a4a

icinga-migration · 2014-08-05T15:59:39Z

Updated by TechIsCool on 2014-08-05 15:59:39 +00:00

still errors for me but its not stopping replication or config migration.

Master Restart of Icinga2 service

Error on Secondary

[2014-08-05 08:50:35 -0700] critical/TlsStream: SSL_read() failed with code 0, "error:00000000:lib(0):func(0):reason(0)"
[2014-08-05 08:50:35 -0700] warning/ApiClient: Error while reading JSON-RPC message for identity 'vitals': /home/administrator/downloads/2.0.1-8-5-14/icinga2/lib/base/tlsstream.cpp(180): Throw in function virtual size_t icinga::TlsStream::Read(void*, size_t)
Dynamic exception type: boost::exception_detail::clone_impl
std::exception::what: std::exception
[icinga::StackTrace*] =
        (0) libbase.so: void boost::throw_exception(icinga::openssl_error const&) (+0xbe) [0x2b578535678e]
        (1) libbase.so: void boost::exception_detail::throw_exception_(icinga::openssl_error const&, char const*, char const*, int) (+0x40) [0x2b5785356850]
        (2) libbase.so: icinga::TlsStream::Read(void*, unsigned long) (+0x2ab) [0x2b578535504b]
        (3) libbase.so: icinga::NetString::ReadStringFromStream(boost::shared_ptr const&, icinga::String*) (+0x4b) [0x2b578531e19b]
        (4) libremote.so: icinga::JsonRpc::ReadMessage(boost::shared_ptr const&) (+0x3e) [0x2b578a826a8e]
        (5) libremote.so: icinga::ApiClient::ProcessMessage() (+0x3b) [0x2b578a7efc5b]
        (6) libremote.so: icinga::ApiClient::MessageThreadProc() (+0x58) [0x2b578a7f0b48]
        (7) libboost_thread.so.1.54.0:  (+0xba4a) [0x2b57863d3a4a]
        (8) libpthread.so.0:  (+0x8182) [0x2b578504a182]
        (9) libc.so.6: clone (+0x6d) [0x2b5785ef930d]


[icinga::ContextTrace*] =

0, "error:00000000:lib(0):func(0):reason(0)"[boost::errinfo_api_function_*] = SSL_read

[2014-08-05 08:50:35 -0700] information/ApiListener: New client connection for identity 'vitals'
[2014-08-05 08:50:38 -0700] critical/TlsStream: SSL_write() failed with code 0, "error:00000000:lib(0):func(0):reason(0)"
[2014-08-05 08:50:38 -0700] warning/ApiClient: Error while sending JSON-RPC message for identity 'vitals'
[2014-08-05 08:50:38 -0700] warning/ApiClient: API client disconnected for identity 'vitals'

Error on Master

[2014-08-05 08:44:48 -0700] information/ApiListener: New client connection for identity 'abydos'
[2014-08-05 08:44:48 -0700] warning/ApiListener: Ignoring config update. 'api' does not accept config.
[2014-08-05 08:49:41 -0700] information/DynamicObject: Dumping program state to file '/var/lib/icinga2/icinga2.state'
[2014-08-05 08:50:35 -0700] information/Application: Shutting down Icinga...
[2014-08-05 08:50:35 -0700] information/CheckerComponent: Checker stopped.
[2014-08-05 08:50:35 -0700] information/LivestatusListener: Created unix socket in '/run/icinga2/cmd/livestatus'.
[2014-08-05 08:50:35 -0700] information/ConfigItem: Activated all objects.
[2014-08-05 08:50:35 -0700] information/ApiListener: New client connection for identity 'abydos'
[2014-08-05 08:50:35 -0700] warning/ApiListener: Ignoring config update. 'api' does not accept config.

Secondary Restart of Icinga2 service

error on Master

[2014-08-05 08:52:21 -0700] critical/TlsStream: SSL_read() failed with code 0, "error:00000000:lib(0):func(0):reason(0)"
[2014-08-05 08:52:21 -0700] warning/ApiClient: Error while reading JSON-RPC message for identity 'abydos': /home/administrator/downloads/2.0.1-8-5-14/icinga2/lib/base/tlsstream.cpp(180): Throw in function virtual size_t icinga::TlsStream::Read(void*, size_t)
Dynamic exception type: boost::exception_detail::clone_impl
std::exception::what: std::exception
[icinga::StackTrace*] =
        (0) libbase.so: void boost::throw_exception(icinga::openssl_error const&) (+0xbe) [0x7f543f42678e]
        (1) libbase.so: void boost::exception_detail::throw_exception_(icinga::openssl_error const&, char const*, char const*, int) (+0x40) [0x7f543f426850]
        (2) libbase.so: icinga::TlsStream::Read(void*, unsigned long) (+0x2ab) [0x7f543f42504b]
        (3) libbase.so: icinga::NetString::ReadStringFromStream(boost::shared_ptr const&, icinga::String*) (+0x4b) [0x7f543f3ee19b]
        (4) libremote.so: icinga::JsonRpc::ReadMessage(boost::shared_ptr const&) (+0x3e) [0x7f5439f3ba8e]
        (5) libremote.so: icinga::ApiClient::ProcessMessage() (+0x3b) [0x7f5439f04c5b]
        (6) libremote.so: icinga::ApiClient::MessageThreadProc() (+0x58) [0x7f5439f05b48]
        (7) libboost_thread.so.1.54.0:  (+0xba4a) [0x7f543e35ea4a]
        (8) libpthread.so.0:  (+0x8182) [0x7f543f6d9182]
        (9) libc.so.6: clone (+0x6d) [0x7f543e86830d]


[icinga::ContextTrace*] =

0, "error:00000000:lib(0):func(0):reason(0)"[boost::errinfo_api_function_*] = SSL_read

[2014-08-05 08:52:21 -0700] information/ApiListener: New client connection for identity 'abydos'
[2014-08-05 08:52:21 -0700] warning/ApiListener: Ignoring config update. 'api' does not accept config.
[2014-08-05 08:52:24 -0700] critical/TlsStream: SSL_write() failed with code 0, "error:00000000:lib(0):func(0):reason(0)"
[2014-08-05 08:52:24 -0700] warning/ApiClient: Error while sending JSON-RPC message for identity 'abydos'
[2014-08-05 08:52:24 -0700] warning/ApiClient: API client disconnected for identity 'abydos'
[2014-08-05 08:52:24 -0700] critical/TlsStream: SSL_write() failed with code 0, "error:140D00CF:SSL routines:SSL_write:protocol is shutdown"
[2014-08-05 08:52:24 -0700] warning/ApiClient: Error while sending JSON-RPC message for identity 'abydos'
[2014-08-05 08:52:24 -0700] warning/ApiClient: API client disconnected for identity 'abydos'

Error on Secondary

[2014-08-05 08:50:35 -0700] information/ApiListener: New client connection for identity 'vitals'
[2014-08-05 08:50:38 -0700] critical/TlsStream: SSL_write() failed with code 0, "error:00000000:lib(0):func(0):reason(0)"
[2014-08-05 08:50:38 -0700] warning/ApiClient: Error while sending JSON-RPC message for identity 'vitals'
[2014-08-05 08:50:38 -0700] warning/ApiClient: API client disconnected for identity 'vitals'
[2014-08-05 08:52:21 -0700] information/Application: Shutting down Icinga...
[2014-08-05 08:52:21 -0700] information/CheckerComponent: Checker stopped.
[2014-08-05 08:52:21 -0700] information/ConfigItem: Activated all objects.
[2014-08-05 08:52:21 -0700] information/ApiListener: New client connection for identity 'vitals'

icinga-migration closed this as completed Aug 5, 2014

icinga-migration added bug Something isn't working area/distributed Distributed monitoring (master, satellites, clients) labels Jan 17, 2017

icinga-migration added this to the 2.0.2 milestone Jan 17, 2017

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[dev.icinga.com #6724] TLS Connections still unstable in 2.0.1 #1800

[dev.icinga.com #6724] TLS Connections still unstable in 2.0.1 #1800

icinga-migration commented Jul 15, 2014

icinga-migration commented Aug 1, 2014

icinga-migration commented Aug 1, 2014

icinga-migration commented Aug 1, 2014

icinga-migration commented Aug 1, 2014

icinga-migration commented Aug 1, 2014

icinga-migration commented Aug 1, 2014

icinga-migration commented Aug 1, 2014

icinga-migration commented Aug 3, 2014

icinga-migration commented Aug 4, 2014

icinga-migration commented Aug 4, 2014

icinga-migration commented Aug 5, 2014

icinga-migration commented Aug 5, 2014

icinga-migration commented Aug 5, 2014

icinga-migration commented Aug 5, 2014

[dev.icinga.com #6724] TLS Connections still unstable in 2.0.1 #1800

[dev.icinga.com #6724] TLS Connections still unstable in 2.0.1 #1800

Comments

icinga-migration commented Jul 15, 2014

icinga-migration commented Aug 1, 2014

icinga-migration commented Aug 1, 2014

icinga-migration commented Aug 1, 2014

icinga-migration commented Aug 1, 2014

icinga-migration commented Aug 1, 2014

icinga-migration commented Aug 1, 2014

icinga-migration commented Aug 1, 2014

icinga-migration commented Aug 3, 2014

icinga-migration commented Aug 4, 2014

icinga-migration commented Aug 4, 2014

icinga-migration commented Aug 5, 2014

icinga-migration commented Aug 5, 2014

icinga-migration commented Aug 5, 2014

icinga-migration commented Aug 5, 2014

Master Restart of Icinga2 service

Secondary Restart of Icinga2 service