DataBaseds crashes #61
Comments
|
Hi @Diego91RA , the "debugger aborting error" might be a bug in the mariaDB version you are using?: https://jira.mariadb.org/browse/MDEV-8316 What MariaDB version are you using? |
|
Hi. On another VM MariaDB was built from sources one year ago, version 10.3.12, but Tango is 9.2.5a, and I don't have such problem. Thanks. |
|
I've updated MariaDB to 10.4.11 from official repos https://downloads.mariadb.org/mariadb/repositories/ And still have exactly the same error :( |
|
Do you have a core file? |
|
Here is gdb backtrace |
|
Thank you very much for the backtrace. |
|
Oops, VM I've shared before was with Debian 10. Here is for Debian 9 https://yadi.sk/d/E4zF1XHcZ9VAlw |
|
You wrote the problem is occurring after about 5+ hours. |
|
No, just executed |
|
I had a look at the logs in /var/log/mysql in the VM you provided and I saw some messages like the following: This seems to indicate that the MariaDB server detected a crash yesterday before you sent the VM. |
|
How much RAM and CPUs did you allocate for your VM? |
|
Hi, I then restarted the mysql (mariadb) daemon and the DataBaseds. So it looks like the crash occurs when the connection between the DataBaseDs and mysql is idle for too long and mysql has closed the idle connection? |
|
Hello. I see lines in the console output for DatabaseDS during the idle:
Does it mean that DS stores data to the DB every 30 minutes? That should be enough for MySQL to keep the connection alive, if default timeout is 8 hours. |
This line indicates that it does store data only if needed. I think when nothing happens as in your use case, there is no new sub device to register, so no query to the DB. |
|
Ok. Btw, is it bug or feature? |
Having a Starter running and a TangoTest started once on your host and stopped should be enough.
I have the feeling it is a bug in libmariadbclient library. |
|
I confirm this bug is a libmariadb2 bug. I reproduced the crash and I got the following backtrace with gdb: I then searched on the web for libmariadb bugs related to The reply from the upstream author is interesting:
I recompiled libmariadb.so.2 with It looks like there is no libmariadb3 package in Debian Stretch so you can recompile manually libmariadb.so.2 using |
|
I did not had any issue on debian unstable with libmariadb3 1:10.3.21-2 (https://packages.debian.org/sid/libmariadb3) and I had the VM instance running for three days with little interaction. |
|
According to the libmariadb upstream author, the problem is not present in libmariadb3 |
|
@bourtemb Should we add a configure check for compatible libmardiadb versions? |
|
Debian 9 has only libmariadb2 in the official repo by default. I think version check and an advice to install MariaDB from their sources will be useful. |
|
I did a test with libmariadbclient18 and it seems like the problem is not present in this version. The problem will not be present in libmariadb3 because all these debug macros which were causing this kind of crashes were removed. |
|
Well, DS is working fine now for 2+ hours, even with wait_timeout and interactive_timeout set to 60 seconds. Thanks for your help. |
|
@bourtemb Will do. |
Reports [1] indicate that versions 2 of the mariadb client package trigger DatabaseDS crashes when the database connection timeout is reached. In order to warn users about potential issues we now about a warning message in this case. [1]: tango-controls#61 The output on a stock debian stretch system looks like Configuration (tango): Source code location: . Version: 9.3.4-rc1 Compiler: gcc,g++ OMNIORB PATH: /usr/local OMNIORB VERSION: ZMQ PATH: /usr ZMQ VERSION: 4.2.1 JAVA PATH: /usr/bin/java JAVA VERSION: 1.8.0_131 Database MARIADB CLIENT LIB: -lmariadb CLIENT VERSION: 2.3.2 (Warning! potentially buggy version) VERSION: 10.1.41-MariaDB CONNECTION: OK build: libraries: yes java application: yes access control server: yes database server: yes database schema create: yes Please check whether the configuration I detected matches what you would like to have.
Reports [1] indicate that versions 2 of the mariadb client package trigger DatabaseDS crashes when the database connection timeout is reached. In order to warn users about potential issues we now about a warning message in this case. [1]: tango-controls#61 The output on a stock debian stretch system looks like Configuration (tango): Source code location: . Version: 9.3.4-rc1 Compiler: gcc,g++ OMNIORB PATH: /usr/local OMNIORB VERSION: ZMQ PATH: /usr ZMQ VERSION: 4.2.1 JAVA PATH: /usr/bin/java JAVA VERSION: 1.8.0_131 Database MARIADB CLIENT LIB: -lmariadb CLIENT VERSION: 2.3.2 (Warning! potentially buggy version) VERSION: 10.1.41-MariaDB CONNECTION: OK build: libraries: yes java application: yes access control server: yes database server: yes database schema create: yes notice "Warning .." message. On debian unstable it looks better Configuration (tango): Source code location: . Version: 9.3.4-rc2 Compiler: gcc,g++ OMNIORB PATH: /usr OMNIORB VERSION: "4.2.2" ZMQ PATH: /usr ZMQ VERSION: 4.3.2 JAVA PATH: /usr/bin/java JAVA VERSION: 1.8.0_242-ea Database MARIADB CLIENT LIB: -lmariadb CLIENT VERSION: 3.1.6 VERSION: 10.3.21-MariaDB CONNECTION: OK build: libraries: yes java application: yes access control server: yes database server: yes database schema create: yes
Hello again.
I've noticed that my TangoDB proccess crashes after some time, about 5+ hours.
Debian 9.11, Tango 9.3.4-rc2, omniORB 4.2.3, two different VMs
Here are last lines of output for
/usr/local/bin/DataBaseds 2 -ORBendPoint giop:tcp::10000 -v5after launching Jive:Happy holidays, sorry for disturbing :)
The text was updated successfully, but these errors were encountered: