How to use database ds with sqlite3?

[reszelaz]

Hi PyTango experts!

I was trying to use the Tango database with sqlite3 both with python2 and python3.

I try to launch the database ds with the following command:

zreszela@pc255:~> python -m tango.databaseds.database --port=30000 --logging_level=2 2
Traceback (most recent call last):
  File "/usr/lib/python2.7/runpy.py", line 174, in _run_module_as_main
    "__main__", fname, loader, pkg_name)
  File "/usr/lib/python2.7/runpy.py", line 72, in _run_code
    exec code in run_globals
  File "/home/zreszela/workspace/pytango/tango/databaseds/database.py", line 1880, in <module>
    main()
  File "/home/zreszela/workspace/pytango/tango/databaseds/database.py", line 1765, in main
    get_plugin(options.db_access)
  File "/home/zreszela/workspace/pytango/tango/databaseds/database.py", line 141, in get_plugin
    return __import__(fullname, None, None, fullname)
ImportError: No module named None.sqlite3

For python2 I must patch this to overcome this error:

diff --git a/tango/databaseds/database.py b/tango/databaseds/database.py
index 6d6aa53..df571bb 100644
--- a/tango/databaseds/database.py
+++ b/tango/databaseds/database.py
@@ -137,7 +137,7 @@ def update_timing_stats(dev, time_before, time_after, cmd_name):
 
 
 def get_plugin(name):
-    fullname = '%s.%s' % (db_access.__package__, name)
+    fullname = '%s.%s' % (db_access.__name__, name)
     return __import__(fullname, None, None, fullname)

Then the server hangs on "Initializing database...":

zreszela@pc255:~> python -m tango.databaseds.database --port=30000 --logging_level=2 2
Dummy-1        DEBUG    2019-03-15 12:52:29,963 tango: server loop started
MainThread     DEBUG    2019-03-15 12:52:29,966 sys/database/2: In init_device()
Thread-2       INFO     2019-03-15 12:52:29,967 Sqlite3Database: Initializing database...
submit get_cursor

I added these prints which points to some problem with concurrent.future.ThreadPoolExecutor. When I don't submit the get_cursor method to the executor and execute it directly it works. But then the next submits to the executor hangs...

diff --git a/tango/databaseds/db_access/sqlite3.py b/tango/databaseds/db_access/sqlite3.py
index 4b809b0..5be9b3e 100644
--- a/tango/databaseds/db_access/sqlite3.py
+++ b/tango/databaseds/db_access/sqlite3.py
@@ -60,7 +60,9 @@ def use_cursor(f):
         has_cursor = 'cursor' in kwargs
         cursor = kwargs.pop('cursor', None)
         if not has_cursor:
+            print("submit get_cursor")
             cursor = Executor.submit(self.get_cursor).result()
+            print("after get_cursor")
         self.cursor = cursor
         try:
             ret = Executor.submit(f, *args, **kwargs).result()

Certainly I must be doing something wrong so any kind of tip/guidance would be appreciated. Thanks!
I take a profit of this post to ask if someone had tried this on Windows? Finally I would like to use it on windows to avoid the necessity to install MySQL and tango :)

Software versions:

• PyTango from develop branch insalled with pip install -e --user
• Tango 9.2.5a with some ALBA patches (I don't think that these are relevant here)

[ajoubertza]

Hi @reszelaz

I haven't used this database server before, so might not be much help. I also don't know of the quality or usefulness of this Python-based database server. It doesn't seem to be actively maintained.

Before I could run your commands above, I also had to pip install futures and gevent. After that, I got the same symptoms you do.

One step forward is to get a copy of a database, instead of letting the code try to create one for you. You can get a copy (maybe old format), from here: https://github.com/ALBA-Synchrotron/pydatabaseds/blob/master/databaseds.8.1.6/empty_database.db. Just copy it into your current folder as tango_database.db. Maybe that repo has some other ideas you could look at, although it says it isn't maintained either. For example, it doesn't include support for pipes.

$ python -m tango.databaseds.database --port=30000 --logging_level=2 2
Dummy-1        DEBUG    2019-03-19 20:01:37,153 tango: server loop started
MainThread     DEBUG    2019-03-19 20:01:37,156 sys/database/2: In init_device()
ThreadPoolExecutor-0_0 INFO     2019-03-19 20:01:37,157 Sqlite3Database: Initializing database...
MainThread     DEBUG    2019-03-19 20:01:37,159 root: post_init_cb()
MainThread     DEBUG    2019-03-19 20:01:37,159 sys/database/2: In DbExportDevice()
ThreadPoolExecutor-0_0 INFO     2019-03-19 20:01:37,159 Sqlite3Database: export_device(dev_name=dserver/databaseds/2, host=vagrant.vm, pid=4899, version=5)
ThreadPoolExecutor-0_0 INFO     2019-03-19 20:01:37,160 Sqlite3Database: export_device(IOR=IOR:010000001700000049444c3a54616e676f2f4465766963655f353a312e3000000100000000000000a7000000010102000a00000031302e302e322e313500307514000000647365727665722f646174616261736564732f320300000000000000080000000100000000545441010000001c00000001000000010001000100000001000105090101000100000009010100025454413b000000010000000800000076616772616e7400270000002f746d702f6f6d6e692d76616772616e742f3030303030343839392d3135353330323536393700)
MainThread     DEBUG    2019-03-19 20:01:37,164 sys/database/2: In DbExportDevice()
ThreadPoolExecutor-0_0 INFO     2019-03-19 20:01:37,164 Sqlite3Database: export_device(dev_name=sys/database/2, host=vagrant.vm, pid=4899, version=5)
ThreadPoolExecutor-0_0 INFO     2019-03-19 20:01:37,164 Sqlite3Database: export_device(IOR=IOR:010000001700000049444c3a54616e676f2f4465766963655f353a312e30000001000000000000009b000000010102000a00000031302e302e322e31350030750800000064617461626173650300000000000000080000000100000000545441010000001c00000001000000010001000100000001000105090101000100000009010100025454413b000000010000000800000076616772616e7400270000002f746d702f6f6d6e692d76616772616e742f3030303030343839392d3135353330323536393700)
Ready to accept request

[reszelaz]

Thanks @ajoubertza for the hint!

I also don't know of the quality or usefulness of this Python-based database server. It doesn't seem to be actively maintained.

I think it should be pretty usable. At least in the bliss project they run this code in an embedded mode.

I think the problem could come from the use of concurrent.futures.ThreadPoolExecutor. I saw that the databaseds is run with green_mode=GreenMode.Gevent. I totally don't know the gevent library, but I suspect that in this case we should use gevent.threadpool.ThreadPoolExecutor, could it be?
I tried this and then I get a bunch of other errors, if someone is interested I could post them..

Anyway, I tried to set the green_mode=GreenMode.Synchronous but also withuout success.

[tiagocoutinho]

Motivated by this issue and the fact that, together with @vxgmichel, we informally agreed a long time ago that pytango database should be a separate project from tango, we (me and @jairomoldes) have created a pytango-db repository.

Your issue should already be solved in this project.

If the @tango-controls/pytango-developers agree:

• mark this module deprecated in pytango or even remove it directly in the next release
• move pytango-db to the official tango-controls group and associate the PyTango developers team to it

We would also need to synchronize with @sergirubio to see if we can meet his needs so he doesn't need to maintain a separate project.