Splitting pymodaq in subpackages: towards pymodaq 5.0.0

[seb5g]

Since I introduced data management in pymodaq 4.0.0 and completing its features, I was more and more thinking that it would be great to be able to use those data objects independently of pymodaq (especially because it can take a while to import pymodaq because of the plugins).

So I went into this, but while trying to extract things specific to data from other bits, I started to realize that both data, GUI and higher features all requires basic functionalities such as logging, configuration management, mathematical tools... So I needed to first extract the basic things that are used everywhere. That is making a lightweight package I called pymodaq_utils.

From there, it was easier to put the things exclusive to data management together. But one of the nice features of my new data management is the interplay between python Data objects and their hdf5 file couter-part. So obviously I included the necessary things for this in the data package that I called pymodaq_data. But then I bumped into another issue. The other nice thing about data management is the ability to plot data in high level data viewers which are GUI using Qt (see https://github.com/PyMoDAQ/notebooks/blob/main/notebooks/plotting_data.ipynb). BUT data management doesn't have to have this explicitly to work, and some people would maybe just prefer use matplotlib and nothing from Qt. Moreover browsing data is done using the H5Browser but it is not mandatory either... So I removed everything from Qt from the pymodaq_data package and moved them into a dedicated pymodaq_gui one. In there are defined the qt data viewers, h5 browser, navigator.... This package can be used to build custom GUI independent from pymodaq instruments and data acquisition.

Finally I simplified the initial pymodaq package to have only high level module in there such as the control modules, extensions and some bits specific to this.

So to summarize, there could be:

• pymodaq_utils

used by:

• pymodaq_data: specific for data management/saving/loading
• pymodaq_gui: specific to create GUI and giving access to the one we know from pymodaq

everything is then used by

• pymodaq

I'd like to know what users would think of this as it makes advanced usage a bit more complex as the developer has to deal with 4 packages. It is fully transparent for the pymodaq user. It is much simpler for the user interested only in data management.

I tried also to make it transparent on the plugins side (by keeping some modules in pymodaq only dedicated to importing things from the other packages) but I expect some bugs still that would need at the end to work a bit again on the plugins to adapt them. By the way I found a solution to use only pyproject.toml in the plugins while keeping the use of the plugin_info.toml simpler file.

I created all this by:

• pymodaq_utils: forked from pymodaq (to keep history) but renamed. I released version 0.0.1 (and 0.0.2) on pypi already

• pymodaq_data : forked from pymodaq (to keep history) but renamed. I released version 0.0.1 (and 0.0.2) on pypi already

• pymodaq_gui : forked from pymodaq (to keep history) but renamed. I released version 0.0.1 on pypi already

• pymodaq: created the 5.0.x_dev branch using the subpackages. Published as 5.0.0 on pypi (but somehow hidden from pip otherwise explicitly required)

• pymodaq_plugins_mock: created the 5.0.x_dev branch

So I'd like to here, you users and developers, your opinion about all this and eventually try it out. THere is a branch in the data plotting notebook : (see https://github.com/PyMoDAQ/notebooks/blob/main/notebooks/plotting_data.ipynb) dedicated to try this branch pmd_5.x.y

[seb5g]

And all tests have also been split in their dedicated package with all of them passing! I also ran a DAQ_Scan and explored the data so no obvious bug so far...

[BenediktBurger]

I think it is a good idea to split a large project into several smaller ones, especially separating GUI-less logic (if it is useful without the GUI) from the GUI itself.

yaq also is separated in different parts.

In order to facilitate working with these separate repositories, you could create a "meta-repository", which contains all the other (main) repositories as submodules.
That allows to work on all the interrelated parts (utils, GUI, data) in one large project folder, pulling changes of all project at once, but commiting to the several repositories individually.
In fact, yaq has such a meta repository as well.

[seb5g]

just read the submodules documentation...sounds tricky not to mess with it... I have the feeling that using pycharm for edition and git stuff and having all the separated packages loaded as project would do the job nicely (its how I did it so far)

[quantumm]

Hi everyone!

In general, I think it is a good idea. Even if I consider not so competent to grasp the full consequences of such a modification. If you think it is a good choice to improve the stability and readability of the code on the long run, go for it!

I still have a few concerns:

1. What will be the consequences about the standard installation procedure? It should be kept as simple and short as possible if we want to get more users. Ideally it should be "pip install pymodaq". Which is btw still not the case because pyqt5 is not declared as a dependency. I already asked you that but I forgot the reason sorry ;)
2. What will be the consequences for the plugin maintainers? Keeping the plugins updated for the maintainers is not a work to neglect (even if it is of course not an amount comparable to your work). Those maintainers are sometimes people not directly involved in the project, and they should not get demotivated to pursue their important mission by having to update their repositories every year. In brief: let's be careful to not make major release to often if we want to keep the users and maintainers!

As for the problem of dealing with multiple packages for developers, as mentioned by @BenediktBurger, I am not so worried.

[seb5g]

I still have a few concerns:

1. What will be the consequences about the standard installation procedure? It should be kept as simple and short as possible if we want to get more users. Ideally it should be "pip install pymodaq". 

The 3 other packages needed: pymodaq_utils, pymodaq_data and pymodaq_gui are in the pymodaq package dependency. So for most users a pip install pymodaq will do the job transparently, only those who want for instance to just analyse the data would need to do a pip install pymodaq_data or only a pip install pymodaq_gui if they need the data viewers + the data management

Which is btw still not the case because pyqt5 is not declared as a dependency. I already asked you that but I forgot the reason sorry ;)

it was left out of the dependencies to let the final user choose the particular back-end they want for the GUI stuff, in particular because Qt5 has various licensing. However for an non lucrative use and no direct incorporation of the source code, I think we can safely add pyqt5 in the dependency. Do you want to do the PR?

The 3 other packages needed: utils, data and gui are in the pymodaq package dependency. So for most users a pip install pymodaq

2. What will be the consequences for the plugin maintainers? Keeping the plugins updated for the maintainers is not a work to neglect (even if it is of course not an amount comparable to your work). Those maintainers are sometimes people not directly involved in the project, and they should not get demotivated to pursue their important mission by having to update their repositories every year. In brief: let's be careful to not make major release to often if we want to keep the users and maintainers!

As I wrote it, pymodaq5 is still fully compatible with current plugins. I left or added in the pymodaq5 packages modules importing the new submodules. So probably all the things imported from pymodaq within plugins are still directly importable, they will jut fire a deprecation message warning in the log. That of course has to be properly tested for existing plugins before releasing pymodaq5.

[loicguilmard]

Should i try to generate an exhaustive list of the displacement of classes and functions ?
from pymodaq.... import => from pymodaq_package... import

[seb5g]

that would actually help a lot and give a clearer view of the whole thing!

[loicguilmard]

Also sometimes it makes sense to import a whole submodule like viewers in the DAQ_Viewer ... :)

What about the "from pymodaq_gui.utils.widgets.qled import QLED" or "from pymodaq_gui.utils.widgets import QLED" ?
Shall we use the first, the second form or it doesn't matter ?

[BenediktBurger]

That could be nice but sometimes things are imported in a module and not used because:

* It will be imported from there by other modules

I already stumbled several times over broken imports due to this nested import (importing something, which is imported by another module). Therefore, I suggest to not do this kind of imports as far as possible.

* It will trigger some actions like registration of parameters, registration of factories.

In those cases, it is good to add comments

[BenediktBurger]

What about the "from pymodaq_gui.utils.widgets.qled import QLED" or "from pymodaq_gui.utils.widgets import QLED" ?
Shall we use the first, the second form or it doesn't matter ?

From the importing side, it does not matter.
It depends on the planned architecture.
Importing it from the module directly (i.e. from widgets.qled) makes it independent from the import line in the widgets module.
I personally would import it from the module directly, but @seb5g has to decide that.

[seb5g]

this is the content of the widgets module:

from .qled import QLED
from .push import PushButtonIcon, EditPush, EditPushRel
from .spinbox import QSpinBox_ro, SpinBox
from .table import TableView, TableModel
from .label import LabelWithFont

it exposes all widgets directly from there while the actual code is within submodules. I thought it would make things easier/clearer

[seb5g]

I already stumbled several times over broken imports due to this nested import (importing something, which is imported by another module). Therefore, I suggest to not do this kind of imports as far as possible.

yes I agree except when it makes things easier to maintain. For instance, I import a few things from main pymodaq modules in the init of the plugins package. These things are then imported a lot within the package. If the import from the main module changes, I only have to modify it once...

[malik-irain]

this discussion has been validated for the next release