This repository has been archived by the owner on Nov 3, 2021. It is now read-only.
Allow SWMR mode and avoid flock problems with NXscanH5_FileRecorder #1457
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sardana H5 recorder opens and closes file at every scan. This prevents other applications e.g. PyMCA, silx to open them for reading in between the scans and keep them opened while next scan is being launched. This is described in sardana-org#1124. Add test to demonstrate this issue.
Sardana H5 recorder opens and closes file at every scan. This prevents other applications e.g. PyMCA, silx to open them for reading in between the scans and keep them opened while next scan is being launched. This is described in sardana-org#1124. - Add singleton handler of H5 files to keep them opened - In NXscanH5_FileRecorder open the file only if it is not already opened - Mode "populate instrument info" and "create NXData" to startRecordList instead of endRecordList (necessary by SWMR) - Adapt test to avoid failure
There are use cases when we need to retrieve the environment variable even if the macro had been stopped/aborterd e.g. h5_write_session context manager. Add _getEnv non-mAPI method to Macro class.
h5storage macros are used to control the h5 write sessions.
cpascual
reviewed
Feb 4, 2021
Importing the h5util module registers an atexit callback. This is an unwanted secondary effect e.g. when building docs. Register and unregister cleanup when opening the fist file and closing the last file.
Fixed Merge Conflicts in: src/sardana/macroserver/recorders/test/test_h5storage.py
|
Strange... I cannot reproduce the problem Maybe I am doing something different... or maybe it is that the problem is not triggered with a more recent stack. Here is what I do: Prepare environmentInstall all dependencies from conda-forge ┬─[cpascual@pt168:~]─[18:33:03]
╰─>$ conda create -n sdn-1457 -c conda-forge taurus silx lxml h5py itango pytango
(...)
The following NEW packages will be INSTALLED:
_libgcc_mutex conda-forge/linux-64::_libgcc_mutex-0.1-conda_forge
_openmp_mutex conda-forge/linux-64::_openmp_mutex-4.5-1_gnu
backcall conda-forge/noarch::backcall-0.2.0-pyh9f0ad1d_0
backports conda-forge/noarch::backports-1.0-py_2
backports.functoo~ conda-forge/noarch::backports.functools_lru_cache-1.6.1-py_0
boost conda-forge/linux-64::boost-1.74.0-py39h5472131_3
boost-cpp conda-forge/linux-64::boost-cpp-1.74.0-hc6e9bd1_2
bzip2 conda-forge/linux-64::bzip2-1.0.8-h7f98852_4
c-ares conda-forge/linux-64::c-ares-1.17.1-h7f98852_1
ca-certificates conda-forge/linux-64::ca-certificates-2020.12.5-ha878542_0
cached-property conda-forge/noarch::cached-property-1.5.2-hd8ed1ab_1
cached_property conda-forge/noarch::cached_property-1.5.2-pyha770c72_1
certifi conda-forge/linux-64::certifi-2020.12.5-py39hf3d152e_1
click conda-forge/noarch::click-7.1.2-pyh9f0ad1d_0
cpptango conda-forge/linux-64::cpptango-9.3.4-hf7cf922_3
cycler conda-forge/noarch::cycler-0.10.0-py_2
dbus conda-forge/linux-64::dbus-1.13.6-hfdff14a_1
decorator conda-forge/noarch::decorator-4.4.2-py_0
expat conda-forge/linux-64::expat-2.2.10-h9c3ff4c_0
fabio conda-forge/linux-64::fabio-0.11.0-py39hce5d2b2_1
fontconfig conda-forge/linux-64::fontconfig-2.13.1-hba837de_1004
freetype conda-forge/linux-64::freetype-2.10.4-h0708190_1
future conda-forge/linux-64::future-0.18.2-py39hf3d152e_3
gettext conda-forge/linux-64::gettext-0.19.8.1-h0b5b191_1005
glib conda-forge/linux-64::glib-2.68.0-h9c3ff4c_1
glib-tools conda-forge/linux-64::glib-tools-2.68.0-h9c3ff4c_1
glymur conda-forge/noarch::glymur-0.9.3-pyhd8ed1ab_0
gst-plugins-base conda-forge/linux-64::gst-plugins-base-1.18.4-h29181c9_0
gstreamer conda-forge/linux-64::gstreamer-1.18.4-h76c114f_0
guidata conda-forge/noarch::guidata-1.7.9-pyh9f0ad1d_0
guiqwt conda-forge/linux-64::guiqwt-3.0.5-py39h9cfe711_1
h5py conda-forge/linux-64::h5py-3.1.0-nompi_py39h25020de_100
hdf5 conda-forge/linux-64::hdf5-1.10.6-nompi_h6a2412b_1114
icu conda-forge/linux-64::icu-68.1-h58526e2_0
importlib-metadata conda-forge/linux-64::importlib-metadata-3.7.3-py39hf3d152e_0
importlib_metadata conda-forge/noarch::importlib_metadata-3.7.3-hd8ed1ab_0
importlib_resourc~ conda-forge/linux-64::importlib_resources-5.1.2-py39hf3d152e_0
ipykernel conda-forge/linux-64::ipykernel-5.5.0-py39hef51801_1
ipython conda-forge/linux-64::ipython-7.21.0-py39hef51801_0
ipython_genutils conda-forge/noarch::ipython_genutils-0.2.0-py_1
itango conda-forge/noarch::itango-0.1.8-pyhd8ed1ab_0
jedi conda-forge/linux-64::jedi-0.18.0-py39hf3d152e_2
jpeg conda-forge/linux-64::jpeg-9d-h36c2ea0_0
jupyter_client conda-forge/noarch::jupyter_client-6.1.12-pyhd8ed1ab_0
jupyter_core conda-forge/linux-64::jupyter_core-4.7.1-py39hf3d152e_0
kiwisolver conda-forge/linux-64::kiwisolver-1.3.1-py39h1a9c180_1
krb5 conda-forge/linux-64::krb5-1.17.2-h926e7f8_0
lcms2 conda-forge/linux-64::lcms2-2.12-hddcbb42_0
ld_impl_linux-64 conda-forge/linux-64::ld_impl_linux-64-2.35.1-hea4e1c9_2
libblas conda-forge/linux-64::libblas-3.9.0-8_openblas
libcblas conda-forge/linux-64::libcblas-3.9.0-8_openblas
libclang conda-forge/linux-64::libclang-11.1.0-default_ha53f305_0
libcurl conda-forge/linux-64::libcurl-7.75.0-hc4aaa36_0
libedit conda-forge/linux-64::libedit-3.1.20191231-he28a2e2_2
libev conda-forge/linux-64::libev-4.33-h516909a_1
libevent conda-forge/linux-64::libevent-2.1.10-hcdb4288_3
libffi conda-forge/linux-64::libffi-3.3-h58526e2_2
libgcc-ng conda-forge/linux-64::libgcc-ng-9.3.0-h2828fa1_18
libgfortran-ng conda-forge/linux-64::libgfortran-ng-9.3.0-hff62375_18
libgfortran5 conda-forge/linux-64::libgfortran5-9.3.0-hff62375_18
libglib conda-forge/linux-64::libglib-2.68.0-h3e27bee_1
libgomp conda-forge/linux-64::libgomp-9.3.0-h2828fa1_18
libiconv conda-forge/linux-64::libiconv-1.16-h516909a_0
liblapack conda-forge/linux-64::liblapack-3.9.0-8_openblas
libllvm11 conda-forge/linux-64::libllvm11-11.1.0-hf817b99_0
libnghttp2 conda-forge/linux-64::libnghttp2-1.43.0-h812cca2_0
libopenblas conda-forge/linux-64::libopenblas-0.3.12-pthreads_h4812303_1
libpng conda-forge/linux-64::libpng-1.6.37-h21135ba_2
libpq conda-forge/linux-64::libpq-13.1-hfd2b0eb_2
libsodium conda-forge/linux-64::libsodium-1.0.18-h36c2ea0_1
libssh2 conda-forge/linux-64::libssh2-1.9.0-ha56f1ee_6
libstdcxx-ng conda-forge/linux-64::libstdcxx-ng-9.3.0-h6de172a_18
libtiff conda-forge/linux-64::libtiff-4.2.0-hdc55705_0
libuuid conda-forge/linux-64::libuuid-2.32.1-h7f98852_1000
libwebp-base conda-forge/linux-64::libwebp-base-1.2.0-h7f98852_2
libxcb conda-forge/linux-64::libxcb-1.13-h7f98852_1003
libxkbcommon conda-forge/linux-64::libxkbcommon-1.0.3-he3ba5ed_0
libxml2 conda-forge/linux-64::libxml2-2.9.10-h72842e0_3
libxslt conda-forge/linux-64::libxslt-1.1.33-h15afd5d_2
lxml conda-forge/linux-64::lxml-4.6.2-py39h107f48f_1
lz4-c conda-forge/linux-64::lz4-c-1.9.3-h9c3ff4c_0
mako conda-forge/noarch::mako-1.1.4-pyh44b312d_0
markupsafe conda-forge/linux-64::markupsafe-1.1.1-py39h3811e60_3
matplotlib conda-forge/linux-64::matplotlib-3.3.4-py39hf3d152e_0
matplotlib-base conda-forge/linux-64::matplotlib-base-3.3.4-py39h2fa2bec_0
mysql-common conda-forge/linux-64::mysql-common-8.0.23-ha770c72_1
mysql-libs conda-forge/linux-64::mysql-libs-8.0.23-h935591d_1
ncurses conda-forge/linux-64::ncurses-6.2-h58526e2_4
nspr conda-forge/linux-64::nspr-4.30-h9c3ff4c_0
nss conda-forge/linux-64::nss-3.63-hb5efdd6_0
numpy conda-forge/linux-64::numpy-1.20.1-py39hdbf815f_0
olefile conda-forge/noarch::olefile-0.46-pyh9f0ad1d_1
omniorb conda-forge/linux-64::omniorb-4.2.4-py39hff7568b_2
openjpeg conda-forge/linux-64::openjpeg-2.4.0-hf7af979_0
openssl conda-forge/linux-64::openssl-1.1.1j-h7f98852_0
packaging conda-forge/noarch::packaging-20.9-pyh44b312d_0
parso conda-forge/noarch::parso-0.8.1-pyhd8ed1ab_0
pcre conda-forge/linux-64::pcre-8.44-he1b5a44_0
pexpect conda-forge/noarch::pexpect-4.8.0-pyh9f0ad1d_2
pickleshare conda-forge/noarch::pickleshare-0.7.5-py_1003
pillow conda-forge/linux-64::pillow-8.1.2-py39hf95b381_0
pint conda-forge/noarch::pint-0.16.1-py_0
pip conda-forge/noarch::pip-21.0.1-pyhd8ed1ab_0
ply conda-forge/noarch::ply-3.11-py_1
prompt-toolkit conda-forge/noarch::prompt-toolkit-3.0.17-pyha770c72_0
pthread-stubs conda-forge/linux-64::pthread-stubs-0.4-h36c2ea0_1001
ptyprocess conda-forge/noarch::ptyprocess-0.7.0-pyhd3deb0d_0
pygments conda-forge/noarch::pygments-2.8.1-pyhd8ed1ab_0
pyparsing conda-forge/noarch::pyparsing-2.4.7-pyh9f0ad1d_0
pyqt conda-forge/linux-64::pyqt-5.12.3-py39hf3d152e_7
pyqt-impl conda-forge/linux-64::pyqt-impl-5.12.3-py39h0fcd23e_7
pyqt5-sip conda-forge/linux-64::pyqt5-sip-4.19.18-py39he80948d_7
pyqtchart conda-forge/linux-64::pyqtchart-5.12-py39h0fcd23e_7
pyqtwebengine conda-forge/linux-64::pyqtwebengine-5.12.1-py39h0fcd23e_7
pytango conda-forge/linux-64::pytango-9.3.3-py39hbcdfc36_1
python conda-forge/linux-64::python-3.9.2-hffdb5ce_0_cpython
python-dateutil conda-forge/noarch::python-dateutil-2.8.1-py_0
python_abi conda-forge/linux-64::python_abi-3.9-1_cp39
pythonqwt conda-forge/noarch::pythonqwt-0.7.0-pyhac0dd68_0
pyzmq conda-forge/linux-64::pyzmq-22.0.3-py39h37b5a0c_1
qt conda-forge/linux-64::qt-5.12.9-hda022c4_4
qtconsole conda-forge/noarch::qtconsole-5.0.3-pyhd8ed1ab_0
qtpy conda-forge/noarch::qtpy-1.9.0-py_0
readline conda-forge/linux-64::readline-8.0-he28a2e2_2
scipy conda-forge/linux-64::scipy-1.6.1-py39hee8e79c_0
setuptools conda-forge/linux-64::setuptools-49.6.0-py39hf3d152e_3
silx conda-forge/noarch::silx-0.15.0-hd8ed1ab_0
silx-base conda-forge/linux-64::silx-base-0.15.0-py39hde0f152_0
sip conda-forge/linux-64::sip-4.19.25-py39he80948d_0
six conda-forge/noarch::six-1.15.0-pyh9f0ad1d_0
sqlite conda-forge/linux-64::sqlite-3.34.0-h74cdb3f_0
taurus conda-forge/noarch::taurus-4.7.0-hd8ed1ab_0
taurus-core conda-forge/noarch::taurus-core-4.7.0-pyhd8ed1ab_0
taurus-qt conda-forge/noarch::taurus-qt-4.7.0-pyhd8ed1ab_0
tk conda-forge/linux-64::tk-8.6.10-h21135ba_1
tornado conda-forge/linux-64::tornado-6.1-py39h3811e60_1
traitlets conda-forge/noarch::traitlets-5.0.5-py_0
tzdata conda-forge/noarch::tzdata-2021a-he74cb21_0
wcwidth conda-forge/noarch::wcwidth-0.2.5-pyh9f0ad1d_2
wheel conda-forge/noarch::wheel-0.36.2-pyhd3deb0d_0
xorg-libxau conda-forge/linux-64::xorg-libxau-1.0.9-h7f98852_0
xorg-libxdmcp conda-forge/linux-64::xorg-libxdmcp-1.1.3-h7f98852_0
xz conda-forge/linux-64::xz-5.2.5-h516909a_1
zeromq conda-forge/linux-64::zeromq-4.3.4-h9c3ff4c_0
zipp conda-forge/noarch::zipp-3.4.1-pyhd8ed1ab_0
zlib conda-forge/linux-64::zlib-1.2.11-h516909a_1010
zstd conda-forge/linux-64::zstd-1.4.9-ha95c52a_0
(...)
Executing transaction: done
┬─[cpascual@pt168:~]─[18:45:15]
╰─>$ conda activate sdn-1457
┬─[cpascual@pt168:~]─[18:45:19]
╰─>$ cd src/sardana (sdn-1457)
┬─[cpascual@pt168:~/src/sardana]─[18:45:36]─[G:develop=]
╰─>$ pip install -e . (sdn-1457)
(...)
Successfully installed sardanaRun Sardana:On a dfferent console, run the sardana server (a pristine new instance) ┬─[cpascual@pt168:~]─[18:33:03]
╰─>$ conda activate sdn-1457
┬─[cpascual@pt168:~]─[18:47:29]
╰─>$ Sardana kk-1457 (sdn-1457)
kk-1457 does not exist. Do you wish to create a new one (Y/n) ?
MainThread ERROR 2021-03-22 18:47:42,805 ModuleManager: Invalid module HklPseudoMotorController
MainThread WARNING 2021-03-22 18:47:46,185 pt168.cells.es:10000.Pool_kk-1457_1: Received elements error event Pool was shutdown or is inaccessible
Ready to accept request
test issue with spock (not reproduced)Use a pristine profile, run sar_demo , run newfile, launch a scan , then silx view (keep open) and then another scan ==> all ok: ┬─[cpascual@pt168:~/src/sardana]─[18:45:49]─[G:develop=]
╰─>$ spock --profile=kk-1457 (sdn-1457)
(...)
IPython profile: kk-1457
Connected to Door_kk-1457_1
Door_kk-1457_1 [1]: %sar_demo
(...)
DONE!
Door_kk-1457_1 [2]: newfile /tmp/test-1457.h5
ScanDir is : /tmp
ScanFile set to : test-1457.h5
Next scan is : #1
Door_kk-1457_1 [3]: ascan mot18 0 1 1 0.1
Operation will be saved in /tmp/test-1457.h5 (HDF5::NXscan from NXscanH5_FileRecorder)
Scan #1 started at Mon Mar 22 18:50:37 2021. It will take at least 0:00:00.482843
#Pt No mot18 ct17 ct18 ct19 ct20 dt
0 0 0.1 0.2 0.3 0.4 0.0502625
1 1 0.1 0.2 0.3 0.4 0.500065
Operation saved in /tmp/test-1457.h5 (HDF5::NXscan)
Scan #1 ended at Mon Mar 22 18:50:38 2021, taking 0:00:00.634184. Dead time 68.5% (motion dead time 50.5%)
Door_kk-1457_1 [4]: !silx view /tmp/test-1457.h5 &
Door_kk-1457_1 [5]: WARNING:silx.gui.data.DataViews:Plot3dView is not available
WARNING:silx.gui.data.DataViews:NXdataVolumeView is not available
WARNING:silx.gui.data.DataViews:3D visualization is not available
WARNING:silx.gui.data.DataViewer:_Plot3dView instantiation failed. View is ignored
Door_kk-1457_1 [5]:
Door_kk-1457_1 [5]: ascan mot18 0 1 1 0.1
Operation will be saved in /tmp/test-1457.h5 (HDF5::NXscan from NXscanH5_FileRecorder)
Scan #2 started at Mon Mar 22 18:51:48 2021. It will take at least 0:00:00.765685
#Pt No mot18 ct17 ct18 ct19 ct20 dt
0 0 0.1 0.2 0.3 0.4 0.32062
1 1 0.1 0.2 0.3 0.4 0.743912
Operation saved in /tmp/test-1457.h5 (HDF5::NXscan)
Scan #2 ended at Mon Mar 22 18:51:49 2021, taking 0:00:00.873439. Dead time 77.1% (motion dead time 68.6%)
Door_kk-1457_1 [6]: |
test_swmr() is prone to a race condition because of reuse of threading event. Fix it by using two events instead of reusing. Also add a test of swmr without h5_write_session context (using HDF5_USE_FILE_LOCKING="FALSE")
|
The reason for not reproducing the issue with a newer stack seems to be related with the reader (in this case silx). |
This variable won't be considered in the libhdf5 version from stretch. So, currently will fail in the CI. |
test_swmr_without_h5_session requires support for HDF5_USE_FILE_LOCKING environment variable. Mark the test as expected failure if hdf5 does not support it.
Well seen @reszelaz ! |
added 7 commits
March 24, 2021 16:30
Provide a more user-friendly explanation when there is a problem handling a hdf5 file file in swmr mode
Print file name(s) , SWMR mode and hdf5 compatibility info
Some docstrings referred to outdated version of the start session macro. Fix them
Do not allow to start a new session if one is already open
This allows to close sessions even if the Scandata/ScanFiles variables changed
- Add macros to start/end h5 sessions based on file paths instead of ScanDir and ScanPath - Support a path argument in the h5_write_session context - Do not block creation of new session even if some sessions is already started (just print hint on how to close)
|
I checked the existing code and added the following UI-related improvements:
(most code done in pair-programming with @reszelaz ) I think it is ok to merge as is. |
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Recently there were several issues somehow related with the H5 data storage e.g. #771, #1296, #1448, etc.. and for a developer it will be much more convenient to not have the problems with the file locking when inspecting the recorded data with tools like silx. This motivated me to dedicate some time to try to advance with #1124.
First, I wanted to dedicate the minimum possible time to achive an improvement and I discarded from the beginning more complex changes e.g. making recorders long lived objects in Sardana.
Second, the issue described in #1124 can be reproduce with automatic test added in 7747c94 or manually.
To reproduce manually:
NXscanH5_FileRecordersilx(I tested with version 0.12) e.g.Now, very briefly, what this PR introduces is a concept of write session for the NXscanH5_FileRecorder. This session can be manually started and ended using macros or programmatically (in a macro code) using a context manager.
The session opening and closing is based on the
ScanFileandScanDirenvironment variables so it respects different levels of environment (door and macro).More details in the documentation: 8941294.
I have added an atexit hook to close the still opened files at the server shutdown in a466bde but most probably due to taurus-org/taurus#982 it is never called.
In case one forgot to end the session and already shutdown the server it is possible to clear the SWMR flag with
h5clear -s <file_name>(not available in Debain stretch, but available in Debian buster and conda).Some ideas on how to improve it:
pre-newfilehook place to newfile in order to allow automatic closing and opening sessions when changing fileScanFileandScanDirlevel or directly using file path.I have a feeling that 2 and 3 are more time consuming, while 1 would be very easy to add. Anyway, I would like to hear your opinion about all this before investing more time to this topic.
If we agree to merge it, I would prefer to mark is as experimental feature (allowing API changes if strictly necessary).