Parameterise S3DXRD notebooks

[jadball]

Addresses #334

Leaving as draft PR as we still have box-beam remaining.

Allows end-to-end automated notebook testing via papermill (currently only working at ESRF) - see test/papermill_test_notebooks.py for an example on how to use it.
I recommend not running it with pytest but instead manually choosing functions to run in main() after cd-ing into the test folder (ImageD11 paths are relative).
If you want to run the FeAu tests, I recommend changing the output directory in inhouse2 into your own folder to stop collisions, e.g currently: tomo_dir = '/data/id11/inhouse2/test_data_3DXRD/S3DXRD/FeAu/PROCESSED_DATA/20250123_JADB/tomo_route'

Other bits and bobs:

• Added papermill to reqs in setup.py (optional)
• Bumped ImageD11 version in README.MD to 2.0.5
• Fixed a bug in ImageD11/parameters.py so we can use a pars.json without any phases present (e.g. to compute geometry) - thanks @abmajith
• A few refinement enhancements in ImageD11/sinograms/point_by_point.py
• Removed tomo_3_refinement_minor_phase.ipynb all the pbp and refinement notebooks (and running on cluster) now append phase strings to filenames by default to avoid collisions in multi-phase. This could be improved and will be soon.
• Added html notebook renders to docs folder - these will be updated with FeAu examples.

[review-notebook-app]

Check out this pull request on 

See visual diffs & provide feedback on Jupyter Notebooks.

---

Powered by ReviewNB

[abmajith]

Hello @jadball and the ID11 team,

I tested the master branch (https://github.com/jadball/ImageD11) locally for geometry correction without phase files. There’s at least one change required:

When initializing the AnalysisSchema class, the get_pars_objects function is called before get_xfab_pars_dict.

A quick solution could involve modifying the loadparameters function in the parameters class. Based on the phase_name value, we could adjust the loading process with a small change to the AnalysisSchema class init function.

[abmajith]

Hello @jadball and the ID11 team,

I checked Dataset (from dataset.py), columnfile (from columnfile.py) and parameters (from parameters.py) python class, in order to do the geometry correction, we could do using this python lines without modifying the existing code, kindly check and comment

cf_2d, cf_3d = process(ds, options)
cf_3d.parameters.loadparameters("/path/to/geometry_tdxrd.par")
cf_3d.updateGeometry()  

Is this equivalent of correction the peaks based on the geometry file?

[jonwright]

Is this equivalent of correction the peaks based on the geometry file?

Not sure what you mean here? In general things are all the same for grains on the rotation axis. For a grain which is moving during rotation you get a different correction in the "updateGeometry" method versus the sinograms.geometry codes.

[abmajith]

Oh, it seems there are many ways to do, I like to discuss further, will do in the following weeks.

parfile = os.path.join(ds.analysisroot, 'pars_tdxrd.json')
ds.parfile = parfile
ds.update_colfile_pars(cf_3d)

When using this method without providing a phases file reference (i.e., {"geometry": {"file": "geometry_tdxrd.par"}} in parfile), I get a "phases attribute missing" error.

However, when I include an empty phases object (i.e., {"geometry": {"file": "geometry_tdxrd.par"}, "phases": {}}), I instead receive a "phase parameter error."
Attempted Solution

cf_2d, cf_3d = process(ds, options)
cf_3d.parameters.loadparameters("/path/to/geometry_tdxrd.par")
cf_3d.updateGeometry()  

[jadball]

Hello @jadball and the ID11 team,

I checked Dataset (from dataset.py), columnfile (from columnfile.py) and parameters (from parameters.py) python class, in order to do the geometry correction, we could do using this python lines without modifying the existing code, kindly check and comment

cf_2d, cf_3d = process(ds, options)
cf_3d.parameters.loadparameters("/path/to/geometry_tdxrd.par")
cf_3d.updateGeometry()  

Is this equivalent of correction the peaks based on the geometry file?

Hi @abmajith - this will work for you guys as a workaround.

[jadball]

Remaining:

• Replace HTML rendered notebooks with FeAu examples
• Finish box-beam parameterisation