split settings for the first pass and multi-pass

[alexlib]

the problem
at the moment windef.piv() takes windef.Settings() with only one type of correlation method. it prevents us from a more robust operation when the first pass is for instance linear and normalized and other passes are faster due to already deformed windows circular and not normalized.

Describe the solution you'd like
the suggestion is to move to YAML type of settings that could be even stored in a file next to the results. then we could have

[first pass]
correlation: linear 
normalized : True

[multi pass]
correlation: circular
normalized: False

it would also help us to get extended search into the first pass using different interrogaton window sizes and
overlaps and also will open the path to the rectangular windows

[first pass]
interrogation size A : [64, 128]
interrogation size B : [128, 256] 

[alexlib]

@eguvep what do you think about changing settings to YAML format and unify those between openpiv and GUI version?

[eguvep]

Hi Alex,
YAML ist just the format, not the structure, right? For example: For the GUI, we are using an OpenPivParams object, which is holding basically just a single dictionary (and some convenient functions). That dictionary could of course be saved as a YAML file. windef.Settings() is an object containing separate member variables, that could also be saved as a YAML file. So the storage format would be the same, the data structure would still be different. Both settings must still be translated to each other, because the structure is different. We earlier discussed a similar topic for the vector data format. OpenPIV/openpiv_tk_gui#22 (comment)
In my point of view, the question is broader:

• Do we want a single vector format?
• Do we want a single settings format?
• Do we want to merge them in a single multi purpose file (like suggested by Erich in the linked thread)?
• Should every contributor care about converting the formats by himself?
• Should we start a separate format conversion project (DaVis to OpenPiv to NetCDF to ...)?

[alexlib]

Thanks for a quick reply.

The parameters issue shall be probably separated from the results storage. Since parameters file is small and it's really for the simple adjustment of those between the runs and for storage for reproducibility, the data storage format is a different beast and it requires some thinking about being a proprietary (our) binary format or some general binary format or maybe ASCII (human readable) format and what it should contain.

As far as I see it is that a working folder will contain the raw image files and the settings parameter file next to it leading to the data in the output folder. The path can be a Python script, Jupyter notebook or GUI, but the result should be the same to a certain extent. If GUI creates a different output depending on additional filters, sub-settings, etc. - we need to have it also in the parameters file, so the Jupyter notebook can be re-created for the same path from input to output.
All the internal things are fine to be adjusted as long as those are transparent for the user.

I believe we have to have a single settings format and more so - to have a single format for Matlab and Python and other versions. Therefore it's best to store in YAML format (or similar, but there are no many that are readable so easy by C, Matlab and Python) .

Let's have the data output format discussion on OpenPIV/openpiv_tk_gui#22 (comment)

[alexlib]

See how nice I can draw :)

[alexlib]

as far as I see your JSON format is very close to YAML format. I do not mind to keep parameters in JSON format as well, but I think that YAML today is a bit more popular and more human readable.
I suggest the following:
GUI reads the piv_parameters.yaml that will be a single format with a list of a) common parameters, b) additional parameters. Since YAML is a flexible format, the reader (either Jupyter notebook or script or GUI) does not have to know a-priori all the settings it might include, but only those which are necessary and common will be checked. All the rest is populated by GUI or shown to the user in the notebook and could be adjusted there (and stored back). I.e. a simple parameters file will include only what is in windef.Settings() and anything below it can be for GUI, but won't be read by the notebook as irrelevant.
The main thing is still that x,y,u,v should be at least within a small error reproducible through the various systems if they all use underneath the same extended_search_piv. It can be different between Matlab and Python, of course, but for the simple set of settings like 32 x 32 no filtering no smoothing we'd expect to get the same sort of result, at least on average.

[alexlib]

I guess we need to figure out first between us what is crucial in the structure of the parameters. It will be then easier to decide if we have the same minimal set with the translators or we can agree about the same structure for the minimal set.

[alexlib]

I have a prototype here
https://github.com/OpenPIV/openpiv-python/blob/windef_yaml/openpiv/examples/notebooks/windef_settings_yaml.ipynb
https://github.com/OpenPIV/openpiv-python/blob/windef_yaml/openpiv/examples/notebooks/default_settings.yaml
in the branch windef_yaml https://github.com/OpenPIV/openpiv-python/tree/windef_yaml

[eguvep]

The GUI is created based on the parameters in a default OpenPivParams object. Still, it would be easy to use a standard Settings-Object:

default.dict → create GUI → replace selected values read from YAML file → ready

Actually, we are using the following default parameters:

>>> from openpivgui.OpenPivParams import OpenPivParams
>>> p = OpenPivParams()
>>> for key in p.default:
...     print(key + ': ' + str(p.help[key]))
... 
general: None
fnames: None
general_frame: None
warnings: Enable popup warning messages (recommended).
multicore_frame: None
manual_select_cores: Mannualy select cores. If not seected, all available cores will be used.
cores: Select amount of cores to be used for PIV evaluations.
frequencing_sub_frame: None
sequence: Select sequence order for evaluation.
skip: Select sequence order jump for evaluation.
Ex: (1+(1+x)),(2+(2+x))
filters_sub_frame: None
navi_pattern: Regular expression patterns for filtering the files in the current directory. Use the back and forward buttons to apply a different filter.
pandas_sub_frame: None
load_settings: Individual settings for loading files using pandas.
skiprows: Number of rows skipped at the beginning of the file.
decimal: Decimal separator for floating point numbers.
sep: Column separator.
header: Read header. If chosen, first line will be interpreted as the header
header_names: Specify comma separated list of column names.Example: x,y,vx,vy,mask,sig2noise
save_sub_frame: None
vec_fname: Filename for vector output. A number and an acronym that indicates the process history are added automatically.
separator: Delimiter.
image_plotting_sub_frame: None
matplot_intensity: Define a reference intensity for the plotting of images.
preproc: None
exclusions_frame: None
crop_ROI: Crop region of interest. Allows images with different sizes to have a uniform size after cropping.
crop_roi-xminmax: Define left/right side of region of interest by 'min,max'.
crop_roi-yminmax: Define top/bottom of region of interest by 'min,max.'
dynamic_mask_spacer: None
dynamic_mask: Dynamic masking for masking of images. 
Warning: This is still in testing and is not recommended for use.
dynamic_mask_type: Defining dynamic mask type.
dynamic_mask_threshold: Defining threshold of dynamic mask.
dynamic_mask_size: Defining size of the masks.
preproc-filtering: None
preprocess_frame: None
data-type: None
Invert_spacer: None
invert: Invert image (see skimage invert()).
background_spacer: None
background_subtract: Subtract background via local sliding windows.
background_type: The algorithm used to generate the background which is subtracted from the piv images.
starting_frame: Defining the starting image of the background subtraction.
ending_frame: Defining the ending image of the background subtraction.
CLAHE_spacer: None
CLAHE: Contrast Limited Adaptive Histogram Equalization filter (see skimage adapthist()).
CLAHE_first: Perform CLAHE filter before Gaussian high pass filters.
CLAHE_auto_kernel: Have the kernel automatically sized to 1/8 width and height of the image.
CLAHE_kernel: Defining the size of the kernel for CLAHE.
high_pass_filter_spacer: None
high_pass_filter: A simple subtracted Gaussian high pass filter.
hp_sigma: Defining the sigma size of the subtracted gaussian filter in the high pass filter (positive ints only).
hp_clip: Set all values less than zero to zero.
intensity_threshold_spacer: None
intensity_cap_filter: Simple global intesity cap filter. Masked pixels are set to the mean pixel intensity.
ic_mult: Multiply the standard deviation of the pixel intensities to get a higher cap value.
Gaussian_lp_spacer: None
gaussian_filter: Standard Gaussian blurring filter (see scipy gaussian_filter()).
gf_sigma: Defining the sigma size for gaussian blur filter.
intensity_clip_spacer: None
intensity_clip: Any intensity less than the threshold is set to zero.
intensity_clip_min: Any intensity less than the threshold is set to zero with respect to the resized image inntensities.
img_int_resize_spacer: None
img_int_resize: Resize the max image intensity to 
a user defined value.
piv: None
piv_frame: None
evaluation_method: Direct Correlation: Direct correlation with extended size of the search area. 
FFT WinDef: Fast Fourier Transforms with window deformation (recommended).
corr_method: Correlation method. Circular is no padding andlinear is zero padding (applies to Windef).
subpixel_method: Fit function for determining the subpixel position of the correlation peak.
sig2noise_method: Calculation method for the signal to noise ratio.
adv_s2n_mask: the half size of the region around the first correlation peak to ignore for finding the second peak. Only used if sig2noise method = 'peak2peak' 
adv_interpolation_order: Interpolation oder of the window deformation. 
»0« yields zero order nearest interpolation 
»1« yields first order linear interpolation 
»2« yields second order quadratic interpolation 
and so on...
calibration_spacer: None
dt: Interframing time in seconds.
scale: Interframing scaling in pix/m 
Important: The scaling ratio is applied before dt, so when getting the oroper scale, set »dt« to 1
flip_spacer: None
flip_u: flip u-component array when saving RAW results.
flip_v: flip v-component array when saving RAW results.
invert_spacer: None
invert_u: Invert (negative) u-component when saving RAW results.
invert_v: Invert (negative) v-component when saving RAW results.
swap_files_spacer: None
swap_files: Swap A/B files when analyzing.
windowing: None
window_frame: None
search_area: Size of square search area in pixel for Single-pass DCC method.
corr_window: Size of the final interrogation windows in pixels.
overlap: Size of the final overlap in pixels.
coarse_factor: Example: A window size of 16 and a number of refinement steps of 3 gives an window size of 64×64 in the first pass, 32×32 in the second pass and 16×16 pixel in the final pass. (Applies to FFT WinDef methods only.)
grid_refinement: Refine the interregationg grid every PIV pass when performing multipass FFT. 
»all passes« refines all passes. 
»2nd pass on« refines second pass on.
sub_window_frame: None
custom_windowing: Enable custom windowing for more advanced techniques.
pass_1_spacer: None
corr_window_1: Interrogation window for the first pass.
overlap_1: Size of the overlap of the first pass in pixels. The overlap will then be calculated for the following passes.
pass_2_spacer: None
pass_2: Enable a second pass in the FFT window deformation evaluation.
corr_window_2: Interrogation window for the second pass.
pass_3_spacer: None
pass_3: Enable a third pass in the FFT window deformation evaluation.
corr_window_3: Interrogation window for the third pass.
pass_4_spacer: None
pass_4: Enable a fourth pass in the FFT window deformation evaluation.
corr_window_4: Interrogation window for the fourth pass.
pass_5_spacer: None
pass_5: Enable a fifth pass in the FFT window deformation evaluation.
corr_window_5: Interrogation window for the fifth pass.
pass_6_spacer: None
pass_6: Enable a sixth pass in the FFT window deformation evaluation.
corr_window_6: Interrogation window for the sixth pass.
pass_7_spacer: None
pass_7: Enable a seventh pass in the FFT window deformation evaluation.
corr_window_7: Interrogation window for the seventh pass.
validation: None
validation_frame: None
piv_sub_frame1: None
fp_local_med: Discard vector, if the absolute difference with the local median is greater than the threshold. 
fp_local_med_size: Local median filter kernel size.
globa_thr_spacer: None
fp_vld_global_threshold: Validate first pass based on set global thresholds.
fp_MinU: Minimum U allowable component.
fp_MaxU: Maximum U allowable component.
fp_MinV: Minimum V allowable component.
fp_MaxV: Maximum V allowable component.
piv_sub_frame2: None
sp_local_med_threshold: Discard vector, if the absolute difference with the local median is greater than the threshold. 
sp_local_med_size: Local median filter kernel size.
glob_std_spacer: None
sp_global_std_threshold: Remove vectors, if the the sum of the squared vector components is larger than the threshold times the standard deviation of the flow field.
glob_thr_spacer: None
sp_MinU: Minimum U allowable component.
sp_MaxU: Maximum U allowable component.
sp_MinV: Minimum V allowable component.
sp_MaxV: Maximum V allowable component.
piv_sub_frame3: None
adv_repl_method: Each NaN element is replaced by a weighed averageof neighbours. Localmean uses a square kernel, disk a uniform circular kernel, and distance a kernel with a weight that is proportional to the distance.
adv_repl_iter: If there are adjacent NaN elements, iterative replacement is needed.
adv_repl_kernel: Diameter of the NaN interpolation kernel.
piv_sub_frame4: None
smoothn_each_pass: Smoothen each pass using openpiv.smoothn.
smoothn_first_more: Double the smoothing strength on the first pass.
robust1: Activate robust in smoothen (minimizes influence of outlying data).
smoothn_val1: Strength of smoothen script. Higher scalar number produces more smoothed data.
vld: None
vld_frame: None
vld_sig2noise: Validate the data based on the signal to nose ratio of the cross correlation.
sig2noise_threshold: Threshold for filtering based on signal to noise ratio.
horizontal_spacer11: None
vld_global_std: Validate the data based on a multiple of the standard deviation.
global_std_threshold: Remove vectors, if the the sum of the squared vector components is larger than the threshold times the standard deviation of the flow field.
horizontal_spacer12: None
vld_local_med: Validate the data based on a local median threshold.
local_median_threshold: Discard vector, if the absolute difference with the local median is greater than the threshold. 
horizontal_spacer13: None
vld_global_thr: Validate the data based on set global thresholds.
MinU: Minimum U allowable component.
MaxU: Maximum U allowable component.
MinV: Minimum V allowable component.
MaxV: Maximum V allowable component.
horizontal_spacer14: None
repl: Replace outliers.
repl_method: Each NaN element is replaced by a weighed averageof neighbours. Localmean uses a square kernel, disk a uniform circular kernel, and distance a kernel with a weight that is proportional to the distance.
repl_iter: If there are adjacent NaN elements, iterative replacement is needed.
repl_kernel: Diameter of the weighting kernel.
horizontal_spacer15: None
smoothn: Smoothn post processed results using openpiv.smoothn.
robust: Activate robust in smoothn (minimizes influence of outlying data).
smoothn_val: Strength of smoothn script. Higher scalar number produces more smoothned data.
average_spacer: None
average_results: Average all results in selected directory. Results in a single file with averaged results.
delimiter_spacer: None
delimiter: Delimiter.
postproc_spacer: None
plt: None
plt_frame: None
plot_type: Select how to plot velocity data.
plot_title: diagram title.
streamline_density: streamline density. Can be one value (e.g. 1) or a couple of values for a range (e.g. 0.5, 1).
integrate_dir: Integrate the streamline in forward, backward or both directions. default is both.
statistics_frame: None
u_data: column name for the u-velocity component. If unknown watch labbook entry.
v_data: column name for v-velocity component. If unknown watch labbook entry. For histogram only the v-velocity component is needed.
plot_scaling: scales the axes. logarithm scaling x-axis --> logx; logarithm scaling y-axis --> logy; logarithm scaling both axes --> loglog.
histogram_type: Choose histogram type. Only available for histogramplot.
histogram_quantity: The absolute value of the velocity (v) or its x- or y-component (v_x or v_y).
histogram_bins: Number of bins (bars) in the histogram.
profiles_orientation: Plot v_y over x (horizontal) or v_x over y (vertical).
plot_xlim: For implementation use (lower_limit, upper_limit).
plot_ylim: For implementation use (lower_limit, upper_limit).
modify_plot_appearance: None
modify_plot_frame: None
vector_subframe: None
vec_scale: Velocity as a fraction of the plot width, e.g.: m/s per plot width. Large values result in shorter vectors.
vec_width: Line width as a fraction of the plot width.
invalid_color: Choose the color of the vectors
valid_color: Choose the color of the vectors
invert_yaxis: Define the top left corner as the origin of the vector plot coordinate sytem, as it is common practice in image processing.
derived_subframe: None
color_map: Color map for streamline- and contour-plot.
extend_cbar: Extend the top and bottom of the colorbar to accept out of range values.
velocity_color: Set colorbar to velocity components.
color_levels: Select the number of color levels for contour plot.
vmin: minimum velocity for colormap (contour plot).
vmax: maximum velocity for colormap (contour plot).
statistics_subframe: None
plot_grid: adds a grid to the diagram.
plot_legend: adds a legend to the diagram.
lab_book: None
lab_book_frame: None
lab_book_content: None
data_information: shows column names, if you choose a file at the right side.
user_func: None
user_func_def: None

[eguvep]

As long as the names are the same, we do not even need the same structure. We could just copy them in a single line loop.

[alexlib]

We can start preparing a YAML settings file(s) that are easy to convert to JSON back and forth.
https://stackoverflow.com/questions/50846431/converting-a-yaml-file-to-python-json-object

The idea is that we have a file that contains the same names for the union set - between windef and GUI and this minimal set has the same names. All the additional things can keep the GUI names, if these are used in the Python script.

BTW, does it make sense that we could run a command-line PIV analysis of the data using the openpivgui.OpenPivParams object ? I.e. a non-GUI script that runs the same set of functions as in the GUI?

[alexlib]

@eguvep I wonder what is the difference between

MinU: Minimum U allowable component.
MaxU: Maximum U allowable component.
MinV: Minimum V allowable component.
MaxV: Maximum V allowable component.

and

sp_MinU: Minimum U allowable component.
sp_MaxU: Maximum U allowable component.
sp_MinV: Minimum V allowable component.
sp_MaxV: Maximum V allowable component.

or

fp_MinU: Minimum U allowable component.
fp_MaxU: Maximum U allowable component.
fp_MinV: Minimum V allowable component.
fp_MaxV: Maximum V allowable component.

[ErichZimmer]

Dear Alex,

The different min/max parameters are for first pass validation (fp), second pass validation (sp), and postprocessing (doesn't have fp or sp). This is to allow a more robust/customizable validation for multipass evaluations, but may be obsolete and confusing/time consuming to use.

Regards,
Erich Zimmer

PS; I got 20 days off for Christmas break, so I will be able to help/test new enhancements :)

[alexlib]

Thanks @ErichZimmer and great to see you here. Enjoy your holidays. And so everyone :) Holiday Season greetings to the community.

I'm still struggling with the robustness of the windef - also in the comparison with PIVLab there is a bit more "noise" and it's probably due to the changes I made in the code. I'm now trying to figure out all the places that were changed from the original windef.py to 0.23.3 and to clarify the differences that might lead to the experience you @ErichZimmer had with the robustness. If you can take a careful look at the differences between windef versions and find a bug - please go ahead.

[alexlib]

My present suspect now is the orientation of the deformation in respect to the velocity field. In the past @TKaeufer used flipped y coordinate and negative vertical displacement for the velocity, but then when we apply image deformation, these probably should be given in the image coordinate system. I'm afraid I am misplaced some of the flips or some of the sign changes (also the fact that rows, cols in image processing are not the opposite order of u,v displacement, u is columnwise) could be a problem.

Another major problem I experience now is the introduction of image masking. This creates a lot of difficult situations where we need to clean/smooth the vector field but we might not want to the image masked region to participate in the interpolation. This might also lead to the loss of smoothing I was looking for.