Improve postprocessing

[greole]

Motivation

This issue collects requirements to add further support for post-processing of OF postProcessing folder. Inside the postProcessing folder we usually find multiple .dat files stored in a subfolder named after the probe or function which generates the file and a time subfolder for the first time step at which the file is created. The data files itself are not standardized and could be in different formats.

Goal

The goal of OBRs post-processing functionality is automated reading and parsing of OpenFOAMs unstructured simulation artifacts like log files and probes and sample data in the postProcessing folder. After OBRs post-processing step, the post-processed data should be accessible as python data structures (ideally records) via OBRs query functionality for further processing like plotting etc.

Approaches

Currently, we have several postprocessing strategies:

1. Explicit obr run -o apply <path/to/postproscript.py>
2. Implicit obr status

Explicit approach

In the explicit approach 1. a handle to all (not filtered) workspace folders is given to the user. The user can use convenience functions to perform the post-processing (eg. access to the statepoint, job_document, owls functions for log and file parsing. The user is responsible for writing the post processing script them self. Improving this approach means:

1. add more parser to owls
2. implement a post-processing function which allows to register parser postPro.register(minMaxU, forceCoeffs ...)
3. Improve the cli interface by adding a obr apply <path/to/postproscript.py> shortcut?

Implicit approach

The implicit approach is used for standard information one can extract from a simulation. Eg.: current time-step, next write to disk ...

Related

This Issues is also related to changing obr status output.

[HendriceH]

postProcessing/minMax_U/<time_>/fieldMinMax.dat

# Field minima and maxima
# Time                  field                   min                     location(min)           processor               max                     location(max)           processor              
3.6745                  mag(U)                  0.000000000000000e+00   (3.300559710475396e+00 -1.847687698003787e-01 1.546702551125981e-01)    72      2.293594108528461e+02   (1.924696857594865e-03 -4.752312297628312e-01 2.198786178471663e+00)    89
3.6749                  mag(U)                  0.000000000000000e+00   (3.300559710475396e+00 -1.847687698003787e-01 1.546702551125981e-01)    72      2.305186418637790e+02   (1.924710728785659e-03 2.153727910134374e+00 -6.495852223928575e-01)    26

postProcessing/forces/<time_>/force.dat
postProcessing/forces/<time_>/moment.dat

# Force                
# CofR                 : (0.000000000000000e+00 0.000000000000000e+00 0.000000000000000e+00)
#
# Time                  total_x total_y total_z pressure_x pressure_y pressure_z        viscous_x viscous_y viscous_z
0.5505                  2.086333634573342e+03 -3.829427035220899e+01 1.111923227572331e+01 2.076552934122240e+03 -3.815014207024837e+01 1.089831554077432e+01 9.780700451102447e+00 -1.441282819606187e-01 2.209167349489938e-01
0.551                   2.086004796690621e+03 -3.846122385913035e+01 1.123902340685886e+01 2.076227602829565e+03 -3.831686265694570e+01 1.101863719192531e+01 9.777193861056723e+00 -1.443612021846450e-01 2.203862149335558e-01

# Moment               
# CofR                 : (0.000000000000000e+00 0.000000000000000e+00 0.000000000000000e+00)
#
# Time                  total_x total_y total_z pressure_x pressure_y pressure_z        viscous_x viscous_y viscous_z
3.2225                  -1.721296269437523e+02 -6.266208666127422e+02 -8.781596066685380e+01 -1.718519543482061e+02 -6.167919281549131e+02 -8.768387921880760e+01 -2.776725955461493e-01 -9.828938457829222e+00 -1.320814480461875e-01
3.2229                  -1.755986470654739e+02 -6.271915725035343e+02 -8.970842248371999e+01 -1.753074233277084e+02 -6.173614224950355e+02 -8.957015722603410e+01 -2.912237377654813e-01 -9.830150008498832e+00 -1.382652576858889e-01

[HendriceH]

postProcessing/sample_cuttingLine/<time_>/<type_>_<name_>_<var1_>_<var2_>_<var3_>.xy (e.g. line_h-wake-survey-1_nut_p_U.xy), one file per unique name

1.0414  1.71323         -0.271523672954795      2.54880095856372e-09    101325  27.2    0       0
1.0414  1.71323         -0.257232953325596      2.54880095856372e-09    101325  27.2    0       0
1.0414  1.71323         -0.242942233696396      2.54880095856372e-09    101325  27.2    0       0
1.0414  1.71323         -0.228651514067196      2.54880095856372e-09    101325  27.2    0       0
1.0414  1.71323         -0.214360794437996      2.54880095856372e-09    101325  27.2    0       0
1.0414  1.71323         -0.200070074808797      2.54880095856372e-09    101325  27.2    0       0

[HendriceH]

postProcessing/solverInfo/<time_>/solverInfo.dat

# Solver information   
# Time                  h_solver                h_initial               h_final                 h_iters                 h_converged             U_solver                Ux_initial              Ux_final                Ux_iters                Uy_initial              Uy_final                Uy_iters                Uz_initial              Uz_final                Uz_iters                U_converged             k_solver                k_initial               k_final                 k_iters                 k_converged             p_solver                p_initial               p_final                 p_iters                 p_converged             omega_solver            omega_initial           omega_final             omega_iters             omega_converged        
0.101025                DILUPBiCGStab   2.151666432088290e-05   1.352760193887710e-09   1       true    DILUPBiCGStab   1.217528032213750e-05   1.537738551410860e-09   1       1.660141124828770e-04   3.462593131181640e-08   1       3.047461485016110e-04   6.166290294007240e-08   1       false   DILUPBiCGStab   2.826083499520860e-06   6.393431631881380e-10   1       true    FDICPCG 5.061087174491460e-04   4.446544182456180e-06   19      true    DILUPBiCGStab   1.897961877915790e-09   5.591413873781000e-13   1       true
0.10105                 DILUPBiCGStab   2.150461005493010e-05   1.351533040597660e-09   1       true    DILUPBiCGStab   1.217504096063520e-05   1.534606218576370e-09   1       1.655860156045340e-04   3.454512253309460e-08   1       3.025206081378640e-04   6.125125789399401e-08   1       false   DILUPBiCGStab   2.825850432873950e-06   6.392169021745950e-10   1       true    FDICPCG 5.060094661166300e-04   4.638420364907130e-06   19      true    DILUPBiCGStab   1.900778379941010e-09   5.600505033607530e-13   1       true

postProcessing/forceCoeffs/<time_>/coefficients.dat

# Force coefficients    
# dragDir               : (1.0000000000000000e+00 0.0000000000000000e+00 0.0000000000000000e+00)
# sideDir               : (0.0000000000000000e+00 1.0000000000000000e+00 0.0000000000000000e+00)
# liftDir               : (0.0000000000000000e+00 0.0000000000000000e+00 1.0000000000000000e+00)
# rollAxis              : (1.0000000000000000e+00 0.0000000000000000e+00 0.0000000000000000e+00)
# pitchAxis             : (0.0000000000000000e+00 1.0000000000000000e+00 0.0000000000000000e+00)
# yawAxis               : (0.0000000000000000e+00 0.0000000000000000e+00 1.0000000000000000e+00)
# magUInf               : 3.8889000000000003e+01
# lRef                  : 2.7861799999999999e+00
# Aref                  : 2.1699999999999999e+00
# CofR                  : (1.4000900000000001e+00 0.0000000000000000e+00 -3.1759999999999999e-01)
#
# Time                          Cd                              Cs                              Cl                              CmRoll                          CmPitch                         CmYaw                           Cd(f)                           Cd(r)                           Cs(f)                           Cs(r)                           Cl(f)                           Cl(r)                   
2.4001                          2.7748632628483905e-01  1.1989317002386207e-02  2.6798852862071825e-02  -7.8200773829396625e-03 -8.3498144923480802e-02 1.2797919469776822e-03  1.3092308575947986e-01  1.4656324052535918e-01  7.2744504481707852e-03  4.7148665542154217e-03  -7.0098718492444895e-02 9.6897571354516709e-02
2.4002                          2.7777021478601704e-01  1.2200265761890223e-02  2.6821060290490598e-02  -7.8654981150179191e-03 -8.3247181483952340e-02 1.3006384451831657e-03  1.3101960927799061e-01  1.4675060550802643e-01  7.4007713261282776e-03  4.7994944357619453e-03  -6.9836651338707034e-02 9.6657711629197646e-02

postProcessing/probes/<time_>/<varname_> (e.g. U or p)

# Probe 0 (-0.7982483442005627 5.3068374670484e-10 0.1499998518516053)
# Probe 1 (-0.6999975425699337 5.145081117091494e-09 0.4063468443279496)
# Probe 2 (-0.6000010007824411 -2.458327056016179e-09 0.4626323047661132)
# Probe 3 (-0.4999975242475813 5.51797649605548e-09 0.4994421575527441)
# Probe 4 (-0.4000011659902881 -3.212150307800182e-09 0.5277046104424136)
# Probe 5 (-0.2999973264091636 8.01211744549547e-09 0.5508374191001341)
# Probe 6 (-0.09999759432107691 7.676789618290838e-09 0.5877151371957703)
# Probe 7 (2.152368662435032e-07 7.533596579539676e-10 0.6030485232702065)
# Probe 8 (0.1000015316487817 -5.606700761777402e-09 0.6168883373133495)
# Probe 9 (0.3000008457825677 -3.630596169915212e-09 0.6408219942754139)
# Probe 10 (0.350000144201042 -5.92294222429301e-10 0.6459285356188823)
# Probe 11 (0.4131077083543568 8.870645740890686e-09 0.6250049780564316)
# Probe 12 (0.4641267914000089 6.136946099177787e-09 0.6250050162292955)
# Probe 13 (0.5431759490651962 7.734922057659956e-09 0.6750064817163424)
# Probe 14 (0.6242528364362485 -5.289855655045509e-09 0.7249953370691977)
# Probe 15 (0.7078781123375553 3.54565870319451e-09 0.7750032100720428)
# Probe 16 (0.8853272016288498 5.71772533005421e-09 0.8750052124507874)
# Probe 17 (1.084898284542428 3.502671772109501e-09 0.9750037169249847)
# Probe 18 (1.198999217824584 1.492611976692156e-09 1.025002540352205)
# Probe 19 (1.299995916707717 1.367731846895517e-08 1.051518580740842)
# Probe 20 (3.752647271136354 8.306782790355373e-08 0.4499926787779487)
# Probe 21 (3.75793958458342 -3.788089434332002e-08 0.3500100059880658)
# Probe 22 (3.804117129718127 -7.317841735811607e-09 0.2499994076551568)
#                 Probe                       0                       1                       2                       3                       4                       5                       6                       7                       8                       9                      10                      11                      12                      13                      14                      15                      16                      17                      18                      19                      20                      21                      22
#                  Time
                 2.4001        717.828837335529      -414.3389057943822      -335.8641605887472      -269.9527346075409      -216.2173770623557      -174.0859335065655      -90.58410152306531      -48.09038513346582       1.930021616898618       146.1340527359809       204.0858829615126       258.2704064268581       217.9850897585612       390.9768262190973       314.0608216567417       171.6089100792254      -38.79791794494701      -275.7116972389834      -617.0509716642457      -531.0561798792939      -31.14431823607596      -37.03421428276486      -59.83377176518714
                 2.4002       717.7695390464291      -413.1690473537882      -335.6245893428572      -269.6966547799292      -216.3084831376605      -174.1776721553891      -90.51532217672421      -48.04750732494712       1.893198373556162       146.1952513778632        204.123677040109       258.4915460571779       217.9855054285146       391.0672003952158       314.2180120806286       171.6100543676108      -38.75684907399806      -275.5828242494968      -617.0231705230655      -531.0606914968135      -31.27182985238947      -37.17551824670435      -60.23222935712623

[greole]

OK let's start with implementing a parser for the probes in Owls. From this we will derive a generic parse for the body (which is a simple pandas/numpy csv parser) and a separate parser for the different header. The header parser allows to get and later set the column names. In a later step we will work on an automatic detection of the format, either by name/folder or the header structure.