Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Different dimensions for dimensions Issue in FSI when turbulence model is used - OpenFOAM adapter #158

Open
dntjudme opened this issue Apr 1, 2021 · 28 comments
Labels
bug Unexpected problems (crashes, numerical issues, etc) FSI Fluid-structure interaction

Comments

@dntjudme
Copy link

dntjudme commented Apr 1, 2021

Hi there,

This issue was found when I was doing a FSI simulation using OpenFOAM v7 / CalculiX v2.15 / preCICE v2.0.0.
Please let me lead you to this discourse-post for more details.

As suggested by @MakisH , a brief introduction to the settings of turbulence model in OpenFOAM is to simply turn on the turbulence by setting /constant/turbulenceProperties file, using:

simulationType RAS;
RAS
{
    RASModel        kOmegaSST;
    turbulence      on;
}

Of course, the corresponding initial conditions such as k, omega, nut, p and U are well defined in the 0/ directory.

When the turbulence model is turned on, the FSI simulation stops at the 2nd iteration of the first time step of the coupling (shown in the figure in the post)

Interestingly however, when I turned off the turbulence (switch to laminar):

simulationType laminar;

The FSI simulation went well and no errors are reported. However, the results are definitely not I'm expecting since the turbulent behaviours cannot be well resolved in my FSI simulation.

As @MakisH said, I do also suspect this issue might be originated from the OpenFOAM adapter side, so I'm looking for help from developers.

Should you requires more information please contact me directly and I'm happy to assists.

@precice-bot
Copy link

This issue has been mentioned on preCICE Forum on Discourse. There might be relevant details there:

https://precice.discourse.group/t/different-dimensions-for-dimensions-issue/535/6

@davidscn
Copy link
Member

I cannot reproduce this issue with OpenFOAM v2012 and since I have at the moment no foundation version installed I cannot try with another OpenFOAM version. Are you tied to use OpenFOAM v7?

@dntjudme
Copy link
Author

Yes, I use OpenFOAM v7.
New updates: I also implemented other 3 types of turbulence models : SA one equation / k-epsilon / k-omega.
None of them throw back any dimension mismatch errors.

@ernestoricciardi
Copy link

I'm having the same issue with OpenFOAM v7 and kOmegaSST for turbolence.

I also tried with kOmega and I get MPI_ERR_TRUNCATE.

There are any news for fixing the dimensions error for kOmegaSST?

@martinsaravia
Copy link

I am having the same issue with OF10 and Calculix using kOmegaSST turbulence model. Is there any news regarding this?

@MakisH
Copy link
Member

MakisH commented May 11, 2023

If I understand correctly, this seems to be specific to the Foundation versions.

As I wrote in the preCICE forum, I would assume that the operator= of some related fields modifies the dimensions. So, every time we call it when reading a checkpoint, we unintentionally modify the dimensions. Of course, the whole checkpointing concept is something that OpenFOAM would not expect under normal operation.

@MakisH MakisH added bug Unexpected problems (crashes, numerical issues, etc) FSI Fluid-structure interaction labels Aug 8, 2023
@MakisH
Copy link
Member

MakisH commented Dec 18, 2023

Another user reported that with OpenFOAM v2212:

  • kOmegaSST fails with this issue
  • kOmega works
  • laminar works

In any case, this only affects checkpointing, i.e., only implicit coupling (explicit should work with any turbulence model). This is only a software issue, which we don't yet understand.

@KariimAhmed
Copy link

KariimAhmed commented Feb 6, 2024

Yes i have the same exact issue with komega sst on OpenFOAM v2312

@KariimAhmed
Copy link

KariimAhmed commented Mar 8, 2024

Hello @MakisH I still have the same problem with the K-omega sst and the OpenFOAM adapter, did you manage to figure out where is the issue, please?

@MakisH
Copy link
Member

MakisH commented Mar 8, 2024

Hi! No, we have not yet looked into it. It is clear that we need to fix it, it is near the top of the list, but we currently have other priorities.

Any clues or contributions in this direction are welcome! 🤗

@MakisH
Copy link
Member

MakisH commented Mar 10, 2024

@KariimAhmed let's look into this together. I think that, by fixing this, we will fix more strange issues. But first, I need to be able to reproduce it.

I modified our flow-over-heated-plate case, but I cannot yet trigger the issue with v2312. Could you please help me modify the case so that it triggers it?

precice/tutorials#476

@precice-bot
Copy link

This issue has been mentioned on preCICE Forum on Discourse. There might be relevant details there:

https://precice.discourse.group/t/fsi-openfoam-my-solver-using-python/1823/2

@KariimAhmed
Copy link

@MakisH Thanks for considering this issue, the problem is triggered with me when I use the K-omega sst with my flapping foil with FSI. Also when using k-omega it simulates some time steps then it diverges and crashes, only the laminar is working fine. Concerning the case you mentioned I don't know exactly how can the issue be triggered, I saw that people are reporting this issue when using turbulent models in different cases but we don't know why this happens.

@MakisH
Copy link
Member

MakisH commented Mar 13, 2024

It is clear that this issue happens to user's cases, but if I don't have a failing example at hand, it is very challenging for me to fix the problem.

@precice-bot
Copy link

This issue has been mentioned on preCICE Forum on Discourse. There might be relevant details there:

https://precice.discourse.group/t/openfoam-ras-models/2047/2

@KariimAhmed
Copy link

KariimAhmed commented Sep 27, 2024

I am running an overset-OpenFOAM coupled with calculix using K-omegaSST turbulence model and after the first time step i faced this error

smoothSolver:  Solving for k, Initial residual = 0.0488651, Final residual = 2.53792e-07, No Iterations 5
bounding k, min: -5.28732e+10 max: 2.06797e+18 average: 1.62944e+14
PIMPLE: converged in 2 iterations
ExecutionTime = 250.98 s  ClockTime = 279 s

(0) 11:09:47 [impl::DataContext]:132 in mapData: Mapping "Force" for t=0.0005 from "Fluid-Mesh" to "Solid_Mesh"
(0) 11:09:48 [impl::DataContext]:132 in mapData: Mapping "Displacement" for t=0.0005 from "Solid_Mesh" to "Fluid-Mesh"
(0) 11:09:48 [impl::ParticipantImpl]:391 in advance: iteration: 2 of 80 (min 1), time-window: 2, time: 0.00025 of 60, time-window-size: 0.00025, max-time-step-size: 0.00025, ongoing: yes, time-window-complete: no, read-iteration-checkpoint 
inverseDistance : detected 2 mesh regions
    zone:0 nCells:33882  voxels:(301 301 1) bb:(-0.800002 -0.600002 -0.0500023) (1.2 0.600002 0.0500023)
    zone:1 nCells:57274  voxels:(301 301 1) bb:(-0.1 -0.0999502 -0.0500004) (0.2 0.100051 0.0500004)
Overset analysis : nCells : 91156
    calculated   : 90678
    interpolated : 284 (from local:284  mixed local/remote:0  remote:0)
    hole         : 194

forces forces_object1 write:
    Sum of forces
        Total    : (3.00445 -113.459 1.69319e-19)
        Pressure : (2.81724 -113.381 1.69327e-19)
        Viscous  : (0.187209 -0.0775872 -7.50285e-24)
    Sum of moments
        Total    : (6.12174e-22 -1.33295e-20 -6.04707)
        Pressure : (6.12136e-22 -1.33303e-20 -6.04349)
        Viscous  : (3.77677e-26 7.73974e-25 -0.00357817)
Courant Number mean: 0.0167492 max: 2.91093
Time = 0.0005

DICPCG:  Solving for cellDisplacementx, Initial residual = 1, Final residual = 9.89894e-07, No Iterations 228
DICPCG:  Solving for cellDisplacementy, Initial residual = 1, Final residual = 9.88574e-07, No Iterations 223
[stack trace]
=============
#1  Foam::sigFpe::sigHandler(int) in /usr/lib/openfoam/openfoam2406/platforms/linux64GccDPInt32Opt/lib/libOpenFOAM.so
#2  ? in /usr/lib/x86_64-linux-gnu/libc.so.6
#3  Foam::cellCellStencils::inverseDistance::index3(Foam::boundBox const&, Foam::Vector<int> const&, Foam::Vector<double> const&) in /usr/lib/openfoam/openfoam2406/platforms/linux64GccDPInt32Opt/lib/liboverset.so
#4  Foam::cellCellStencils::inverseDistance::fill(Foam::PackedList<2u>&, Foam::boundBox const&, Foam::Vector<int> const&, Foam::boundBox const&, unsigned int) in /usr/lib/openfoam/openfoam2406/platforms/linux64GccDPInt32Opt/lib/liboverset.so
#5  Foam::cellCellStencils::inverseDistance::markBoundaries(Foam::fvMesh const&, Foam::Vector<double> const&, Foam::boundBox const&, Foam::Vector<int> const&, Foam::PackedList<2u>&, Foam::List<int> const&, Foam::List<int>&) in /usr/lib/openfoam/openfoam2406/platforms/linux64GccDPInt32Opt/lib/liboverset.so
#6  Foam::cellCellStencils::inverseDistance::update() in /usr/lib/openfoam/openfoam2406/platforms/linux64GccDPInt32Opt/lib/liboverset.so
#7  Foam::fvMesh::movePoints(Foam::Field<Foam::Vector<double> > const&) in /usr/lib/openfoam/openfoam2406/platforms/linux64GccDPInt32Opt/lib/libfiniteVolume.so
#8  Foam::dynamicMultiMotionSolverFvMesh::update() in /usr/lib/openfoam/openfoam2406/platforms/linux64GccDPInt32Opt/lib/libdynamicFvMesh.so
#9  Foam::dynamicOversetZoneFvMesh::update() in /usr/lib/openfoam/openfoam2406/platforms/linux64GccDPInt32Opt/lib/liboverset.so
#10  ? in /usr/lib/openfoam/openfoam2406/platforms/linux64GccDPInt32Opt/bin/overPimpleDyMFoam
#11  ? in /usr/lib/x86_64-linux-gnu/libc.so.6
#12  __libc_start_main in /usr/lib/x86_64-linux-gnu/libc.so.6
#13  ? in /usr/lib/openfoam/openfoam2406/platforms/linux64GccDPInt32Opt/bin/overPimpleDyMFoam

@c7888026
Copy link

It is clear that this issue happens to user's cases, but if I don't have a failing example at hand, it is very challenging for me to fix the problem.

Hi makis, I have made a case related to this problem, you may run this case to catch the error.
3000_40Mpa_orig.zip

@martinsaravia
Copy link

martinsaravia commented Oct 10, 2024

Hey guys, I have a student that states that eliminating the cache of grad(U) from the fvSolution dictionary solves the problem. Well, we didn't dig into this but I just wanted to let you know this because it may help you tracing the root cause.

@KariimAhmed
Copy link

KariimAhmed commented Oct 11, 2024

Hey guys, I have a student that states that eliminating the cache of grad(U) from the fvSolution dictionary solves the problem. Well, we didn't dig into this but I just wanted to let you know this because it may help you tracing the root cause.

Unfortunately, I never used the cache of grad(U) and I have the same problem

@KariimAhmed
Copy link

@MakisH Hello, any updates regarding this issue, I see @c7888026 has posted a case that triggered the same error as mine.

@MakisH
Copy link
Member

MakisH commented Oct 22, 2024

@KariimAhmed we now have a student looking into this, but we cannot give a timeline at the moment.

Thanks a lot for following up on this and for uploading cases and helpful details.

@vidulejs
Copy link
Collaborator

Hi all,

I am the student that Makis mentioned. First of all, thank you, Charlie (@c7888026), for providing the failing case, which helped with debugging the issue. I've found a workaround you can use for now, but I will try to submit a bugfix pull request ASAP.

The error occurs with the preCICE implicit coupling method, which stores past timesteps of fields called checkpoints. Also, it occurs with the kOmegaSST turbulence model, but not with the kOmega or laminar. Upon logging the fields checkpointed, I observed that the amount of fields was different initially and at the end of the 1st timestep. This doesn't cause a problem in and of itself, because the same is true for the kOmega case. However, the difference between the two cases is that the volTensorField grad(U) is checkpointed with kOmegaSST.

[0] ---[preciceAdapter] [DEBUG] Writing a checkpoint...
[0] ---[preciceAdapter] [DEBUG] 5 : volScalarField : k nu nut omega p 
[0] ---[preciceAdapter] [DEBUG] 8 : volVectorField : k nu nut omega p Force U cellDisplacement 
[0] ---[preciceAdapter] [DEBUG] 9 : volTensorField : k nu nut omega p Force U cellDisplacement grad(U) 
[0] ---[preciceAdapter] [DEBUG] 9 : volSymmTensorField : k nu nut omega p Force U cellDisplacement grad(U) 
[0] ---[preciceAdapter] [DEBUG] 11 : surfaceScalarField : k nu nut omega p Force U cellDisplacement grad(U) faceDiffusivity phi 
[0] ---[preciceAdapter] [DEBUG] 12 : surfaceVectorField : k nu nut omega p Force U cellDisplacement grad(U) faceDiffusivity phi Uf 
[0] ---[preciceAdapter] [DEBUG] 12 : surfaceTensorField : k nu nut omega p Force U cellDisplacement grad(U) faceDiffusivity phi Uf 
[0] ---[preciceAdapter] [DEBUG] 12 : pointScalarField : k nu nut omega p Force U cellDisplacement grad(U) faceDiffusivity phi Uf 
[0] ---[preciceAdapter] [DEBUG] 13 : pointVectorField : k nu nut omega p Force U cellDisplacement grad(U) faceDiffusivity phi Uf pointDisplacement 
[0] ---[preciceAdapter] [DEBUG] 13 : pointTensorField : k nu nut omega p Force U cellDisplacement grad(U) faceDiffusivity phi Uf pointDisplacement

...

[0] ---[preciceAdapter] [DEBUG] Reading a checkpoint...
[0] ---[preciceAdapter] [DEBUG] 7 : volScalarField : k k_0 nu nut omega omega_0 p 
[0] ---[preciceAdapter] [DEBUG] 15 : volVectorField : k k_0 nu nut omega omega_0 p Force U U_0 cellDisplacement force forceCoeff moment momentCoeff 
[0] ---[preciceAdapter] [DEBUG] 15 : volTensorField : k k_0 nu nut omega omega_0 p Force U U_0 cellDisplacement force forceCoeff moment momentCoeff 
[0] ---[preciceAdapter] [DEBUG] 15 : volSymmTensorField : k k_0 nu nut omega omega_0 p Force U U_0 cellDisplacement force forceCoeff moment momentCoeff 
[0] ---[preciceAdapter] [DEBUG] 17 : surfaceScalarField : k k_0 nu nut omega omega_0 p Force U U_0 cellDisplacement force forceCoeff moment momentCoeff faceDiffusivity phi 
[0] ---[preciceAdapter] [DEBUG] 19 : surfaceVectorField : k k_0 nu nut omega omega_0 p Force U U_0 cellDisplacement force forceCoeff moment momentCoeff faceDiffusivity phi Uf Uf_0 
[0] ---[preciceAdapter] [DEBUG] 19 : surfaceTensorField : k k_0 nu nut omega omega_0 p Force U U_0 cellDisplacement force forceCoeff moment momentCoeff faceDiffusivity phi Uf Uf_0 
[0] ---[preciceAdapter] [DEBUG] 19 : pointScalarField : k k_0 nu nut omega omega_0 p Force U U_0 cellDisplacement force forceCoeff moment momentCoeff faceDiffusivity phi Uf Uf_0 
[0] ---[preciceAdapter] [DEBUG] 20 : pointVectorField : k k_0 nu nut omega omega_0 p Force U U_0 cellDisplacement force forceCoeff moment momentCoeff faceDiffusivity phi Uf Uf_0 pointDisplacement 
[0] ---[preciceAdapter] [DEBUG] 20 : pointTensorField : k k_0 nu nut omega omega_0 p Force U U_0 cellDisplacement force forceCoeff moment momentCoeff faceDiffusivity phi Uf Uf_0 pointDisplacement 

Somehow the grad(U) is present initially but disappears after the 1st timestep. I believe that the indices in Adapter::readCheckpoint() are then mixed up. This leads to reading the wrong field and a dimension mismatch, since grad(U) has the dimension [0 0 -1 0 0 0 0], as seen in the error:

[1] --> FOAM FATAL ERROR: (openfoam-2406)
[1] Different dimensions for '(a = b)'
     dimensions : [0 2 -2 0 0 0 0] != [0 0 -1 0 0 0 0]

A possible solution is to call Adapter::setupCheckpointing() after the 1st timestep. I've tried this, and it works, but I must do some more testing to ensure this doesn't cause another bug. I also need to investigate why the grad(U) is present only in the initialization.

Workaround

For the time being, I will post a workaround. The workaround is to run the OpenFOAM case until the first timestep is written, and then restart the case from that timestep. You need to change the startTime and writeInterval in the controlDict. In your example, it will write the first timestep to the directory 5e-6; then we have to set the startTime to 5e-6.

FoamFile
{
    version     2.0;
    format      ascii;
    class       dictionary;
    object      controlDict;
}
// * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * //

application     pimpleFoam;

startFrom       startTime;

startTime       5e-6; // Restart from 1st timestep

stopAt          endTime;

endTime         1;

deltaT          5e-6;

writeControl    adjustable;

writeInterval   5e-6;

@martinsaravia
Copy link

Hi Daniels,

Thanks very much for posting. Your workaround appears to be consistent which what we have seen. I normally initialize the flow with a transient simulation, and then run the FSI reading the last converged solution and kOmegaSST is not giving any problem. We never realized that this was the reason why the case was running succesfully.

@KariimAhmed
Copy link

KariimAhmed commented Nov 11, 2024

Hello @vidulejs ,
Thank you for posting this.
I tried the way you described in my case, I ran a time step with no precice , just OpenFOAM with K-omega SST , then I restarted the simulation from this time step with adding precice , However it also crashed with this error :

Courant Number mean: 0.0145696 max: 1.93691
Time = 0.00055

DICPCG: Solving for cellDisplacementx, Initial residual = 1, Final residual = 9.47425e-07, No Iterations 287
DICPCG: Solving for cellDisplacementy, Initial residual = 1, Final residual = 9.46109e-07, No Iterations 288
DILUPBiCGStab: Solving for yPsi, Initial residual = 0.00149323, Final residual = 5.42746e-08, No Iterations 372
[0] [stack trace]
[0] =============
[2] [stack trace]
[2] =============
[3] [stack trace][1] [stack trace]
[1] =============

[3] =============
[5] [stack trace]
[5] =============
[4] [stack trace]
[4] =============
[0] #1 [2] #1 [1] #1 Foam::sigFpe::sigHandler(int)[3] #1 Foam::sigFpe::sigHandler(int)Foam::sigFpe::sigHandler(int)Foam::sigFpe::sigHandler(int)[5] #1 Foam::sigFpe::sigHandler(int)[4] #1 Foam::sigFpe::sigHandler(int) in /opt/soft/OpenFOAM-v2306/platforms/linux64GccDPInt32Opt/lib/libOpenFOAM.so
in /opt/soft/OpenFOAM-v2306/platforms/linux64GccDPInt32Opt/lib/libOpenFOAM.so
in /opt/soft/OpenFOAM-v2306/platforms/linux64GccDPInt32Opt/lib/libOpenFOAM.so
in /opt/soft/OpenFOAM-v2306/platforms/linux64GccDPInt32Opt/lib/libOpenFOAM.so
[2] #2 ?[0] #2 ?[1] #2 ?[3] #2 ? in /opt/soft/OpenFOAM-v2306/platforms/linux64GccDPInt32Opt/lib/libOpenFOAM.so
in [4] #2 ?/opt/soft/OpenFOAM-v2306/platforms/linux64GccDPInt32Opt/lib/libOpenFOAM.so
[5] #2 ? in /lib/x86_64-linux-gnu/libc.so.6
in /lib/x86_64-linux-gnu/libc.so.6

Interestingly, When I ran the same simulation in serial ( no mpirun ) just one core for OpenFOAM ( parallel implicit for precice) it ran fines for the K-omega-SST, I can send you the test case if you want.

@vidulejs
Copy link
Collaborator

vidulejs commented Nov 11, 2024

Hi @KariimAhmed,

Thanks for the information. Could you provide a test case and/or full log? Your error seems to be different than the one I observed. Did you get a 'Different dimensions' error?

@KariimAhmed
Copy link

log.txt

The log is attached above , I found that the variable yPSi that is calculated in the fvSolution and required by the method of the wall distance in the fvSchemes is the problem, so it behaves as the gradU as you described and not passed correctly through the check point . I changed the wallDist method to meshWave as @c7888026 did in his case , and it worked fine even with K-omega sst and in parallel setting using mpirun , so it is matter of variables that are not passing well ( may be ) through the checkpoint step.

@MakisH
Copy link
Member

MakisH commented Nov 11, 2024

@KariimAhmed we already discussed that this would probably also hit function objects creating objects later on. This is useful feedback, because it means that the fix we were planning will not solve this case. Thank you!

@vidulejs we then really need to change the way we store checkpoints, not change the time setupCheckpointing() is called. The map you suggested should be the right direction.

@MakisH
Copy link
Member

MakisH commented Jan 14, 2025

The fix for this is now merged in develop. Everyone affected, please test and let us know here. A bugfix release will follow.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Unexpected problems (crashes, numerical issues, etc) FSI Fluid-structure interaction
Projects
None yet
Development

No branches or pull requests

9 participants