[WIP] Check Guard Cells v. Valid Cells in PML

[EZoni]

Following up on #1873, we now check that the number of guard cells does not exceed the number of valid cells also in the PML region (rather than only in the regular domain).

We also make the checks more robust, by verifying that each MultiFab pointer is actually allocated, before passing it the function that performs the check (both for regular grid and PML).

[EZoni]

Just leaving two comments that will need to be discussed when the PR is reviewed.

[EZoni]

Here I needed to decrease the spectral order as well as the number of guard cells from 16 to 8. The check was otherwise failing and my understanding is that this was happening because we have 10 PML cells and the 16 guard cells of the regular grid along the PML interface would span beyond the 10 cells of the PML grid. Does it make sense to you, too?

[EZoni]

Here I had to increase pml_ncell and pml_delta from 10 (default) to 36. The check was otherwise failing and my understanding is that this was happening because we have a mesh refinement ratio of 4, 8 guard cells, and need to make sure that when we construct the coarse BoxArray for the PML,

WarpX/Source/BoundaryConditions/PML.cpp

Lines 776 to 781 in 2476d50

	const IntVect cncells = IntVect(ncell)/ref_ratio;
	const IntVect cdelta = IntVect(delta)/ref_ratio;
	// Assuming that refinement ratio is equal in all dimensions
	const BoxArray& cba = MakeBoxArray(is_single_box_domain, cdomain, *cgeom, grid_cba_reduced,
	cncells, do_pml_in_domain, do_pml_Lo, do_pml_Hi);

the ratios of ncell and delta over the refinement ratio are still larger than the number of guard cells.

This seems to require using a lot of cells for the PML grid when using mesh refinement with high refinement ratios. So I'm not sure that I'm fixing this correctly, it would need to be double checked.

Also, we might not need to increase both pml_ncell and pml_delta, pml_ncell might be sufficient (but I'm not sure about what happens algorithmically if pml_delta is smaller than pml_ncell, e.g., what kinds of consequences this has on the absorption rate of the PML).

[EZoni]

@prkkumar These are the PR and the comment on guard cells v. PML width with MR that we talked about before, just for reference. Your feedback on this is welcome as well, since you're trying out similar things on your LWFA tests with MR.

[EZoni]

HIP warning/error addressed in #2840.

[RemiLehe]

I think that I agree in principle with the motivation ; however, that has some implications for the users of the code (i.e. with the default code parameters, I think that the user will get an error everytime they try to use PSATD with PML).
Maybe we could have a quick offline meeting to discuss how to solve this.

[EZoni]

Update after an offline discussion with Remi and Jean-Luc: for now we should restrict this check to the directions that are tangential to the PML boundary (as we do not decompose the domain in the normal direction), if that does not prove to be still too stringent for corner cases.