From 5f652487eb741719aaa095d3c6423c8d0199a2e2 Mon Sep 17 00:00:00 2001 From: JordiManyer Date: Fri, 22 Mar 2024 15:48:09 +1100 Subject: [PATCH 01/30] Added paper repository --- joss_paper/paper.bib | 0 joss_paper/paper.md | 119 +++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 119 insertions(+) create mode 100644 joss_paper/paper.bib create mode 100644 joss_paper/paper.md diff --git a/joss_paper/paper.bib b/joss_paper/paper.bib new file mode 100644 index 00000000..e69de29b diff --git a/joss_paper/paper.md b/joss_paper/paper.md new file mode 100644 index 00000000..657f416a --- /dev/null +++ b/joss_paper/paper.md @@ -0,0 +1,119 @@ +--- +title: 'Gala: A Python package for galactic dynamics' +tags: + - Python + - astronomy + - dynamics + - galactic dynamics + - milky way +authors: + - name: Adrian M. Price-Whelan + orcid: 0000-0000-0000-0000 + equal-contrib: true + affiliation: "1, 2" # (Multiple affiliations must be quoted) + - name: Author Without ORCID + equal-contrib: true # (This is how you can denote equal contributions between multiple authors) + affiliation: 2 + - name: Author with no affiliation + corresponding: true # (This is how to denote the corresponding author) + affiliation: 3 + - given-names: Ludwig + dropping-particle: van + surname: Beethoven + affiliation: 3 +affiliations: + - name: Lyman Spitzer, Jr. Fellow, Princeton University, USA + index: 1 + - name: Institution Name, Country + index: 2 + - name: Independent Researcher, Country + index: 3 +date: 13 August 2017 +bibliography: paper.bib + +# Optional fields if submitting to a AAS journal too, see this blog post: +# https://blog.joss.theoj.org/2018/12/a-new-collaboration-with-aas-publishing +aas-doi: 10.3847/xxxxx <- update this with the DOI from AAS once you know it. +aas-journal: Astrophysical Journal <- The name of the AAS journal. +--- + +# Summary + +The forces on stars, galaxies, and dark matter under external gravitational +fields lead to the dynamical evolution of structures in the universe. The orbits +of these bodies are therefore key to understanding the formation, history, and +future state of galaxies. The field of "galactic dynamics," which aims to model +the gravitating components of galaxies to study their structure and evolution, +is now well-established, commonly taught, and frequently used in astronomy. +Aside from toy problems and demonstrations, the majority of problems require +efficient numerical tools, many of which require the same base code (e.g., for +performing numerical orbit integration). + +# Statement of need + +`Gala` is an Astropy-affiliated Python package for galactic dynamics. Python +enables wrapping low-level languages (e.g., C) for speed without losing +flexibility or ease-of-use in the user-interface. The API for `Gala` was +designed to provide a class-based and user-friendly interface to fast (C or +Cython-optimized) implementations of common operations such as gravitational +potential and force evaluation, orbit integration, dynamical transformations, +and chaos indicators for nonlinear dynamics. `Gala` also relies heavily on and +interfaces well with the implementations of physical units and astronomical +coordinate systems in the `Astropy` package [@astropy] (`astropy.units` and +`astropy.coordinates`). + +`Gala` was designed to be used by both astronomical researchers and by +students in courses on gravitational dynamics or astronomy. It has already been +used in a number of scientific publications [@Pearson:2017] and has also been +used in graduate courses on Galactic dynamics to, e.g., provide interactive +visualizations of textbook material [@Binney:2008]. The combination of speed, +design, and support for Astropy functionality in `Gala` will enable exciting +scientific explorations of forthcoming data releases from the *Gaia* mission +[@gaia] by students and experts alike. + +# Mathematics + +Single dollars ($) are required for inline mathematics e.g. $f(x) = e^{\pi/x}$ + +Double dollars make self-standing equations: + +$$\Theta(x) = \left\{\begin{array}{l} +0\textrm{ if } x < 0\cr +1\textrm{ else} +\end{array}\right.$$ + +You can also use plain \LaTeX for equations +\begin{equation}\label{eq:fourier} +\hat f(\omega) = \int_{-\infty}^{\infty} f(x) e^{i\omega x} dx +\end{equation} +and refer to \autoref{eq:fourier} from text. + +# Citations + +Citations to entries in paper.bib should be in +[rMarkdown](http://rmarkdown.rstudio.com/authoring_bibliographies_and_citations.html) +format. + +If you want to cite a software repository URL (e.g. something on GitHub without a preferred +citation) then you can do it with the example BibTeX entry below for @fidgit. + +For a quick reference, the following citation commands can be used: +- `@author:2001` -> "Author et al. (2001)" +- `[@author:2001]` -> "(Author et al., 2001)" +- `[@author1:2001; @author2:2001]` -> "(Author1 et al., 2001; Author2 et al., 2002)" + +# Figures + +Figures can be included like this: +![Caption for example figure.\label{fig:example}](figure.png) +and referenced from text using \autoref{fig:example}. + +Figure sizes can be customized by adding an optional second parameter: +![Caption for example figure.](figure.png){ width=20% } + +# Acknowledgements + +We acknowledge contributions from Brigitta Sipocz, Syrtis Major, and Semyeong +Oh, and support from Kathryn Johnston during the genesis of this project. + +# References \ No newline at end of file From c7e35517f6a99c8c1c852fd27c1c341ba34b198a Mon Sep 17 00:00:00 2001 From: JordiManyer Date: Mon, 8 Apr 2024 00:16:13 +1000 Subject: [PATCH 02/30] Started writing paper --- joss_paper/paper.md | 162 ++++++++++++++++++++++---------------------- 1 file changed, 80 insertions(+), 82 deletions(-) diff --git a/joss_paper/paper.md b/joss_paper/paper.md index 657f416a..684c2ebf 100644 --- a/joss_paper/paper.md +++ b/joss_paper/paper.md @@ -1,119 +1,117 @@ --- -title: 'Gala: A Python package for galactic dynamics' +title: 'GridapSolvers.jl: A Julia package for scalable FE solvers' tags: - - Python + - Julia - astronomy - dynamics - galactic dynamics - milky way authors: - - name: Adrian M. Price-Whelan - orcid: 0000-0000-0000-0000 + - name: Jordi Manyer + orcid: 0000-0002-0178-3890 + corresponding: true equal-contrib: true - affiliation: "1, 2" # (Multiple affiliations must be quoted) - - name: Author Without ORCID - equal-contrib: true # (This is how you can denote equal contributions between multiple authors) - affiliation: 2 - - name: Author with no affiliation - corresponding: true # (This is how to denote the corresponding author) - affiliation: 3 - - given-names: Ludwig - dropping-particle: van - surname: Beethoven - affiliation: 3 + affiliation: "1" + - name: Alberto Martín-Huertas + orcid: 0000-0001-5751-4561 + equal-contrib: true + affiliation: "2" + - name: Santiago Badia + orcid: 0000-0003-2391-4086 + corresponding: true + affiliation: "1,3" affiliations: - - name: Lyman Spitzer, Jr. Fellow, Princeton University, USA + - name: School of Mathematics, Monash University, Clayton, Victoria, 3800, Australia. index: 1 - - name: Institution Name, Country + - name: School of Computing, Australian National University, Autonomous territories of Canberra, Australia index: 2 - - name: Independent Researcher, Country + - name: Centre Internacional de Mètodes Numèrics en Enginyeria, Esteve Terrades 5, E-08860 Castelldefels, Spain. index: 3 -date: 13 August 2017 +date: XXX April 2024 bibliography: paper.bib -# Optional fields if submitting to a AAS journal too, see this blog post: -# https://blog.joss.theoj.org/2018/12/a-new-collaboration-with-aas-publishing -aas-doi: 10.3847/xxxxx <- update this with the DOI from AAS once you know it. -aas-journal: Astrophysical Journal <- The name of the AAS journal. +aas-journal: Journal of Open Source Software --- -# Summary +# Summary and statement of need + +The ever-increasing demand for resolution and accuracy in mathematical models of physical processes governed by systems of Partial Differential Equations (PDEs) can only be addressed using fully-parallel advanced numerical discretization methods and scalable solution methods, thus able to exploit the vast amount of computational resources in state-of-the-art supercomputers. + +One of the biggest scalability bottlenecks within Finite Element (FE) parallel codes is the solution of linear systems arising from the discretization of PDEs. +The implementation of exact factorization-based solvers in parallel environments is an extremely challenging task, and even state-of-the-art libraries such as MUMPS [@MUMPS] or PARDISO [@PARDISO] have severe limitations in terms of scalability and memory consumption above a certain number of CPU cores. +Hence the use of iterative methods is crucial to maintain scalability of FE codes. Unfortunately, the convergence of iterative methods is not guaranteed and rapidly deteriorates as the size of the linear system increases. To retain performance, the use of highly scalable preconditioners is mandatory. +For most problems, algebraic solvers and preconditioners (i.e based uniquelly on the algebraic system) are enough to obtain robust convergence. Many well-known libraries providing algebraic solvers already exist, such as PETSc [@petsc-user-ref], Trilinos [@trilinos], or Hypre [@hypre]. However, algebraic solvers are not always suited to deal with some of the most challenging problems. +In these cases, geometric solvers (i.e., solvers that exploit the geometry and physics of the particular problem) are required. To this end, GridapSolvers is a registered Julia [@Bezanson2017] software package which provides highly scalable geometric solvers tailored for the FE numerical solution of PDEs on parallel computers. + +This library builds on top of the well-established Gridap [@Badia2020] ecosystem of Julia packages. The core functionality for FE discretization of PDEs is provided by Gridap [@Badia2020]. GridapDistributed and PartitionedArrays provide the distributed-memory layer for parallel computing, while mirroring as far as possible the serial API. + +There are a number of high quality open source parallel finite element packages available in the literature. Some examples are deal.II [@dealII93], libMesh [@libMeshPaper], MFEM [@mfem], FEMPAR [@Badia2017], FEniCS [@fenics-book], or FreeFEM++ [@freefem], to name a few. All these packages have their own set of features, potentials, and limitations. Among these, FEniCS and FreeFEM++ are perhaps the closest ones in scope and spirit to the packages in the Gridap ecosystem. A hallmark of Gridap ecosystem packages compared to FreeFEM++ and FEniCS is that a very expressive and compact (yet efficient) syntax is transformed into low-level code using the Julia JIT compiler and thus they do not need a sophisticated compiler of variational forms nor a more intricate workflow (e.g., a Python front-end and a C/C++ back-end). + +# Building blocks and composability -The forces on stars, galaxies, and dark matter under external gravitational -fields lead to the dynamical evolution of structures in the universe. The orbits -of these bodies are therefore key to understanding the formation, history, and -future state of galaxies. The field of "galactic dynamics," which aims to model -the gravitating components of galaxies to study their structure and evolution, -is now well-established, commonly taught, and frequently used in astronomy. -Aside from toy problems and demonstrations, the majority of problems require -efficient numerical tools, many of which require the same base code (e.g., for -performing numerical orbit integration). +\autoref{fig:packages} depicts the relation among GridapDistributed and other packages in the Julia package ecosystem. The interaction of GridapDistributed and its dependencies is mainly designed with separation of concerns in mind towards high composability and modularity. On the one hand, Gridap provides a rich set of abstract types/interfaces suitable for the FE solution of PDEs (see @Verdugo:2021 for more details). It also provides realizations (implementations) of these abstractions tailored to serial/multi-threaded computing environments. GridapDistributed **implements** these abstractions for parallel distributed-memory computing environments. To this end, GridapDistributed also leverages (**uses**) the serial realizations in Gridap and associated methods to handle the local portion on each parallel task. (See \autoref{fig:packages} arrow labels.) On the other hand, GridapDistributed relies on PartitionedArrays [@parrays] in order to handle the parallel execution model (e.g., message-passing via the Message Passing Interface (MPI) [@mpi40]), global data distribution layout, and communication among tasks. PartitionedArrays also provides a parallel implementation of partitioned global linear systems (i.e., linear algebra vectors and sparse matrices) as needed in grid-based numerical simulations. While PartitionedArrays is an stand-alone package, segregated from GridapDistributed, it was designed with parallel FE packages such as GridapDistributed in mind. In any case, GridapDistributed is designed so that a different distributed linear algebra library from PartitionedArrays might be used as well, as far as it is able to provide the same functionality. -# Statement of need +![GridapDistributed and its relation to other packages in the Julia package ecosystem. In this diagram, each rectangle represents a Julia package, while the (directed) arrows represent relations (dependencies) among packages. Both the direction of the arrow and the label attached to the arrows are used to denote the nature of the relation. Thus, e.g., GridapDistributed depends on Gridap and PartitionedArrays, and GridapPETSc depends on Gridap and PartitionedArrays. Note that, in the diagram, the arrow direction is relevant, e.g., GridapP4est depends on GridapDistributed but not conversely. \label{fig:packages}](packages.png){ width=60% } -`Gala` is an Astropy-affiliated Python package for galactic dynamics. Python -enables wrapping low-level languages (e.g., C) for speed without losing -flexibility or ease-of-use in the user-interface. The API for `Gala` was -designed to provide a class-based and user-friendly interface to fast (C or -Cython-optimized) implementations of common operations such as gravitational -potential and force evaluation, orbit integration, dynamical transformations, -and chaos indicators for nonlinear dynamics. `Gala` also relies heavily on and -interfaces well with the implementations of physical units and astronomical -coordinate systems in the `Astropy` package [@astropy] (`astropy.units` and -`astropy.coordinates`). +As mentioned earlier, GridapDistributed offers a built-in Cartesian-like mesh generator, and does not provide, by now, built-in highly scalable solvers. To address this, as required by real-world applications, one can combine GridapDistributed with GridapP4est [@gridap4est] and GridapPETSc [@gridapetsc] (see \autoref{fig:packages}). The former provides a mesh data structure that leverages the p4est library as highly scalable mesh generation engine [@Burstedde2011]. This engine can mesh domains that can be expressed as a forest of adaptive octrees. The latter enables the usage of the highly scalable solvers (e.g., algebraic multigrid) in the PETSc library [@petsc-user-ref] to be combined with GridapDistributed. -`Gala` was designed to be used by both astronomical researchers and by -students in courses on gravitational dynamics or astronomy. It has already been -used in a number of scientific publications [@Pearson:2017] and has also been -used in graduate courses on Galactic dynamics to, e.g., provide interactive -visualizations of textbook material [@Binney:2008]. The combination of speed, -design, and support for Astropy functionality in `Gala` will enable exciting -scientific explorations of forthcoming data releases from the *Gaia* mission -[@gaia] by students and experts alike. +# Usage example -# Mathematics +In order to confirm our previous claims on expressiveness, conciseness and productivity (e.g., a very small number of lines of code), the example Julia script below illustrates how one may use GridapDistributed in order to solve, in parallel, a 2D Poisson problem defined on the unit square. +(In order to fully understand the code snippet, familiarity with the high level API of Gridap is assumed.) +The domain is discretized using the parallel Cartesian-like mesh generator built-in in GridapDistributed. The only minimal burden posed on the programmer versus Gridap is a call to the `prun` function of PartitionedArrays right at the beginning of the program. With this function, the programmer sets up the PartitionedArrays communication backend (i.e., MPI communication backend in the example), specifies the number of parts and their layout (i.e., `(2,2)` 2D layout in the example), and provides a function (using Julia do-block syntax for function arguments in the example) to be run on each part. This function is equivalent to a sequential Gridap script, except for the `CartesianDiscreteModel` call, which, in GridapDistributed, also requires the `parts` argument passed back by the `prun` function. In a typical cluster environment, this example would be executed on 4 MPI tasks from a terminal as `mpirun -n 4 julia --project=. example.jl`. -Single dollars ($) are required for inline mathematics e.g. $f(x) = e^{\pi/x}$ +```julia +using Gridap +using GridapDistributed +using PartitionedArrays +partition = (2,2) +prun(mpi,partition) do parts + domain = (0,1,0,1) + mesh_partition = (4,4) + model = CartesianDiscreteModel(parts,domain,mesh_partition) + order = 2 + u((x,y)) = (x+y)^order + f(x) = -Δ(u,x) + reffe = ReferenceFE(lagrangian,Float64,order) + V = TestFESpace(model,reffe,dirichlet_tags="boundary") + U = TrialFESpace(u,V) + Ω = Triangulation(model) + dΩ = Measure(Ω,2*order) + a(u,v) = ∫( ∇(v)·∇(u) )dΩ + l(v) = ∫( v*f )dΩ + op = AffineFEOperator(a,l,U,V) + uh = solve(op) + writevtk(Ω,"results",cellfields=["uh"=>uh,"grad_uh"=>∇(uh)]) +end +``` + -Double dollars make self-standing equations: +# Parallel scaling benchmark -$$\Theta(x) = \left\{\begin{array}{l} -0\textrm{ if } x < 0\cr -1\textrm{ else} -\end{array}\right.$$ +\autoref{fig:scaling} reports the strong (left) and weak scaling (right) of GridapDistributed when applied to an standard elliptic benchmark PDE problem, namely the 3D Poisson problem. In strong form this problem reads: find $u$ such that $-{\boldsymbol{\nabla}} \cdot (\boldsymbol{\kappa} {\boldsymbol{\nabla}} u) = f$ in $\Omega=[0,1]^3$, with $u = u_{{\rm D}}$ on ${\Gamma_{\rm D}}$ (Dirichlet boundary) and $\partial_{\boldsymbol{n}} u = g_{\rm N}$ on ${\Gamma_{\rm N}}$ (Neumann Boundary); $\boldsymbol{n}$ is the outward unit normal to ${\Gamma_{\rm N}}$. The domain was discretized using the built-in Cartesian-like mesh generator in GridapDistributed. The code was run on the NCI@Gadi Australian supercomputer (3024 nodes, 2x 24-core Intel Xeon Scalable *Cascade Lake* cores and 192 GB of RAM per node) with Julia 1.7 and OpenMPI 4.1.2. For the strong scaling test, we used a fixed **global** problem size resulting from the trilinear FE discretization of the domain using a 300x300x300 hexaedra mesh (26.7 MDoFs) and we scaled the number of cores up to 21.9K cores. For the weak scaling test, we used a fixed **local** problem size of 32x32x32 hexaedra, and we scaled the number of cores up to 16.5K cores. A global problem size of 0.54 billion DoFs was solved for this number of cores. The reported wall clock time includes: (1) Mesh generation; (2) Generation of global FE space; (3) Assembly of distributed linear system; (4) Interpolation of a manufactured solution; (5) Computation of the residual (includes a matrix-vector product) and its norm. Note that the linear solver time (GAMG built-in solver in PETSc) was not included in the total computation time as it is actually external to GridapDistributed. -You can also use plain \LaTeX for equations -\begin{equation}\label{eq:fourier} -\hat f(\omega) = \int_{-\infty}^{\infty} f(x) e^{i\omega x} dx -\end{equation} -and refer to \autoref{eq:fourier} from text. +![Strong (left) and weak (right) scaling of GridapDistributed when applied to 3D Poisson problem on the Australian Gadi@NCI supercomputer.\label{fig:scaling}](strong_and_weak_scaling.png) -# Citations +\autoref{fig:scaling} shows, on the one hand, an efficient reduction of computation times with increasing number of cores, even far beyond a relatively small load of 25K DoFs per CPU core. +On the other hand, an asymptotically constant time-to-solution (i.e., perfect weak scaling) when the number of cores is increased in the same proportion of global problem size with a local problem size of 32x32x32 trilinear FEs. -Citations to entries in paper.bib should be in -[rMarkdown](http://rmarkdown.rstudio.com/authoring_bibliographies_and_citations.html) -format. +# Demo application -If you want to cite a software repository URL (e.g. something on GitHub without a preferred -citation) then you can do it with the example BibTeX entry below for @fidgit. +To highlight the ability of GridapDistributed and associated packages (see \autoref{fig:packages}) to tackle real-world problems, and the potential behind its composable architecture, we consider a demo application with interest in the geophysical fluid dynamics community. +This application solves the so-called non-linear rotating shallow water equations on the sphere, +i.e., a surface PDE posed on a two-dimensional manifold immersed in three-dimensional space. This complex system of PDEs describes the dynamics of a single incompressible thin layer of constant density fluid with a free surface under rotational effects. It is often used as a test bed for horizontal discretisations with application to numerical weather prediction and ocean modelling. We in particular considered the synthetic benchmark proposed in [@Galewsky2016], which is characterized by its ability to generate a complex and realistic flow. -For a quick reference, the following citation commands can be used: -- `@author:2001` -> "Author et al. (2001)" -- `[@author:2001]` -> "(Author et al., 2001)" -- `[@author1:2001; @author2:2001]` -> "(Author1 et al., 2001; Author2 et al., 2002)" +For the geometrical discretization of the sphere, the software uses the so-called cubed sphere mesh [@Ronchi1996], which was implemented using GridapP4est. The spatial discretization of the equations relies on GridapDistributed to build a **compatible** set of FE spaces [@Gibson2019] for the system unknowns (fluid velocity, fluid depth, potential vorticity and mass flux) grounded on Raviart-Thomas and Lagrangian FEs defined on the manifold [@Rognes2013]. Compatible FEs are advanced discretization techniques that preserve at the discrete level physical properties of the continuous equations. In order to stabilize the spatial discretization we use the most standard stabilization method in the geophysical flows literature, namely the so-called Anticipated Potential Vorticity Method (APVM) [@Rognes2013]. We stress that other stabilisation techniques, e.g., Streamline Upwind Petrov–Galerkin (SUPG)-like methods, have also been implemented with these tools [@Lee2022]. Time integration is based on a fully-implicit trapezoidal rule, and thus a fully-coupled nonlinear problem has to be solved at each time step. In order to solve this nonlinear problem, we leveraged a Newton-GMRES solver preconditioned with an algebraic preconditioner provided by GridapPETSc (on top of PETSc 3.16). The *exact* Jacobian of the shallow water system was computed/assembled at each nonlinear iteration. -# Figures +\autoref{fig:galewsky_scaling} shows the magnitude of the vorticity field after 6.5 simulation days (left) and the results of a strong scaling study of the model on the Australian Gadi@NCI supercomputer (right). The spurious ringing artifacts in the magnitude of the vorticity field are well-known in the APVM method at coarse resolutions and can be corrected using a more effective stabilization method, such as, e.g., SUPG-like stabilization [@Lee2022]. The reported times correspond to the *total* wall time of the first 10 time integration steps; these were the only ones (out of 3600 time steps, i.e., 20 simulation days with a time step size of 480 secs.) that we could afford running for all points in the plot due to limited computational budget reasons. We considered two different problem sizes, corresponding to 256x256 and 512x512 quadrilaterals/panel cubed sphere meshes, resp. We stress that the time discretization is fully implicit. Thus we can afford larger time step sizes than with explicit methods. Besides, the purpose of the experiment is to evaluate the scalability of the framework, and not necessarily to obtain physically meaningful simulation results. Overall, \autoref{fig:galewsky_scaling} +confirms a remarkable ability of the ecosystem of Julia packages at hand to efficiently reduce computation times with increasing number of CPU cores for a complex, real-world computational model. -Figures can be included like this: -![Caption for example figure.\label{fig:example}](figure.png) -and referenced from text using \autoref{fig:example}. -Figure sizes can be customized by adding an optional second parameter: -![Caption for example figure.](figure.png){ width=20% } +![Magnitude of the vorticity field after 6.5 simulation days with a coarser 48x48 quadrilaterals/panel cubed sphere mesh (left) and strong scaling (right) of the non-linear rotating shallow water equations solver on the Australian Gadi@NCI supercomputer.\label{fig:galewsky_scaling}](galewsky_visualization_and_scaling.png) # Acknowledgements -We acknowledge contributions from Brigitta Sipocz, Syrtis Major, and Semyeong -Oh, and support from Kathryn Johnston during the genesis of this project. +This research was partially funded by the Australian Government through the Australian Research Council (project number DP210103092), the European Commission under the FET-HPC ExaQUte project (Grant agreement ID: 800898) within the Horizon 2020 Framework Program and the project RTI2018-096898-B-I00 from the “FEDER/Ministerio de Ciencia e Innovación (MCIN) – Agencia Estatal de Investigación (AEI)”. F. Verdugo acknowledges support from the “Severo Ochoa Program for Centers of Excellence in R&D (2019-2023)" under the grant CEX2018-000797-S funded by MCIN/AEI/10.13039/501100011033. This work was also supported by computational resources provided by the Australian Government through NCI under the National Computational Merit Allocation Scheme (NCMAS). # References \ No newline at end of file From 5e280ee09aa9c8d1c5b7f95e02740efc78d7b4b7 Mon Sep 17 00:00:00 2001 From: JordiManyer Date: Mon, 8 Apr 2024 16:43:08 +1000 Subject: [PATCH 03/30] Added demo code --- joss_paper/demo.jl | 123 ++++++++++++++++++++++++++++++++++++++++ joss_paper/docker.sh | 1 + joss_paper/packages.dot | 22 +++++++ joss_paper/packages.png | Bin 0 -> 25269 bytes joss_paper/packages.sh | 3 + joss_paper/paper.md | 72 +++++------------------ 6 files changed, 163 insertions(+), 58 deletions(-) create mode 100644 joss_paper/demo.jl create mode 100755 joss_paper/docker.sh create mode 100644 joss_paper/packages.dot create mode 100644 joss_paper/packages.png create mode 100755 joss_paper/packages.sh diff --git a/joss_paper/demo.jl b/joss_paper/demo.jl new file mode 100644 index 00000000..4449efdf --- /dev/null +++ b/joss_paper/demo.jl @@ -0,0 +1,123 @@ +using Gridap +using Gridap.ReferenceFEs, Gridap.Algebra, Gridap.Geometry, Gridap.FESpaces +using Gridap.CellData, Gridap.MultiField, Gridap.Algebra +using PartitionedArrays +using GridapDistributed +using GridapP4est + +using GridapSolvers +using GridapSolvers.LinearSolvers, GridapSolvers.MultilevelTools, GridapSolvers.PatchBasedSmoothers +using GridapSolvers.BlockSolvers: LinearSystemBlock, BiformBlock, BlockTriangularSolver + +function get_patch_smoothers(mh,tests,biform,patch_decompositions,qdegree) + patch_spaces = PatchFESpace(tests,patch_decompositions) + nlevs = num_levels(mh) + smoothers = map(view(tests,1:nlevs-1),patch_decompositions,patch_spaces) do tests, PD, Ph + Vh = get_fe_space(tests) + Ω = Triangulation(PD) + dΩ = Measure(Ω,qdegree) + ap = (u,v) -> biform(u,v,dΩ) + patch_smoother = PatchBasedLinearSolver(ap,Ph,Vh) + return RichardsonSmoother(patch_smoother,10,0.2) + end + return smoothers +end + +function get_bilinear_form(mh_lev,biform,qdegree) + model = get_model(mh_lev) + Ω = Triangulation(model) + dΩ = Measure(Ω,qdegree) + return (u,v) -> biform(u,v,dΩ) +end + +nc = (8,8) +fe_order = 2 + +np = 4 +np_per_level = [np,1] + +with_mpi() do distribute + parts = distribute(LinearIndices((prod(np),))) + + # Coarse geometry + cmodel = CartesianDiscreteModel((0,1,0,1),nc) + labels = get_face_labeling(cmodel) + add_tag_from_tags!(labels,"top",[3,4,6]) + add_tag_from_tags!(labels,"walls",[1,5,7]) + add_tag_from_tags!(labels,"right",[2,8]) + + # Mesh refinement using GridapP4est + cparts = generate_subparts(parts,np_per_level[end]) + coarse_model = OctreeDistributedDiscreteModel(cparts,cmodel,0) + mh = ModelHierarchy(parts,coarse_model,np_per_level) + model = get_model(mh,1) + + # FE spaces + qdegree = 2*(fe_order+1) + reffe_u = ReferenceFE(lagrangian,VectorValue{Dc,Float64},fe_order) + reffe_p = ReferenceFE(lagrangian,Float64,fe_order-1;space=:P) + + tests_u = TestFESpace(mh,reffe_u,dirichlet_tags=["walls","top"]); + trials_u = TrialFESpace(tests_u,[VectorValue(0.0,0.0),VectorValue(1.0,0.0)]); + U, V = get_fe_space(trials_u,1), get_fe_space(tests_u,1) + Q = TestFESpace(model,reffe_p;conformity=:L2) + + mfs = Gridap.MultiField.BlockMultiFieldStyle() + X = MultiFieldFESpace([U,Q];style=mfs) + Y = MultiFieldFESpace([V,Q];style=mfs) + + # Weak formulation + α = 1.e2 + f = VectorValue(1.0,1.0) + Π_Qh = LocalProjectionMap(QUAD,lagrangian,Float64,fe_order-1;quad_order=qdegree,space=:P) + graddiv(u,v,dΩ) = ∫(α*Π_Qh(divergence(u))⋅Π_Qh(divergence(v)))dΩ + biform_u(u,v,dΩ) = ∫(∇(v)⊙∇(u))dΩ + graddiv(u,v,dΩ) + biform((u,p),(v,q),dΩ) = biform_u(u,v,dΩ) - ∫(divergence(v)*p)dΩ - ∫(divergence(u)*q)dΩ + liform((v,q),dΩ) = ∫(v⋅f)dΩ + + # Finest level + Ω = Triangulation(model) + dΩ = Measure(Ω,qdegree) + a(u,v) = biform(u,v,dΩ) + l(v) = liform(v,dΩ) + op = AffineFEOperator(a,l,X,Y) + A, b = get_matrix(op), get_vector(op); + + # GMG preconditioner for the velocity block + biforms = map(mhl -> get_bilinear_form(mhl,biform_u,qdegree),mh) + patch_decompositions = PatchDecomposition(mh) + smoothers = get_patch_smoothers( + mh,tests_u,biform_u,patch_decompositions,qdegree + ) + restrictions = setup_restriction_operators( + tests_u,qdegree;mode=:residual,solver=IS_ConjugateGradientSolver(;reltol=1.e-6) + ) + prolongations = setup_patch_prolongation_operators( + tests_u,biform_u,graddiv,qdegree + ) + solver_u = GMGLinearSolver( + mh,trials_u,tests_u,biforms, + prolongations,restrictions, + pre_smoothers=smoothers, + post_smoothers=smoothers, + coarsest_solver=LUSolver(), + maxiter=2,mode=:preconditioner,verbose=i_am_main(parts) + ) + + # PCG solver for the pressure block + solver_p = CGSolver(JacobiLinearSolver();maxiter=20,atol=1e-14,rtol=1.e-6,verbose=i_am_main(parts)) + + # Block triangular preconditioner + blocks = [LinearSystemBlock(), LinearSystemBlock(); + LinearSystemBlock(), BiformBlock((p,q) -> ∫(-1.0/α*p*q)dΩ,Q,Q)] + coeffs = [1.0 1.0; + 0.0 1.0] + P = BlockTriangularSolver(blocks,[solver_u,solver_p],coeffs,:upper) + solver = FGMRESSolver(20,P;atol=1e-14,rtol=1.e-8,verbose=i_am_main(parts)) + + ns = numerical_setup(symbolic_setup(solver,A),A) + x = allocate_in_domain(A); fill!(x,0.0) + solve!(x,ns,b) + uh, ph = FEFunction(X,x) + writevtk(Ω,"results",cellfields=["uh"=>uh,"ph"=>ph]) +end diff --git a/joss_paper/docker.sh b/joss_paper/docker.sh new file mode 100755 index 00000000..fd54f5d1 --- /dev/null +++ b/joss_paper/docker.sh @@ -0,0 +1 @@ +sudo docker run -it --rm -v $(pwd):/data -u $(id -u):$(id -g) openjournals/inara -o pdf,crossref paper.md \ No newline at end of file diff --git a/joss_paper/packages.dot b/joss_paper/packages.dot new file mode 100644 index 00000000..799f3141 --- /dev/null +++ b/joss_paper/packages.dot @@ -0,0 +1,22 @@ +# https://sketchviz.com/@JordiManyer/a34cdc3942aab5c0e011c05ce12bc2e5 + +digraph G { + graph [fontname = "Handlee"]; + node [fontname = "Handlee"]; + edge [fontname = "Handlee",arrowhead=vee,arrowtail=vee]; + + bgcolor=transparent; + + "Gridap.jl" -> "GridapDistributed.jl" [dir=back] + "Gridap.jl" -> "GridapPETSC.jl" [dir=back] + "PartitionedArrays.jl" -> "GridapPETSC.jl" [dir=back] + "PartitionedArrays.jl" -> "GridapDistributed.jl" [dir=back] + "GridapDistributed.jl" -> "GridapP4est.jl" [dir=back] + "GridapDistributed.jl" -> "GridapSolvers.jl" [dir=back] + "GridapPETSC.jl" -> "GridapSolvers.jl" [dir=back,style=dashed] + "GridapP4est.jl" -> "GridapSolvers.jl" [dir=back,style=dashed] + + "Gridap.jl" [shape=rectangle]; + "PartitionedArrays.jl" [shape=rectangle] + "GridapSolvers.jl" [style=filled;] +} \ No newline at end of file diff --git a/joss_paper/packages.png b/joss_paper/packages.png new file mode 100644 index 0000000000000000000000000000000000000000..56721ab864ade953442df62235ebe5b47e78bef3 GIT binary patch literal 25269 zcmb@uby(AH^fyij$S8%;4bmVjU851Ckq$?Lq*4N-JEcQ%QqmxuB4g4epwyJ^hLQWt z&-eFS&-48MT)W=8w|non&mHGJ=e%BLu{y6*i2yVJ3=9k+bu}eD3=B*p`Y?Npi>{F< z%9leIxNlynDq*1hy^6Za(=afYFw~V44E*x(*=`{+*SEi>TzQL12fJHGgHblpKCg}Crr z=21w+?3Qk&7p*?`bDW3NxgUEok`4c!r4dcug)a3Yjr(cQ4>pYXfBKc0yAHb+g93|o zD(=nM6lrWBMhg$ZxGB^P(S^AwCj&<%VU6noKCVAMH+4?U*=87=p*VkTfu;GpNjg%t zNh1hWJ_BiQmw}(vc|A^NuBzfS>(A?-CcL9)Bl2UcXBEJz!RrzBAoLLQ2+}ys8MdVg z$Qs9DfFd;>bPP-&UzM)Egzog&^l#1MtZLy*XSZOKy}8EspsB}fdx8|+^?zS%d24`B zGB6=Ymyoyc9ONa);Py2_a(KKuZpRVFKvLI}c@Q603jNUC_O71W3YP4sWyEyAKEmrM zD3iB?Ruxo|cTp4GVl&DQ-|HY!^Y015-nry#2xUwJoUa6uwqfR|YF$?-5?$H#ZH4&a zu1+Xs66z}~D7oVfQ1@T51n^w-h%*?H7J|xp7en7y{3N^{>VOrmd?&TmPeEw~?n(E{b0}X>h&Cf zf7w;3FHa@!N>sJ%TG-L*&RUxT$DLf(o2-qOp%B8q5aoUuKVd3%+{G=j%Vm|Lu&=7G z3et2V8whCF1GE^oZp7a z;EWqx2+1C8gJ94K!5vLEgmtQZo?u&MNhBm|x8enA=9dN=3lCk9MTEa-c zT)~wll74dM13A7w=okFB{Y83f@ikodUcuu~-+Nq&r|oh?N1Gl|0|%V?K+jciuKs|| z-Kh1wfxpj8!vNba@NeB%d&5Y~F0N9{)!DzMA1QYyEt8$Dg1oeCMUa zcLWT!M#rvE$zffcgTbBTqH4H%R+|i6mHw9aEmF_+rNSz*ye$e)(d47@*|FRq9Hf`` zGJjgbr;zJE;CXVbMagQ&2`&6nUM2Fu1k7P4+zGkZl^l=UrRZ4h>wf7O&`QG_A+p5c z)x7$nx&~=ax0movfd$xyvCdM#P0G}gBPETl_=-Ty#80IuUf8Dvctq6S+gTP06oi77 z_X2ML{$+0+jRqN%%5EuUOM7Dr5|6c6yIU|sD`Wxtd8#c{Lx2^+s1IQWd3e_-@ z^=gf3#F#bmZT-DS{p;9?uNVsjw)$%)_`OH)_sIm^i7UTVuKD~jxB--p$pqD z6DQov*#z6w%MeObl1rf&!wwA9V1k@9tL={aK?LE+hkjh#;MSrYw>4dUKN=4#ZdRKd zPUzAsS>nZ!me&CDd8tI=-|a;7pvUXx+>uwg2df=*Y`BT&+fXzD5|oK@Ydn=?^-Bq1 zl`i5z+P`*G_;%{Z$QO`H3(w*NwGR3GZT%lh>~ztL&<>yB;mT^cd8f3>wIA5d%c;+- zf20}}9=l2s#fT?M!uc^#J$W~vQz~Yaqn68_^~|M>U-s)4V*kY}!_99i{>&SjZHNzt z&0XdXy~eILI7fUMg~@-|&%TrcNNh+m-nBgC zf-x2~)StzU=7-Bn?CCl#eeS=C4oy?@q39siVhbQleEVEs{Do@qiax0?rn|g;*o)%f z4ZZbAefVsyNPdTNC|dZycPxWG9J#Ly_wU|1n*P3ahkPJEnV!? zDRngTxQp!jTA>+{_!%^;wJn{+pREZe@jfMq?3bN6ez*rL&zoDuR|C&thEFXj{#@^1 zt$UliFjV|}NfoUD=}rjsvh~P+GnP~LpH-1J{O7h}K&R>#-{A9+%1Z=&{7QDQAdQKT zyk#dW`|nH$FDs)0@w?9+!8IL?u4LVU;akfU^8qUL-4Yg_36&b-ir@{1>WBf#9e-Rm z!|oRVeR;!E98n31#8?`z#=wZ-kR zI(V+?W>3J}_&k1OLbNrwnacLq-qRE%nh7d#x0=l_-oJLW@=XElrkutf^)e!5j#aHa z)!DSlfJeI6^>sKxYq)JBTLJ_eaUU&8*k-Iwr#bRblYS^3g+ zxDnd#*5^EHc98VB#rpQ~vA=b6`uYe~C-y2%=G58Bv(L|G(6{mIqotvsvT% z$#viz^p(nccl@OSJEzj?Cd;hQevmYbo!$_K)_25=p3avxT)M62dC+{dupz-iZ+h|D zvXP6&1>HQNJB|G$sVId7`8&`5E>E1t;@NrY{Hd@^hZ@m+ZeV_viW*gb7rU_@KScyu zztIpX>lZ=15qVie-$v1vTR@Tg9FgAqX!MEqwv6jZj{%zyY?jbnpFQo5nGm!SGBu&B=mlS&pL;SIG=Eda0z!5r{|F-NEl#Q;sD-GFo2FyBv!a? z5SzI^@_D8E9IQ#?@nZM=;8@j(2tQiywzPO`L2zV?+6wV%euKk}hY3 z-@scLJ*XRFcrnMZVOgeaEW#Ax1b$B}j`aH|)q0ZZuNb(mnns?1**qju*K8b__Y#77 zMMQs~t@R%s>lbvd*GaIgRLIY9S?uJ**r=P-u&2)ZU2sz%QqUzUWtr6xoylz8CdS>G zI@PAfwYW)TM{V?s>;cI=I?k9p35?T=b#^bu-OJ0^KM9sqEzm^j9;nE~%RRYJ;m4vV z){xwPEB!Gn!Ngwt*nO5@u`&deA4T)h^DM}y#H6KUrjX|bTa4jsl)*xzaniOWFh!N(*#IUKr!XVxarFVe8mBDU~PG-B9o7+P9O z5x@h6!KXFa?8PhBO-geSO*)P%AZlFcWV$C>?w2gioGAC>#j72YG9P85MmI!UqlQC5 zB|rrm@3NtndeHU3XTS9<78s&#FPV!i2UKCjks?_>l1dc|9(a;W%duZ$l^A7dRJ?Ly zsVIFI9g3p;D_E5yJSl7unNNYa5?Y|}lGlo?vEqFZZkr$sm?*5fe}T;2>rvu?=GYq|F%P@>Eg07!U=SZ&_11eF-ch z;xu(JzZgl1ic^-1A4BkZa<}Bbq24EcU4mtgyKpY3(z^Wku?{)w#Mxm~h_pWM3cO%p z(^Nk7hQl!6b))f0T4j>FRE8R>mu29~Z#_JzltESrTpdcV?A*0i%^@13szu65njN#O zgTcbi2{MU5)SOZ=xO4K|SAvw*7WJ%dUGddtvQhSYz?q3}RNw#)3>sEx-iZg?W|68N z#J5Zi?JuCSQXvq%z#(a2k#TwE;Isw2j82&mjwHddIi&Et)Rezyy{iDO1bjI*eE*2li?>Wi5R zb7&m$hXc@Vva%i4=`p=?3WdHo2&Wr8Czp!D#m8GJ6zZu=oIq#YKZu7)V2jqQ%rsz4 z7F42>zJ!mtIAk2V-HFTqI zeue@X6&s!OmK8v6HvHLkKy-I8-@!B%?WOseZ;IL{Q7?rRsO$rxs$caEtkx{lPSh*KZa9+WA}R2->?_O{Ys_R|&8L1GdO}V-S*F$wP}^zh$nbUo>)eJg_L8+{iDL&S>u4I?#C>G&d<4V7p5Z&du2xuH-40ZWWwqJKd zZ8GTl#!NYmN+vK}iFg$)zR)U;)1;S~uYgcktQk204DAAOC8Q<4*P|7{ejF@xHNAz& zP+t1ozI{M+QjL*43(p|;yMru@&T?+u_a02nOo!+bVA;d#^L_e`4_~?R8{oG00#ylH z#RaZ>k|mMxE7Gw67eO_;?lQCqm%z+Z|X7-6`IE`{=8Mm&T zC#IL92rL^@m?kkNSn#ca&?~p?g`aBb2YF>W)VQk|M{XVhjfAfbrKMt79qu)RV=)b{|Q5+(E zsat0H2CFf%E9S6wzT)UdU5MF#qESl|ny0or;Yy>GykuLSYzF6oeu|@KgfH{iy`rPB ztx_(?E<`x=!q&S63#=I>PHb54b@y2dkKX!7WzuBBPmRFDo6p-hTCWUC&KHO>e*JPH zc$4tsMJonM%`8SpI3L~CM9mXQ4E-74FkwurBhr?7isk~ZoJ$8VTa3HI@o%Q3SzM}Y zZktlCE|EyM)j(T7Lq@FVCM4_Rxy+)-kwO2aSmOu^o8fZ@xjB{OB{TGKffBL#8v2&N zjLE=NVqOKAJ7-9bmC8#(TCgU+k39L`5Biy<1`);J`5EDNuA*Z0&YuoaydW8li8tLd zz>0`2Vo_t6YTJaJ!9hnFJ<%KyGhO$4L4#|V&@A2>CxfiD^~dYs$A|yc zuWh_hbHY|RdbT`9-sUWpdhkyC2H25G@(3Gvu#I%!K)#gRK7(kK|QE)bs3Lmdg~m;Cl_Br8CjdU z1n&iD{74!t52C`h%?a;vT)X#bwu2a7P6XIInsQE#E`{Bgg&(PZk@qQu1zPpCmNquy znDoEEE!4`X57SQmQ&Io5q+w~QqCQapJ${6jk%dE#{8d;6Wr$^Z8!GyU%THN6BXQ&` zxvJT#2Ch!#)9$PGY>3N zrEN;bnhQfk@k2GBvoA{(B@EZeJO#!M)m2Ko7IY+v;H_uiLWx$K^s=jcc~j^;!!4pJ zr)iI2+)o)@qx*+8dG2(Dg6qI2fcpe_6*%lir1zXQ7C6Tqz|&bzNk@_S0(vYzTqlKG z-DFz*y&wS^%uRk@G4x4k;bubBO%W`zyG+6=0TiCq-}+Uu`CP_-xKHYgDl z1R6#0!Oc6kwM}CWQF*%Q-JknDD+JUg?+KOZ_+X9W%EtT!-4VAv)~bAz@i`U_4*D<) zC)`UuQOKoo^^T5_)csU1sn!kx)UXFK$TP(f6j}ywI(|)WbbL*%Vz6CS((p@N35Wkd z!tL--h5t9;lqVayI{(o;MKc&QqGClsgfmJML{R7-gXX-CxUq5x}vINq4z&eit$D(KYkzHcTXZ zc2d&c&MfwY5=gkoYR`_CC(wL+YnR@cOP#!X&ZQ6sDM%=IxlnJa0NpRdZ_&m+i{ULH ze<{j?WIvZL=YW3FeJN>6?);Na0f+DEjHSBSCC6x%DL-PRaA`t6cJ$+0rS6C&k;31* z{)V(QeTRM3)}4$1KQh_ddWP6?f5(-(TrUEb8(XP!I`8)byd}&qHqOmKX^}68SSG^+ z@;Vyeo#Xn~4prAq9=*e2{T~341FixJ+(ovwA-kyYXR~Q}scB`bE>GZuom5;m^?x6? z8ouSOmWN^EeV-0ivT@F0{e172vRUHL)^cT|895abWq1kduqCn9woP?XB0_TLREG%2 zsH;eYWIz$o#kAdGGUPYS)+qLOr2`tmwWG@cFPx@9+MgNSW1=#lSv!@-jEX0B2V+(N zG^to}kGam~OQNXZwjbn#zDy8JPXld5cBT^ou0AkrEn(|Set8O)@FWh`I>B+*?f=vRIt8D~IM zW;FZMkFJeu4BShp{X^*5Nj~vhM~*&>j+-_m5}9eNF$l^c$C0Dv^YY-Vm)or*QC!O) znQ02kOV-*g>!?(exnIIPG+sZEBe|e@AX*CLCYuo>o1BG1+c-It0LIZm{vG$Ac{J-b@2c;z^C82x$kzK z)UrLPy=%on@&Od~C1KF9mA?5&9WybZWfX-dR^uC8#wRl&$(Rt;8Gz(u+gy8vo<^Lz zYXY@&Vqa+Amag_!q6=a-9NG9yr`N1nYN03`iz5I+wPM~t>t%cH?i(6_Q=pvQCBY;O zM%oOpik}^`5Su{(s}AP@qpQ#{E>^HiIstnH$2>IMQcP$S^2za1ovn$y5#zZsaV$!H zesO8OFyRN&hZ4iP%+m4Wi@yF!4WIyF&g#J2nFgY~Pq1GWta&5C1j#}v>QYHf8J}|* ztgtAPO_J8Qq?RsSb1tL^W{e9U=pETr#Tt9)bE25iLXV1`B-rd-%PgF`Y0f!;7&hiv zgdd!nMGi+rB55#6Lr}`hN>&PU-=H|!36;1>@`=Z8B3<1OiB}!i$1csz^DiAWgES;w zEs@kImcb((3yw|KBOzqR_#&%*MWZTW7yla%5(L`gZpD|s=d8UT>XUdz5(^Bl7-ZS$ z8p7)hsv2zsrp-_zy|p{=oMe*4$wDTIPI`$E&LWI*E_BIn?(3A+GM$X+ox78Mh1K4Z z1|*SjY>a93Dk!SlZ^W0c;terIMrAOtb{u(6k##X zT*#CFk>|aS6Wt4QOL#kr7*5~BT``d0&I^{{SnSR8_r%}8&9?pBMw(Dd)+mDQ-yBSP zV{U!a{U|TKSG`ODmRq>~(WSVsg6&T|)E5Z^YzIP}e2tCoF$x)W`)MY#UsF2wyUEOA zQ{>#oGOWvT4SP!NQ1+PjFs|olqsl5mt#MZKW-4WI7^kt&nmjg3FL*U~c~bmV#?j}M zDY`4hXv2bhiJ6Y%C%nxH?!_M6t_t+6McO&D0v;Q>HtN~SMGtSz-ww~d9$XC>};Iwq1@Gonb7!%C;`UHqPy^O$kK19wj(Cx#K>k%N2U4BN>(b0Y?Ce2Y|f;%vY{2`_L@-`kHB|$9_y&qw@K9JhFUCViAU(5?=Sv*hOs;OYU z@|pcW6&x;U55A^7#Z^Pq{u)%+3S=kx!#a}jYImwvB3=Zdg*)EA5lnJ8Kzzo>@WT(F+GXx!Gub+^7IXRLB?Q>&={2?%Fx=NA8TTDCo zUtEAemdXqterNTgf)u&@FWI-GN71IE{8LTjtBG$;qa#k2vX~~`iM*ERnf@tnHj1KO zzx86Af6wEUyTcE4umb-MZ!e6r^k$YpKCE|>U%dGU^KrmIbuuAkv$qPA*v@fp)-;-_ z47jc_^~6nz=k~JwoE78FNCRFaGQ>~@^$1N{k;vL|(+J(jq^4Q<$*jG(z6r35)Z?cp_1az+St*4jc&PRY|OOkpl zdCag#eJw|f4c_#mdr}ye(;?rfzokyJMlNwB9H#;i3|#PivK992&_S)C&Yx*NDUifc zDXxdiu-ZatLM$P{XxqEa{(-Ryp^|k>*Q85#U#+wEepD{D)|1MV=wRU7Rr(oP_d+z` zR*(6MJTH7%drRnb#pkgn1;A>^E+4wT*W9rX7D%Nf(ZDktIYM%T9UDaz)e@QOsx%yx zHT?*w{EmM@7035c$Lf3dqu0*4q2Ta|Ac;KbxKZ!&;<;;^3L$?iEiSGIy_E@%UkIpg zi_TWvF%^*lpzvjvLs}#ii3w{pS=^M=@Q^XfP!%sil08p4IqG-Eq<|TGx&IRCMUc+& zCG4PWiC#U!jyBgfS1L1IIhfW}E^)Ig^CM2*<+HqrByHqd$v9dMhGX71=xn0tk87g~ z$xPd`U*r9PfInCw3>PBzA5cC;@4xfQzv7cB!63_mokjp!7QT2Bo#!fUe4G{1j{CWk z)uqi$^4D|y$TNa{5LEaIXjD2ukcQp8z;-|bWyY4y__U0m(=t5GJC3&{8B7>4*6f`#E`49ns6{dik6v(HN zQ@z!n^A{BnI+Rj+@;W=M$Uoc$W9U^0Pj$-y+i-Cxb|=37On?opB*C)fB5SYcSnRUC zFZV%;8@8T?<`1zBHMKcx4qJbq8oY30*cP22XhD+`rk=uGZ)5_Jx=EB)UQS7Q5uA?F zCkKFLbmTMSp32Oi#rZlNv$7#q+OVSMNyg1#V$!*cR(y9@MD#tFqUKShr-5`rQ{kCr zL^0{du`=MCNOWzS#%eJX$Qz<{v?(GP3B1(+!w((g+d zcIRU)Zyv&`fndOh*(v<=i5E+NXr@^D776f0u> z*L1Tt>dw`LZ4&sO&H$tW}Pq)8$s;&@@2X#Nn5ZkFB&g-~A}7`izQ; zDmNX+-76COzr(#Z8_!ja5uF#n~1nVxF1K zg8U+<@!Bg4&5>1kQ}EMCt&Q~JwaMzZ*WnWz_Ea?BW%+|gKyJ2LFo1Q$&WE(ovKAyu zu4vCY^^NRmQut8`wdC`_pCg^Iu)8^onw_{A{-%_)w_H~&wC~RygYRY3D*17HLeqnkrxRcgJ@NSg-@e8dsx*Xa zvzo5VQq71h7OR|U>y)+=aJ59Iew0hy|L)B%CRIYUPEvH+NTDq9fdroi&1AvdO;vqp z)jc0@6G;!^a*lYB{uk;LFp>mXbBr$9@rf>J`1aGWavBfJIPOij3d<3ypi zom}*%UP|}qyZYj4@Z|9V zLAvrmcIy2cf)sv4soz#f_)7Gm^REg(n@L09#Yp3mJXd{fDBPE%rAf!O$|H7DSt;?i zM8A)6J5G&Piz4|+s@l#`h~~}N)EnL_)=1Q2wnL|!I51#0CCW(-_{D~`k#IRK?3;|* zk>T#J${cwhDPIN4l1$5|NPq*qsk)=AssFYK^tg~{ngHt8q@|^s%pFP%HxOf06V{y& zq+=wR_avsR)Go@npG!BfEM<6YY|~5eo)=akxgm(qJrCT(buy8BlPff>Is0T)SnamR z_cK|>uUoLwbLwfqj0(h=Ab{(xomU3R0 z^sJ&CLm7SSW?gOhmXuhbTJa)XzQwLCcpt-7eF=MkAn|ipW=jyE%!q%5{t2OOH70uL zDI^}%WA<I=>^_) zQGH#=LU>L%I{*Q>jeL45X|cd-Ti}EZi|%)r+xd#Lsd~C((s&LD42B$32 z{(PxWK|@ZHzleBnQ)t8my7vwD`W=g1kAD59C+mAkpZeM>{!q9TmSIIZIV<;HDYFW- zYAR%0T~=TvLz2X>Zhw^)3tkU!U@zeGLoC#}6Tq3VXO z+4}3JbE|1PCRx!GUnf6yGBL}jl4scQAVa6rW=r#8EX(jE3OR5Jhi0U(bfr=oC$i@7 z785Hl8`zO=(ZShpx)A3?BdCWei}3Gl#2Kf7Qxc2b%C(+bQJR~rsaYiV%%GY{FDY~b zEnje5TW04Ey2YYb-+=+#MNa)bQUOuV-g4{;bdlb65lWcV5>MwY_z+1nT{ zCc`v;55!0h4(!oCy=&!yi%}SYPQYvpG!`EqiRE88`@ZwOgGgfi`fHR^i}3sNL~B79 zf^$)y`IKnJK#O42Hfl10j=Kh5tvE6HzK4iA*(_DgdV+%Gb1(5v>b#iU2qRlVq%b$$ zNs?l=SQd0#*|=cs`-qg%`$5Oq9ctQjr^F3<6)60yCcd7lt)M=%3CMNgx3=}udiAA$ zZ83X8*aSn+h0#C|I2K@BVUU*YR3X63ZN9#)&SKD}{+$-sjy{-+D+Md38eSk7UcC4g zv=??n8dYb*dvJ z8RXKe0#LG@-d_dvL>wovl19IB7Tci|*K?uoL9dFG8Z6XLnflEIJ2=0Qcku7L(y(-a zo1zRIzvi2&$WJ=&dR_(e?73^{bN-7b6`$19>n$Xd(HcpXrnI_NthW{kcD(}Xor>_6t61eUJ(j)=U4AJ&TEwC%5L1sUaik6AKTzDpIq9*=DqsRDw?| z1H?{26MVh%&}XHB?5s;5+rKXHL5E5_NW+oy;S)Ibl-8+G(gzltRbq2Z5y)ybuK@m- z0|`0O6xt^j$E(+?6xSOMPu|R{RC7-F@#w=7!^bC{U2T;|ZL?4w*&$L3_i2MZNd)gK zt!YR#EexuTeDz!*@r0j+@uMH2taR$by$_4SuUClKMh*1!z+RQ%E^HiV8&OT}IcWdN zY=*Q}$vlR;kN-T?LCAx?N(7?*P+q5LvWtrl{zZ~KZ88(LB&^i%688F6a2@s zl7budX3b6h(B9IO0{|4!G1-{&^P2$TDIyeeJqQmFdrjw|gl{KgUvHiOYK;56%+t>U@6WiwN*Uij3J z_|D2JSdM}V=}c=X3?ocQPL(G9^aY_H_`7ow@b)L{7tUrO;Eqf*py$=m=deUB%2mb0 zb1sQqZDqULzl|?#6VyN9BB&815kaep3RYecco&vtq?d+5r8_%`(oaBjn-`8>+v(%^ z(gp92O>GUOgl(+oO&RUgqE| z@fc^)fZPwp(!VzQKxa*@Q?1kJqJx+H^O##UP4Dub!hyfZU86tqBvoSega&L2-@wnL ztl#*5M+1M6Q|B_}o%}c?de&?4|L|t+dg8o&EvH|!*O0)j!oWiAa9&t8pi~aW{j`Zz zfO3#TT+q%$^Suy*gzsf5=QQzWN;G4^$+({+@#!{$59XGhjOQJi)n*`WbUgTn*O~yp zxU1}vAjmf4VnM6V5S;Wknvr-ZQL7A=O$x?w*Su~cQ|@w6^*l;eGepx44FA>n`#KG$liCC?S`?MZJ|TpqbzZui;IB1YYuUIj z)@W`-KcFXYAD0)apy=vjVVREv{k1b)W+NS%o;LvK>C#l;3FpJ|(J-M%lO>yK>zwxz zg8}Jzkf%-L9*oCW0y5}6>A1u9BX&d+>-7Eo7`g<1p_xy-#{dsnqtma$bA1eb(IFRH zBj`4Omcq4@Jvsj_vvjSy*8Ios_wpoJ$=+|I5y$l-M^Xy#S$Jv!-a+|7D*sUgf4uk- zx)%DLv-!YhCl1XM1pcd&9nlrYCq_`LM>r%12E9S|nq57R>pHbh^nb`~jY;T~h>U@e zKHAW3o&5lQ9dN>r_YqooplMht4$)Yd{12Q}Z?;1pWyhVV=obS$I0o-oN?WZz5i zcNm&PN5X@2ogrKbPmfs3c+IeiGl%$zohZm8V3^;FL^9>u zY61<0%j9fr&$Na$wHJe#Rd6a#Zd!;OHV0zoHEfpG=y+#3Yz|pK165ax)y9J16>^YX zv?#VrwtAT>Lp8;UtT|nDv_FVWzJiJj1hz-W9E)GTu~J?xWQSBJ2;w@*Cnl=c3K5?r zem=q-j>%th2Mq_S*INL(NTHM=%6;+uU#U+%0OjpS&Uq6BMWe_Aot$v(u{s|ua-AD& z*EDpwgk()l7xA+yXF_Lkr7Nqp(f z8-M=GniioBd^}We0h?t)FH;*dmU)nnem$;`8yC=E(f0=GQJ^}^VIXgF09O+(c7cLA7wr@6f9duL)$8>dWfcw z<u5w3+^fHBp}vlOhHZ=qv;;G`b>*0j0Hl zzeGaxGy7DS-j5K;h=8D3owHAjj)R1;6TwkVz9KzcI9%bXy8S`C$SxLo!7(Oky#3g@ z*Q-)$ADHAqi4j}~%h}s*G33gpIB2+oqyQz7{kSIGu2Ok71cH&K8yGf*UO}2~rytV+ z0x@ewx`2k8W;!@1NZ*H4tmE@2@ZHOG!5j*Wj+Z-$ZVQ(3qTA~My&=_L@dYv1vMq_4`I&pn+w+{~FBJ z6T-|na|bCzzF@Y4(Cy@&B+F+;f%Vdhjr?d&bI~d;)(DZN1cqL*&UAi?*EDq{3dCq5 zGD$AL>50I)fcCHIXs-yPS(4=a2J7<8PyGq3!m>t1et};%KOO~$aNG|gTD=-coE^K0 zN4G>_*d;M2chDb8 zRQ&mCP2n<~U0v76Yscyb2TtUxN{J^#Hr=&^TS90ylD9zBlh>@}?VCb{sP}MI2r~_- zF)iF%pJ~DehtB0imE#1h^}|f~-4Q2(PSM8RqsQ;}KYZ=R*ZA)3KE&ksnKsJ~HSoqf zN&|a$ieF(yY{W*km#Di5MwKS1)#sb4ex@>fM6{}@9ht#D_76O(K&&}&T^A1MI#!D% zV7(^BfjD~9)=?Z6@^T_mYNpx(#o~SnG#>3M{>?^6<7FJveGRLLh7pDeRHS*d5+u3ctNRTVRk~pU!IlQKp#5Ga zG!~YobDV$6RnvMO)K~b@@6Pnk*0u$Dbl{IgqVMgd&k|M()!L8JJ`$!7;V_;?S+9P= zzwJp3xH7ZBku$b0QJDpwo1gV$cM+I;_u-s6aS5Ggefd*r20aNi*1ZshKmuvL;4tnx zcObHNEgTgDNL*e`Dm#g^zYC1BrOTzSeAmHWEk9T%dib#q~YOIK4vyE)=`AzAthOpP={(!{kJf_ky-W z3*0Fi#|sb;9@9Ds| znDmUUGRk(p^%>4;R2TrAvKYIu%)qyKHkH4&sejyB0`SuN_I@1j;H|)WG;`i7p!1a* zdBFAdy#0#@A`j%Kq7diS&kWIu`muZS6XUjo%P3M)iiiCFM{Pm@{`4OI(w^e{nMRf3 zZek;XAlk?SYmulRq1^DoFOSkPtGc)p3L@2F$JO8{?GhKdC|&?#nZw#2)jo4XIX+7J zjj!#P-&3xeSJ3YrF||R?=tQA_*OWz3gQ`=vx;$MG_H6sFOV&_S7i#n?{?>HKG!~53 zh>hRbjcIl`a=1Aknsh?Gt&II2c}<@`$U4&{$WMGF#zAo5nH977lPCZrSJqag;7!lb zZmaw^-#TWs;?WZZUGZL|F0kod#8yU}P}HYD{#DS|J{l2eUBIFIQQX)V_L`(M2T*w? zd-S6nPVb|f?=PwM!9{m37J_NUApC3uI!8=DlnU2f#px3x3!{y=7ZZRxga-I*p1eQU zZ$5>Cfy34-rR6DjC~=m-g1o-INO%1l;Zm_BPU_F7t^7)(_>?j3=O66~ z5Q+@IN#@%q#m$T&fL9*6&JNI`fk%n?jSoF|OoLRCzvuzx;oMhp#GuXBKkA^2M5hVF zJ-&}t2h`rS7CA^WTr8Ahaun^6)LpqXuk`WS7E{I3NST#Jsl(CWigA6NsW_F)8YJ_W zE~^yx!jnxpS`c;@{UtT)Jw*Wdz|IHL*J65>RvdcOC;nUNKydETf5-<78RUPE55nMF z^Wt++t;2bLTM~W8hJ_{7f_U(dtNdBVnO54O=4{{NA025H^m_a1n2~W~^zc99RQfo! z?RX@|#~!=!hNCMkFU{s zH2%j2N)L9L7x+6$iG@6JQ~^;kzgFg6Soq<8&P?ej=y)&MhKx2 z+_&xi{M-(oO%Y`uD^xGWSZxPL;MrC7?gk5U7!H3{0*j6NRl8^tXI8^u$fiDg&Wf=S zsMrTv-qU4q(a&cNvD!uOl+OSG+k_uB;--O6Yzck(oLX@hVb1+jGwgYD)oS*;!9OZp zukJ*E`8$xC5yu2B*;}$m^LxtNTUSlC27XiLW&#t^#%W_2f-z!hw7t%yi+6`TmUP1X z0S1NJsP+4no-|(RLHO@%IW zH142J$QITrT<`;smO8Wa1}goK1pC#rt9t4V>~5ILj+pa9S^jep0-ESOp4UIVNI8@r z7fQRmNfTtV#uD;fjR^Q&hp}4b;I~|TN)8m^`a+bVHx*OHX>UVB2-akvNqK#`QK7sN z`6d^Vxo&5b$c4MejPtbCw-+-KHyCef*Ds^*!Exb>ki4iGyVNYEREcBk2kZ~UkL~qSf8FD+4|uw#Y?iayoBU$%d#aXT z;rPF}05#Iv1zm#VXh#B-Aio#R__zz#@J@==eUYedP3@N)t~5QAvf=N9P}-F6628nt zcdGC&s`I4+ zAI8+CKgF^%*C-G*55-gqd2aiomw<7s%v)3dwv;S(q;3PAqdv_=@c6#7K7xj3Dr1IcW*Dc!3}b?; z#5t4YMa3DF4M%^{oB1kKWeY2cFWwyonMg#|>bz3a2&QvxGa^7|DI7jucK~btw861r z*q!v-Y-ZA8)>Dysw1~uF{iGBZjGz0PGT1V9^P&#W@&&xgliJ`IJL@Uo!~+~EV$O62 zrTlgAc9+jgRj!x9)_$ZxbnJ>xXT{4pl%NG?ac?wZ* zAsXNk4Q$AXW5iiK!c_5d!8x(pO$q!i ztoKA}(7h)S8!kqK5dQ^LhVweWHFO zH)IDfs!W#@O?O}u3Hgz!Z<$1{@1Jcn-bshw3G2ai`TUGtx5)K(K)eAD4u5ogd>+GZ6B7P*I*|Oy7j1B56YMbM8ic|Vg`K_Rq2tfMVe`4;5fACol zcJDz?q8qxB3ssNd4&1my_m$NSML%-qskJ@-zaGgwf7o9roy6~x)ExAWe#HNT>2-e& zURuqas(JYRzf-scT5)xF{k(`&wW$>)Nn<&0jOAlJ7F|ftM~3FWa+ZVN z`Iy)VZq1a8@Ec=XE!*njlj(Sx@!wec4a^j@{RpZ_S)W=)^l_Y;;~U6JYksdoqgco0 z*$`HULUWPT#DQ={f|^%>a^@>t?1gaB)=QmmF7kMHP(Wi!_uwJbEn#_~`wF(7i}pTC zNd*TCP3^ZPC=UMTtA^;fy(y`KOx`rJ<{yJ$|1^^R%PpW-_GYyOpLI-oukip9bGmqL zELskRPPrnmNT27VmsQ2Gw(4wFJX>chy04`2dC4Jb0w9f$i?-XOJbzID_5!<2gJS4X z&?$OByT=g)vvl-QbWt!V;~!6$E+BBXK@tD`ghMZ&kAB}|JOZ$93RfkC0I&7)2+m7$ zd2KXEacGS&9AQ8~i_g!1Kvfz|I9R&D;$!*$>Z#ag$0XC91UGA3@lPp*;!L8*p*iLf1R){!`o&-^Fm0;?iV zviBD6fN?E40hAeg2*U2S^D^4OyHPbpf^ZHU>&h}e7)F|(W4M%r%5?<&+O zd$SGrbHBOudn_t0iq3lS17lc{$1TWX?Rq9a&UCR0njHOD+i9hQ3xJ_O@C=ml7G0VD z9=c$SV{VXcxqm?u-+(_|o2aIbe7 zJ7p~zb2{8k7z}^`VHN06t4R=r2&JU81^r1s*%IhUM#F95WPsJ66K5|9w4rJJeNu=^ zVqqkGUPXZB9Z*o)1%;uDM)A~m{Un*rM-f;_f0tX+ZbFY*93zl zz36VEc6-PgQ%!zJ>_oJ=MXm~hT`EFmtX5+< z={QsuzCBi=+2}yetX^txx;<8^Rl6egKGocXfj=l~;joVc^sOAd&?>o)94G8l`lB zO3S}_vI>u1DVAn&7-G!Wz*tv$qfM8)!mO$X_#&t@w%}c+Np*F6iR}7=fAi%uo1B>7 zN$eZnHS`QGkJg&6kKopFNo>15r(+sv@8IG9B*8sBJyO=z*2gdyjOT9>vo3c=5#gm^%q18e|@d{sanVHorV}PeXm$>RCQ?6x?SOl0J22X=WzyqqWt?#nbc3&2N2ueb5|H;JjA_ zXX`ZSPwqI`LZom)aAs`Vyq6nlf6}U#>ka%0!q;rDdp;z3#WEy(EZ@sLB#fwaSo>p$ z2FyU4l8P$J!qO6kn~W(On&9i1>X0^#+D$=$ScrVG4GOh!0pmdjFj_%pP)MyY-)}j$ z$>gYfrydee0B%1>ttA#ul;GBd^Uu2&njWzMh?(%>lV1wY0ToP#a_Q z%(*VK*o>EI?zEiG*VNUWxw1eWZEbHGE&U4WPvehvKVCnaghba4WQnz914w3PXB%SL z=>ywWXxDA`Zb-5Xk9gAj6I&Xy7tPkp!a}gPrNz4!P{3Y>>wM#`A10Amridrs)gG+; zdn41EjEoH7%hh<{lVJ(rgLce0V$lnOjV5O#5E2#t_}JLo-2D8l>tgxu*rDX-hS;{n z-&5M}M@B~4f*}bKm?ZgBQ^j*| z@2N&ZfHZKl43Cey5Pm#!T{QDGdGNU-&BDlZF#$AW{2d>!TlpPJCqyK6eekc7SgiSW zSc2G@cZB8g18x0 zS$TK^6nIrzS67$ewj9Kd0vSjWP4}04+*WK-TTjOC5l@P7RW_B=m)q?Xi z&@|A_3s|%I_&`S=-_(F^jzRCXRyHEB* zc_(xbt!5)R1>hwGfV_aqh8?$pm6qCrH~;C;X}fk>YHI7~{Jf}VdCQrF@50smgwty5 zdoHhw1D`u9`PzfkcqNAc-id<^vAf3Vk`hY@z&f%2Q;hOF*0eq%)L`5+R8k^If_2Lrf@)4YU|8ubp=}w#3Vj@bH%O5zPr**SJ-3t>n@C7-%@0{UFANf%+QuIEgRS_ zABx$)S-3mj@46l<*^R2aN3XcMy*f|3YeGU-z_?A;`%?hneUkKVwl%JIZG9oP*Ow6#+`(awUlF~YRLY*nG>(Jj&ra}rUD%lnWe!};pSS+I z|CnKPC_~6~ADBe6i?eeX25nD=lSOI|*c6U|c%a>PVq(JA1>8O3vKcwe1ii`QICfjqQzBK@ZG=Mi7w~I^uuq5dS?1yzRbhVe>~{g8vj^PAbYD&?8Y4NL__B=9yp3S3tL+; z8aE#6OL;*6TNbXepFg*zfow%Y&l`t$o58GYW#IOER0|h46Z*a)bkY%vfEPA4bE8d; zR{MY#thSosLV;$teqnIA`x~xTKd|cdpqZ#!$b| zk}s^=c;Oc-eswgi6LZ7ElG?(JCO4$j{j*t_A~S&}eg$U@DbvsEaid)X%@W!zQob6o zEn`ZjAXTBy7)uDdUiQy6y88GI_X}beYYPVY=x@w&cb@y~&kkkad+l|eU$waYZftB6 z6LULQr0wLmC3BlwsJyR{C3EoZM;L#KyfFwC_*+l@$`#z+TwSro03wcXUGOF=8T*rF z+uEnVKex&)#=sDqoJ<}UAAdu#^j$?w4ZD3MT1?TxqPSj|jV-F7K>(2NYWeV2IFH?; zxSsEgGeZCFpT^Pg@dSpqKmBc#Ttkzj?~CU#U(@6ZqC*HM9yQs_365p;(fp{Wc;H9D zZBiq*w~*i5oc{GI*1QLj^=xkr4j5ckQIUa}8Q%=1lok`y&RpKST&?5i$nEa#j&kmK zc`vJ~su-D>kxn94XS)#KtK^e8{2$Uu$0R29%r-i{0fQ+#{PgKlOh$&n$B&Q9&COZ# zn#t1!)>Yd~hceNyaB=m|_=Kp5)XTJ_q@{7f!ou?M@;HDhU3RBWGvK>(up}^`L^j>& zrEBY%y4aW)aUY-7`u<&)<2~4Z9O%`-flF3a7UfAn?6c@{w~=^!zKh$*fkLoqp`tOc zHqd3v_wU_6n6j3Z$a5P<(<@Hp-&Ot1?#FH?Bk$>D<1%s9FN7GDwjY@uyldx3k+%&2 zFQ{J2G|9aoMKQOE-SM@u`uTtQtomwd-GHgTLF+|mua{RHnL;`{N;Rtmye=y0cZRe7 z^9tA78f_LDcVPPc>2DB){U+5a>gv+&?r#8{QGyxnE^AgAVFw$!O*GoCNT2O53;}nn zP;WDbCZnWOn3wnURi;}2a9o@Yt1|n|QD&(yCf0oL3m?Td$3Kzo91+FE@>!xS2C5m#He_;thho3xgS!NOKYjiYc49M4jVF?K&dOnxN zb=EUI{+PG`RV!1bS@evIzc(Vzm)&gIZd}5@eVeK_CAMuo#0kV9T!yo^ec7F^o%eXm zAa4Nph+51i9k4@OT-+}}!tTIlTmUj@#=S>N#b>qH?vHu0JCl^8?~^hIb3I-PdCFxv zNc~dQ2+U@Fzlts;C54BVx6=E{CF>|S7zY4QN=u8<``-yVY8uPSMmU6&UBzl8jO^?H zx+2YhExUSqRh5+&ML4cnZz{lrrRvN>Vt2wY*ZqR3sF+IFYh^7MGLzriAYn z4U8t!>qyqf$f!cAEP|XfAW6^TetcgS?o+QtRad~{R#R0$Qr}29eXiY{bTZPCGdUyzSbaY5cO2)*;OPH825QPK+^fVx>vjps4gy&S4BN(!d ze*Jn!(D{s;8&N2qXbeIz2@8wY{IhnL>LuU1x+F8**D3s5aPcyBLkZCs(kSD=}-VRezJhQQQ@JBRNPDZyPNYg9tAl}qZ^uNJLgcjlXv1V#_sZ3T3X^a zqQOBy_ItB$wzs#ZMT*v!UuQp(#HRbf z+?;7hnT`)YFyPK0-FQer5jL}1P*D+6 zQNi)|@84$UXd0nlL-JXd8S8njA$r##WD0j{V54*0Q&Kja-`AJNDIgC4x!-e9(M%A< zaenz7qX^mk^YPePqz9MX--W|_8kqutWEcTZqzJ~^-{#&5!%j3o`T3-kU@Dsn zrq08|Y0L6#bILbw-XtU>RJiWziQb&37MGNSMMk!~Kt@DFeCdm$2MiZQBjmMyGil=( z|4QkM!P)Q6pAQxm7R{#Tw3#n$$b(%SU~{6oRBR%fTUKHK%+>D4CV(VD!^4g4Zf{Vo z{tL*SX!Ovxwn?Nqj4C(9*-_x7L4mj93s=U^pFa;>$wh?;VJMdoMg7J>YU0V`#}eQL zySf-2r@a1HTU+}zI2dJVIk~x z0nPJOo111+<5E*UxVbf)S6%l+QlU4jL@{w%S!(}a-J)k;Z~z*t0O3QO&0P4L%|>M}Eo^6K^EHUm8|{_?@UX_8rrLFf zVzFak6(Tf0fBKZ?t|f>YkS!6y1;Pn7orZQ0e|>K8;05NHUt9Ymr1Pz}w+LWaG1;TG z-MSe&*j#^>SXOvA9_rjJM4|h?t-=}IOx)1W(8ZGSR-?{f!pU8i9Ty=X*S(joUKQ%n z2}EyiThMy$a4Kd9O3BILU!Cup&NsU^=&uAzD65^8KC3AOc8i0uO%ZR<1uSMHrZLYe*XOeYt+OwYWDO}wf8-S&B*fa`HF*62Cz$fbTMx1o@vAtII-M*Gc zq<7%^FBch-D?vY3{+ev_ZEJ%Za-|Kh=r$!>Uwipn%`7|u3@~MdqBAnG-hJOYPd?ju zc}2y+Xb+;6DJv7;pybo|B8P{S`%Sn}Q|%_xKcP?VBgW#H*fL55idP;tOzNHbU z0(eH-iKJP`#lcdkUP~&Vad6f1=g+^Gm>l&`*UU{ale zj)}togu-Vt`y1tC%F2shUtV6WuC6|yqWV@`?6g0hp^__IX)#u;c-5}|nu{xLB=&^lMe)^`NX&pp-QT z+-S=}FT2knK7cEz8!Q=n2fr#1Uw0Br5%X;=sjWR_i9w=7_o~8|I|vzB%J@fGP0hC+ zU<6U5l6y@v#G4J2pPo*Q;zM9&*EEB?xgZ(v({l(Ax)g-DgaT=1ut`A)RUKwgx36C3gzPZ`=H3gF= zAtil8P3<*Rj|C-FK%lZi)dG3@vGH+Qhen;S8WAeJhCvi2`F(jueDJdaFVd!(! zm#`U|`YR!}w`lD~l^jl+$|y(pEOJ3vMWq9nWz8BqNhn=$1HqPLNn)g&hD-#z=VA#` zH1vMqVglb`r!l1zUbnKPx0}KTT2~1${K!g^+ zjx_=U0u+TZ25Z|(+H3iDG7sWGCk>>MXoyd08TsZ~@661K&fxmD9~I=}=-#}c_}0C= ze}T1xE9A064h{p5PiQ$i*UBau!wku(2s+*HC#AF@pWbdJ$Q=BtT_;^sR8xCpk;*5p zrUn@;QrQC}mgKdRNLOq>n`8WxDl6Ft2YrKpzxCN2@dMYeR1r79q7eXm1o zH8r(_q$Er5tE#{}ytw#1ufhzLRc={FI=n3wPxt%95Q7eTfZwNQb6>b3xd|W%hUDup zF%L`~9IBW&IaL&8WDE`#JBo_8_df?&4ySON3OH>(05i?4s`_{Z${FTd1QLK<3V9S} zOvt4v4j$%srX4(f2n45xjZSAScLABRaV>@-)umJ-K+ZP-;(`F1)=LxA5qlE!w6wJ~ zQzC@OeslP_=-strOH1_F^*{62~LcTb{($twjcmw=mHT0!5SN<;`^1FbfCWE z!Ifl_?`L8v^{?JoEeSq<)cnjz5eEk|tldsF?$IZ;n?}S|1lG&=zAs%}T>`kov{tk{ zW-%mywmy9#QBhGj*&YL8!xy7aKopgf0UWrVDfcUFrbO>kJCaKhYTCw*qn) z-_sEFiY{{B#`mY)P{8cGAK%w|;L~8R$S|$7vj8||=!q0Xfw|V#nO$|XQuUG&{Wc#D zE2~mU8X8v5N4}DhSPvgQ6dlqhyOx`99?TL`0`9y}DXXBaE^&HV>-MzV*BnHT4q0Np zdd{%9M}B=@EE0_`D-c`4_oy*MafvVsuNw&ih9AZ*VlNfuVsMJ5o5sB~|503g*aYp} zZlc}!X;%4J2P6w4bT3}S!ewXB&@iM?-w}||>V1{^tFv=!4#vNLp|PGk@18g8c_fI!&tWZ?D|}O67nJ5fT!D8dDDti6bu$&+z2rrov*}%dNL>-#UYB z=A9dL-utopKPFR%^qGD6_Tnu7F(6y4P>dXZnIBIdH9+Y1L^SoghLp4bL}RHQr`uBHdM)?v-_HhO21z)g#nUw)FtA1SnaI*kuZ6XBw@p1D z)CGh38T0=hKTKVr%rEX?dl$Fj{b6lQ56;m1MB&dYu zEIv?QUdZlIuD@$d@|F4ROSiZ8Sk$Mb*O8cCb$L#4Jeb4(zE?@OCmZL1Z1Ty4BIf|6 zLs)NENWYbUEJxbTu42Zh(x6t0zIuv9vrN6)iVv<{@iRcx}quU2s}gKQ8>Px9Ua9I2h|Ci`kAZg(a}{9-R|}DNOy7&C>`8#-P1 zeo#pu)l5t?))XyAdbGgT1ToWLW)=;rH*G@ysNXiIhheeb)75wVu_1*Z6fc+i(=_68 zU~qZXqN$QVTg9SHFN>G>vQJ)n2IEQlt_yRpA(npAi~5O*xFLGzmh+ znyn4Z_c;hUFYxRvQ_pOtgLP`Xg%)tg7sf5~%DOMRRS{ChUCLzv&+h#{Kfj32@80cU W8=iAq`~}}bfhH@ZC<%RM5b!^OMncs9 literal 0 HcmV?d00001 diff --git a/joss_paper/packages.sh b/joss_paper/packages.sh new file mode 100755 index 00000000..657116b2 --- /dev/null +++ b/joss_paper/packages.sh @@ -0,0 +1,3 @@ +#!/bin/bash + +dot -Tpng -opackages.png packages.dot diff --git a/joss_paper/paper.md b/joss_paper/paper.md index 684c2ebf..9b2ddc27 100644 --- a/joss_paper/paper.md +++ b/joss_paper/paper.md @@ -43,75 +43,31 @@ Hence the use of iterative methods is crucial to maintain scalability of FE code For most problems, algebraic solvers and preconditioners (i.e based uniquelly on the algebraic system) are enough to obtain robust convergence. Many well-known libraries providing algebraic solvers already exist, such as PETSc [@petsc-user-ref], Trilinos [@trilinos], or Hypre [@hypre]. However, algebraic solvers are not always suited to deal with some of the most challenging problems. In these cases, geometric solvers (i.e., solvers that exploit the geometry and physics of the particular problem) are required. To this end, GridapSolvers is a registered Julia [@Bezanson2017] software package which provides highly scalable geometric solvers tailored for the FE numerical solution of PDEs on parallel computers. -This library builds on top of the well-established Gridap [@Badia2020] ecosystem of Julia packages. The core functionality for FE discretization of PDEs is provided by Gridap [@Badia2020]. GridapDistributed and PartitionedArrays provide the distributed-memory layer for parallel computing, while mirroring as far as possible the serial API. - -There are a number of high quality open source parallel finite element packages available in the literature. Some examples are deal.II [@dealII93], libMesh [@libMeshPaper], MFEM [@mfem], FEMPAR [@Badia2017], FEniCS [@fenics-book], or FreeFEM++ [@freefem], to name a few. All these packages have their own set of features, potentials, and limitations. Among these, FEniCS and FreeFEM++ are perhaps the closest ones in scope and spirit to the packages in the Gridap ecosystem. A hallmark of Gridap ecosystem packages compared to FreeFEM++ and FEniCS is that a very expressive and compact (yet efficient) syntax is transformed into low-level code using the Julia JIT compiler and thus they do not need a sophisticated compiler of variational forms nor a more intricate workflow (e.g., a Python front-end and a C/C++ back-end). - # Building blocks and composability -\autoref{fig:packages} depicts the relation among GridapDistributed and other packages in the Julia package ecosystem. The interaction of GridapDistributed and its dependencies is mainly designed with separation of concerns in mind towards high composability and modularity. On the one hand, Gridap provides a rich set of abstract types/interfaces suitable for the FE solution of PDEs (see @Verdugo:2021 for more details). It also provides realizations (implementations) of these abstractions tailored to serial/multi-threaded computing environments. GridapDistributed **implements** these abstractions for parallel distributed-memory computing environments. To this end, GridapDistributed also leverages (**uses**) the serial realizations in Gridap and associated methods to handle the local portion on each parallel task. (See \autoref{fig:packages} arrow labels.) On the other hand, GridapDistributed relies on PartitionedArrays [@parrays] in order to handle the parallel execution model (e.g., message-passing via the Message Passing Interface (MPI) [@mpi40]), global data distribution layout, and communication among tasks. PartitionedArrays also provides a parallel implementation of partitioned global linear systems (i.e., linear algebra vectors and sparse matrices) as needed in grid-based numerical simulations. While PartitionedArrays is an stand-alone package, segregated from GridapDistributed, it was designed with parallel FE packages such as GridapDistributed in mind. In any case, GridapDistributed is designed so that a different distributed linear algebra library from PartitionedArrays might be used as well, as far as it is able to provide the same functionality. - -![GridapDistributed and its relation to other packages in the Julia package ecosystem. In this diagram, each rectangle represents a Julia package, while the (directed) arrows represent relations (dependencies) among packages. Both the direction of the arrow and the label attached to the arrows are used to denote the nature of the relation. Thus, e.g., GridapDistributed depends on Gridap and PartitionedArrays, and GridapPETSc depends on Gridap and PartitionedArrays. Note that, in the diagram, the arrow direction is relevant, e.g., GridapP4est depends on GridapDistributed but not conversely. \label{fig:packages}](packages.png){ width=60% } - -As mentioned earlier, GridapDistributed offers a built-in Cartesian-like mesh generator, and does not provide, by now, built-in highly scalable solvers. To address this, as required by real-world applications, one can combine GridapDistributed with GridapP4est [@gridap4est] and GridapPETSc [@gridapetsc] (see \autoref{fig:packages}). The former provides a mesh data structure that leverages the p4est library as highly scalable mesh generation engine [@Burstedde2011]. This engine can mesh domains that can be expressed as a forest of adaptive octrees. The latter enables the usage of the highly scalable solvers (e.g., algebraic multigrid) in the PETSc library [@petsc-user-ref] to be combined with GridapDistributed. - -# Usage example - -In order to confirm our previous claims on expressiveness, conciseness and productivity (e.g., a very small number of lines of code), the example Julia script below illustrates how one may use GridapDistributed in order to solve, in parallel, a 2D Poisson problem defined on the unit square. -(In order to fully understand the code snippet, familiarity with the high level API of Gridap is assumed.) -The domain is discretized using the parallel Cartesian-like mesh generator built-in in GridapDistributed. The only minimal burden posed on the programmer versus Gridap is a call to the `prun` function of PartitionedArrays right at the beginning of the program. With this function, the programmer sets up the PartitionedArrays communication backend (i.e., MPI communication backend in the example), specifies the number of parts and their layout (i.e., `(2,2)` 2D layout in the example), and provides a function (using Julia do-block syntax for function arguments in the example) to be run on each part. This function is equivalent to a sequential Gridap script, except for the `CartesianDiscreteModel` call, which, in GridapDistributed, also requires the `parts` argument passed back by the `prun` function. In a typical cluster environment, this example would be executed on 4 MPI tasks from a terminal as `mpirun -n 4 julia --project=. example.jl`. - -```julia -using Gridap -using GridapDistributed -using PartitionedArrays -partition = (2,2) -prun(mpi,partition) do parts - domain = (0,1,0,1) - mesh_partition = (4,4) - model = CartesianDiscreteModel(parts,domain,mesh_partition) - order = 2 - u((x,y)) = (x+y)^order - f(x) = -Δ(u,x) - reffe = ReferenceFE(lagrangian,Float64,order) - V = TestFESpace(model,reffe,dirichlet_tags="boundary") - U = TrialFESpace(u,V) - Ω = Triangulation(model) - dΩ = Measure(Ω,2*order) - a(u,v) = ∫( ∇(v)·∇(u) )dΩ - l(v) = ∫( v*f )dΩ - op = AffineFEOperator(a,l,U,V) - uh = solve(op) - writevtk(Ω,"results",cellfields=["uh"=>uh,"grad_uh"=>∇(uh)]) -end -``` - - -# Parallel scaling benchmark +\autoref{fig:packages} depicts the relation among GridapDistributed and other packages in the Julia package ecosystem. -\autoref{fig:scaling} reports the strong (left) and weak scaling (right) of GridapDistributed when applied to an standard elliptic benchmark PDE problem, namely the 3D Poisson problem. In strong form this problem reads: find $u$ such that $-{\boldsymbol{\nabla}} \cdot (\boldsymbol{\kappa} {\boldsymbol{\nabla}} u) = f$ in $\Omega=[0,1]^3$, with $u = u_{{\rm D}}$ on ${\Gamma_{\rm D}}$ (Dirichlet boundary) and $\partial_{\boldsymbol{n}} u = g_{\rm N}$ on ${\Gamma_{\rm N}}$ (Neumann Boundary); $\boldsymbol{n}$ is the outward unit normal to ${\Gamma_{\rm N}}$. The domain was discretized using the built-in Cartesian-like mesh generator in GridapDistributed. The code was run on the NCI@Gadi Australian supercomputer (3024 nodes, 2x 24-core Intel Xeon Scalable *Cascade Lake* cores and 192 GB of RAM per node) with Julia 1.7 and OpenMPI 4.1.2. For the strong scaling test, we used a fixed **global** problem size resulting from the trilinear FE discretization of the domain using a 300x300x300 hexaedra mesh (26.7 MDoFs) and we scaled the number of cores up to 21.9K cores. For the weak scaling test, we used a fixed **local** problem size of 32x32x32 hexaedra, and we scaled the number of cores up to 16.5K cores. A global problem size of 0.54 billion DoFs was solved for this number of cores. The reported wall clock time includes: (1) Mesh generation; (2) Generation of global FE space; (3) Assembly of distributed linear system; (4) Interpolation of a manufactured solution; (5) Computation of the residual (includes a matrix-vector product) and its norm. Note that the linear solver time (GAMG built-in solver in PETSc) was not included in the total computation time as it is actually external to GridapDistributed. +At the core, Gridap provides all necessary abstraction and interfaces needed for the FE solution of PDEs (see @Verdugo:2021). It also provides realizations (implementations) of these abstractions tailored to serial/multi-threaded computing environments. +GridapDistributed provides distributed-memory counterparts for these abstractions, while leveraging the serial implementations in Gridap and associated methods to handle the local portion on each parallel task. GridapDistributed relies on PartitionedArrays [@parrays] in order to handle the parallel execution model (e.g., message-passing via the Message Passing Interface (MPI) [@mpi40]), global data distribution layout, and communication among tasks. PartitionedArrays also provides a parallel implementation of partitioned global linear systems (i.e., linear algebra vectors and sparse matrices) as needed in grid-based numerical simulations. +This parallel framework does however not include any performant solver for the resulting linear systems. This was delegated to GridapPETSc, which provides a plethora of highly-scalable and efficient algebraic solvers through a high-level interface to the Portable, Extensible Toolkit for Scientific Computation (PETSc) [@petsc-user-ref]. -![Strong (left) and weak (right) scaling of GridapDistributed when applied to 3D Poisson problem on the Australian Gadi@NCI supercomputer.\label{fig:scaling}](strong_and_weak_scaling.png) +GridapSolvers complements GridapPETSc with a modular and extensible interface for the design of geometric solvers. Some of the highlights of the library are: -\autoref{fig:scaling} shows, on the one hand, an efficient reduction of computation times with increasing number of cores, even far beyond a relatively small load of 25K DoFs per CPU core. -On the other hand, an asymptotically constant time-to-solution (i.e., perfect weak scaling) when the number of cores is increased in the same proportion of global problem size with a local problem size of 32x32x32 trilinear FEs. +- A set of HPC-first implementations for popular Krylov-based iterative solvers. These solvers extend Gridap's API and are fully compatible with PartitionedArrays. +- A modular, high-level interface for designing block-based preconditioners for multiphysics problems. These preconditioners can be used together with any solver compliant with Gridap's API, including those provided by GridapPETSc. +- A generic interface to handle multi-level distributed meshes, with full support for Adaptative Mesh Refinement (AMR) through GridapP4est. It also provides a modular implementation of geometric multigrid (GMG) solvers, allowing different types of smoothers and restriction/prolongation operators. +- A generic interface for patch-based subdomain decomposition methods, and an implementation of patch-based smoothers for geometric multigrid solvers. -# Demo application +![GridapSolvers and its relation to other packages in the Julia package ecosystem. In this diagram, each node represents a Julia package, while the (directed) arrows represent relations (dependencies) among packages. Dashed arrows mean the package can be used, but is not necessary. \label{fig:packages}](packages.png){ width=60% } -To highlight the ability of GridapDistributed and associated packages (see \autoref{fig:packages}) to tackle real-world problems, and the potential behind its composable architecture, we consider a demo application with interest in the geophysical fluid dynamics community. -This application solves the so-called non-linear rotating shallow water equations on the sphere, -i.e., a surface PDE posed on a two-dimensional manifold immersed in three-dimensional space. This complex system of PDEs describes the dynamics of a single incompressible thin layer of constant density fluid with a free surface under rotational effects. It is often used as a test bed for horizontal discretisations with application to numerical weather prediction and ocean modelling. We in particular considered the synthetic benchmark proposed in [@Galewsky2016], which is characterized by its ability to generate a complex and realistic flow. +# Demo -For the geometrical discretization of the sphere, the software uses the so-called cubed sphere mesh [@Ronchi1996], which was implemented using GridapP4est. The spatial discretization of the equations relies on GridapDistributed to build a **compatible** set of FE spaces [@Gibson2019] for the system unknowns (fluid velocity, fluid depth, potential vorticity and mass flux) grounded on Raviart-Thomas and Lagrangian FEs defined on the manifold [@Rognes2013]. Compatible FEs are advanced discretization techniques that preserve at the discrete level physical properties of the continuous equations. In order to stabilize the spatial discretization we use the most standard stabilization method in the geophysical flows literature, namely the so-called Anticipated Potential Vorticity Method (APVM) [@Rognes2013]. We stress that other stabilisation techniques, e.g., Streamline Upwind Petrov–Galerkin (SUPG)-like methods, have also been implemented with these tools [@Lee2022]. Time integration is based on a fully-implicit trapezoidal rule, and thus a fully-coupled nonlinear problem has to be solved at each time step. In order to solve this nonlinear problem, we leveraged a Newton-GMRES solver preconditioned with an algebraic preconditioner provided by GridapPETSc (on top of PETSc 3.16). The *exact* Jacobian of the shallow water system was computed/assembled at each nonlinear iteration. +Code in `demo.jl`. -\autoref{fig:galewsky_scaling} shows the magnitude of the vorticity field after 6.5 simulation days (left) and the results of a strong scaling study of the model on the Australian Gadi@NCI supercomputer (right). The spurious ringing artifacts in the magnitude of the vorticity field are well-known in the APVM method at coarse resolutions and can be corrected using a more effective stabilization method, such as, e.g., SUPG-like stabilization [@Lee2022]. The reported times correspond to the *total* wall time of the first 10 time integration steps; these were the only ones (out of 3600 time steps, i.e., 20 simulation days with a time step size of 480 secs.) that we could afford running for all points in the plot due to limited computational budget reasons. We considered two different problem sizes, corresponding to 256x256 and 512x512 quadrilaterals/panel cubed sphere meshes, resp. We stress that the time discretization is fully implicit. Thus we can afford larger time step sizes than with explicit methods. Besides, the purpose of the experiment is to evaluate the scalability of the framework, and not necessarily to obtain physically meaningful simulation results. Overall, \autoref{fig:galewsky_scaling} -confirms a remarkable ability of the ecosystem of Julia packages at hand to efficiently reduce computation times with increasing number of CPU cores for a complex, real-world computational model. - - -![Magnitude of the vorticity field after 6.5 simulation days with a coarser 48x48 quadrilaterals/panel cubed sphere mesh (left) and strong scaling (right) of the non-linear rotating shallow water equations solver on the Australian Gadi@NCI supercomputer.\label{fig:galewsky_scaling}](galewsky_visualization_and_scaling.png) +# Parallel scaling benchmark # Acknowledgements This research was partially funded by the Australian Government through the Australian Research Council (project number DP210103092), the European Commission under the FET-HPC ExaQUte project (Grant agreement ID: 800898) within the Horizon 2020 Framework Program and the project RTI2018-096898-B-I00 from the “FEDER/Ministerio de Ciencia e Innovación (MCIN) – Agencia Estatal de Investigación (AEI)”. F. Verdugo acknowledges support from the “Severo Ochoa Program for Centers of Excellence in R&D (2019-2023)" under the grant CEX2018-000797-S funded by MCIN/AEI/10.13039/501100011033. This work was also supported by computational resources provided by the Australian Government through NCI under the National Computational Merit Allocation Scheme (NCMAS). -# References \ No newline at end of file +# References From 707fb07e15f3b3da0f3e0084f7ad4c828edf0c2b Mon Sep 17 00:00:00 2001 From: JordiManyer Date: Thu, 16 May 2024 17:18:50 +1000 Subject: [PATCH 04/30] Simplified demo --- joss_paper/demo.jl | 99 ++++++++++++++------------------------------- joss_paper/paper.md | 21 ++++++---- 2 files changed, 43 insertions(+), 77 deletions(-) diff --git a/joss_paper/demo.jl b/joss_paper/demo.jl index 4449efdf..3bf24f37 100644 --- a/joss_paper/demo.jl +++ b/joss_paper/demo.jl @@ -1,27 +1,6 @@ -using Gridap -using Gridap.ReferenceFEs, Gridap.Algebra, Gridap.Geometry, Gridap.FESpaces -using Gridap.CellData, Gridap.MultiField, Gridap.Algebra -using PartitionedArrays -using GridapDistributed -using GridapP4est - -using GridapSolvers -using GridapSolvers.LinearSolvers, GridapSolvers.MultilevelTools, GridapSolvers.PatchBasedSmoothers -using GridapSolvers.BlockSolvers: LinearSystemBlock, BiformBlock, BlockTriangularSolver - -function get_patch_smoothers(mh,tests,biform,patch_decompositions,qdegree) - patch_spaces = PatchFESpace(tests,patch_decompositions) - nlevs = num_levels(mh) - smoothers = map(view(tests,1:nlevs-1),patch_decompositions,patch_spaces) do tests, PD, Ph - Vh = get_fe_space(tests) - Ω = Triangulation(PD) - dΩ = Measure(Ω,qdegree) - ap = (u,v) -> biform(u,v,dΩ) - patch_smoother = PatchBasedLinearSolver(ap,Ph,Vh) - return RichardsonSmoother(patch_smoother,10,0.2) - end - return smoothers -end +using Gridap, Gridap.Algebra, Gridap.MultiField +using PartitionedArrays, GridapDistributed, GridapSolvers +using GridapSolvers.LinearSolvers, GridapSolvers.MultilevelTools, GridapSolvers.BlockSolvers function get_bilinear_form(mh_lev,biform,qdegree) model = get_model(mh_lev) @@ -30,35 +9,30 @@ function get_bilinear_form(mh_lev,biform,qdegree) return (u,v) -> biform(u,v,dΩ) end -nc = (8,8) -fe_order = 2 +function add_labels!(labels) + add_tag_from_tags!(labels,"top",[3,4,6]) + add_tag_from_tags!(labels,"bottom",[1,2,5]) +end np = 4 np_per_level = [np,1] +nc = (10,10) +fe_order = 2 with_mpi() do distribute parts = distribute(LinearIndices((prod(np),))) - # Coarse geometry - cmodel = CartesianDiscreteModel((0,1,0,1),nc) - labels = get_face_labeling(cmodel) - add_tag_from_tags!(labels,"top",[3,4,6]) - add_tag_from_tags!(labels,"walls",[1,5,7]) - add_tag_from_tags!(labels,"right",[2,8]) - - # Mesh refinement using GridapP4est - cparts = generate_subparts(parts,np_per_level[end]) - coarse_model = OctreeDistributedDiscreteModel(cparts,cmodel,0) - mh = ModelHierarchy(parts,coarse_model,np_per_level) + # Geometry + mh = CartesianModelHierarchy(parts,np_per_level,(0,1,0,1),nc;add_labels!) model = get_model(mh,1) # FE spaces qdegree = 2*(fe_order+1) - reffe_u = ReferenceFE(lagrangian,VectorValue{Dc,Float64},fe_order) + reffe_u = ReferenceFE(lagrangian,VectorValue{2,Float64},fe_order) reffe_p = ReferenceFE(lagrangian,Float64,fe_order-1;space=:P) - tests_u = TestFESpace(mh,reffe_u,dirichlet_tags=["walls","top"]); - trials_u = TrialFESpace(tests_u,[VectorValue(0.0,0.0),VectorValue(1.0,0.0)]); + tests_u = TestFESpace(mh,reffe_u,dirichlet_tags=["bottom","top"]) + trials_u = TrialFESpace(tests_u,[VectorValue(0.0,0.0),VectorValue(1.0,0.0)]) U, V = get_fe_space(trials_u,1), get_fe_space(tests_u,1) Q = TestFESpace(model,reffe_p;conformity=:L2) @@ -67,33 +41,22 @@ with_mpi() do distribute Y = MultiFieldFESpace([V,Q];style=mfs) # Weak formulation - α = 1.e2 f = VectorValue(1.0,1.0) - Π_Qh = LocalProjectionMap(QUAD,lagrangian,Float64,fe_order-1;quad_order=qdegree,space=:P) - graddiv(u,v,dΩ) = ∫(α*Π_Qh(divergence(u))⋅Π_Qh(divergence(v)))dΩ - biform_u(u,v,dΩ) = ∫(∇(v)⊙∇(u))dΩ + graddiv(u,v,dΩ) - biform((u,p),(v,q),dΩ) = biform_u(u,v,dΩ) - ∫(divergence(v)*p)dΩ - ∫(divergence(u)*q)dΩ + biform_u(u,v,dΩ) = ∫(∇(v)⊙∇(u))dΩ + biform((u,p),(v,q),dΩ) = biform_u(u,v,dΩ) - ∫((∇⋅v)*p)dΩ - ∫((∇⋅u)*q)dΩ liform((v,q),dΩ) = ∫(v⋅f)dΩ # Finest level Ω = Triangulation(model) dΩ = Measure(Ω,qdegree) - a(u,v) = biform(u,v,dΩ) - l(v) = liform(v,dΩ) - op = AffineFEOperator(a,l,X,Y) - A, b = get_matrix(op), get_vector(op); + op = AffineFEOperator((u,v)->biform(u,v,dΩ),v->liform(v,dΩ),X,Y) + A, b = get_matrix(op), get_vector(op) # GMG preconditioner for the velocity block biforms = map(mhl -> get_bilinear_form(mhl,biform_u,qdegree),mh) - patch_decompositions = PatchDecomposition(mh) - smoothers = get_patch_smoothers( - mh,tests_u,biform_u,patch_decompositions,qdegree - ) - restrictions = setup_restriction_operators( - tests_u,qdegree;mode=:residual,solver=IS_ConjugateGradientSolver(;reltol=1.e-6) - ) - prolongations = setup_patch_prolongation_operators( - tests_u,biform_u,graddiv,qdegree + smoothers = map(mhl -> RichardsonSmoother(JacobiLinearSolver(),10,2.0/3.0), view(mh,1:num_levels(mh)-1)) + restrictions, prolongations = setup_transfer_operators( + trials_u, qdegree; mode=:residual, solver=CGSolver(JacobiLinearSolver()) ) solver_u = GMGLinearSolver( mh,trials_u,tests_u,biforms, @@ -101,23 +64,21 @@ with_mpi() do distribute pre_smoothers=smoothers, post_smoothers=smoothers, coarsest_solver=LUSolver(), - maxiter=2,mode=:preconditioner,verbose=i_am_main(parts) + maxiter=2,mode=:preconditioner ) # PCG solver for the pressure block - solver_p = CGSolver(JacobiLinearSolver();maxiter=20,atol=1e-14,rtol=1.e-6,verbose=i_am_main(parts)) + solver_p = CGSolver(JacobiLinearSolver();maxiter=20,atol=1e-14,rtol=1.e-6) # Block triangular preconditioner - blocks = [LinearSystemBlock(), LinearSystemBlock(); - LinearSystemBlock(), BiformBlock((p,q) -> ∫(-1.0/α*p*q)dΩ,Q,Q)] - coeffs = [1.0 1.0; - 0.0 1.0] - P = BlockTriangularSolver(blocks,[solver_u,solver_p],coeffs,:upper) - solver = FGMRESSolver(20,P;atol=1e-14,rtol=1.e-8,verbose=i_am_main(parts)) - + blocks = [LinearSystemBlock() LinearSystemBlock(); + LinearSystemBlock() BiformBlock((p,q) -> ∫(p*q)dΩ,Q,Q)] + P = BlockTriangularSolver(blocks,[solver_u,solver_p]) + solver = FGMRESSolver(10;Pr=P,rtol=1.e-8,verbose=i_am_main(parts)) ns = numerical_setup(symbolic_setup(solver,A),A) + x = allocate_in_domain(A); fill!(x,0.0) solve!(x,ns,b) uh, ph = FEFunction(X,x) - writevtk(Ω,"results",cellfields=["uh"=>uh,"ph"=>ph]) -end + writevtk(Ω,"joss_paper/demo",cellfields=["uh"=>uh,"ph"=>ph]) +end \ No newline at end of file diff --git a/joss_paper/paper.md b/joss_paper/paper.md index 9b2ddc27..3bc07238 100644 --- a/joss_paper/paper.md +++ b/joss_paper/paper.md @@ -1,11 +1,11 @@ --- -title: 'GridapSolvers.jl: A Julia package for scalable FE solvers' +title: 'GridapSolvers.jl: Scalable multiphysics finite element solvers in Julia' tags: - Julia - - astronomy - - dynamics - - galactic dynamics - - milky way + - pdes + - finite elements + - hpc + - solvers authors: - name: Jordi Manyer orcid: 0000-0002-0178-3890 @@ -41,14 +41,13 @@ One of the biggest scalability bottlenecks within Finite Element (FE) parallel c The implementation of exact factorization-based solvers in parallel environments is an extremely challenging task, and even state-of-the-art libraries such as MUMPS [@MUMPS] or PARDISO [@PARDISO] have severe limitations in terms of scalability and memory consumption above a certain number of CPU cores. Hence the use of iterative methods is crucial to maintain scalability of FE codes. Unfortunately, the convergence of iterative methods is not guaranteed and rapidly deteriorates as the size of the linear system increases. To retain performance, the use of highly scalable preconditioners is mandatory. For most problems, algebraic solvers and preconditioners (i.e based uniquelly on the algebraic system) are enough to obtain robust convergence. Many well-known libraries providing algebraic solvers already exist, such as PETSc [@petsc-user-ref], Trilinos [@trilinos], or Hypre [@hypre]. However, algebraic solvers are not always suited to deal with some of the most challenging problems. -In these cases, geometric solvers (i.e., solvers that exploit the geometry and physics of the particular problem) are required. To this end, GridapSolvers is a registered Julia [@Bezanson2017] software package which provides highly scalable geometric solvers tailored for the FE numerical solution of PDEs on parallel computers. +In these cases, geometric solvers (i.e., solvers that exploit the geometry and physics of the particular problem) are required. This is the case of many multiphysics problems, such as Navier-Stokes, Darcy or MHD. To this end, GridapSolvers is a registered Julia [@Bezanson2017] software package which provides highly scalable geometric solvers tailored for the FE numerical solution of PDEs on parallel computers. # Building blocks and composability \autoref{fig:packages} depicts the relation among GridapDistributed and other packages in the Julia package ecosystem. -At the core, Gridap provides all necessary abstraction and interfaces needed for the FE solution of PDEs (see @Verdugo:2021). It also provides realizations (implementations) of these abstractions tailored to serial/multi-threaded computing environments. -GridapDistributed provides distributed-memory counterparts for these abstractions, while leveraging the serial implementations in Gridap and associated methods to handle the local portion on each parallel task. GridapDistributed relies on PartitionedArrays [@parrays] in order to handle the parallel execution model (e.g., message-passing via the Message Passing Interface (MPI) [@mpi40]), global data distribution layout, and communication among tasks. PartitionedArrays also provides a parallel implementation of partitioned global linear systems (i.e., linear algebra vectors and sparse matrices) as needed in grid-based numerical simulations. +The core library Gridap provides all necessary abstraction and interfaces needed for the FE solution of PDEs (see @Verdugo:2021) for serial computing. GridapDistributed provides distributed-memory counterparts for these abstractions, while leveraging the serial implementations in Gridap to handle the local portion on each parallel task. GridapDistributed relies on PartitionedArrays [@parrays] in order to handle the parallel execution model (e.g., message-passing via the Message Passing Interface (MPI) [@mpi40]), global data distribution layout, and communication among tasks. PartitionedArrays also provides a parallel implementation of partitioned global linear systems (i.e., linear algebra vectors and sparse matrices) as needed in grid-based numerical simulations. This parallel framework does however not include any performant solver for the resulting linear systems. This was delegated to GridapPETSc, which provides a plethora of highly-scalable and efficient algebraic solvers through a high-level interface to the Portable, Extensible Toolkit for Scientific Computation (PETSc) [@petsc-user-ref]. GridapSolvers complements GridapPETSc with a modular and extensible interface for the design of geometric solvers. Some of the highlights of the library are: @@ -62,7 +61,13 @@ GridapSolvers complements GridapPETSc with a modular and extensible interface fo # Demo +The following code snippet shows how to solve a 2D Stokes cavity problem in a cartesian domain $\Omega = [0,1]^2$. We discretize the velocity and pressure in $H^1(\Omega)$ and $L^2(\Omega)$ respectively, and use the well known stable element pair $Q_k \times P_{k-1}$ with $k=2$. For the cavity problem, we fix the velocity to $u_b = \vec{0}$ and $u_t = \hat{x}$ on the bottom and top boundaries respectively, and homogeneous Neumann boundary conditions elsewhere. +The system is block-assembled and solved using a GMRES solver, right-preconditioned with block-triangular Shur-complement-based preconditioner. The Shur complement is approximated by a mass matrix, and solved using a CG solver with Jacobi preconditioner. The eliminated velocity block is approximated by a 2-level V-cycle Geometric Multigrid solver. +The code is setup to run in parallel with 4 MPI tasks and can be executed with the following command: `mpiexec -n 4 julia --project=. demo.jl`. + +```julia Code in `demo.jl`. +``` # Parallel scaling benchmark From 60249d5408c14d141d842eaa57ca20fb2e2e5848 Mon Sep 17 00:00:00 2001 From: JordiManyer Date: Fri, 17 May 2024 07:59:50 +1000 Subject: [PATCH 05/30] Added gadi drivers --- .gitignore | 2 +- joss_paper/scalability/Project.toml | 3 + joss_paper/scalability/data/.gitignore | 2 + joss_paper/scalability/driver.jl | 118 +++++++++++++++++++++++++ joss_paper/scalability/jobs/.gitignore | 2 + joss_paper/scalability/modules.sh | 22 +++++ joss_paper/scalability/preparejobs.jl | 78 ++++++++++++++++ joss_paper/scalability/template.sh | 22 +++++ 8 files changed, 248 insertions(+), 1 deletion(-) create mode 100644 joss_paper/scalability/Project.toml create mode 100644 joss_paper/scalability/data/.gitignore create mode 100644 joss_paper/scalability/driver.jl create mode 100644 joss_paper/scalability/jobs/.gitignore create mode 100644 joss_paper/scalability/modules.sh create mode 100644 joss_paper/scalability/preparejobs.jl create mode 100644 joss_paper/scalability/template.sh diff --git a/.gitignore b/.gitignore index 2f4409be..74470240 100644 --- a/.gitignore +++ b/.gitignore @@ -1,4 +1,4 @@ .vscode Manifest.toml LocalPreferences.toml -data \ No newline at end of file +data/* \ No newline at end of file diff --git a/joss_paper/scalability/Project.toml b/joss_paper/scalability/Project.toml new file mode 100644 index 00000000..13a89595 --- /dev/null +++ b/joss_paper/scalability/Project.toml @@ -0,0 +1,3 @@ +[deps] +DrWatson = "634d3b9d-ee7a-5ddf-bec9-22491ea816e1" +Mustache = "ffc61752-8dc7-55ee-8c37-f3e9cdd09e70" diff --git a/joss_paper/scalability/data/.gitignore b/joss_paper/scalability/data/.gitignore new file mode 100644 index 00000000..c96a04f0 --- /dev/null +++ b/joss_paper/scalability/data/.gitignore @@ -0,0 +1,2 @@ +* +!.gitignore \ No newline at end of file diff --git a/joss_paper/scalability/driver.jl b/joss_paper/scalability/driver.jl new file mode 100644 index 00000000..f309684f --- /dev/null +++ b/joss_paper/scalability/driver.jl @@ -0,0 +1,118 @@ + +using Gridap, Gridap.Algebra, Gridap.MultiField +using PartitionedArrays, GridapDistributed, GridapSolvers +using GridapSolvers.LinearSolvers, GridapSolvers.MultilevelTools, GridapSolvers.BlockSolvers + +function get_bilinear_form(mh_lev,biform,qdegree) + model = get_model(mh_lev) + Ω = Triangulation(model) + dΩ = Measure(Ω,qdegree) + return (u,v) -> biform(u,v,dΩ) +end + +function add_labels!(labels) + add_tag_from_tags!(labels,"top",[3,4,6]) + add_tag_from_tags!(labels,"bottom",[1,2,5]) +end + +function driver(parts,np_per_level,nc) + t = PTimer(parts;verbose=true) + + tic!(t;barrier=true) + # Geometry + mh = CartesianModelHierarchy(parts,np_per_level,(0,1,0,1),nc;add_labels!) + model = get_model(mh,1) + + # FE spaces + fe_order = 2 + qdegree = 2*(fe_order+1) + reffe_u = ReferenceFE(lagrangian,VectorValue{2,Float64},fe_order) + reffe_p = ReferenceFE(lagrangian,Float64,fe_order-1;space=:P) + + tests_u = TestFESpace(mh,reffe_u,dirichlet_tags=["bottom","top"]) + trials_u = TrialFESpace(tests_u,[VectorValue(0.0,0.0),VectorValue(1.0,0.0)]) + U, V = get_fe_space(trials_u,1), get_fe_space(tests_u,1) + Q = TestFESpace(model,reffe_p;conformity=:L2) + + mfs = Gridap.MultiField.BlockMultiFieldStyle() + X = MultiFieldFESpace([U,Q];style=mfs) + Y = MultiFieldFESpace([V,Q];style=mfs) + + # Weak formulation + f = VectorValue(1.0,1.0) + biform_u(u,v,dΩ) = ∫(∇(v)⊙∇(u))dΩ + biform((u,p),(v,q),dΩ) = biform_u(u,v,dΩ) - ∫((∇⋅v)*p)dΩ - ∫((∇⋅u)*q)dΩ + liform((v,q),dΩ) = ∫(v⋅f)dΩ + + # Finest level + Ω = Triangulation(model) + dΩ = Measure(Ω,qdegree) + op = AffineFEOperator((u,v)->biform(u,v,dΩ),v->liform(v,dΩ),X,Y) + A, b = get_matrix(op), get_vector(op) + + # GMG preconditioner for the velocity block + biforms = map(mhl -> get_bilinear_form(mhl,biform_u,qdegree),mh) + smoothers = map(mhl -> RichardsonSmoother(JacobiLinearSolver(),10,2.0/3.0), view(mh,1:num_levels(mh)-1)) + restrictions, prolongations = setup_transfer_operators( + trials_u, qdegree; mode=:residual, solver=CGSolver(JacobiLinearSolver()) + ) + solver_u = GMGLinearSolver( + mh,trials_u,tests_u,biforms, + prolongations,restrictions, + pre_smoothers=smoothers, + post_smoothers=smoothers, + coarsest_solver=LUSolver(), + maxiter=2,mode=:preconditioner + ) + + # PCG solver for the pressure block + solver_p = CGSolver(JacobiLinearSolver();maxiter=20,atol=1e-14,rtol=1.e-6) + + # Block triangular preconditioner + blocks = [LinearSystemBlock() LinearSystemBlock(); + LinearSystemBlock() BiformBlock((p,q) -> ∫(p*q)dΩ,Q,Q)] + P = BlockTriangularSolver(blocks,[solver_u,solver_p]) + solver = FGMRESSolver(10;Pr=P,rtol=1.e-8,verbose=i_am_main(parts)) + ns = numerical_setup(symbolic_setup(solver,A),A) + toc!(t,"Setup") + + tic!(t;barrier=true) + x = allocate_in_domain(A); fill!(x,0.0) + solve!(x,ns,b) + toc!(t,"Solver") + + # Postprocess + output = Dict{String,Any}() + map_main(t.data) do timer_data + output["np"] = np_per_level[1] + output["nc"] = nc + output["niter"] = solver.log.num_iters + output["np_per_level"] = np_per_level + merge!(output,timer_data) + end + return output +end + +function main( + nr = 1, + np = 1, + np_per_level = [np,1], + nc = (10,10), + petsc_options = "-ksp_monitor -ksp_error_if_not_converged true -ksp_converged_reason", + title = "stokes" +) + parts = with_mpi() do distribute + distribute(LinearIndices((prod(np),))) + end + GridapPETSc.with(;args=split(petsc_options)) do + driver(parts,np_per_level,nc) + for ir in 1:nr + title_ir = "$(title)_$(ir)" + output = driver(parts,np_per_level,nc) + map_main(parts) do p + output["ir"] = ir + save("$title_ir.bson",output) + end + end + end +end diff --git a/joss_paper/scalability/jobs/.gitignore b/joss_paper/scalability/jobs/.gitignore new file mode 100644 index 00000000..c96a04f0 --- /dev/null +++ b/joss_paper/scalability/jobs/.gitignore @@ -0,0 +1,2 @@ +* +!.gitignore \ No newline at end of file diff --git a/joss_paper/scalability/modules.sh b/joss_paper/scalability/modules.sh new file mode 100644 index 00000000..284ce35f --- /dev/null +++ b/joss_paper/scalability/modules.sh @@ -0,0 +1,22 @@ +module purge +module load pbs + +module load intel-compiler-llvm/2023.2.0 +module load intel-mpi/2021.10.0 +module load intel-mkl/2023.2.0 + +export P4EST_VERSION='2.8.5' +export PETSC_VERSION='3.19.5' +export PROJECT="np01" + +SCRATCH="/scratch/$PROJECT/$USER" +export JULIA_DEPOT_PATH="$SCRATCH/.julia" +export MPI_VERSION="intel-$INTEL_MPI_VERSION" +export JULIA_MPI_PATH=$INTEL_MPI_ROOT +export JULIA_PETSC_LIBRARY="$HOME/bin/petsc/$PETSC_VERSION-$MPI_VERSION/lib/libpetsc" +export P4EST_ROOT_DIR="$HOME/bin/p4est/$P4EST_VERSION-$MPI_VERSION" + +export UCX_ERROR_SIGNALS="SIGILL,SIGBUS,SIGFPE" +export HCOLL_ML_DISABLE_SCATTERV=1 +export HCOLL_ML_DISABLE_BCAST=1 +export ZES_ENABLE_SYSMAN=1 \ No newline at end of file diff --git a/joss_paper/scalability/preparejobs.jl b/joss_paper/scalability/preparejobs.jl new file mode 100644 index 00000000..4bdc2bad --- /dev/null +++ b/joss_paper/scalability/preparejobs.jl @@ -0,0 +1,78 @@ + +using Mustache +using DrWatson + +function jobname(args...) + s = savename(args...;connector="_") + s = replace(s,"="=>"_") + s = "stokes_$s" + return s +end +driverdir(args...) = normpath(projectdir("../../",args...)) + +function clean_params(d) + o = Dict() + for k in keys(d) + if isa(d[k],Real) + o[k] = Int(d[k]) + elseif isa(d[k],Symbol) || isa(d[k],String) || isa(d[k],Number) + o[k] = d[k] + elseif isa(d[k],Tuple) + o[k] = prod(d[k]) + end + end + return o +end + +function jobdict(params) + np = params[:np] + nc = params[:nc] + fparams = clean_params(params) + Dict( + "project" => haskey(ENV,"PROJECT") ? ENV["PROJECT"] : "", + "q" => "normal", + "o" => datadir(jobname(fparams,"o.txt")), + "e" => datadir(jobname(fparams,"e.txt")), + "walltime" => "01:00:00", + "ncpus" => prod(np), + "nnodes" => prod(np)÷48, + "mem" => "$(prod(np)*4)gb", + "name" => jobname(fparams), + "np" => prod(np), + "nc" => nc, + "np_per_level" => Int[prod(np),prod(np)], + "driver" => projectdir("driver.jl"), + "projectdir" => driverdir(), + "datadir" => datadir(), + "modules" => projectdir("modules.sh"), + "title" => datadir(jobname(fparams)), + "sysimage" => projectdir("GridapSolvers.so") + ) +end + +function generate_dictionaries(num_cells,num_procs) + dicts = Dict[] + for (nc,np) in zip(num_cells,num_procs) + aux = Dict( + :np => np, + :nc => nc + ) + push!(dicts,aux) + end + return dicts +end + +########################################### + +num_cells = [(6,6,3), (10,10,5),(20,20,5)] +num_procs = [(2,2) , (4,4) ,(6,8)] +dicts = generate_dictionaries(num_cells,num_procs) + +template = read(projectdir("template.sh"),String) +for params in dicts + fparams = clean_params(params) + jobfile = projectdir("jobs/",jobname(fparams,"sh")) + open(jobfile,"w") do io + render(io,template,jobdict(params)) + end +end diff --git a/joss_paper/scalability/template.sh b/joss_paper/scalability/template.sh new file mode 100644 index 00000000..cc895162 --- /dev/null +++ b/joss_paper/scalability/template.sh @@ -0,0 +1,22 @@ +#!/bin/bash +#PBS -P np01 +#PBS -q {{q}} +#PBS -l walltime={{walltime}} +#PBS -l ncpus={{ncpus}} +#PBS -l mem={{mem}} +#PBS -N {{{name}}} +#PBS -l wd +#PBS -o {{{o}}} +#PBS -e {{{e}}} + +source {{{modules}}} + +mpiexec -n {{ncpus}} julia --project={{{projectdir}}} -O3 --check-bounds=no -e\ + 'include({{{driver}}}); + main(; + nr={{nr}}, + np={{np}}, + nc={{nc}}, + np_per_level={{np_per_level}}, + title="{{title}}", + )' From c10a1767860eac77914ff8de86e8e532491083d3 Mon Sep 17 00:00:00 2001 From: JordiManyer Date: Fri, 17 May 2024 08:59:59 +1000 Subject: [PATCH 06/30] Fixed Gadi drivers --- joss_paper/scalability/Project.toml | 2 ++ joss_paper/scalability/driver.jl | 29 +++++++++++++++++++-------- joss_paper/scalability/preparejobs.jl | 1 + joss_paper/scalability/template.sh | 19 ++++++++++-------- 4 files changed, 35 insertions(+), 16 deletions(-) diff --git a/joss_paper/scalability/Project.toml b/joss_paper/scalability/Project.toml index 13a89595..7b735c93 100644 --- a/joss_paper/scalability/Project.toml +++ b/joss_paper/scalability/Project.toml @@ -1,3 +1,5 @@ [deps] +BSON = "fbb218c0-5317-5bc6-957e-2ee96dd4b1f0" DrWatson = "634d3b9d-ee7a-5ddf-bec9-22491ea816e1" +FileIO = "5789e2e9-d7fb-5bc7-8068-2c6fae9b9549" Mustache = "ffc61752-8dc7-55ee-8c37-f3e9cdd09e70" diff --git a/joss_paper/scalability/driver.jl b/joss_paper/scalability/driver.jl index f309684f..68e4ca0f 100644 --- a/joss_paper/scalability/driver.jl +++ b/joss_paper/scalability/driver.jl @@ -1,6 +1,6 @@ - +using FileIO using Gridap, Gridap.Algebra, Gridap.MultiField -using PartitionedArrays, GridapDistributed, GridapSolvers +using PartitionedArrays, GridapDistributed, GridapSolvers, GridapPETSc using GridapSolvers.LinearSolvers, GridapSolvers.MultilevelTools, GridapSolvers.BlockSolvers function get_bilinear_form(mh_lev,biform,qdegree) @@ -62,17 +62,19 @@ function driver(parts,np_per_level,nc) pre_smoothers=smoothers, post_smoothers=smoothers, coarsest_solver=LUSolver(), - maxiter=2,mode=:preconditioner + maxiter=2,mode=:preconditioner,verbose=i_am_main(parts) ) + solver_u.log.depth = 4 # PCG solver for the pressure block - solver_p = CGSolver(JacobiLinearSolver();maxiter=20,atol=1e-14,rtol=1.e-6) + solver_p = CGSolver(JacobiLinearSolver();maxiter=20,atol=1e-14,rtol=1.e-6,verbose=i_am_main(parts)) + solver_p.log.depth = 4 # Block triangular preconditioner blocks = [LinearSystemBlock() LinearSystemBlock(); LinearSystemBlock() BiformBlock((p,q) -> ∫(p*q)dΩ,Q,Q)] P = BlockTriangularSolver(blocks,[solver_u,solver_p]) - solver = FGMRESSolver(10;Pr=P,rtol=1.e-8,verbose=i_am_main(parts)) + solver = FGMRESSolver(10,P;rtol=1.e-8,verbose=i_am_main(parts)) ns = numerical_setup(symbolic_setup(solver,A),A) toc!(t,"Setup") @@ -82,24 +84,30 @@ function driver(parts,np_per_level,nc) toc!(t,"Solver") # Postprocess + ncells = num_cells(model) + ndofs_u = num_free_dofs(U) + ndofs_p = num_free_dofs(Q) output = Dict{String,Any}() map_main(t.data) do timer_data output["np"] = np_per_level[1] output["nc"] = nc - output["niter"] = solver.log.num_iters output["np_per_level"] = np_per_level + output["ncells"] = ncells + output["ndofs_u"] = ndofs_u + output["ndofs_p"] = ndofs_p + output["niter"] = solver.log.num_iters merge!(output,timer_data) end return output end -function main( +function main(; nr = 1, np = 1, np_per_level = [np,1], nc = (10,10), petsc_options = "-ksp_monitor -ksp_error_if_not_converged true -ksp_converged_reason", - title = "stokes" + title = "data/stokes_np_$(np)_nc_$(prod(nc))" ) parts = with_mpi() do distribute distribute(LinearIndices((prod(np),))) @@ -107,6 +115,11 @@ function main( GridapPETSc.with(;args=split(petsc_options)) do driver(parts,np_per_level,nc) for ir in 1:nr + if i_am_main(parts) + println(repeat('-',28)) + println(" ------- ITERATION $(ir) ------- ") + println(repeat('-',28)) + end title_ir = "$(title)_$(ir)" output = driver(parts,np_per_level,nc) map_main(parts) do p diff --git a/joss_paper/scalability/preparejobs.jl b/joss_paper/scalability/preparejobs.jl index 4bdc2bad..fd76bc2c 100644 --- a/joss_paper/scalability/preparejobs.jl +++ b/joss_paper/scalability/preparejobs.jl @@ -45,6 +45,7 @@ function jobdict(params) "projectdir" => driverdir(), "datadir" => datadir(), "modules" => projectdir("modules.sh"), + "driverdir" => projectdir(), "title" => datadir(jobname(fparams)), "sysimage" => projectdir("GridapSolvers.so") ) diff --git a/joss_paper/scalability/template.sh b/joss_paper/scalability/template.sh index cc895162..453c896f 100644 --- a/joss_paper/scalability/template.sh +++ b/joss_paper/scalability/template.sh @@ -12,11 +12,14 @@ source {{{modules}}} mpiexec -n {{ncpus}} julia --project={{{projectdir}}} -O3 --check-bounds=no -e\ - 'include({{{driver}}}); - main(; - nr={{nr}}, - np={{np}}, - nc={{nc}}, - np_per_level={{np_per_level}}, - title="{{title}}", - )' + ' + push!(Base.LOAD_PATH, "{{driverdir}}"); + include({{{driver}}}); + main(; + nr={{nr}}, + np={{np}}, + nc={{nc}}, + np_per_level={{np_per_level}}, + title="{{title}}", + ) + ' From 3785967322537c881ebb5269e2321c96c1755109 Mon Sep 17 00:00:00 2001 From: JordiManyer Date: Fri, 14 Jun 2024 09:57:56 +1000 Subject: [PATCH 07/30] Reformated driver into a real module --- joss_paper/scalability/Project.toml | 9 ++++ joss_paper/scalability/postprocess.jl | 47 +++++++++++++++++++ joss_paper/scalability/preparejobs.jl | 4 +- joss_paper/scalability/src/Scalability.jl | 15 ++++++ .../scalability/{driver.jl => src/stokes.jl} | 26 +++------- joss_paper/scalability/src/utils.jl | 12 +++++ joss_paper/scalability/template.sh | 5 +- 7 files changed, 93 insertions(+), 25 deletions(-) create mode 100644 joss_paper/scalability/postprocess.jl create mode 100644 joss_paper/scalability/src/Scalability.jl rename joss_paper/scalability/{driver.jl => src/stokes.jl} (86%) create mode 100644 joss_paper/scalability/src/utils.jl diff --git a/joss_paper/scalability/Project.toml b/joss_paper/scalability/Project.toml index 7b735c93..ed7e839e 100644 --- a/joss_paper/scalability/Project.toml +++ b/joss_paper/scalability/Project.toml @@ -1,5 +1,14 @@ +name = "Scalability" +authors = ["Jordi Manyer "] +version = "0.1.0" + [deps] BSON = "fbb218c0-5317-5bc6-957e-2ee96dd4b1f0" DrWatson = "634d3b9d-ee7a-5ddf-bec9-22491ea816e1" FileIO = "5789e2e9-d7fb-5bc7-8068-2c6fae9b9549" +Gridap = "56d4f2e9-7ea1-5844-9cf6-b9c51ca7ce8e" +GridapDistributed = "f9701e48-63b3-45aa-9a63-9bc6c271f355" +GridapPETSc = "bcdc36c2-0c3e-11ea-095a-c9dadae499f1" +GridapSolvers = "6d3209ee-5e3c-4db7-a716-942eb12ed534" Mustache = "ffc61752-8dc7-55ee-8c37-f3e9cdd09e70" +PartitionedArrays = "5a9dfac6-5c52-46f7-8278-5e2210713be9" diff --git a/joss_paper/scalability/postprocess.jl b/joss_paper/scalability/postprocess.jl new file mode 100644 index 00000000..302ebd4d --- /dev/null +++ b/joss_paper/scalability/postprocess.jl @@ -0,0 +1,47 @@ +### A Pluto.jl notebook ### +# v0.19.9 + +using Markdown +using InteractiveUtils + +# ╔═╡ f0aae79e-13d9-11ef-263f-b720d8f10878 +begin + using Pkg + Pkg.activate(Base.current_project()) + Pkg.instantiate() + + using Plots, DrWatson + using DataFrames, BSON, CSV +end + +# ╔═╡ 4725bb5d-8973-436b-af96-74cc90e7f354 +begin + raw = collect_results(datadir()) + + dfgr = groupby(raw,[:np]) + df = combine(dfgr) do df + return ( + t_solver = minimum(map(x->x[:max],df.Solver)), + t_setup = minimum(map(x->x[:max],df.Setup)), + n_iter = df.niter[1], + n_levels = length(df.np_per_level[1]), + n_dofs_u = df.ndofs_u[1], + n_dofs_p = df.ndofs_p[1], + n_dofs = df.ndofs_u[1] + df.ndofs_p[1], + n_cells = df.nc[1] + ) + end + sort!(df,:np) +end + +# ╔═╡ a10880e9-680b-46f0-9bda-44e53a7196ce +begin + plt = plot(xlabel="N processors",ylabel="walltime (s)",legend=false) + plot!(df[!,:np],df[!,:t_solver][1]*df[!,:np]./df[!,:np][1]) # Perfect scaling + plot!(df[!,:np],df[!,:t_solver]) +end + +# ╔═╡ Cell order: +# ╠═f0aae79e-13d9-11ef-263f-b720d8f10878 +# ╠═4725bb5d-8973-436b-af96-74cc90e7f354 +# ╠═a10880e9-680b-46f0-9bda-44e53a7196ce diff --git a/joss_paper/scalability/preparejobs.jl b/joss_paper/scalability/preparejobs.jl index fd76bc2c..4f60699f 100644 --- a/joss_paper/scalability/preparejobs.jl +++ b/joss_paper/scalability/preparejobs.jl @@ -8,7 +8,6 @@ function jobname(args...) s = "stokes_$s" return s end -driverdir(args...) = normpath(projectdir("../../",args...)) function clean_params(d) o = Dict() @@ -41,8 +40,7 @@ function jobdict(params) "np" => prod(np), "nc" => nc, "np_per_level" => Int[prod(np),prod(np)], - "driver" => projectdir("driver.jl"), - "projectdir" => driverdir(), + "projectdir" => projectdir(), "datadir" => datadir(), "modules" => projectdir("modules.sh"), "driverdir" => projectdir(), diff --git a/joss_paper/scalability/src/Scalability.jl b/joss_paper/scalability/src/Scalability.jl new file mode 100644 index 00000000..a47daff6 --- /dev/null +++ b/joss_paper/scalability/src/Scalability.jl @@ -0,0 +1,15 @@ +module Scalability + +using FileIO + +using Gridap, PartitionedArrays, GridapDistributed, GridapSolvers, GridapPETSc +using Gridap.Algebra, Gridap.MultiField + +using GridapSolvers.LinearSolvers, GridapSolvers.MultilevelTools, GridapSolvers.BlockSolvers + +include("utils.jl") +include("stokes.jl") + +export stokes_main + +end # module \ No newline at end of file diff --git a/joss_paper/scalability/driver.jl b/joss_paper/scalability/src/stokes.jl similarity index 86% rename from joss_paper/scalability/driver.jl rename to joss_paper/scalability/src/stokes.jl index 68e4ca0f..39498d68 100644 --- a/joss_paper/scalability/driver.jl +++ b/joss_paper/scalability/src/stokes.jl @@ -1,21 +1,9 @@ -using FileIO -using Gridap, Gridap.Algebra, Gridap.MultiField -using PartitionedArrays, GridapDistributed, GridapSolvers, GridapPETSc -using GridapSolvers.LinearSolvers, GridapSolvers.MultilevelTools, GridapSolvers.BlockSolvers -function get_bilinear_form(mh_lev,biform,qdegree) - model = get_model(mh_lev) - Ω = Triangulation(model) - dΩ = Measure(Ω,qdegree) - return (u,v) -> biform(u,v,dΩ) -end - -function add_labels!(labels) - add_tag_from_tags!(labels,"top",[3,4,6]) - add_tag_from_tags!(labels,"bottom",[1,2,5]) -end - -function driver(parts,np_per_level,nc) +function stokes_driver( + parts, + np_per_level, + nc +) t = PTimer(parts;verbose=true) tic!(t;barrier=true) @@ -101,7 +89,7 @@ function driver(parts,np_per_level,nc) return output end -function main(; +function stokes_main(; nr = 1, np = 1, np_per_level = [np,1], @@ -128,4 +116,4 @@ function main(; end end end -end +end \ No newline at end of file diff --git a/joss_paper/scalability/src/utils.jl b/joss_paper/scalability/src/utils.jl new file mode 100644 index 00000000..1b608023 --- /dev/null +++ b/joss_paper/scalability/src/utils.jl @@ -0,0 +1,12 @@ + +function get_bilinear_form(mh_lev,biform,qdegree) + model = get_model(mh_lev) + Ω = Triangulation(model) + dΩ = Measure(Ω,qdegree) + return (u,v) -> biform(u,v,dΩ) +end + +function add_labels!(labels) + add_tag_from_tags!(labels,"top",[3,4,6]) + add_tag_from_tags!(labels,"bottom",[1,2,5]) +end diff --git a/joss_paper/scalability/template.sh b/joss_paper/scalability/template.sh index 453c896f..961e52c3 100644 --- a/joss_paper/scalability/template.sh +++ b/joss_paper/scalability/template.sh @@ -13,9 +13,8 @@ source {{{modules}}} mpiexec -n {{ncpus}} julia --project={{{projectdir}}} -O3 --check-bounds=no -e\ ' - push!(Base.LOAD_PATH, "{{driverdir}}"); - include({{{driver}}}); - main(; + using Scalability; + stokes_main(; nr={{nr}}, np={{np}}, nc={{nc}}, From 9698fbfca9abf3f16fe4e87b4688c98acfaf95e6 Mon Sep 17 00:00:00 2001 From: JordiManyer Date: Fri, 14 Jun 2024 12:11:39 +1000 Subject: [PATCH 08/30] Minor --- joss_paper/scalability/preparejobs.jl | 32 +++++++++++++----- joss_paper/scalability/src/stokes.jl | 48 ++++++++++++++++----------- joss_paper/scalability/src/utils.jl | 16 +++++++++ joss_paper/scalability/template.sh | 2 +- 4 files changed, 69 insertions(+), 29 deletions(-) diff --git a/joss_paper/scalability/preparejobs.jl b/joss_paper/scalability/preparejobs.jl index 4f60699f..20e9ea19 100644 --- a/joss_paper/scalability/preparejobs.jl +++ b/joss_paper/scalability/preparejobs.jl @@ -12,7 +12,9 @@ end function clean_params(d) o = Dict() for k in keys(d) - if isa(d[k],Real) + if k == :nc + #o[k] = "$(prod(d[k])÷1000)k" + elseif isa(d[k],Real) o[k] = Int(d[k]) elseif isa(d[k],Symbol) || isa(d[k],String) || isa(d[k],Number) o[k] = d[k] @@ -26,17 +28,19 @@ end function jobdict(params) np = params[:np] nc = params[:nc] + nl = params[:nl] fparams = clean_params(params) Dict( "project" => haskey(ENV,"PROJECT") ? ENV["PROJECT"] : "", "q" => "normal", - "o" => datadir(jobname(fparams,"o.txt")), - "e" => datadir(jobname(fparams,"e.txt")), + "o" => datadir(jobname(fparams,"o")), + "e" => datadir(jobname(fparams,"e")), "walltime" => "01:00:00", "ncpus" => prod(np), "nnodes" => prod(np)÷48, "mem" => "$(prod(np)*4)gb", "name" => jobname(fparams), + "nr" => 5, "np" => prod(np), "nc" => nc, "np_per_level" => Int[prod(np),prod(np)], @@ -49,12 +53,13 @@ function jobdict(params) ) end -function generate_dictionaries(num_cells,num_procs) +function generate_dictionaries(n_procs,n_cells,n_levels) dicts = Dict[] - for (nc,np) in zip(num_cells,num_procs) + for (np,nc,nl) in zip(n_procs,n_cells,n_levels) aux = Dict( :np => np, - :nc => nc + :nc => nc, + :nl => nl ) push!(dicts,aux) end @@ -63,9 +68,18 @@ end ########################################### -num_cells = [(6,6,3), (10,10,5),(20,20,5)] -num_procs = [(2,2) , (4,4) ,(6,8)] -dicts = generate_dictionaries(num_cells,num_procs) +n = 4 +n_nodes = [4^i for i in 0:n] +n_procs = [1,12,48 .* n_nodes...] +n_levels = [2,2,[i+2 for i in 0:n]...] + +c = (60,60) +n_cells_global = [c.*(2,2),[c.*(3,4).*(2^(i+1),2^(i+1)) for i in 0:n+1]...] +n_cells_coarse = [c,c.*(3,4),[c.*(6,8) for i in 0:n]...] +@assert all(r -> r == prod(n_cells_global[1])/n_procs[1],map(prod,n_cells_global)./n_procs) +@assert all(map((N,n,nl) -> prod(N) == prod(n)*4^(nl-1), n_cells_global,n_cells_coarse,n_levels)) + +dicts = generate_dictionaries(n_procs,n_cells_coarse,n_levels) template = read(projectdir("template.sh"),String) for params in dicts diff --git a/joss_paper/scalability/src/stokes.jl b/joss_paper/scalability/src/stokes.jl index 39498d68..cf6c5bea 100644 --- a/joss_paper/scalability/src/stokes.jl +++ b/joss_paper/scalability/src/stokes.jl @@ -1,14 +1,8 @@ -function stokes_driver( - parts, - np_per_level, - nc -) +function stokes_driver(parts,mh) t = PTimer(parts;verbose=true) tic!(t;barrier=true) - # Geometry - mh = CartesianModelHierarchy(parts,np_per_level,(0,1,0,1),nc;add_labels!) model = get_model(mh,1) # FE spaces @@ -42,15 +36,15 @@ function stokes_driver( biforms = map(mhl -> get_bilinear_form(mhl,biform_u,qdegree),mh) smoothers = map(mhl -> RichardsonSmoother(JacobiLinearSolver(),10,2.0/3.0), view(mh,1:num_levels(mh)-1)) restrictions, prolongations = setup_transfer_operators( - trials_u, qdegree; mode=:residual, solver=CGSolver(JacobiLinearSolver()) + trials_u, qdegree; mode=:residual, solver=CGSolver(JacobiLinearSolver(),verbose=i_am_main(parts)) ) solver_u = GMGLinearSolver( mh,trials_u,tests_u,biforms, prolongations,restrictions, pre_smoothers=smoothers, post_smoothers=smoothers, - coarsest_solver=LUSolver(), - maxiter=2,mode=:preconditioner,verbose=i_am_main(parts) + coarsest_solver=PETScLinearSolver(petsc_mumps_setup), + maxiter=3,mode=:preconditioner,verbose=i_am_main(parts) ) solver_u.log.depth = 4 @@ -71,15 +65,24 @@ function stokes_driver( solve!(x,ns,b) toc!(t,"Solver") + # Cleanup PETSc C-allocated objects + nlevs = num_levels(mh) + cparts = get_parts(get_model(mh,nlevs)) + if i_am_in(cparts) + P_ns = ns.Pr_ns + gmg_ns = P_ns.block_ns[1] + finalize(gmg_ns.coarsest_solver_cache.X) + finalize(gmg_ns.coarsest_solver_cache.B) + finalize(gmg_ns.coarsest_solver_cache) + GridapPETSc.gridap_petsc_gc() + end + # Postprocess ncells = num_cells(model) ndofs_u = num_free_dofs(U) ndofs_p = num_free_dofs(Q) output = Dict{String,Any}() map_main(t.data) do timer_data - output["np"] = np_per_level[1] - output["nc"] = nc - output["np_per_level"] = np_per_level output["ncells"] = ncells output["ndofs_u"] = ndofs_u output["ndofs_p"] = ndofs_p @@ -100,20 +103,27 @@ function stokes_main(; parts = with_mpi() do distribute distribute(LinearIndices((prod(np),))) end + + mh = CartesianModelHierarchy(parts,np_per_level,(0,1,0,1),nc;add_labels!) + GridapPETSc.with(;args=split(petsc_options)) do - driver(parts,np_per_level,nc) + driver(parts,mh) for ir in 1:nr - if i_am_main(parts) + map_main(parts) do p println(repeat('-',28)) println(" ------- ITERATION $(ir) ------- ") println(repeat('-',28)) end - title_ir = "$(title)_$(ir)" - output = driver(parts,np_per_level,nc) + output = stokes_driver(parts,mh) map_main(parts) do p output["ir"] = ir - save("$title_ir.bson",output) + output["np"] = np + output["nc"] = nc + output["nl"] = length(np_per_level) + output["np_per_level"] = np_per_level + save("$(title)_$(ir).bson",output) end + GridapPETSc.gridap_petsc_gc() end end -end \ No newline at end of file +end diff --git a/joss_paper/scalability/src/utils.jl b/joss_paper/scalability/src/utils.jl index 1b608023..8a66ee8d 100644 --- a/joss_paper/scalability/src/utils.jl +++ b/joss_paper/scalability/src/utils.jl @@ -10,3 +10,19 @@ function add_labels!(labels) add_tag_from_tags!(labels,"top",[3,4,6]) add_tag_from_tags!(labels,"bottom",[1,2,5]) end + +function petsc_mumps_setup(ksp) + pc = Ref{GridapPETSc.PETSC.PC}() + mumpsmat = Ref{GridapPETSc.PETSC.Mat}() + @check_error_code GridapPETSc.PETSC.KSPSetType(ksp[],GridapPETSc.PETSC.KSPPREONLY) + @check_error_code GridapPETSc.PETSC.KSPGetPC(ksp[],pc) + @check_error_code GridapPETSc.PETSC.PCSetType(pc[],GridapPETSc.PETSC.PCLU) + @check_error_code GridapPETSc.PETSC.PCFactorSetMatSolverType(pc[],GridapPETSc.PETSC.MATSOLVERMUMPS) + @check_error_code GridapPETSc.PETSC.PCFactorSetUpMatSolverType(pc[]) + @check_error_code GridapPETSc.PETSC.PCFactorGetMatrix(pc[],mumpsmat) + #@check_error_code GridapPETSc.PETSC.MatMumpsSetIcntl(mumpsmat[], 4, 1) + @check_error_code GridapPETSc.PETSC.MatMumpsSetIcntl(mumpsmat[], 7, 0) +# @check_error_code GridapPETSc.PETSC.MatMumpsSetIcntl(mumpsmat[], 28, 2) +# @check_error_code GridapPETSc.PETSC.MatMumpsSetIcntl(mumpsmat[], 29, 2) + @check_error_code GridapPETSc.PETSC.KSPView(ksp[],C_NULL) +end diff --git a/joss_paper/scalability/template.sh b/joss_paper/scalability/template.sh index 961e52c3..21d234b3 100644 --- a/joss_paper/scalability/template.sh +++ b/joss_paper/scalability/template.sh @@ -19,6 +19,6 @@ mpiexec -n {{ncpus}} julia --project={{{projectdir}}} -O3 --check-bounds=no -e\ np={{np}}, nc={{nc}}, np_per_level={{np_per_level}}, - title="{{title}}", + title="{{{title}}}", ) ' From 4c4c83f6d1b4b8d2832f07154ec6bcc7d05d99cc Mon Sep 17 00:00:00 2001 From: JordiManyer Date: Fri, 14 Jun 2024 19:14:40 +1000 Subject: [PATCH 09/30] More fixes to driver --- joss_paper/paper.md | 1 - joss_paper/scalability/Project.toml | 4 ++++ joss_paper/scalability/compile/compile.jl | 10 ++++++++++ joss_paper/scalability/compile/warmup.jl | 10 ++++++++++ joss_paper/scalability/preparejobs.jl | 4 ++-- joss_paper/scalability/src/stokes.jl | 7 +++---- joss_paper/scalability/template.sh | 2 +- 7 files changed, 30 insertions(+), 8 deletions(-) create mode 100644 joss_paper/scalability/compile/compile.jl create mode 100644 joss_paper/scalability/compile/warmup.jl diff --git a/joss_paper/paper.md b/joss_paper/paper.md index 3bc07238..c8edb4d0 100644 --- a/joss_paper/paper.md +++ b/joss_paper/paper.md @@ -18,7 +18,6 @@ authors: affiliation: "2" - name: Santiago Badia orcid: 0000-0003-2391-4086 - corresponding: true affiliation: "1,3" affiliations: - name: School of Mathematics, Monash University, Clayton, Victoria, 3800, Australia. diff --git a/joss_paper/scalability/Project.toml b/joss_paper/scalability/Project.toml index ed7e839e..d6c4d8a5 100644 --- a/joss_paper/scalability/Project.toml +++ b/joss_paper/scalability/Project.toml @@ -11,4 +11,8 @@ GridapDistributed = "f9701e48-63b3-45aa-9a63-9bc6c271f355" GridapPETSc = "bcdc36c2-0c3e-11ea-095a-c9dadae499f1" GridapSolvers = "6d3209ee-5e3c-4db7-a716-942eb12ed534" Mustache = "ffc61752-8dc7-55ee-8c37-f3e9cdd09e70" +PackageCompiler = "9b87118b-4619-50d2-8e1e-99f35a4d4d9d" PartitionedArrays = "5a9dfac6-5c52-46f7-8278-5e2210713be9" + +[extras] +MPIPreferences = "3da0fdf6-3ccc-4f1b-acd9-58baa6c99267" diff --git a/joss_paper/scalability/compile/compile.jl b/joss_paper/scalability/compile/compile.jl new file mode 100644 index 00000000..bf1ac4a0 --- /dev/null +++ b/joss_paper/scalability/compile/compile.jl @@ -0,0 +1,10 @@ +using PackageCompiler + +create_sysimage( + [ + "Gridap","GridapDistributed","GridapPETSc","GridapSolvers","PartitionedArrays", + "BSON","DrWatson","FileIO" + ], + sysimage_path=joinpath(@__DIR__,"..","Scalability.so"), + precompile_execution_file=joinpath(@__DIR__,"warmup.jl") +) diff --git a/joss_paper/scalability/compile/warmup.jl b/joss_paper/scalability/compile/warmup.jl new file mode 100644 index 00000000..27385909 --- /dev/null +++ b/joss_paper/scalability/compile/warmup.jl @@ -0,0 +1,10 @@ + +using Scalability + +stokes_main(; + nr=1, + np=1, + nc=(5,5), + np_per_level=[1,1], + title="data/compile", +) diff --git a/joss_paper/scalability/preparejobs.jl b/joss_paper/scalability/preparejobs.jl index 20e9ea19..5bc8574c 100644 --- a/joss_paper/scalability/preparejobs.jl +++ b/joss_paper/scalability/preparejobs.jl @@ -43,13 +43,13 @@ function jobdict(params) "nr" => 5, "np" => prod(np), "nc" => nc, - "np_per_level" => Int[prod(np),prod(np)], + "np_per_level" => fill(prod(np),nl), "projectdir" => projectdir(), "datadir" => datadir(), "modules" => projectdir("modules.sh"), "driverdir" => projectdir(), "title" => datadir(jobname(fparams)), - "sysimage" => projectdir("GridapSolvers.so") + "sysimage" => projectdir("Scalability.so") ) end diff --git a/joss_paper/scalability/src/stokes.jl b/joss_paper/scalability/src/stokes.jl index cf6c5bea..dcbd8478 100644 --- a/joss_paper/scalability/src/stokes.jl +++ b/joss_paper/scalability/src/stokes.jl @@ -36,7 +36,7 @@ function stokes_driver(parts,mh) biforms = map(mhl -> get_bilinear_form(mhl,biform_u,qdegree),mh) smoothers = map(mhl -> RichardsonSmoother(JacobiLinearSolver(),10,2.0/3.0), view(mh,1:num_levels(mh)-1)) restrictions, prolongations = setup_transfer_operators( - trials_u, qdegree; mode=:residual, solver=CGSolver(JacobiLinearSolver(),verbose=i_am_main(parts)) + trials_u, qdegree; mode=:residual, solver=CGSolver(JacobiLinearSolver(),verbose=false) ) solver_u = GMGLinearSolver( mh,trials_u,tests_u,biforms, @@ -67,8 +67,7 @@ function stokes_driver(parts,mh) # Cleanup PETSc C-allocated objects nlevs = num_levels(mh) - cparts = get_parts(get_model(mh,nlevs)) - if i_am_in(cparts) + GridapSolvers.MultilevelTools.with_level(mh,nlevs) do mhl P_ns = ns.Pr_ns gmg_ns = P_ns.block_ns[1] finalize(gmg_ns.coarsest_solver_cache.X) @@ -107,7 +106,7 @@ function stokes_main(; mh = CartesianModelHierarchy(parts,np_per_level,(0,1,0,1),nc;add_labels!) GridapPETSc.with(;args=split(petsc_options)) do - driver(parts,mh) + stokes_driver(parts,mh) for ir in 1:nr map_main(parts) do p println(repeat('-',28)) diff --git a/joss_paper/scalability/template.sh b/joss_paper/scalability/template.sh index 21d234b3..45eb3a30 100644 --- a/joss_paper/scalability/template.sh +++ b/joss_paper/scalability/template.sh @@ -11,7 +11,7 @@ source {{{modules}}} -mpiexec -n {{ncpus}} julia --project={{{projectdir}}} -O3 --check-bounds=no -e\ +mpiexec -n {{ncpus}} julia --project={{{projectdir}}} -O3 --check-bounds=no -J{{{sysimage}}} -e\ ' using Scalability; stokes_main(; From 30562d1d36facbb6ff5561be99724dcbc32e2fd8 Mon Sep 17 00:00:00 2001 From: JordiManyer Date: Fri, 14 Jun 2024 19:27:26 +1000 Subject: [PATCH 10/30] Added gitignore --- joss_paper/scalability/.gitignore | 1 + 1 file changed, 1 insertion(+) create mode 100644 joss_paper/scalability/.gitignore diff --git a/joss_paper/scalability/.gitignore b/joss_paper/scalability/.gitignore new file mode 100644 index 00000000..f48b12f5 --- /dev/null +++ b/joss_paper/scalability/.gitignore @@ -0,0 +1 @@ +Scalability.so \ No newline at end of file From c1832e994943f42b78b6c38ca932d89405c9e5b5 Mon Sep 17 00:00:00 2001 From: JordiManyer Date: Wed, 19 Jun 2024 11:04:48 +1000 Subject: [PATCH 11/30] Further changes to scalability --- joss_paper/paper.bib | 210 ++++++++++++++++++++++ joss_paper/paper.md | 9 +- joss_paper/scalability/compile/compile.sh | 3 + joss_paper/scalability/template.sh | 9 +- 4 files changed, 227 insertions(+), 4 deletions(-) create mode 100644 joss_paper/scalability/compile/compile.sh diff --git a/joss_paper/paper.bib b/joss_paper/paper.bib index e69de29b..351a0c3b 100644 --- a/joss_paper/paper.bib +++ b/joss_paper/paper.bib @@ -0,0 +1,210 @@ +@techreport{petsc-user-ref, + author = {Satish Balay and Shrirang Abhyankar and Mark~F. Adams and Steven Benson and Jed Brown + and Peter Brune and Kris Buschelman and Emil Constantinescu and Lisandro Dalcin and Alp Dener + and Victor Eijkhout and William~D. Gropp and V\'{a}clav Hapla and Tobin Isaac and Pierre Jolivet + and Dmitry Karpeev and Dinesh Kaushik and Matthew~G. Knepley and Fande Kong and Scott Kruger + and Dave~A. May and Lois Curfman McInnes and Richard Tran Mills and Lawrence Mitchell and Todd Munson + and Jose~E. Roman and Karl Rupp and Patrick Sanan and Jason Sarich and Barry~F. Smith + and Stefano Zampini and Hong Zhang and Hong Zhang and Junchao Zhang}, + title = {{PETSc/TAO} Users Manual}, + institution = {Argonne National Laboratory}, + number = {ANL-21/39 - Revision 3.16}, + year = {2021}, +} + +@manual{mpi40, + author = "{Message Passing Interface Forum}", + title = "{MPI}: A Message-Passing Interface Standard Version 4.0", + url = "https://www.mpi-forum.org/docs/mpi-4.0/mpi40-report.pdf", + year = 2021, + month = jun +} + +@article{Verdugo:2021, + doi = {10.1016/j.cpc.2022.108341}, + url = {https://doi.org/10.1016/j.cpc.2022.108341}, + year = {2022}, + month = jul, + publisher = {Elsevier {BV}}, + volume = {276}, + pages = {108341}, + author = {Francesc Verdugo and Santiago Badia}, + title = {The software design of {G}ridap: a finite element package based on the {J}ulia {JIT} compiler}, + journal = {Computer Physics Communications} +} + +@misc{gridapetsc, + author = {Verdugo, F. and Sande, V. and Martin, A. F.}, + title = {GridapPETSc}, + year = {2021}, + publisher = {GitHub}, + journal = {GitHub repository}, + url = {https://github.com/gridap/GridapPETSc.jl} +} + +@misc{gridap4est, + author = {Martin, A. F.}, + title = {GridapP4est}, + year = {2021}, + publisher = {GitHub}, + journal = {GitHub repository}, + url = {https://github.com/gridap/GridapP4est.jl} +} + +@misc{parrays, + author = {Verdugo, F.}, + title = {PartitionedArrays}, + year = {2021}, + publisher = {GitHub}, + journal = {GitHub repository}, + url = {https://github.com/fverdugo/PartitionedArrays.jl} +} + +@article{Bezanson2017, +abstract = {Bridging cultures that have often been distant, Julia combines expertise from the diverse fields of computer science and computational science to create a new approach to numerical computing. Julia...}, +archivePrefix = {arXiv}, +arxivId = {1411.1607}, +author = {Bezanson, Jeff and Edelman, Alan and Karpinski, Stefan and Shah, Viral B.}, +doi = {10.1137/141000671}, +eprint = {1411.1607}, +issn = {00361445}, +journal = {SIAM Review}, +keywords = {65Y05,68N15,97P40,Julia,numerical,parallel,scientific computing}, +month = {feb}, +number = {1}, +pages = {65--98}, +publisher = {Society for Industrial and Applied Mathematics}, +title = {{Julia: a fresh approach to numerical computing}}, +volume = {59}, +year = {2017} +} + +@article{Badia2020, +abstract = {Gridap is a new Finite Element (FE) framework, exclusively written in the Julia programming language, for the numerical simulation of a wide range of mathematical models governed by partial differential equations (PDEs). The library provides a feature-rich set of discretization techniques, including continuous and discontinuous FE methods with Lagrangian, Raviart-Thomas, or N{\'{e}}d{\'{e}}lec interpolations, and supports a wide range of problem types including linear, nonlinear, single-field, and multi-field PDEs (see (Badia, Mart{\'{i}}n, {\&} Principe, 2018, Section 3) for a detailed presentation of the mathematical abstractions behind the implementation of these FE methods). Gridap is designed to help application experts to easily simulate real-world problems, to help researchers improve productivity when developing new FE-related techniques, and also for its usage in numerical PDE courses. The main motivation behind Gridap is to find an improved balance between computational performance, user-experience, and work-flow productivity when working with FE libraries. Previous FE frameworks, e.g., FEniCS (Alnaes et al., 2015) or Deal.II (Bangerth, Hartmann, {\&} Kanschat, 2007) usually provides a high-level user front-end to facilitate the use of the library and a computational back-end to achieve performance. The user front-end is usually programmable in an interpreted language like Python, whereas the computational back-end is usually coded in a compiled language like C/C++ or Fortran. Users can benefit from the high-level front-end (i.e., for rapid prototyping) and simultaneously enjoy the performance of the compiled back-end. This approach reaches a compromise between performance and productivity when the back-end provides all the functionality required by the user. However, it does not satisfactorily address the needs of researchers on numerical methods willing to extend the library with new techniques or features. These extensions usually need to be done at the level of the computational back-end for performance reasons.}, +author = {Badia, Santiago and Verdugo, Francesc}, +doi = {10.21105/JOSS.02520}, +file = {:home/amartin/.local/share/data/Mendeley Ltd./Mendeley Desktop/Downloaded/Badia, Verdugo - 2020 - Gridap An extensible Finite Element toolbox in Julia.pdf:pdf}, +issn = {2475-9066}, +journal = {Journal of Open Source Software}, +month = {aug}, +number = {52}, +pages = {2520}, +publisher = {The Open Journal}, +title = {{Gridap: an extensible finite element toolbox in Julia}}, +url = {https://joss.theoj.org/papers/10.21105/joss.02520}, +volume = {5}, +year = {2020} +} + +@article{Badia2020a, +abstract = {In this work we formally derive and prove the correctness of the algorithms and data structures in a parallel, distributed-memory, generic finite element framework that supports {\$}h{\$}-adaptivity on c...}, +archivePrefix = {arXiv}, +arxivId = {1907.03709}, +author = {Badia, Santiago and Mart{\'{i}}n, Alberto F. and Neiva, Eric and Verdugo, Francesc}, +doi = {10.1137/20M1328786}, +eprint = {1907.03709}, +issn = {10957197}, +journal = {SIAM Journal on Scientific Computing}, +keywords = {65M50,65N30,65Y05,65Y20,adaptive mesh refinement,finite elements,forest of trees,parallel algorithms,partial differential equations,scientific software}, +month = {dec}, +number = {6}, +pages = {C436--C468}, +publisher = {Society for Industrial and Applied Mathematics}, +title = {{A generic finite element framework on parallel tree-based adaptive meshes}}, +volume = {42}, +year = {2020} +} + +@article{p4est, +abstract = {We present scalable algorithms for parallel adaptive mesh refinement and coarsening (AMR), partitioning, and 2:1 balancing on computational domains composed of multiple connected two-dimensional qu...}, +author = {Burstedde, Carsten and Wilcox, Lucas C. and Ghattas, Omar}, +doi = {10.1137/100791634}, +issn = {10648275}, +journal = {SIAM Journal on Scientific Computing}, +keywords = {65D18,65M50,65Y05,68W10,Morton code,forest of octrees,large-scale scientific computing,parallel adaptive mesh refinement,scalable algorithms}, +month = {may}, +number = {3}, +pages = {1103--1133}, +publisher = {Society for Industrial and Applied Mathematics}, +title = {{p4est: scalable algorithms for parallel adaptive mesh refinement on forests of octrees}}, +volume = {33}, +year = {2011} +} + +@article{mfem, + title = {{MFEM}: A modular finite element methods library}, + author = {R. Anderson and J. Andrej and A. Barker and J. Bramwell and J.-S. Camier and + J. Cerveny V. Dobrev and Y. Dudouit and A. Fisher and Tz. Kolev and W. Pazner and + M. Stowell and V. Tomov and I. Akkerman and J. Dahm and D. Medina and S. Zampini}, + journal = {Computers \& Mathematics with Applications}, + doi = {10.1016/j.camwa.2020.06.009}, + volume = {81}, + pages = {42-74}, + year = {2021} +} + +@article {freefem, + doi = {10.1515/jnum-2012-0013}, + AUTHOR = {Hecht, F.}, TITLE = {New development in {FreeFem++}}, + JOURNAL = {J. Numer. Math.}, FJOURNAL = {Journal of Numerical Mathematics}, + VOLUME = {20}, YEAR = {2012}, + NUMBER = {3-4}, PAGES = {251--265}, + ISSN = {1570-2820}, MRCLASS = {65Y15}, MRNUMBER = {3043640}, +} + +@Article{libMeshPaper, + doi = {10.1007/s00366-006-0049-3}, + author = {B.~S.~Kirk and J.~W.~Peterson and R.~H.~Stogner and G.~F.~Carey}, + title = {{\texttt{libMesh}: A C++ library for parallel adaptive mesh refinement/coarsening simulations}}, + journal = {Engineering with Computers}, + volume = 22, + number = {3--4}, + pages = {237--254}, + year = 2006, + note = {\url{http://dx.doi.org/10.1007/s00366-006-0049-3}} +} + +@article{dealII93, + title = {The \texttt{deal.II} Library, Version 9.3}, + author = {Daniel Arndt and Wolfgang Bangerth and Bruno Blais and + Marc Fehling and Rene Gassm{\"o}ller and Timo Heister + and Luca Heltai and Uwe K{\"o}cher and Martin + Kronbichler and Matthias Maier and Peter Munch and + Jean-Paul Pelteret and Sebastian Proell and Konrad + Simon and Bruno Turcksin and David Wells and Jiaqi + Zhang}, + journal = {Journal of Numerical Mathematics}, + year = {2021}, + url = {https://dealii.org/deal93-preprint.pdf}, + doi = {10.1515/jnma-2021-0081}, + volume = {29}, + number = {3}, + pages = {171--186} +} + +@article{Kirby2006, +abstract = {As a key step towards a complete automation of the finite element method, we present a new algorithm for automatic and efficient evaluation of multilinear variational forms. The algorithm has been implemented in the form of a compiler, the FEniCS Form Compiler (FFC). We present benchmark results for a series of standard variational forms, including the incompressible Navier-Stokes equations and linear elasticity. The speedup compared to the standard quadrature-based approach is impressive; in some cases the speedup is as large as a factor of 1000. {\textcopyright} 2006 ACM.}, +archivePrefix = {arXiv}, +arxivId = {1112.0402}, +author = {Kirby, Robert C and Logg, Anders}, +doi = {10.1145/1163641.1163644}, +eprint = {1112.0402}, +file = {:home/fverdugo/.local/share/data/Mendeley Ltd./Mendeley Desktop/Downloaded/Kirby - Unknown - A Compiler for Variational Forms.pdf:pdf}, +issn = {00983500}, +journal = {ACM Transactions on Mathematical Software}, +keywords = {Automation,Compiler,Finite element,Variational form}, +number = {3}, +pages = {417--444}, +title = {{A compiler for variational forms}}, +volume = {32}, +year = {2006} +} + +@book{fenics-book, + doi = {10.1007/978-3-642-23099-8}, + url = {https://doi.org/10.1007/978-3-642-23099-8}, + year = {2012}, + publisher = {Springer Berlin Heidelberg}, + editor = {Anders Logg and Kent-Andre Mardal and Garth Wells}, + title = {Automated Solution of Differential Equations by the Finite Element Method} +} \ No newline at end of file diff --git a/joss_paper/paper.md b/joss_paper/paper.md index c8edb4d0..4ca09413 100644 --- a/joss_paper/paper.md +++ b/joss_paper/paper.md @@ -53,7 +53,8 @@ GridapSolvers complements GridapPETSc with a modular and extensible interface fo - A set of HPC-first implementations for popular Krylov-based iterative solvers. These solvers extend Gridap's API and are fully compatible with PartitionedArrays. - A modular, high-level interface for designing block-based preconditioners for multiphysics problems. These preconditioners can be used together with any solver compliant with Gridap's API, including those provided by GridapPETSc. -- A generic interface to handle multi-level distributed meshes, with full support for Adaptative Mesh Refinement (AMR) through GridapP4est. It also provides a modular implementation of geometric multigrid (GMG) solvers, allowing different types of smoothers and restriction/prolongation operators. +- A generic interface to handle multi-level distributed meshes, with full support for Adaptative Mesh Refinement (AMR) using p4est [p4est] through GridapP4est. +- A modular implementation of geometric multigrid (GMG) solvers, allowing different types of smoothers and restriction/prolongation operators. - A generic interface for patch-based subdomain decomposition methods, and an implementation of patch-based smoothers for geometric multigrid solvers. ![GridapSolvers and its relation to other packages in the Julia package ecosystem. In this diagram, each node represents a Julia package, while the (directed) arrows represent relations (dependencies) among packages. Dashed arrows mean the package can be used, but is not necessary. \label{fig:packages}](packages.png){ width=60% } @@ -61,7 +62,7 @@ GridapSolvers complements GridapPETSc with a modular and extensible interface fo # Demo The following code snippet shows how to solve a 2D Stokes cavity problem in a cartesian domain $\Omega = [0,1]^2$. We discretize the velocity and pressure in $H^1(\Omega)$ and $L^2(\Omega)$ respectively, and use the well known stable element pair $Q_k \times P_{k-1}$ with $k=2$. For the cavity problem, we fix the velocity to $u_b = \vec{0}$ and $u_t = \hat{x}$ on the bottom and top boundaries respectively, and homogeneous Neumann boundary conditions elsewhere. -The system is block-assembled and solved using a GMRES solver, right-preconditioned with block-triangular Shur-complement-based preconditioner. The Shur complement is approximated by a mass matrix, and solved using a CG solver with Jacobi preconditioner. The eliminated velocity block is approximated by a 2-level V-cycle Geometric Multigrid solver. +The system is block-assembled and solved using a GMRES solver, right-preconditioned with block-triangular Shur-complement-based preconditioner. The Shur complement is approximated by a mass matrix, and solved using a CG solver with Jacobi preconditioner. The eliminated velocity block is approximated by a 2-level V-cycle Geometric Multigrid solver. The coarsest-level system is solved exactly using MUMPS [@MUMPS], provided by PETSc [@petsc-user-ref] though the package GridapPETSc.jl. The code is setup to run in parallel with 4 MPI tasks and can be executed with the following command: `mpiexec -n 4 julia --project=. demo.jl`. ```julia @@ -70,8 +71,10 @@ Code in `demo.jl`. # Parallel scaling benchmark +The following section shows scalability results for the demo problem discussed above. We run our code on the Gadi supercomputer, which is part of the Australian National Computational Infrastructure. We use Intel's Cascade Lake 2x24-core Xeon Platinum 8274 nodes. Scalability is shown for up to XXX cores, for a fixed local problem size of XXX quadrangle cells per processor. This amounts to a maximum size of XXX cells and XXX degrees of freedom distributed amongst XXX processors. The code used to create these results can be found together with the submiteed paper (LINK). + # Acknowledgements -This research was partially funded by the Australian Government through the Australian Research Council (project number DP210103092), the European Commission under the FET-HPC ExaQUte project (Grant agreement ID: 800898) within the Horizon 2020 Framework Program and the project RTI2018-096898-B-I00 from the “FEDER/Ministerio de Ciencia e Innovación (MCIN) – Agencia Estatal de Investigación (AEI)”. F. Verdugo acknowledges support from the “Severo Ochoa Program for Centers of Excellence in R&D (2019-2023)" under the grant CEX2018-000797-S funded by MCIN/AEI/10.13039/501100011033. This work was also supported by computational resources provided by the Australian Government through NCI under the National Computational Merit Allocation Scheme (NCMAS). +This research was partially funded by the Australian Government through the Australian Research Council (project number DP210103092). This work was also supported by computational resources provided by the Australian Government through NCI under the National Computational Merit Allocation Scheme (NCMAS). # References diff --git a/joss_paper/scalability/compile/compile.sh b/joss_paper/scalability/compile/compile.sh new file mode 100644 index 00000000..802aa5a6 --- /dev/null +++ b/joss_paper/scalability/compile/compile.sh @@ -0,0 +1,3 @@ +#!/bin/bash + +julia --project=. -O3 compile/compile.jl diff --git a/joss_paper/scalability/template.sh b/joss_paper/scalability/template.sh index 45eb3a30..06b3427f 100644 --- a/joss_paper/scalability/template.sh +++ b/joss_paper/scalability/template.sh @@ -11,7 +11,14 @@ source {{{modules}}} -mpiexec -n {{ncpus}} julia --project={{{projectdir}}} -O3 --check-bounds=no -J{{{sysimage}}} -e\ +julia --project={{{projectdir}}} -O3 -J{{{sysimage}}} -e\ + ' + using Scalability + using Gridap, GridapDistributed, PartitionedArrays, GridapSolvers, GridapPETSc + using FileIO, BSON + ' + +mpiexec -n {{ncpus}} julia --project={{{projectdir}}} -O3 -J{{{sysimage}}} -e\ ' using Scalability; stokes_main(; From 98bd6aa32a8e121e1c34cf85cde76d3c74eeddd3 Mon Sep 17 00:00:00 2001 From: JordiManyer Date: Wed, 19 Jun 2024 11:08:17 +1000 Subject: [PATCH 12/30] Minor --- joss_paper/demo.jl | 2 +- joss_paper/paper.bib | 4 +--- joss_paper/paper.md | 2 +- joss_paper/scalability/postprocess.jl | 3 +-- 4 files changed, 4 insertions(+), 7 deletions(-) diff --git a/joss_paper/demo.jl b/joss_paper/demo.jl index 3bf24f37..76d88aea 100644 --- a/joss_paper/demo.jl +++ b/joss_paper/demo.jl @@ -74,7 +74,7 @@ with_mpi() do distribute blocks = [LinearSystemBlock() LinearSystemBlock(); LinearSystemBlock() BiformBlock((p,q) -> ∫(p*q)dΩ,Q,Q)] P = BlockTriangularSolver(blocks,[solver_u,solver_p]) - solver = FGMRESSolver(10;Pr=P,rtol=1.e-8,verbose=i_am_main(parts)) + solver = GMRESSolver(10;Pr=P,rtol=1.e-8,verbose=i_am_main(parts)) ns = numerical_setup(symbolic_setup(solver,A),A) x = allocate_in_domain(A); fill!(x,0.0) diff --git a/joss_paper/paper.bib b/joss_paper/paper.bib index 351a0c3b..ef63c707 100644 --- a/joss_paper/paper.bib +++ b/joss_paper/paper.bib @@ -78,7 +78,6 @@ @article{Bezanson2017 volume = {59}, year = {2017} } - @article{Badia2020, abstract = {Gridap is a new Finite Element (FE) framework, exclusively written in the Julia programming language, for the numerical simulation of a wide range of mathematical models governed by partial differential equations (PDEs). The library provides a feature-rich set of discretization techniques, including continuous and discontinuous FE methods with Lagrangian, Raviart-Thomas, or N{\'{e}}d{\'{e}}lec interpolations, and supports a wide range of problem types including linear, nonlinear, single-field, and multi-field PDEs (see (Badia, Mart{\'{i}}n, {\&} Principe, 2018, Section 3) for a detailed presentation of the mathematical abstractions behind the implementation of these FE methods). Gridap is designed to help application experts to easily simulate real-world problems, to help researchers improve productivity when developing new FE-related techniques, and also for its usage in numerical PDE courses. The main motivation behind Gridap is to find an improved balance between computational performance, user-experience, and work-flow productivity when working with FE libraries. Previous FE frameworks, e.g., FEniCS (Alnaes et al., 2015) or Deal.II (Bangerth, Hartmann, {\&} Kanschat, 2007) usually provides a high-level user front-end to facilitate the use of the library and a computational back-end to achieve performance. The user front-end is usually programmable in an interpreted language like Python, whereas the computational back-end is usually coded in a compiled language like C/C++ or Fortran. Users can benefit from the high-level front-end (i.e., for rapid prototyping) and simultaneously enjoy the performance of the compiled back-end. This approach reaches a compromise between performance and productivity when the back-end provides all the functionality required by the user. However, it does not satisfactorily address the needs of researchers on numerical methods willing to extend the library with new techniques or features. These extensions usually need to be done at the level of the computational back-end for performance reasons.}, author = {Badia, Santiago and Verdugo, Francesc}, @@ -95,7 +94,6 @@ @article{Badia2020 volume = {5}, year = {2020} } - @article{Badia2020a, abstract = {In this work we formally derive and prove the correctness of the algorithms and data structures in a parallel, distributed-memory, generic finite element framework that supports {\$}h{\$}-adaptivity on c...}, archivePrefix = {arXiv}, @@ -207,4 +205,4 @@ @book{fenics-book publisher = {Springer Berlin Heidelberg}, editor = {Anders Logg and Kent-Andre Mardal and Garth Wells}, title = {Automated Solution of Differential Equations by the Finite Element Method} -} \ No newline at end of file +} diff --git a/joss_paper/paper.md b/joss_paper/paper.md index 4ca09413..5d2addd8 100644 --- a/joss_paper/paper.md +++ b/joss_paper/paper.md @@ -57,7 +57,7 @@ GridapSolvers complements GridapPETSc with a modular and extensible interface fo - A modular implementation of geometric multigrid (GMG) solvers, allowing different types of smoothers and restriction/prolongation operators. - A generic interface for patch-based subdomain decomposition methods, and an implementation of patch-based smoothers for geometric multigrid solvers. -![GridapSolvers and its relation to other packages in the Julia package ecosystem. In this diagram, each node represents a Julia package, while the (directed) arrows represent relations (dependencies) among packages. Dashed arrows mean the package can be used, but is not necessary. \label{fig:packages}](packages.png){ width=60% } +![GridapSolvers and its relation to other packages in the Julia package ecosystem. In this diagram, each node represents a Julia package, while the (directed) arrows represent relations (dependencies) among packages. Dashed arrows mean the package can be used, but is not required. \label{fig:packages}](packages.png){ width=60% } # Demo diff --git a/joss_paper/scalability/postprocess.jl b/joss_paper/scalability/postprocess.jl index 302ebd4d..4a8d00c3 100644 --- a/joss_paper/scalability/postprocess.jl +++ b/joss_paper/scalability/postprocess.jl @@ -28,7 +28,7 @@ begin n_dofs_u = df.ndofs_u[1], n_dofs_p = df.ndofs_p[1], n_dofs = df.ndofs_u[1] + df.ndofs_p[1], - n_cells = df.nc[1] + n_cells = df.ncells[1] ) end sort!(df,:np) @@ -37,7 +37,6 @@ end # ╔═╡ a10880e9-680b-46f0-9bda-44e53a7196ce begin plt = plot(xlabel="N processors",ylabel="walltime (s)",legend=false) - plot!(df[!,:np],df[!,:t_solver][1]*df[!,:np]./df[!,:np][1]) # Perfect scaling plot!(df[!,:np],df[!,:t_solver]) end From f899259690b21b0d4ff24e7c65bfdd7b106c2bc1 Mon Sep 17 00:00:00 2001 From: JordiManyer Date: Mon, 24 Jun 2024 21:36:42 +1000 Subject: [PATCH 13/30] Minor --- joss_paper/scalability/postprocess.jl | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/joss_paper/scalability/postprocess.jl b/joss_paper/scalability/postprocess.jl index 4a8d00c3..70244611 100644 --- a/joss_paper/scalability/postprocess.jl +++ b/joss_paper/scalability/postprocess.jl @@ -40,7 +40,11 @@ begin plot!(df[!,:np],df[!,:t_solver]) end +# ╔═╡ 8d1b090e-2548-48fe-afb9-92f044442a80 + + # ╔═╡ Cell order: # ╠═f0aae79e-13d9-11ef-263f-b720d8f10878 # ╠═4725bb5d-8973-436b-af96-74cc90e7f354 # ╠═a10880e9-680b-46f0-9bda-44e53a7196ce +# ╠═8d1b090e-2548-48fe-afb9-92f044442a80 From 7cc0448a0585b3a6188385cc45d80d11ff6cfd85 Mon Sep 17 00:00:00 2001 From: JordiManyer Date: Thu, 18 Jul 2024 16:29:23 +1000 Subject: [PATCH 14/30] More modifications to the drivers --- joss_paper/scalability/Project.toml | 1 + joss_paper/scalability/compile/compile.sh | 13 +- joss_paper/scalability/compile/warmup.jl | 23 +++- joss_paper/scalability/modules.sh | 10 +- joss_paper/scalability/preparejobs.jl | 51 ++++++-- joss_paper/scalability/src/Scalability.jl | 3 +- joss_paper/scalability/src/stokes.jl | 57 ++------- joss_paper/scalability/src/stokes_gmg.jl | 145 ++++++++++++++++++++++ joss_paper/scalability/src/utils.jl | 42 ++++++- joss_paper/scalability/template.sh | 5 +- joss_paper/scalability/template_gmg.sh | 35 ++++++ 11 files changed, 318 insertions(+), 67 deletions(-) create mode 100644 joss_paper/scalability/src/stokes_gmg.jl create mode 100644 joss_paper/scalability/template_gmg.sh diff --git a/joss_paper/scalability/Project.toml b/joss_paper/scalability/Project.toml index d6c4d8a5..d3a61157 100644 --- a/joss_paper/scalability/Project.toml +++ b/joss_paper/scalability/Project.toml @@ -8,6 +8,7 @@ DrWatson = "634d3b9d-ee7a-5ddf-bec9-22491ea816e1" FileIO = "5789e2e9-d7fb-5bc7-8068-2c6fae9b9549" Gridap = "56d4f2e9-7ea1-5844-9cf6-b9c51ca7ce8e" GridapDistributed = "f9701e48-63b3-45aa-9a63-9bc6c271f355" +GridapP4est = "c2c8e14b-f5fd-423d-9666-1dd9ad120af9" GridapPETSc = "bcdc36c2-0c3e-11ea-095a-c9dadae499f1" GridapSolvers = "6d3209ee-5e3c-4db7-a716-942eb12ed534" Mustache = "ffc61752-8dc7-55ee-8c37-f3e9cdd09e70" diff --git a/joss_paper/scalability/compile/compile.sh b/joss_paper/scalability/compile/compile.sh index 802aa5a6..da4c3f19 100644 --- a/joss_paper/scalability/compile/compile.sh +++ b/joss_paper/scalability/compile/compile.sh @@ -1,3 +1,14 @@ #!/bin/bash +#PBS -P np01 +#PBS -q normal +#PBS -l walltime=01:00:00 +#PBS -l ncpus=4 +#PBS -l mem=16gb +#PBS -N compile +#PBS -l wd +#PBS -o /scratch/np01/jm3247/GridapSolvers.jl/joss_paper/scalability/compile/compile.o +#PBS -e /scratch/np01/jm3247/GridapSolvers.jl/joss_paper/scalability/compile/compile.e -julia --project=. -O3 compile/compile.jl +source /scratch/np01/jm3247/GridapSolvers.jl/joss_paper/scalability/modules.sh + +julia --project=/scratch/np01/jm3247/GridapSolvers.jl/joss_paper/scalability -O3 /scratch/np01/jm3247/GridapSolvers.jl/joss_paper/scalability/compile/compile.jl diff --git a/joss_paper/scalability/compile/warmup.jl b/joss_paper/scalability/compile/warmup.jl index 27385909..f03d9628 100644 --- a/joss_paper/scalability/compile/warmup.jl +++ b/joss_paper/scalability/compile/warmup.jl @@ -3,8 +3,25 @@ using Scalability stokes_main(; nr=1, - np=1, - nc=(5,5), - np_per_level=[1,1], + np=(1,1), + nc=(4,4), title="data/compile", ) + +stokes_gmg_main(; + nr=1, + np=(1,1), + nc=(4,4), + np_per_level=[(1,1),(1,1)], + title="data/compile_gmg", + mode = :mpi +) + +stokes_gmg_main(; + nr=0, + np=(2,1), + nc=(4,4), + np_per_level=[(2,1),(1,1)], + title="data/compile_gmg", + mode = :debug +) diff --git a/joss_paper/scalability/modules.sh b/joss_paper/scalability/modules.sh index 284ce35f..6e21db7a 100644 --- a/joss_paper/scalability/modules.sh +++ b/joss_paper/scalability/modules.sh @@ -4,6 +4,14 @@ module load pbs module load intel-compiler-llvm/2023.2.0 module load intel-mpi/2021.10.0 module load intel-mkl/2023.2.0 +export MPI_VERSION="intel-$INTEL_MPI_VERSION" +export JULIA_MPI_PATH=$INTEL_MPI_ROOT + +#module load gcc +#module load openmpi/4.1.5 +#module load intel-mkl +#export MPI_VERSION="ompi-$OMPI_VERSION" +#export JULIA_MPI_PATH=$OMPI_ROOT export P4EST_VERSION='2.8.5' export PETSC_VERSION='3.19.5' @@ -11,8 +19,6 @@ export PROJECT="np01" SCRATCH="/scratch/$PROJECT/$USER" export JULIA_DEPOT_PATH="$SCRATCH/.julia" -export MPI_VERSION="intel-$INTEL_MPI_VERSION" -export JULIA_MPI_PATH=$INTEL_MPI_ROOT export JULIA_PETSC_LIBRARY="$HOME/bin/petsc/$PETSC_VERSION-$MPI_VERSION/lib/libpetsc" export P4EST_ROOT_DIR="$HOME/bin/p4est/$P4EST_VERSION-$MPI_VERSION" diff --git a/joss_paper/scalability/preparejobs.jl b/joss_paper/scalability/preparejobs.jl index 5bc8574c..a94a592b 100644 --- a/joss_paper/scalability/preparejobs.jl +++ b/joss_paper/scalability/preparejobs.jl @@ -25,6 +25,14 @@ function clean_params(d) return o end +function get_np_per_level(np,nl) + a = fill(np,nl) + if prod(np) > 768 + a[end] = (24,32) # 768 + end + return a +end + function jobdict(params) np = params[:np] nc = params[:nc] @@ -41,9 +49,9 @@ function jobdict(params) "mem" => "$(prod(np)*4)gb", "name" => jobname(fparams), "nr" => 5, - "np" => prod(np), + "np" => np, "nc" => nc, - "np_per_level" => fill(prod(np),nl), + "np_per_level" => get_np_per_level(np,nl), "projectdir" => projectdir(), "datadir" => datadir(), "modules" => projectdir("modules.sh"), @@ -68,20 +76,45 @@ end ########################################### +# Even number of nodes + n = 4 -n_nodes = [4^i for i in 0:n] -n_procs = [1,12,48 .* n_nodes...] +n_nodes = [(2,2).^i for i in 0:n] +n_procs = [(1,1),(3,4),[(6,8).*nn for nn in n_nodes]...] n_levels = [2,2,[i+2 for i in 0:n]...] -c = (60,60) -n_cells_global = [c.*(2,2),[c.*(3,4).*(2^(i+1),2^(i+1)) for i in 0:n+1]...] +c = (6,8) .* 8 n_cells_coarse = [c,c.*(3,4),[c.*(6,8) for i in 0:n]...] -@assert all(r -> r == prod(n_cells_global[1])/n_procs[1],map(prod,n_cells_global)./n_procs) +n_cells_global = map((nc,nl) -> nc .* (2^(nl-1),2^(nl-1)),n_cells_coarse,n_levels) +@assert all(r -> r == prod(n_cells_global[1])/prod(n_procs[1]),map(prod,n_cells_global)./map(prod,n_procs)) @assert all(map((N,n,nl) -> prod(N) == prod(n)*4^(nl-1), n_cells_global,n_cells_coarse,n_levels)) -dicts = generate_dictionaries(n_procs,n_cells_coarse,n_levels) +dicts_even_gmg = generate_dictionaries(n_procs,n_cells_coarse,n_levels) +dicts_even_petsc = generate_dictionaries(n_procs,n_cells_global,n_levels) + +# Odd number of nodes + +n = 3 +n_nodes = [(2,1).*(2,2).^i for i in 0:n] +n_procs = [(6,8).*nn for nn in n_nodes] +n_levels = [i+2 for i in 0:n] + +n_cells_coarse = [c.*(12,8) for i in 0:n] +n_cells_global = map((nc,nl) -> nc .* (2^(nl-1),2^(nl-1)),n_cells_coarse,n_levels) + +dicts_odd_gmg = generate_dictionaries(n_procs,n_cells_coarse,n_levels) +dicts_odd_petsc = generate_dictionaries(n_procs,n_cells_global,n_levels) + +gmg = true +if gmg + template_file = "template_gmg.sh" + dicts = vcat(dicts_odd_gmg,dicts_even_gmg) +else + template_file = "template.sh" + dicts = vcat(dicts_odd_petsc,dicts_even_petsc) +end -template = read(projectdir("template.sh"),String) +template = read(projectdir(template_file),String) for params in dicts fparams = clean_params(params) jobfile = projectdir("jobs/",jobname(fparams,"sh")) diff --git a/joss_paper/scalability/src/Scalability.jl b/joss_paper/scalability/src/Scalability.jl index a47daff6..e45c3c0d 100644 --- a/joss_paper/scalability/src/Scalability.jl +++ b/joss_paper/scalability/src/Scalability.jl @@ -9,7 +9,8 @@ using GridapSolvers.LinearSolvers, GridapSolvers.MultilevelTools, GridapSolvers. include("utils.jl") include("stokes.jl") +include("stokes_gmg.jl") -export stokes_main +export stokes_main, stokes_gmg_main end # module \ No newline at end of file diff --git a/joss_paper/scalability/src/stokes.jl b/joss_paper/scalability/src/stokes.jl index dcbd8478..dc80146b 100644 --- a/joss_paper/scalability/src/stokes.jl +++ b/joss_paper/scalability/src/stokes.jl @@ -1,9 +1,8 @@ -function stokes_driver(parts,mh) +function stokes_driver(parts,model) t = PTimer(parts;verbose=true) tic!(t;barrier=true) - model = get_model(mh,1) # FE spaces fe_order = 2 @@ -11,9 +10,8 @@ function stokes_driver(parts,mh) reffe_u = ReferenceFE(lagrangian,VectorValue{2,Float64},fe_order) reffe_p = ReferenceFE(lagrangian,Float64,fe_order-1;space=:P) - tests_u = TestFESpace(mh,reffe_u,dirichlet_tags=["bottom","top"]) - trials_u = TrialFESpace(tests_u,[VectorValue(0.0,0.0),VectorValue(1.0,0.0)]) - U, V = get_fe_space(trials_u,1), get_fe_space(tests_u,1) + V = TestFESpace(model,reffe_u,dirichlet_tags=["bottom","top"]) + U = TrialFESpace(V,[VectorValue(0.0,0.0),VectorValue(1.0,0.0)]) Q = TestFESpace(model,reffe_p;conformity=:L2) mfs = Gridap.MultiField.BlockMultiFieldStyle() @@ -32,31 +30,15 @@ function stokes_driver(parts,mh) op = AffineFEOperator((u,v)->biform(u,v,dΩ),v->liform(v,dΩ),X,Y) A, b = get_matrix(op), get_vector(op) - # GMG preconditioner for the velocity block - biforms = map(mhl -> get_bilinear_form(mhl,biform_u,qdegree),mh) - smoothers = map(mhl -> RichardsonSmoother(JacobiLinearSolver(),10,2.0/3.0), view(mh,1:num_levels(mh)-1)) - restrictions, prolongations = setup_transfer_operators( - trials_u, qdegree; mode=:residual, solver=CGSolver(JacobiLinearSolver(),verbose=false) - ) - solver_u = GMGLinearSolver( - mh,trials_u,tests_u,biforms, - prolongations,restrictions, - pre_smoothers=smoothers, - post_smoothers=smoothers, - coarsest_solver=PETScLinearSolver(petsc_mumps_setup), - maxiter=3,mode=:preconditioner,verbose=i_am_main(parts) - ) - solver_u.log.depth = 4 - - # PCG solver for the pressure block - solver_p = CGSolver(JacobiLinearSolver();maxiter=20,atol=1e-14,rtol=1.e-6,verbose=i_am_main(parts)) + solver_u = ASMSolver() + solver_p = CGSolver(JacobiLinearSolver();maxiter=20,atol=1e-14,rtol=1.e-10,verbose=i_am_main(parts)) solver_p.log.depth = 4 # Block triangular preconditioner blocks = [LinearSystemBlock() LinearSystemBlock(); LinearSystemBlock() BiformBlock((p,q) -> ∫(p*q)dΩ,Q,Q)] P = BlockTriangularSolver(blocks,[solver_u,solver_p]) - solver = FGMRESSolver(10,P;rtol=1.e-8,verbose=i_am_main(parts)) + solver = FGMRESSolver(15,P;rtol=1.e-8,verbose=i_am_main(parts)) ns = numerical_setup(symbolic_setup(solver,A),A) toc!(t,"Setup") @@ -65,17 +47,6 @@ function stokes_driver(parts,mh) solve!(x,ns,b) toc!(t,"Solver") - # Cleanup PETSc C-allocated objects - nlevs = num_levels(mh) - GridapSolvers.MultilevelTools.with_level(mh,nlevs) do mhl - P_ns = ns.Pr_ns - gmg_ns = P_ns.block_ns[1] - finalize(gmg_ns.coarsest_solver_cache.X) - finalize(gmg_ns.coarsest_solver_cache.B) - finalize(gmg_ns.coarsest_solver_cache) - GridapPETSc.gridap_petsc_gc() - end - # Postprocess ncells = num_cells(model) ndofs_u = num_free_dofs(U) @@ -93,36 +64,34 @@ end function stokes_main(; nr = 1, - np = 1, - np_per_level = [np,1], + np = (1,1), nc = (10,10), petsc_options = "-ksp_monitor -ksp_error_if_not_converged true -ksp_converged_reason", - title = "data/stokes_np_$(np)_nc_$(prod(nc))" + title = "data/stokes_np_$(np)_nc_$(prod(nc))", ) parts = with_mpi() do distribute distribute(LinearIndices((prod(np),))) end - mh = CartesianModelHierarchy(parts,np_per_level,(0,1,0,1),nc;add_labels!) + model = CartesianDiscreteModel(parts,np,(0,1,0,1),nc) + map(add_labels!,local_views(get_face_labeling(model))) GridapPETSc.with(;args=split(petsc_options)) do - stokes_driver(parts,mh) + stokes_driver(parts,model) for ir in 1:nr map_main(parts) do p println(repeat('-',28)) println(" ------- ITERATION $(ir) ------- ") println(repeat('-',28)) end - output = stokes_driver(parts,mh) + output = stokes_driver(parts,model) map_main(parts) do p output["ir"] = ir output["np"] = np output["nc"] = nc - output["nl"] = length(np_per_level) - output["np_per_level"] = np_per_level save("$(title)_$(ir).bson",output) end GridapPETSc.gridap_petsc_gc() end end -end +end \ No newline at end of file diff --git a/joss_paper/scalability/src/stokes_gmg.jl b/joss_paper/scalability/src/stokes_gmg.jl new file mode 100644 index 00000000..d0296971 --- /dev/null +++ b/joss_paper/scalability/src/stokes_gmg.jl @@ -0,0 +1,145 @@ + +function stokes_gmg_driver(parts,mh) + t = PTimer(parts;verbose=true) + + tic!(t;barrier=true) + model = get_model(mh,1) + + # FE spaces + fe_order = 2 + qdegree = 2*(fe_order+1) + reffe_u = ReferenceFE(lagrangian,VectorValue{2,Float64},fe_order) + reffe_p = ReferenceFE(lagrangian,Float64,fe_order-1;space=:P) + + tests_u = TestFESpace(mh,reffe_u,dirichlet_tags=["bottom","top"]) + trials_u = TrialFESpace(tests_u,[VectorValue(0.0,0.0),VectorValue(1.0,0.0)]) + U, V = get_fe_space(trials_u,1), get_fe_space(tests_u,1) + Q = TestFESpace(model,reffe_p;conformity=:L2) + + mfs = Gridap.MultiField.BlockMultiFieldStyle() + X = MultiFieldFESpace([U,Q];style=mfs) + Y = MultiFieldFESpace([V,Q];style=mfs) + toc!(t,"FESpaces") + + # Weak formulation + tic!(t;barrier=true) + f = VectorValue(1.0,1.0) + biform_u(u,v,dΩ) = ∫(∇(v)⊙∇(u))dΩ + biform((u,p),(v,q),dΩ) = biform_u(u,v,dΩ) - ∫((∇⋅v)*p)dΩ - ∫((∇⋅u)*q)dΩ + liform((v,q),dΩ) = ∫(v⋅f)dΩ + + # Finest level + Ω = Triangulation(model) + dΩ = Measure(Ω,qdegree) + op = AffineFEOperator((u,v)->biform(u,v,dΩ),v->liform(v,dΩ),X,Y) + A, b = get_matrix(op), get_vector(op) + toc!(t,"Integration") + + tic!(t;barrier=true) + # GMG preconditioner for the velocity block + biforms = map(mhl -> get_bilinear_form(mhl,biform_u,qdegree),mh) + smoothers = map(mhl -> RichardsonSmoother(JacobiLinearSolver(),10,2.0/3.0), view(mh,1:num_levels(mh)-1)) + restrictions, prolongations = setup_transfer_operators( + trials_u, qdegree; mode=:residual, solver=CGSolver(JacobiLinearSolver(),verbose=false) + ) + solver_u = GMGLinearSolver( + mh,trials_u,tests_u,biforms, + prolongations,restrictions, + pre_smoothers=smoothers, + post_smoothers=smoothers, + coarsest_solver=PETScLinearSolver(petsc_mumps_setup), + maxiter=3,mode=:preconditioner,verbose=i_am_main(parts) + ) + solver_u.log.depth = 4 + + # PCG solver for the pressure block + solver_p = CGSolver(JacobiLinearSolver();maxiter=20,atol=1e-14,rtol=1.e-10,verbose=i_am_main(parts)) + solver_p.log.depth = 4 + + # Block triangular preconditioner + blocks = [LinearSystemBlock() LinearSystemBlock(); + LinearSystemBlock() BiformBlock((p,q) -> ∫(p*q)dΩ,Q,Q)] + P = BlockTriangularSolver(blocks,[solver_u,solver_p]) + solver = FGMRESSolver(15,P;rtol=1.e-8,maxiter=20,verbose=i_am_main(parts)) + ns = numerical_setup(symbolic_setup(solver,A),A) + toc!(t,"SolverSetup") + + tic!(t;barrier=true) + x = allocate_in_domain(A); fill!(x,0.0) + solve!(x,ns,b) + toc!(t,"Solver") + + # Cleanup PETSc C-allocated objects + nlevs = num_levels(mh) + GridapSolvers.MultilevelTools.with_level(mh,nlevs) do mhl + P_ns = ns.Pr_ns + gmg_ns = P_ns.block_ns[1].ns + finalize(gmg_ns.coarsest_solver_cache.X) + finalize(gmg_ns.coarsest_solver_cache.B) + finalize(gmg_ns.coarsest_solver_cache) + GridapPETSc.gridap_petsc_gc() + end + + # Postprocess + ncells = num_cells(model) + ndofs_u = num_free_dofs(U) + ndofs_p = num_free_dofs(Q) + output = Dict{String,Any}() + map_main(t.data) do timer_data + output["ncells"] = ncells + output["ndofs_u"] = ndofs_u + output["ndofs_p"] = ndofs_p + output["niter"] = solver.log.num_iters + merge!(output,timer_data) + end + return output +end + +function stokes_gmg_main(; + nr = 1, + np = (1,1), + np_per_level = [np,np], + nc = (10,10), + petsc_options = "-ksp_monitor -ksp_error_if_not_converged true -ksp_converged_reason", + title = "data/stokes_np_$(prod(np))_nc_$(prod(nc))", + mesher = :gridap, + mode = :mpi +) + @assert mesher ∈ [:gridap,:p4est] + @assert mode ∈ [:mpi,:debug] + with_mode = (mode == :mpi) ? with_mpi : with_debug + parts = with_mode() do distribute + distribute(LinearIndices((prod(np),))) + end + + t = PTimer(parts;verbose=true) + tic!(t;barrier=true) + if mesher == :gridap + mh = CartesianModelHierarchy(parts,np_per_level,(0,1,0,1),nc;add_labels!) + elseif mesher == :p4est + np_per_level = map(prod,np_per_level) + mh = P4estCartesianModelHierarchy(parts,np_per_level,(0,1,0,1),nc;add_labels!) + end + toc!(t,"Geometry") + + GridapPETSc.with(;args=split(petsc_options)) do + stokes_gmg_driver(parts,mh) + for ir in 1:nr + map_main(parts) do p + println(repeat('-',28)) + println(" ------- ITERATION $(ir) ------- ") + println(repeat('-',28)) + end + output = stokes_gmg_driver(parts,mh) + map_main(parts) do p + output["ir"] = ir + output["np"] = np + output["nc"] = nc + output["nl"] = length(np_per_level) + output["np_per_level"] = np_per_level + save("$(title)_$(ir).bson",output) + end + GridapPETSc.gridap_petsc_gc() + end + end +end diff --git a/joss_paper/scalability/src/utils.jl b/joss_paper/scalability/src/utils.jl index 8a66ee8d..40dc5ef2 100644 --- a/joss_paper/scalability/src/utils.jl +++ b/joss_paper/scalability/src/utils.jl @@ -11,6 +11,8 @@ function add_labels!(labels) add_tag_from_tags!(labels,"bottom",[1,2,5]) end +MUMPSSolver() = PETScLinearSolver(petsc_mumps_setup) + function petsc_mumps_setup(ksp) pc = Ref{GridapPETSc.PETSC.PC}() mumpsmat = Ref{GridapPETSc.PETSC.Mat}() @@ -20,9 +22,41 @@ function petsc_mumps_setup(ksp) @check_error_code GridapPETSc.PETSC.PCFactorSetMatSolverType(pc[],GridapPETSc.PETSC.MATSOLVERMUMPS) @check_error_code GridapPETSc.PETSC.PCFactorSetUpMatSolverType(pc[]) @check_error_code GridapPETSc.PETSC.PCFactorGetMatrix(pc[],mumpsmat) - #@check_error_code GridapPETSc.PETSC.MatMumpsSetIcntl(mumpsmat[], 4, 1) - @check_error_code GridapPETSc.PETSC.MatMumpsSetIcntl(mumpsmat[], 7, 0) -# @check_error_code GridapPETSc.PETSC.MatMumpsSetIcntl(mumpsmat[], 28, 2) -# @check_error_code GridapPETSc.PETSC.MatMumpsSetIcntl(mumpsmat[], 29, 2) + @check_error_code GridapPETSc.PETSC.MatMumpsSetIcntl(mumpsmat[], 4, 1) + @check_error_code GridapPETSc.PETSC.MatMumpsSetIcntl(mumpsmat[], 28, 2) + @check_error_code GridapPETSc.PETSC.MatMumpsSetIcntl(mumpsmat[], 29, 1) + @check_error_code GridapPETSc.PETSC.MatMumpsSetCntl(mumpsmat[], 1, 0.00001) + @check_error_code GridapPETSc.PETSC.KSPView(ksp[],C_NULL) +end + +ASMSolver() = PETScLinearSolver(petsc_asm_setup) + +function petsc_asm_setup(ksp) + rtol = PetscScalar(1.e-9) + atol = GridapPETSc.PETSC.PETSC_DEFAULT + dtol = GridapPETSc.PETSC.PETSC_DEFAULT + maxits = GridapPETSc.PETSC.PETSC_DEFAULT + + pc = Ref{GridapPETSc.PETSC.PC}() + @check_error_code GridapPETSc.PETSC.KSPSetType(ksp[],GridapPETSc.PETSC.KSPGMRES) + @check_error_code GridapPETSc.PETSC.KSPSetTolerances(ksp[], rtol, atol, dtol, maxits) + @check_error_code GridapPETSc.PETSC.KSPGetPC(ksp[],pc) + @check_error_code GridapPETSc.PETSC.PCSetType(pc[],GridapPETSc.PETSC.PCASM) + @check_error_code GridapPETSc.PETSC.KSPView(ksp[],C_NULL) +end + +AMGSolver() = PETScLinearSolver(petsc_amg_setup) + +function petsc_amg_setup(ksp) + rtol = PetscScalar(1.e-6) + atol = GridapPETSc.PETSC.PETSC_DEFAULT + dtol = GridapPETSc.PETSC.PETSC_DEFAULT + maxits = GridapPETSc.PETSC.PETSC_DEFAULT + + pc = Ref{GridapPETSc.PETSC.PC}() + @check_error_code GridapPETSc.PETSC.KSPSetType(ksp[],GridapPETSc.PETSC.KSPRICHARDSON) + @check_error_code GridapPETSc.PETSC.KSPGetPC(ksp[],pc) + @check_error_code GridapPETSc.PETSC.PCSetType(pc[],GridapPETSc.PETSC.PCGAMG) + @check_error_code GridapPETSc.PETSC.KSPSetTolerances(ksp[], rtol, atol, dtol, maxits) @check_error_code GridapPETSc.PETSC.KSPView(ksp[],C_NULL) end diff --git a/joss_paper/scalability/template.sh b/joss_paper/scalability/template.sh index 06b3427f..260fae12 100644 --- a/joss_paper/scalability/template.sh +++ b/joss_paper/scalability/template.sh @@ -11,21 +11,20 @@ source {{{modules}}} -julia --project={{{projectdir}}} -O3 -J{{{sysimage}}} -e\ +julia --project={{{projectdir}}} -O3 -e\ ' using Scalability using Gridap, GridapDistributed, PartitionedArrays, GridapSolvers, GridapPETSc using FileIO, BSON ' -mpiexec -n {{ncpus}} julia --project={{{projectdir}}} -O3 -J{{{sysimage}}} -e\ +mpiexec -n {{ncpus}} julia --project={{{projectdir}}} -O3 -e\ ' using Scalability; stokes_main(; nr={{nr}}, np={{np}}, nc={{nc}}, - np_per_level={{np_per_level}}, title="{{{title}}}", ) ' diff --git a/joss_paper/scalability/template_gmg.sh b/joss_paper/scalability/template_gmg.sh new file mode 100644 index 00000000..797983b3 --- /dev/null +++ b/joss_paper/scalability/template_gmg.sh @@ -0,0 +1,35 @@ +#!/bin/bash +#PBS -P np01 +#PBS -q {{q}} +#PBS -l walltime={{walltime}} +#PBS -l ncpus={{ncpus}} +#PBS -l mem={{mem}} +#PBS -N {{{name}}} +#PBS -l wd +#PBS -o {{{o}}} +#PBS -e {{{e}}} + +source {{{modules}}} + +julia --project={{{projectdir}}} -O3 -J{{{sysimage}}} -e\ + ' + @time "Preloading libraries (serial)" begin + using Scalability; + using Gridap, GridapDistributed, PartitionedArrays, GridapSolvers, GridapPETSc; + using FileIO, BSON; + end + ' + +mpiexec -n {{ncpus}} julia --project={{{projectdir}}} -O3 -J{{{sysimage}}} -e\ + ' + @time "Loading libraries (MPI)" begin + using Scalability; + end + stokes_gmg_main(; + nr={{nr}}, + np={{np}}, + nc={{nc}}, + np_per_level={{np_per_level}}, + title="{{{title}}}", + ) + ' From ecdb14dbf6cf4b8e1c1552bf366d57face8163f8 Mon Sep 17 00:00:00 2001 From: JordiManyer Date: Thu, 25 Jul 2024 17:10:29 +1000 Subject: [PATCH 15/30] Minor --- joss_paper/scalability/preparejobs.jl | 6 +++--- joss_paper/scalability/src/stokes_gmg.jl | 2 +- joss_paper/scalability/src/utils.jl | 4 ++-- joss_paper/scalability/template_gmg.sh | 14 ++++++++++++++ 4 files changed, 20 insertions(+), 6 deletions(-) diff --git a/joss_paper/scalability/preparejobs.jl b/joss_paper/scalability/preparejobs.jl index a94a592b..caebfec4 100644 --- a/joss_paper/scalability/preparejobs.jl +++ b/joss_paper/scalability/preparejobs.jl @@ -27,9 +27,9 @@ end function get_np_per_level(np,nl) a = fill(np,nl) - if prod(np) > 768 - a[end] = (24,32) # 768 - end + #if prod(np) > 768 + # a[end] = (24,32) # 768 + #end return a end diff --git a/joss_paper/scalability/src/stokes_gmg.jl b/joss_paper/scalability/src/stokes_gmg.jl index d0296971..b0a579ae 100644 --- a/joss_paper/scalability/src/stokes_gmg.jl +++ b/joss_paper/scalability/src/stokes_gmg.jl @@ -47,7 +47,7 @@ function stokes_gmg_driver(parts,mh) prolongations,restrictions, pre_smoothers=smoothers, post_smoothers=smoothers, - coarsest_solver=PETScLinearSolver(petsc_mumps_setup), + coarsest_solver=PETScLinearSolver(), maxiter=3,mode=:preconditioner,verbose=i_am_main(parts) ) solver_u.log.depth = 4 diff --git a/joss_paper/scalability/src/utils.jl b/joss_paper/scalability/src/utils.jl index 40dc5ef2..87c4b3fb 100644 --- a/joss_paper/scalability/src/utils.jl +++ b/joss_paper/scalability/src/utils.jl @@ -38,7 +38,7 @@ function petsc_asm_setup(ksp) maxits = GridapPETSc.PETSC.PETSC_DEFAULT pc = Ref{GridapPETSc.PETSC.PC}() - @check_error_code GridapPETSc.PETSC.KSPSetType(ksp[],GridapPETSc.PETSC.KSPGMRES) + @check_error_code GridapPETSc.PETSC.KSPSetType(ksp[],GridapPETSc.PETSC.KSPCG) @check_error_code GridapPETSc.PETSC.KSPSetTolerances(ksp[], rtol, atol, dtol, maxits) @check_error_code GridapPETSc.PETSC.KSPGetPC(ksp[],pc) @check_error_code GridapPETSc.PETSC.PCSetType(pc[],GridapPETSc.PETSC.PCASM) @@ -54,7 +54,7 @@ function petsc_amg_setup(ksp) maxits = GridapPETSc.PETSC.PETSC_DEFAULT pc = Ref{GridapPETSc.PETSC.PC}() - @check_error_code GridapPETSc.PETSC.KSPSetType(ksp[],GridapPETSc.PETSC.KSPRICHARDSON) + @check_error_code GridapPETSc.PETSC.KSPSetType(ksp[],GridapPETSc.PETSC.KSPCG) @check_error_code GridapPETSc.PETSC.KSPGetPC(ksp[],pc) @check_error_code GridapPETSc.PETSC.PCSetType(pc[],GridapPETSc.PETSC.PCGAMG) @check_error_code GridapPETSc.PETSC.KSPSetTolerances(ksp[], rtol, atol, dtol, maxits) diff --git a/joss_paper/scalability/template_gmg.sh b/joss_paper/scalability/template_gmg.sh index 797983b3..32a87427 100644 --- a/joss_paper/scalability/template_gmg.sh +++ b/joss_paper/scalability/template_gmg.sh @@ -25,11 +25,25 @@ mpiexec -n {{ncpus}} julia --project={{{projectdir}}} -O3 -J{{{sysimage}}} -e\ @time "Loading libraries (MPI)" begin using Scalability; end + + petsc_options = """ + -ksp_type cg + -ksp_rtol 1.0e-5 + -ksp_atol 1.0e-14 + -ksp_converged_reason + -pc_type asm + -pc_asm_overlap 10 + -pc_asm_type restrict + -sub_ksp_type preonly + -sub_pc_type lu + """ + stokes_gmg_main(; nr={{nr}}, np={{np}}, nc={{nc}}, np_per_level={{np_per_level}}, + petsc_options=petsc_options, title="{{{title}}}", ) ' From ae2d28046c7008244024efc07fdef774101d435a Mon Sep 17 00:00:00 2001 From: JordiManyer Date: Tue, 30 Jul 2024 15:10:07 +1000 Subject: [PATCH 16/30] more changes --- joss_paper/.gitignore | 3 + joss_paper/demo.jl | 22 ++-- joss_paper/demo.pvtu | 20 ++++ joss_paper/paper.bib | 94 +++++++++++++++- joss_paper/paper.md | 156 ++++++++++++++++++++++---- joss_paper/scalability/postprocess.jl | 27 ++++- joss_paper/weakScalability.png | Bin 0 -> 18538 bytes 7 files changed, 287 insertions(+), 35 deletions(-) create mode 100644 joss_paper/.gitignore create mode 100755 joss_paper/demo.pvtu create mode 100644 joss_paper/weakScalability.png diff --git a/joss_paper/.gitignore b/joss_paper/.gitignore new file mode 100644 index 00000000..e6ec4d31 --- /dev/null +++ b/joss_paper/.gitignore @@ -0,0 +1,3 @@ +paper.crossref +paper.pdf +demo diff --git a/joss_paper/demo.jl b/joss_paper/demo.jl index 76d88aea..cbc14783 100644 --- a/joss_paper/demo.jl +++ b/joss_paper/demo.jl @@ -1,6 +1,9 @@ using Gridap, Gridap.Algebra, Gridap.MultiField using PartitionedArrays, GridapDistributed, GridapSolvers -using GridapSolvers.LinearSolvers, GridapSolvers.MultilevelTools, GridapSolvers.BlockSolvers + +using GridapSolvers.LinearSolvers +using GridapSolvers.MultilevelTools +using GridapSolvers.BlockSolvers function get_bilinear_form(mh_lev,biform,qdegree) model = get_model(mh_lev) @@ -14,8 +17,8 @@ function add_labels!(labels) add_tag_from_tags!(labels,"bottom",[1,2,5]) end -np = 4 -np_per_level = [np,1] +np = (2,2) +np_per_level = [np,(1,1)] nc = (10,10) fe_order = 2 @@ -53,8 +56,12 @@ with_mpi() do distribute A, b = get_matrix(op), get_vector(op) # GMG preconditioner for the velocity block - biforms = map(mhl -> get_bilinear_form(mhl,biform_u,qdegree),mh) - smoothers = map(mhl -> RichardsonSmoother(JacobiLinearSolver(),10,2.0/3.0), view(mh,1:num_levels(mh)-1)) + biforms = map(mh) do mhl + get_bilinear_form(mhl,biform_u,qdegree) + end + smoothers = map(view(mh,1:num_levels(mh)-1)) do mhl + RichardsonSmoother(JacobiLinearSolver(),10,2.0/3.0) + end restrictions, prolongations = setup_transfer_operators( trials_u, qdegree; mode=:residual, solver=CGSolver(JacobiLinearSolver()) ) @@ -77,8 +84,9 @@ with_mpi() do distribute solver = GMRESSolver(10;Pr=P,rtol=1.e-8,verbose=i_am_main(parts)) ns = numerical_setup(symbolic_setup(solver,A),A) - x = allocate_in_domain(A); fill!(x,0.0) + x = allocate_in_domain(A) + fill!(x,0.0) solve!(x,ns,b) uh, ph = FEFunction(X,x) - writevtk(Ω,"joss_paper/demo",cellfields=["uh"=>uh,"ph"=>ph]) + writevtk(Ω,"demo",cellfields=["uh"=>uh,"ph"=>ph]) end \ No newline at end of file diff --git a/joss_paper/demo.pvtu b/joss_paper/demo.pvtu new file mode 100755 index 00000000..6c0ea990 --- /dev/null +++ b/joss_paper/demo.pvtu @@ -0,0 +1,20 @@ + + + + + + + + + + + + + + + + + + + + diff --git a/joss_paper/paper.bib b/joss_paper/paper.bib index ef63c707..631c3931 100644 --- a/joss_paper/paper.bib +++ b/joss_paper/paper.bib @@ -20,7 +20,7 @@ @manual{mpi40 month = jun } -@article{Verdugo:2021, +@article{Verdugo2021, doi = {10.1016/j.cpc.2022.108341}, url = {https://doi.org/10.1016/j.cpc.2022.108341}, year = {2022}, @@ -28,7 +28,7 @@ @article{Verdugo:2021 publisher = {Elsevier {BV}}, volume = {276}, pages = {108341}, - author = {Francesc Verdugo and Santiago Badia}, + author = {Verdugo, F. and Badia, S.}, title = {The software design of {G}ridap: a finite element package based on the {J}ulia {JIT} compiler}, journal = {Computer Physics Communications} } @@ -206,3 +206,93 @@ @book{fenics-book editor = {Anders Logg and Kent-Andre Mardal and Garth Wells}, title = {Automated Solution of Differential Equations by the Finite Element Method} } + +@book{Elman2014, + author = {Elman, Howard and Silvester, David and Wathen, Andy}, + title = "{Finite Elements and Fast Iterative Solvers: with Applications in Incompressible Fluid Dynamics}", + publisher = {Oxford University Press}, + year = {2014}, + month = {06}, + abstract = "{The subject of this book is the efficient solution of partial differential equations (PDEs) that arise when modelling incompressible fluid flow. The first part (Chapters 1 through 5) covers the Poisson equation and the Stokes equations. For each PDE, there is a chapter concerned with finite element discretization and a companion chapter concerned with efficient iterative solution of the algebraic equations obtained from discretization. Chapter 5 describes the basics of PDE-constrained optimization. The second part of the book (Chapters 6 to 11) is a more advanced introduction to the numerical analysis of incompressible flows. It starts with four chapters on the convection–diffusion equation and the steady Navier–Stokes equations, organized by equation with a chapter describing discretization coupled with a companion concerned with iterative solution algorithms. The book concludes with two chapters describing discretization and solution methods for models of unsteady flow and buoyancy-driven flow.}", + isbn = {9780199678792}, + doi = {10.1093/acprof:oso/9780199678792.001.0001}, + url = {https://doi.org/10.1093/acprof:oso/9780199678792.001.0001}, +} + +@Manual{trilinos, + title = {The {T}rilinos {P}roject {W}ebsite}, + author = {The {T}rilinos {P}roject {T}eam}, + year = {2020}, + url = {https://trilinos.github.io} +} + +@misc{hypre, + key = {hypre}, + title = {{\sl hypre}: High Performance Preconditioners}, + note = {\url{https://llnl.gov/casc/hypre}, \url{https://github.com/hypre-space/hypre}} +} + +@Inbook{PARDISO, + author="Schenk, Olaf and G{\"a}rtner, Klaus", + editor="Padua, David", + title="PARDISO", + bookTitle="Encyclopedia of Parallel Computing", + year="2011", + publisher="Springer US", + address="Boston, MA", + pages="1458--1464", + isbn="978-0-387-09766-4", + doi="10.1007/978-0-387-09766-4_90", + url="https://doi.org/10.1007/978-0-387-09766-4_90" +} + +@article{MUMPS1, + title = {A Fully Asynchronous Multifrontal Solver Using Distributed Dynamic Scheduling}, + author = {P.R. Amestoy and I. S. Duff and J. Koster and J.-Y. L'Excellent}, + journal = {SIAM Journal on Matrix Analysis and Applications}, + volume = {23}, + number = {1}, + year = {2001}, + pages = {15-41} + } + +@article{MUMPS2, + title = {{Performance and Scalability of the Block Low-Rank Multifrontal + Factorization on Multicore Architectures}}, + author = {P.R. Amestoy and A. Buttari and J.-Y. L'Excellent and T. Mary}, + journal = {ACM Transactions on Mathematical Software}, + volume = 45, + issue = 1, + pages = {2:1--2:26}, + year={2019}, +} + +@article{gridapdistributed, + doi = {10.21105/joss.04157}, + url = {https://doi.org/10.21105/joss.04157}, + year = {2022}, + publisher = {The Open Journal}, + volume = {7}, + number = {74}, + pages = {4157}, + author = {Santiago Badia and Alberto F. Martín and Francesc Verdugo}, + title = {GridapDistributed: a massively parallel finite element toolbox in Julia}, + journal = {Journal of Open Source Software} +} + +@inproceedings{Arnold1, + title={Preconditing in H(div) and applications}, + author={D. N. Arnold and Richard S. Falk and Ragnar Winther}, + year={1997}, + url={https://api.semanticscholar.org/CorpusID:12559456} +} + +@article{Arnold2, + title={Multigrid in H (div) and H (curl)}, + author={Douglas N. Arnold and Richard S. Falk and Ragnar Winther}, + journal={Numerische Mathematik}, + year={2000}, + volume={85}, + pages={197-217}, + url={https://api.semanticscholar.org/CorpusID:14301688} +} \ No newline at end of file diff --git a/joss_paper/paper.md b/joss_paper/paper.md index 5d2addd8..8e824552 100644 --- a/joss_paper/paper.md +++ b/joss_paper/paper.md @@ -12,69 +12,185 @@ authors: corresponding: true equal-contrib: true affiliation: "1" - - name: Alberto Martín-Huertas + - name: Alberto F. Martín orcid: 0000-0001-5751-4561 equal-contrib: true affiliation: "2" - name: Santiago Badia orcid: 0000-0003-2391-4086 - affiliation: "1,3" + affiliation: "1" affiliations: - name: School of Mathematics, Monash University, Clayton, Victoria, 3800, Australia. index: 1 - name: School of Computing, Australian National University, Autonomous territories of Canberra, Australia index: 2 - - name: Centre Internacional de Mètodes Numèrics en Enginyeria, Esteve Terrades 5, E-08860 Castelldefels, Spain. - index: 3 -date: XXX April 2024 +date: 1 August 2024 bibliography: paper.bib aas-journal: Journal of Open Source Software --- -# Summary and statement of need +## Summary and statement of need The ever-increasing demand for resolution and accuracy in mathematical models of physical processes governed by systems of Partial Differential Equations (PDEs) can only be addressed using fully-parallel advanced numerical discretization methods and scalable solution methods, thus able to exploit the vast amount of computational resources in state-of-the-art supercomputers. One of the biggest scalability bottlenecks within Finite Element (FE) parallel codes is the solution of linear systems arising from the discretization of PDEs. -The implementation of exact factorization-based solvers in parallel environments is an extremely challenging task, and even state-of-the-art libraries such as MUMPS [@MUMPS] or PARDISO [@PARDISO] have severe limitations in terms of scalability and memory consumption above a certain number of CPU cores. +The implementation of exact factorization-based solvers in parallel environments is an extremely challenging task, and even state-of-the-art libraries such as MUMPS [@MUMPS1] [@MUMPS2] or PARDISO [@PARDISO] have severe limitations in terms of scalability and memory consumption above a certain number of CPU cores. Hence the use of iterative methods is crucial to maintain scalability of FE codes. Unfortunately, the convergence of iterative methods is not guaranteed and rapidly deteriorates as the size of the linear system increases. To retain performance, the use of highly scalable preconditioners is mandatory. -For most problems, algebraic solvers and preconditioners (i.e based uniquelly on the algebraic system) are enough to obtain robust convergence. Many well-known libraries providing algebraic solvers already exist, such as PETSc [@petsc-user-ref], Trilinos [@trilinos], or Hypre [@hypre]. However, algebraic solvers are not always suited to deal with some of the most challenging problems. -In these cases, geometric solvers (i.e., solvers that exploit the geometry and physics of the particular problem) are required. This is the case of many multiphysics problems, such as Navier-Stokes, Darcy or MHD. To this end, GridapSolvers is a registered Julia [@Bezanson2017] software package which provides highly scalable geometric solvers tailored for the FE numerical solution of PDEs on parallel computers. +For simple problems, algebraic solvers and preconditioners (i.e based uniquelly on the algebraic system) are enough to obtain robust convergence. Many well-known libraries providing algebraic solvers already exist, such as PETSc [@petsc-user-ref], Trilinos [@trilinos], or Hypre [@hypre]. However, algebraic solvers are not always suited to deal with more challenging problems. + +In these cases, solvers that exploit the geometry and physics of the particular problem are required. This is the case of many multiphysics problems involving differential operators with a large kernel such as the divergence [@arnold1] and the curl [@arnold2]. Examples of this are highly relevant problems such as Navier-Stokes, Maxwell or Darcy. Scalable solvers for this type of multiphysics problems rely on exploiting the block structure of such systems to find a spectrally equivalent block-preconditioner, and are often tied to specific discretizations of the underlying equations. +To this end, GridapSolvers is a registered Julia [@Bezanson2017] software package which provides highly scalable geometric solvers tailored for the FE numerical solution of PDEs on parallel computers. Emphasis is put on the modular design of the library, which easily allows new preconditioners to be designed from the user's specific problem. -# Building blocks and composability +## Building blocks and composability \autoref{fig:packages} depicts the relation among GridapDistributed and other packages in the Julia package ecosystem. -The core library Gridap provides all necessary abstraction and interfaces needed for the FE solution of PDEs (see @Verdugo:2021) for serial computing. GridapDistributed provides distributed-memory counterparts for these abstractions, while leveraging the serial implementations in Gridap to handle the local portion on each parallel task. GridapDistributed relies on PartitionedArrays [@parrays] in order to handle the parallel execution model (e.g., message-passing via the Message Passing Interface (MPI) [@mpi40]), global data distribution layout, and communication among tasks. PartitionedArrays also provides a parallel implementation of partitioned global linear systems (i.e., linear algebra vectors and sparse matrices) as needed in grid-based numerical simulations. +The core library Gridap [@Badia2020] provides all necessary abstraction and interfaces needed for the FE solution of PDEs [@Verdugo2021] for serial computing. GridapDistributed [@gridapdistributed] provides distributed-memory counterparts for these abstractions, while leveraging the serial implementations in Gridap to handle the local portion on each parallel task. GridapDistributed relies on PartitionedArrays [@parrays] in order to handle the parallel execution model (e.g., message-passing via the Message Passing Interface (MPI) [@mpi40]), global data distribution layout, and communication among tasks. PartitionedArrays also provides a parallel implementation of partitioned global linear systems (i.e., linear algebra vectors and sparse matrices) as needed in grid-based numerical simulations. This parallel framework does however not include any performant solver for the resulting linear systems. This was delegated to GridapPETSc, which provides a plethora of highly-scalable and efficient algebraic solvers through a high-level interface to the Portable, Extensible Toolkit for Scientific Computation (PETSc) [@petsc-user-ref]. GridapSolvers complements GridapPETSc with a modular and extensible interface for the design of geometric solvers. Some of the highlights of the library are: - A set of HPC-first implementations for popular Krylov-based iterative solvers. These solvers extend Gridap's API and are fully compatible with PartitionedArrays. - A modular, high-level interface for designing block-based preconditioners for multiphysics problems. These preconditioners can be used together with any solver compliant with Gridap's API, including those provided by GridapPETSc. -- A generic interface to handle multi-level distributed meshes, with full support for Adaptative Mesh Refinement (AMR) using p4est [p4est] through GridapP4est. +- A generic interface to handle multi-level distributed meshes, with full support for Adaptative Mesh Refinement (AMR) using p4est [@p4est] through GridapP4est. - A modular implementation of geometric multigrid (GMG) solvers, allowing different types of smoothers and restriction/prolongation operators. - A generic interface for patch-based subdomain decomposition methods, and an implementation of patch-based smoothers for geometric multigrid solvers. ![GridapSolvers and its relation to other packages in the Julia package ecosystem. In this diagram, each node represents a Julia package, while the (directed) arrows represent relations (dependencies) among packages. Dashed arrows mean the package can be used, but is not required. \label{fig:packages}](packages.png){ width=60% } -# Demo +## Demo + +The following code snippet shows how to solve a 2D incompressible Stokes cavity problem in a cartesian domain $\Omega = [0,1]^2$. We discretize the velocity and pressure in $H^1(\Omega)$ and $L^2(\Omega)$ respectively, and use the well known stable element pair $Q_k \times P_{k-1}^{-}$ with $k=2$. For the cavity problem, we fix the velocity to $u_t = \hat{x}$ on the top boundary, and homogeneous Dirichlet boundary conditions elsewhere. We impose a zero-mean pressure constraint to have a solvable system of equations. Given discrete spaces $V \times Q$, we find $(u,p) \in V \times Q$ such that + +$$ + \int_{\Omega} \nabla v : \nabla u - (\nabla \cdot v) p - (\nabla \cdot u) q = \int_{\Omega} v \cdot f \quad \forall v \in V_0, q \in Q +$$ + +where $V_0$ is the space of velocity functions with homogeneous boundary conditions everywhere. -The following code snippet shows how to solve a 2D Stokes cavity problem in a cartesian domain $\Omega = [0,1]^2$. We discretize the velocity and pressure in $H^1(\Omega)$ and $L^2(\Omega)$ respectively, and use the well known stable element pair $Q_k \times P_{k-1}$ with $k=2$. For the cavity problem, we fix the velocity to $u_b = \vec{0}$ and $u_t = \hat{x}$ on the bottom and top boundaries respectively, and homogeneous Neumann boundary conditions elsewhere. -The system is block-assembled and solved using a GMRES solver, right-preconditioned with block-triangular Shur-complement-based preconditioner. The Shur complement is approximated by a mass matrix, and solved using a CG solver with Jacobi preconditioner. The eliminated velocity block is approximated by a 2-level V-cycle Geometric Multigrid solver. The coarsest-level system is solved exactly using MUMPS [@MUMPS], provided by PETSc [@petsc-user-ref] though the package GridapPETSc.jl. +The system is block-assembled and solved using a GMRES solver, together with a block-triangular Shur-complement-based preconditioner. We eliminate the velocity block and approximate the resulting Shur complement by a pressure mass matrix. A more detailed overview of this preconditioner as well as it's spectral analysis can be found in [@Elman2014]. The resulting block structure for the system matrix $\mathcal{A}$ and our preconditioner $\mathcal{P}$ is + +$$ +\mathcal{A} = \begin{bmatrix} + A & B^T \\ + B & 0 +\end{bmatrix} +,\quad +\mathcal{P} = \begin{bmatrix} + A & B^T \\ + 0 & -M +\end{bmatrix} +$$ + +with $A$ the velocity laplacian block, and $M$ a pressure mass matrix. + +The mass matrix is approximated by a CG solver with Jacobi preconditioner. The eliminated velocity block is approximated by a 2-level V-cycle Geometric Multigrid solver. The coarsest-level system is solved exactly using a Conjugate Gradient solver preconditioned by an Additive Schwarz solver, provided by PETSc [@petsc-user-ref] through the package GridapPETSc.jl. The code is setup to run in parallel with 4 MPI tasks and can be executed with the following command: `mpiexec -n 4 julia --project=. demo.jl`. ```julia -Code in `demo.jl`. +using Gridap, Gridap.Algebra, Gridap.MultiField +using PartitionedArrays, GridapDistributed, GridapSolvers + +using GridapSolvers.LinearSolvers +using GridapSolvers.MultilevelTools +using GridapSolvers.BlockSolvers + +function get_bilinear_form(mh_lev,biform,qdegree) + model = get_model(mh_lev) + Ω = Triangulation(model) + dΩ = Measure(Ω,qdegree) + return (u,v) -> biform(u,v,dΩ) +end + +function add_labels!(labels) + add_tag_from_tags!(labels,"top",[3,4,6]) + add_tag_from_tags!(labels,"bottom",[1,2,5]) +end + +np = (2,2) +np_per_level = [np,(1,1)] +nc = (10,10) +fe_order = 2 + +with_mpi() do distribute + parts = distribute(LinearIndices((prod(np),))) + + # Geometry + mh = CartesianModelHierarchy(parts,np_per_level,(0,1,0,1),nc;add_labels!) + model = get_model(mh,1) + + # FE spaces + qdegree = 2*(fe_order+1) + reffe_u = ReferenceFE(lagrangian,VectorValue{2,Float64},fe_order) + reffe_p = ReferenceFE(lagrangian,Float64,fe_order-1;space=:P) + + tests_u = TestFESpace(mh,reffe_u,dirichlet_tags=["bottom","top"]) + trials_u = TrialFESpace(tests_u,[VectorValue(0.0,0.0),VectorValue(1.0,0.0)]) + U, V = get_fe_space(trials_u,1), get_fe_space(tests_u,1) + Q = TestFESpace(model,reffe_p;conformity=:L2) + + mfs = Gridap.MultiField.BlockMultiFieldStyle() + X = MultiFieldFESpace([U,Q];style=mfs) + Y = MultiFieldFESpace([V,Q];style=mfs) + + # Weak formulation + f = VectorValue(1.0,1.0) + biform_u(u,v,dΩ) = ∫(∇(v)⊙∇(u))dΩ + biform((u,p),(v,q),dΩ) = biform_u(u,v,dΩ) - ∫((∇⋅v)*p)dΩ - ∫((∇⋅u)*q)dΩ + liform((v,q),dΩ) = ∫(v⋅f)dΩ + + # Finest level + Ω = Triangulation(model) + dΩ = Measure(Ω,qdegree) + op = AffineFEOperator((u,v)->biform(u,v,dΩ),v->liform(v,dΩ),X,Y) + A, b = get_matrix(op), get_vector(op) + + # GMG preconditioner for the velocity block + biforms = map(mh) do mhl + get_bilinear_form(mhl,biform_u,qdegree) + end + smoothers = map(view(mh,1:num_levels(mh)-1)) do mhl + RichardsonSmoother(JacobiLinearSolver(),10,2.0/3.0) + end + restrictions, prolongations = setup_transfer_operators( + trials_u, qdegree; mode=:residual, solver=CGSolver(JacobiLinearSolver()) + ) + solver_u = GMGLinearSolver( + mh,trials_u,tests_u,biforms, + prolongations,restrictions, + pre_smoothers=smoothers, + post_smoothers=smoothers, + coarsest_solver=LUSolver(), + maxiter=2,mode=:preconditioner + ) + + # PCG solver for the pressure block + solver_p = CGSolver(JacobiLinearSolver();maxiter=20,atol=1e-14,rtol=1.e-6) + + # Block triangular preconditioner + blocks = [LinearSystemBlock() LinearSystemBlock(); + LinearSystemBlock() BiformBlock((p,q) -> ∫(p*q)dΩ,Q,Q)] + P = BlockTriangularSolver(blocks,[solver_u,solver_p]) + solver = GMRESSolver(10;Pr=P,rtol=1.e-8,verbose=i_am_main(parts)) + ns = numerical_setup(symbolic_setup(solver,A),A) + + x = allocate_in_domain(A) + fill!(x,0.0) + solve!(x,ns,b) + uh, ph = FEFunction(X,x) + writevtk(Ω,"joss_paper/demo",cellfields=["uh"=>uh,"ph"=>ph]) +end ``` -# Parallel scaling benchmark +## Parallel scaling benchmark + +The following section shows scalability results for the demo problem discussed above. We run our code on the Gadi supercomputer, which is part of the Australian National Computational Infrastructure. We use Intel's Cascade Lake 2x24-core Xeon Platinum 8274 nodes. Scalability is shown for up to 64 nodes, for a fixed local problem size of 48x64 quadrangle cells per processor. This amounts to a maximum size of 37M cells and 415M degrees of freedom distributed amongst 3072 processors. The code used to create these results can be found together with the submited paper (LINK). -The following section shows scalability results for the demo problem discussed above. We run our code on the Gadi supercomputer, which is part of the Australian National Computational Infrastructure. We use Intel's Cascade Lake 2x24-core Xeon Platinum 8274 nodes. Scalability is shown for up to XXX cores, for a fixed local problem size of XXX quadrangle cells per processor. This amounts to a maximum size of XXX cells and XXX degrees of freedom distributed amongst XXX processors. The code used to create these results can be found together with the submiteed paper (LINK). +![Weak scalability for a Stokes problem in 2D. Time is given per Conjugate Gradient iteration, as a function of the number of processors. \label{fig:packages}](weakScalability.png){ width=60% } -# Acknowledgements +## Acknowledgements This research was partially funded by the Australian Government through the Australian Research Council (project number DP210103092). This work was also supported by computational resources provided by the Australian Government through NCI under the National Computational Merit Allocation Scheme (NCMAS). -# References +## References diff --git a/joss_paper/scalability/postprocess.jl b/joss_paper/scalability/postprocess.jl index 70244611..4c670e45 100644 --- a/joss_paper/scalability/postprocess.jl +++ b/joss_paper/scalability/postprocess.jl @@ -16,35 +16,50 @@ end # ╔═╡ 4725bb5d-8973-436b-af96-74cc90e7f354 begin - raw = collect_results(datadir()) + raw = collect_results(datadir("run_asm")) dfgr = groupby(raw,[:np]) df = combine(dfgr) do df return ( t_solver = minimum(map(x->x[:max],df.Solver)), - t_setup = minimum(map(x->x[:max],df.Setup)), + t_setup = minimum(map(x->x[:max],df.SolverSetup)), n_iter = df.niter[1], n_levels = length(df.np_per_level[1]), n_dofs_u = df.ndofs_u[1], n_dofs_p = df.ndofs_p[1], n_dofs = df.ndofs_u[1] + df.ndofs_p[1], - n_cells = df.ncells[1] + n_cells = df.ncells[1], + n_procs = prod(df.np[1]) ) end - sort!(df,:np) + sort!(df,:n_procs) + + f(row) = row.n_procs ∉ [] + filter!(f,df) end # ╔═╡ a10880e9-680b-46f0-9bda-44e53a7196ce begin - plt = plot(xlabel="N processors",ylabel="walltime (s)",legend=false) - plot!(df[!,:np],df[!,:t_solver]) + plt = plot(xlabel="Number of processors",ylabel="time (s)",legend=false) + plot!(df[!,:n_procs],df[!,:t_solver]./df[!,:n_iter],marker=:circ) + #savefig(plt,datadir("weakScalability")) end # ╔═╡ 8d1b090e-2548-48fe-afb9-92f044442a80 +begin + plt2 = plot(xlabel="N processors",ylabel="N Iters",legend=false) + plot!(plt2,df[!,:n_procs],df[!,:n_iter]) +end +# ╔═╡ f63c1451-e3ad-41bb-851b-0366eb71c0cc +begin + plt3 = plot(xlabel="N processors",ylabel="N Cell per proc",legend=false) + plot!(plt3,df[!,:n_procs],df[!,:n_cells]./df[!,:n_procs]) +end # ╔═╡ Cell order: # ╠═f0aae79e-13d9-11ef-263f-b720d8f10878 # ╠═4725bb5d-8973-436b-af96-74cc90e7f354 # ╠═a10880e9-680b-46f0-9bda-44e53a7196ce # ╠═8d1b090e-2548-48fe-afb9-92f044442a80 +# ╠═f63c1451-e3ad-41bb-851b-0366eb71c0cc diff --git a/joss_paper/weakScalability.png b/joss_paper/weakScalability.png new file mode 100644 index 0000000000000000000000000000000000000000..182ef9a60b9df5c966a4651bba87255dc01970a8 GIT binary patch literal 18538 zcmb8XbyU=C^e;L{I3UP?2q;PjqI82wqcj2{sWj5v9g->_?a+uww@9ZTB@NOgD1vk& zefIeN&ROT)b^p1%Yw?0J^L(G!@!6mKd3dd?C{09godAJA5Xs6sQb8atXd)0;k;n`1 zFQ*!>``~|ghVs&n5a*cx-q&ZxAP}?&*+=54ZpmxYu3}F$W-o5WNRCAY3ctv`JKyv& z<8GC8$}6?BXWu6bLiPsotcTh2MJ6mPb;>6idM)xt@|=H?YgZ{N+t2B5+$4=o+Bvhg z4nvXghW8y^1djpZt4E@AlMNCbkM=>O;U!cx4sws5ML z=;-BYt`<%YHarEa{ZHc=v4*lTWu~#j4n{=H^wMt`?%Wx5G|E-Y8#Qbq)8|O4UtYGz zR54#=7HbUK7Pl$DJoWt*qyz#{V)lQ3_cEe@xhZo51qIzvOz!C7Kw4iVB_a7TXXmv2 zd5Po#LX3WM<>)aT$!Xh}yV0Q5gVo7B>zcjpj*k9Lo(yRiHEVq^Q$FqdbA{P9{;d+m zK(eUH(2W%DwY@P*(}>2CnE&2n)0s#l>1|TV)bT-F$I*R$veNedye82b>A=Jm8Fk@W zXZ{CV2bH_@xZECh@23!2J~R4hfzFDc7PgqFJCtv@>yJVpcHfe!(9#ESKcl6`b$E7{ z-kn>8HY0*+fuZruudbZ1QkPQo8cjzAiFuP2`%*@BiNUGYL*GX;-Y=1m9Gi(`zyF(_ zoxRvK8cd6W@I$L)GPDkV)>M46z)<_fM9rc)frH-bG70``G+X=7yH^GGZqbq8%U%(D zi@w=D^n2^Epm9)4#=|wV*}+zP`cn$Bi|e%7Ge5JsS9%vBo|RkdZST<|u@LVSTOFcr zXSQy)kH2H{UN$LBexW*Oag_w0#&q&$@|9?@i*GxEUIpH$XdOTJ+V)_TV%&d|fcn@CHitL}VZ9ChHSd~;8W#M$-nP3N21 zuVgw5Y&|CJlM4x-Y*Gwm*WEW9YX}ITexbWE!s500quQ&%<`y9wzrMQeZ!zOnJlPQk zG1zg`{5Ohheiy6+H#VM2Zzz2~9J&P)zaG9wozoqwa)0OJLDt(E3PELaI#eJ6(Kfnt zSvBzr4jsaeI$`UjT-i(5JvZ@t%I4IVm-*(zG9|@P7yNMD+g{2!f4|y+B)-VZh7Ysu z#<{H89Ey1@*QnN>SYDhwCNPcg z$#5zG4`P~_SyO(VGx_Gb8`{%*8t9xI>HUT#=ZWi1?^Ry3Zhi73b%w`>Q~0Ca+f$uy zg(aUC+@48a?OA3w;#r+$^gY|#C$i)fXn7^)S3`@$y3Q3Rw@1tKJjI`NBJcbxNRRlz zN7l#~u+`!ZYJ^e9*P(pvt%+5dj#E~z9?AIMxX(WF=+>{ANz`jTdlCn$O>YD_zVBZ| zc)Ut3L*28*@q0H|s25M)31MC)6`*>1MX?p@I|POPS6c8zay#lB>iOcGLiYpzx@kPV zFZUn#d3^ce<0oZx?5*a$@n?kUPUM^B_Q$OW^P;oQxm_!stXwct=%>dqiTI<|&khM(<7wKeu$D;uQBB{cS<_4JP-<8@ z7*O_gk;mLZI1!#dyAaPV*O92I7E%9gENcoI`>%{R!dol?)$~X>@4an`;YT?K6VAqj zE4NT$day)J$;plU2W#ps2R|5Saj|v@g(VO}Dk_f|Qw*XB{=7Up*=K(xhNyoC0|b6G z9vl&jb@7V6N5+KIr-euskK>YcvL51OL{^-A=N78T4yN0Nj#p|=)t`8US;MXL(tNBE zPKb_1Dt)nf1L0I_5l^oor2J@cI1`g@JFVhN`6p~6q=L@Hu)@TIpPoJLj=jjqzl+2o zWXpb2=Pjpow_U*<5#vWq=O=F6SZr}n`5}g7kyi!P#QR<%#XiAOl$Y(1&*Fxqyqr1eY$gG&fgGwcF<_-zB|r#nHhbk>7fv!g_Lo zn2DLW#`nD8$i&XBq`toXj2Va1kNI|jW6*e_lz@Jgke}?JJZh%FyY_jd&G7K>!oM%8 zcqTFeV&aJR@8A3R`ThO-Hzy~jCrQBZo{~4m>TIL$;78fVN&LpPMLM+=@WW9Ks)>{R z*U;D%1uCw^LCywBo&_w0XkGvQp`o3rYR8e0k%oqbWhv?i3K|+3Qc?+IXI~#NA)$hd z%+gG~XLx(V_BWQ}{k0)!HAL{&-q>@lpun6HqqFv1=wc63)3 zE!|g`>dcHTbKHoJm9g={>nnGJgoF|WY>Py)2>Po&#tnh-2#Qn-g>6`ew4@^#d+udO z30RHkTUtJ^6_i2BDk-H2*y>tXX_tiThOP#^b%UW=B)s;eOnRT-6I6Z9#ARoUPs-3;NzX&az1i&`5E{bM%nJ!V!RXZ*k8XEH3&S(qb;?r3XkV+I36wJ8& z43`WgAKnx4@$vCK-VYBCFYF8DCN#3NJT}CAXk@TkUw7KD{FA5q@9(MVn{F41;z(Gy zG=!*$#m)8s6Y!g%bgLbV$4U$=M~l`bD~oHQRG4Ert*opptdgUn=NA@?JGK$J3g&!= zyiu?F(vOdJ1ohw2iWy~~XqU`$EzU z3=E_Qx;D6O=)ym4Z*Obs=-hdn_zS*%{&d`19tKmpDdeV1vw*eKAEj{QBE-`a;%GoQ zdzpm~i&w(q@Q1^_8&NV0q_4@@wPlb7#KxO$b$ueo3XVMcVmbNwE77;*#{!x&q$5Ne ze#ywmy=iRpb#QRd*MBea4Z9!z+QWwGZmXJyzH7anJrp;^{o3Rr0*rSpR>w;BM5Q%# zb=QCVc)t9Mf;-{Rsz*Cf~4m zeUK7KzK4KtBLcz7tzn9J+eRqnx#ShR zodA129mI}5HC>bAiL>7DzOCe|R}jIvNj(A)&W5JnXnH;f9&PH`OXFOKzXJi~vtXj+ zDOl4}!611g-X~YZS&^0}c4qZyR6R&@#x16#82ON|Y;g9;k z6&&3Gg?K6-NKL0%bn`%M?M(f%6C-{=lt1F#C1Ht@ zN5%x)KZP6_Vr({gA~wZvE@<-ZBEQThcV^yn5lLl=Aj3hx@BXlT#czKQS@{&7Taj_VbX4 zKy|;AkR;b$ELM721e3S%_uW9e$3g}6FX?b(*{15BSXGY(U(zlJ*C9gi^8cN^cAZPE zV~|X(&-)wkGY0|Zd?|tpO_v~*UC$4>>NF)Qm3wP3cpLlXy@iD~b|m6D9CG@T2{-9s zbM&?v%{uJ0*u}6=EI*16YC2x-L+))|A;;omYM<~~az`AjADQXW$nefDgx4#cj(<(n zz7Y%ypT>+H;((iQwxMuH9Z!!$)C9nXFTi4seNDg7<6?qOAxI!Af;@&3w~`3|Lw=wLcrtR^x#q`2wb%qt?Pspm7Iyt`_KKObqIbY1MJ zUm4+h;C(t+Bt+LYwugl<3!ueGsdipbFPG<;|B*MIv;1Ix{`JE)W_-ktYElifJ)X3=P z59JkFs`%6k*R+zmPu}HuWaA#p$h`bXHS5c_QL9Jc{QGa912e%_?Zb`dK14_)r%M6? z;XEiGu{l<1RPV9p;Nr4AR+?dQJYHscdV1=)yEM13Q0}h9GlFVSsls-4 zW-RUkLMRk~$iRR-fEGwAgH|7772`SQ@{1d4TsLM$3ZKt8~2av42e>7ov$6n;fRgrC|&9f-giBgv$4cXR^-W0M*o$%{h6o3qSD~d8#4B>~@ zFOPCr9kc(tIj5_un;4E}*TIY_>LWz=QKMKF;E(6Qdf`^ntnaxuOhlVqgO$AT@OM&@ zfbEF1`(|^V+Kb0|dE@-8nHM}fIDL)CIGV5=e__41=b5_Se0;ER5f3j;15}_t%EL`J6$*GSB z39vnECu81j7akTstBJpT;p^)oSU`e}_}y$LuBonWVzl!MnV6EIV{UF9PINbO+1kSP z{h$4{N$Ch`ol-+w&!K^VEqIo?y81JLqMRHz8+XE%y3?)CF|ON}Cg0xjT2Yff;62>0 z75243g|_y27q~xt@<}k@*!okGnv90Cii@KfiNv?ZwkG=C``o-14%NwB3nE6zH{-EV zd?O1B3)7ATjH+mz935Ghn4ZkCva?^lbZKZxjFo)n@FY5#R6sz$1>4p2peD6YEnod< zmcnbmFN^u2u7t}GUA}Kw9CO_kDfUgw$0DgZk454|JkR%MR}M){{>U(FVB)2y81_(&8K(tApx^4$1a%PP8L9Nt*Kdb&rji`uTZ7iC1!&HEvsmN&HGZWxu!F z3ZIqN*v(z=xldoee%;#I3M9tC?rzwtS4bpsX(+qzS7(fAZ*sNy5Mz|g z_k0Zwi4F{)wi^FrJ;^W85klj04_^N9<41RQcj5EO)ccN7Gz;cxPqbvMQY?8LQl>)R zyaCuts2htT z`=8wjJVr)FNqm+SzUMwJci&`WWOyWZ)owIA*VPp^P*-Yf2JOX})Q zsxPJp*qj+{+g_BP)qHp_qVX&Uzp;V6uG)NW8(mn=Ri!1-*zoZ|LdFq8tibKqNlK0c z)wFuT@azBKsYv^vqxQD9hkZS@2j{k0eILjJC|{;5%?uPhzA;*GG(6_~N9Wo76h|#* z&WC&%M79qh;w)*m(lFLysA3Yn)G3p1SrNQvbeF{053t{MpRi z&$lb}C=_gaBq7h>HVe!?t*ftJFrYuX($x8cvX;!7IW|UeG=5e^%_z~KI099biw)#2uR1y|zrYPu6zX@yXr7IDNiA6CSAOgOCKe(ieeSQ{36FX`d4 zyQ(jyt%oZ$R1HWSmehQxO7GV(ArLy^D3ROpazfVlbPNj3)hhi<0m_n{Z!hL0P^e;E ze&MR8dlB7s>H?$}miS+?3TYXn`JFj!rjJI6*K6NXNa7!^G%R<8GE-P?`%b$@r2_Cl ztf2yxI!BgB*c-CAUG8%%$uOR-UT9g2luW4(NQxhDS41F$#o(#W1s+-5KDyU z=X~a`%#PletlWU$BP%P5i;L?eGTYKZFCy^n5GBpNY<0~BN$VEj?Q8WE5 zV50!UuxXgOi|k7g6mZ+r*U%U>AIc)45rI*ECr&cOef#+FBh;6UGj;A=XpV@82v=uk zKDY8CwP%n2-Y`7aD28K9aG5c~mS)Plqm)|b!!tZIT?o^-74^!RBQmPBv)eW|&-Izz zMmX84z&2)}^%R8gC91X0cKSryzkao=e%8yqb$IgjrXVK!tWQ-po@|F|vkT});eAMb z)Is3%D^`8z!ZrI+ey7{S)Ix67zs7Dn=_2jB%pb>ex$L+1!9sR`beOJ7ikACBzNxe% zZiyc+`6RFqPIIy-DJiLuMJ#@P{-@cB!_S4=Vs#|ztkR}_p$65vtDimc^t`tF`EKU! zFo|te!x8qDr4llV^w!$pEb$^ya^j4kGJ53o!$Yjgzr7O45lv0Yf?qjBMfHlEt+TG2~66V4<_aNd^ z?>a(le1zZae$v3(w{II88$%Ab6{xDJ`n9#SCso+og*UWw7t2Hp^K zQD8?OY|N&i<~KH!ke$7~`}_O+X5qonY?-MQN~fl@Kce+b6&*;{R!ypS9JSe9`)F1? zzJ{@wPoTFpzH;=C>P+m8U;GItZ&q?0fnW`JN1dLZpAQg5Tl)sxuY*}%;8?6CJ~PoV zu&{)p=2us(evlF|K9>`+?2$FQ!{6+dki{l(qd|yIpyrft@z(;0I-2;D&wB6}uLuj# zBw_Z4!~qDW7)I}tt#6Kw={Ky&D&@>!067mv5CuC6MH96|%d!snyhhmT)N zN0KhZ*zr$uEgf-l2eZ~>9c9yuC}2aOuqNIyXV(@%BkFVF*3+gWdiou8^_r}DA#8Ys zVeg({o+?Rz&alCuvl@kfVm!_cE*64tuCr25P%2C0Zsy)>RcS{N--}pd4%!))k?CI_9K?O&G;`b5ve-o9<{I^5=|P|sEM++W2fCRUP{AM&-vi-WV^ zpAa-yBfMAWBl{*VLZC97a#|dLz;<^q`)oVg5JF6|4hi1O%uGQ+VX-$QARwUCuv8_N z_3m962v;&PV-4P(Po4~XHXrJ??NF$QbsINt46Wy!i_h{XPXC<9+SvP*xNSHZ=DpbI zT2xzm985%&^*E`*u=NelHEOD=-^acKjkZ=fd-fxMxUR0-?sd#+-UH7-1Jd4uQ9>?V z6MJr+uMt$Mf$QKmbKh6J;87+$iF_;?MRj&_&A>PTm4b8OWk`s0wqgo&9gN%1*?sS# zBOjQ^d=i$Bd!;+;^)jrWCcTD}5VNv?!LxD}6tc#71;DWYwuaH8a&+7i$1=JRq3s<1 zqwy6kg2vi!VVCFU=Sg7)g9lzm4C|Vy#x!s|jCv|LGNbN2LtcxGXzJhWIo4bYC9xnF z+6lrxM}G~o?cK{QFCs!9mZ zNNACe<+9&+j0suNDhs}*=1{oDA-m@zU*uW1*8uj3;FOArtbwUrf*{9a_A;zWbLDR) zP3({kv1)E{(jq|-G@FQpB_vW^;c3lu9}D9h%ViEynu|H|MF{4Ef99~lOw>}u|AGPR zE)6`q6$MOZ3EQmq!891!i=1Y{R4kFnrr~8 zER2o$UUZ-&!*G-K2Ku0fNfHW|6FoDNZ zF;(cWBZ@vz!|=_kR{-0)y1RL`orAZh90LPya%2oq8_019Clnst#lakh(cm^)a#B)K zTpXW4bKvzgYHI4h$htbAhu@7g$JP4oLz2h0|1FgFUS(RxXE`)9uWxdDb>Jorf|Hp| z+|1Zm5*gL`f^ULG#7hQg@8G~KriBN=Q@3x0grJIHEFq>KV~ZP2u+)0tK`Cxc00JSE zwGgVwN*)pzcyhAS7pt>>K5bs*AO%hRr8hy zDlCOyjCRZXoE>u0CZJ#7Oy68h&kZne|Q)_c0P1QV5(mf{ulAsq5^Lis{8BD_Y2 zGnX6dI=rS+ErV{rI%KU>B30Y)iExcsVb-kCtMSZQ6pv2GG{63=f5w7CuatBS9g0&zU<$~- z*JnS@cn0hbb|Nb)iq>wo8pB& zF8BNkNgSeUjf0Bavdk7u`vg}LTqXI6GZM}jq3y`TCfp46bzAYn;qwWL{mql*3lLe2D&4NY@?2vzMHAHhJzM zKRoq%cT)_rqh#jz&7VGT2T8zb-Ai%3Xxbg0u9AzSBM7?12yAZbQ~c{E|62>d#K+&! z+sn?vk|gRY;yE-q`MeVM`#ZJqk}H9Mk%)_mLDA$L%jJrdVRVat8v+4h=sQ+!iPmJL=F}Hkvj)pL+WnI;IqDsJq#a#}cbU%mYw|~l ziT2||;**7}ynsvDWmNyRU9PK>>vBk$&2$aqW$Jr&?`vwF(G=GJ_^9znhGYwE_OAZ) zhv!Et9Nu+KL`2Wjr3zC=jq+5ZH;TDtotg+e;FwznYBUw>>^6ZDf5Ck%qTSKS>3dy% zP7WQmv5<#_RCYkqOb&t5*TcJljZgc|AK7rD)Y=R$s}UwBZ=Buz6OTKR=#y>K5Z$7+=;f z>Bx_r;?|~%w?m8Ws@5(Nqvdp=doTUqoc8(i=c2C0lxwc-ZEfnRs!Jtgk5@*EHzvyS zM_8sSZD&)3sV7T~+5ypn$qio(OixeWXgoj5d6F@fj z=Cv(GllLb7`tU&M$~)DVG+S=Hpt)5DkGYTQk| zr=|XM@V$u%k!$Z&o;=wa)${e<=@Ee7W78>*GnX=FCskrbi0c~?{|4>C}}jk z_1GtFi&eJ-c4xwir)SJO-4Rs3USZ;Jt|zyq$?4IaZT;Q|Bs1_zM`!|0O=%Yrb2i1r zGqbS3@>A=30Kp&Xljum-VeB0P$w%e z=I7+unBQ!A@6GQxkMu=WSvh3yx*W^x5Z_aRw+GO<-Qo){L0ded0~o>}>hh zix7d}e3$KSH=L)I=CzG~TsCsGl!8@L|J4UVDCldd>i@R1Xy3aOlbXy>_gu&c2>c4bj&5wE`o>%#c> zTH+*M0%qNH9CeLOOP}I1(OG40Cw1kWSIoaIO}&l~%h*;$VLk^=6BxO?yu7qz#v_Qz zt8hEa+t9FOCQN(3?XwUPm`|l4c~Woh?3S%a1kVpg7I8X-$82 zM4B*@xabFo+sum9rUn{;wk)e636HTpsXcz+Fpn=aZAG zDP;HngN^J%q_DBEeV_9#23$LpPuz>S@A2BP*dum!vet+PJ?VFjA1Fbm$CN+*jIEsH z*`H792t>aq%!*VZ__D{^e$Z3;i;sv1KTYne33i%B2!JipeS|WzIVl90(QoCXFmZNd z>FuO-7cgtuZ9F=&kWd9^;*8^iC zC9c_f4|EM-BmBlo`e|9Cu26}Te(HB6Mj(_l&B#D*^)Gelmx{-01VwQp5Mr-d@eCTm zYM%WQeZ8X!{MyqtCW%1!;lszjrayrPXqlnr`(e=~to+CUdK?5og7qGf$poH(C>^8i z&c9`YQJwj<;?a>`#|8BqY)x%{&LzAq7{&PC&yl994-kkZ98E)%;W`YEqh<2n4?-rd zF%ywjf>({+Q3n1S2&5`dN+-fc-^F1e{MuQ$k);LTy(e9y0h~}xAP|4ixA6=_*rhq>lEh6=dRmeS7ivkcaU0_h07yQ6W-T2(g%j|M?uzJBmQ=V>l@@NO=nw(m2)n_glA>0DV>b_}(n zgoK2X?!)vOm%=X~ny%ojk+ADEXy>ZFd2}mE=7p)LhG&4V=l<8WHsF0~YHOt*J$hkc zvRvU8ZE@=tspY?QopYN-M54CWYd7#NUL1Aam~RbT1ST2al7x(d!*(K{C8W!Y7LC5* z<|%Lrw>Rm1Bt_cQ4kHCx024mMj#P!Jc$}RaLRV_v1S(tT?l>PW-^17sALVE1duIK# zqN1V+3j3XR(oRkfW}$5OURRluBg2lat*R;!(9zK;D=S;^$b)9ebd|jk1dG+jEG+qH zzGvg#zR9h$L1MjeIRv$x6Uaio$;|ws#$8lj?z`u;?zn$qi3=~A#)AB)|A1V=+PY^^ z^v)l`>w(~8#=evSHrirhO$AzI&pw-_H;lHm1!MFXJu|VLFDfc3oQ30{1gorc1#RJ2 z5&+!Z-d<X3D)+3@cj)r)0h z%yj7&Rh@HVV3E{#Yp`cMU{q&Y!2LxZKQi$0#-bd~&rTnPTrOSX;^Io;w+5<8L&Fv# zJoKV!3bDGrfu9ituaKmQ`ksXg%$N;jg=@1bDJq^FE~ZUSPvg^JIy?a9_Etvm>B>y| z-iC#(f-b|yr=hmiQ&`Yz56wY;Q`_k{7Ob@D8aonFzbAB;YF#%7f|?^EQF~1FPco#4 zC^`L3$7g3n`S|#_xCTZ>DquWre4gF;1@k5DJ=E65 z$jz;ygsXo9j`>|*YdXX0n)(zT!Y^A>6Qyo|i2!bIaB&UH`AD`){`py}zdvX3o0|V# zO2oyblftaen*8d2p2er7iZbi~hy4CR9OK*=k5Z0-w_OIK7qf1wJ_>w!m7Kh2vgA+N z71sxqgdHyK(?df;V^?8@xt!{&aRA8m2>rK*SU9 zY><#&z4|!$%y{Nw%h$HoD2b$ljtwD?w)3O?w_!Q;%fQlW(!#r_@41x}Os`lGh%lBg z;ir{&FWUUP3IU4oYj*%A&mE=T$~)W#a;C*0TGdjy7ZC`B@Pvy7J7T^e&CmI+2UX`5 zCgM_NQbQ=_GocPVJw9-)agO{o>+=AQRL8`Y%E#!<;rG`l?TLZzV)h@Kjp ziOV48px-6v?A%KPN~K#f9I%UN32F4IFiDA-4~ccg(1pgTc`QOV;= z+`t6hLR26;Xn^$&lBpLX1GdJzC0KLEPE8j=Zl}us#55Mrgd#L~FA+#2LooN7V7-mx zihzjw`BUtF0l)v7iobHNGMyCWyfDxC7nRk?$pR@^A?BBB$)OBOARe0;Gl{?Y7cZ~= z7qQ`t|E~ud4p_rud;VXK4aJwZjd_zFIc9cOK}~~rAC!sn&p7eQa~9RUxR;Yy9JX(iGotU~JUPm_%H&uvs57V()t+8DIz$ zlkY{L(G*ZDT)o;l_g%;q4Fb#P=noC!xlixhxs#3d)xm7HUkxQY@+v_PM5ITL9zoy& zUiJbu_QC3S%c7`SQxv^)mSW08k&fX0Z%|u1|IcZ?zZ`r?f;s}7n7F*Q_87^;#(0CF^0%^Q=NznPiqb&K?Sl-H|EB{%)>DT$>E*V6XF*QGGCbBm*B0u%{ozBFe z4cm8G#wCGThhzrn!`|LrBE6(d=lrnG7i4Wjn4ax*K8x=Vy+QW)aZi?vKoV$sdwVX= zr+3Mi&cN!c?xIx7S4o(gz|G-83l#igM-n|m2X_$ z;CsvZRn!LV?uSsWLGmdz?s^CNhLUItyEX)MIW+$TEJvWs>Kh(*o~p8kYy`*;GzG$5 ze>Vp*q@A}1<1xnR%%zQbrlkdqV#Ttcn~#>)*Bk7AN=ZvgOLPDs44k8uvT{Gv&2cOm zHPzLBer4vXeeEU4zyOU)Ex75*cfFuuw5$}ocP~e2uiSh{YNfgb&0zEjqdUbIeA_Lf ze?Ao|i9<&|4I#QF-W|o5EHa%tYU$gs^KErbNA%3eMN%@Tlm!+wWSAUhOlY(>K~$!0 z!*sngtO`#(H`cQE+=upqt4i||Pft%HG#)Ut%~qa)1VQupL84^!Whqs7pbkarO*?x@ zC2@(e>KU&N0?Y}Fw?^qU`lRriE8*}bhHvvfJh&-rTo8?cbqUreZo-FE<(E$!+zB&y zF=7|}58eF8hY}JU0hb=_sjhC>-dNw0L)TElw8%TG0{Yw9e*biByKn7-mZ8y0_xGgl z{n@>pkmhAI)eIwkjNi!_;1E4!=&+qcUkZN5qAC>k$?>>ClxCcTJTzPfEkf`z)l=%K^D!9xT@K2Rwj$)N1an-fkozDQu}>zjc$t~kY6<#_joROWp-}ym z>=8n0ou6*_IyyRr>)2$anXPs-Wnp3Q*1&C3QCEMLn0TF< z+GbTkECMJRdKB{H^fX)txikPIG|)JvmX`VXccArgw%J0QR{t1oh6Dx%e)#ag@fs7)uQeA&^giGmyUi+HZ?U&3^y> z2Y4?9hb|csr>LlC*`->$0h~3If&5(FN{K=ngx{d;_zA3_0v0WeAm$SQ6{ z>A%R$%`GZwaNC*(KE-CH?$6RvPIbViPoIELi%U(_{mJ9G{i_rBipjCDs;a6S0pMPN zpFRMj1hOR%>-H{Zn5+T+;_T|0cF*1rq(&lVe~paL*hJ(vZ(75bt*t1?cQ8=^(m8aq ze?sZYMji@?@>|6n28LIUKHy(rbXuF>14NXrT(V|cwQEa9H!ZP&lYdF1p0c4DG%T_gw3yxsJCz5LhLh~ymIAAtKk+< zH1xM_K|R5U?i$~(obg~`V0bbA^)>A9^9m~is;5t%>S$|YLKTD-=og{m;{N0bIbMbq z_4@jHuaCTmsp;gz#6KsvYZiZzm>R$-G_K9?=D>JPPfk9nFl9OcD*|}cXcAz3J!o&_ za<$oQ2R^hyo?*%azKb*z#51k2gN=CwX`A5`H!Um)}q z!GX$UMynhRYv=(FoIK6Qz<_ClX2-sZi>t1#26^#Y6^!l+lRwYPc09FnY%o*qcoqWtLY!y9w0p%ig)#(k;6_;k>&fu%v-N-GWZ4+cUt-y87)ED`(S z<x%(QGl*!^dDhK4fIz(?Picq~6nw1sFu@ z`SUWW_V#x4yLZ>u*uL_wN3_iLZ;jJqbY&_$suB zV^y4M;TjJ2f^5yX>_+`(GLTsXyNeU7c2oOpq7bmV|k(6zAb@9$qwz*?$VpsOP1 z$dNCwzq$Ei=WwhmD+}BW99gH>9h@X#OO9yjGtn{-gG^5g^YA=ni~(Ff zGNXB?`#$no@PKx9Nc+#yb8`u!c1}X{@h>IxblK{<`iA}9deN3?)8fvSseU4ce}$eO zsGLd#_zt)yIaD`{_a5>deEOY+6VmcUm!A)qROs=H$1**|x$p&`6h0kc3rb@7Frm=W zD1~#8c6WET8-#bW-90^fySu!G!tYrUJuB+!w3*{P+}+*X+?3^Zc6P>H_SQ$HG&5Be z78gC;-LGX@xdp>R3pGpUq2{x(v9SvM{kE&8XJO|MegTsKb0-LrnAY{=D5HYgdDp=p zM7)o}0Oaia?g1r9aZb)A?2p~O#566JV)OOKa^||ascI_35C?lLVW>VUJ@>x%O zpPoL2un~yLRB?yS*Pia%(9n$7PgIpQ)6X2ch2vvSPQ22U&8)52$Ovkk{x~f4(^Uww zv$HcZzd?~+z3ROEOSW2D5qaPDOen1n>R7%>T~DLw{^1;|q1{#rz0X$T4j|+Js^SK( zS!8L~OT0;FQ$duZK%x3`pJpG-cyfq{q^dBZyPlU@@HzbI0B`hkb1N}IGG#)Pw6nKA zJUia_PyGg$J!F~Ga#g`asqWeRh!8zn+SoWBQWCwkb1KFLsy(>S0k*esx&pQzJNV(l zEgG6M*I94ZFaEe7(doG__vc$2i?_G8@87u)aG~Ikkg}inzz_qFQyC=zG3CT}!pi*5nALWC!d7 zt%wN-5=DI*M0B|X1wA3hfqdi0ZTqOBqXT*QGBuas*OS8?s44u=Ujz(o38Ct1VP`K;b9QzHuoa%lPWKBe5Y$V1vf>E} z?~;sJoP*?U>I8|92v$t~Dow#dxPgyVlU zTFK1B)Y;YL**2&F!luN==eeqhNlBsk9A7Spq5Ko#p^pq@2#|W^Gqs>@f>?ZbS~Pwu zrW1%|NR%_6U4|M|K%i2+@Y!^oI}b!9Fj7b_Fh#h#10`2tpO&gBuhWt;)Eyw!g&Sya zXGn5qM$hLk6NE$%+(=&oXQjHMpqnP*ZD?eqQJ^^oegJo%0HKVqSLNj`XGDlV!cR?2 z1tbEaLK+D~<*G81heDIM4{oWw27QgD7u+Yh9-(%Z%&_z?2xZ-l*DB#Im9UTyq>=9$ z4zz(`R02g%B*FXl;A?X;p2zeu9Rp-nA)zTa6rg;%x-7>(!HnI3&;m<;@Zf=mhsU$$ z&kvw&{PAO8{R${NAc7iiM9VPnnD)NlxG9afw+0e}D!4lZC`Al zSXo&CUDNO1CYF}a8q4=71XKHLJ*mz>S`Dh6qoYq0_Qqfs!QNe9aAdC^t$r(&etW}w zI4=%@HVju@?hGNa#DRe~tVn$WkMmuc&(UiX^eFb{LTDz&$8$$N-U(Dlc@TGcc)Zbg zC$L0=IcK`rMSW^|x;>l<9T{0-+DC0`y8-#%_0OO&1PX{>`5MI)pFSntcVs-)bh&Bw z`q4HF#(4WSBpBR_7i(*3J_%FqcLG>jO!M{Gnr{UtNObeTcc_qH^{e%Bb8~QJbaZr+ z71n}|zZJlz)&wA?%DfaSF&p>*ql2*F2?J?RaPYSm+z{|F`uBQ|!{cMvCiuApp2J7| z9Ej9+aQ&_XRF_a;fc`mM+3obm*(D%8p0ZSPbz`H^pd}dg7{Kix2p!Nm2CfR!H6X;E z55&KMq=S;!s8UHu3FMR3gC9GgPAUfRCo~Y5o@SNU?gLl?yAl#Q0SO4G8{pk#rKN|v zLrTLNAHeJ|uoCbP)EAKLTp+*5{Ta^JfJK8c1su(Grmh<7xVZ5w(Ypg)2$C>xbNdUu z<=!+=QIB0Si4M?Xjd_k=syRA5WYaA90eFqbsxbv;mCU*9U%1RoD5s+*k zVR*x(F_;>334_mXzFd$%Nn~VSZU%fKC|C;=rOBrJ?kx3UBzij;L5N6Xvr>}pX=!=5lLc7+SQ2z#@<^=`{Vy^^F!!Mw_Hc3aEKClX zza1j!H9>ugwY9YZW+*G21#`9^zzrpGNF%qlw{HcO6vwBfZT$S1v8U$X=*Xc{^YKve z{Gjn1)Ij_*5Y4Y%xw6y6(de}N9U`M6On;Swj+J#4@X%d0HcY_|jRL3x;U4SCQ`xoo z+oYW!TS!SxW>iXDU0IogbwQCie2m5sL;9n-!^+BbveSHOpjNvcb28Qt^*gc|G>B-g z!s|K-fd2NON8kNVO`5G!<0M@=7V*>-CY*|hERAL(%b{s2l_|oxfJO& zHhliP3-zF}G`4UpX<=*G{lv7ib0`Ys8KPm2%G%|zmf>%0{5FC59vB=w@4Yw^dQUFP zo`+c+jkH}5TZTnNA(%lx{3is45?zfJg%P#|rB%zL*#l8M3EMzPawil3INtM(h8lFh z_o9+q<{(HtATo;lybAj8?D3U}TGwJg=Q1)#ItCEBL0AP3-p8h=r-!-kuNLKr(PA(H zm=?H(4*iF^x;jjXEYb!INdi=1>JBG=?R#c+z(nio>UfDKEkMr&B_0rrQcS|WXD4L> zc>O>TIRd!l0U0Y`WHWxut>cjq7YGQT|E_{KcLxLr+bhjCfI)#g!okK?4_PkBwgIku zgY;2H;uT04AuvMnJcV5ocH4wXc^P~i$_`WMSKp-}E`#&(LDYl|kBy6KZEtTpKRbpi zPOGSI9v&dWgKBAeXNQNI8-%-hfCt>Sf7S4(avRGzJMTeq#T0YO%A7C&JQ!0y3qLv; zH%%jv2qwJd@Von3uIdqFj!b#zM zmG1(#>HcRh^YYCHC<)B%8K*ykz(E83JT)~%M8%H`lGvYp1-ksJ6ciEHco^B(*x*tQ z=;xJWWa7Duf{%8si69~+gD?{!iUK1G8{6puS`SHwx#WD#L4e4RN*{O{ab2zQF4>zO zKlDFvS;VT;7_{I)mf!CadGH`nI3nl11x8)I98OFlqJWfDP*AP(0LS6_?t;H4B#g) z9Ny(%*i5fOAK_|^fW!9}^)t_5(eM7@zcporq4qUfSzz-^b$??SdU zHZF%m4<`gb^(q-z$xct&u~@JHWHqKZx#!T2O_;ZEU7vy^3*-VmAz^|2C=lZSc%dzU zec=MH?^S@afByV|&cW=g9xHkNq#E-u=I*I{a0HQiD2p5xCzaG@@70e8?mW<57XKw nC&KjRn?Ar_9{E4tPer3d8*T*Jw1CnCfsmC{d{iW%@BhC5ib Date: Tue, 6 Aug 2024 15:58:54 +1000 Subject: [PATCH 17/30] Minor --- joss_paper/scalability/compile/compile.sh | 0 joss_paper/scalability/compile/warmup.jl | 9 --------- joss_paper/scalability/preparejobs.jl | 2 +- joss_paper/scalability/src/stokes.jl | 6 +++--- joss_paper/scalability/src/stokes_gmg.jl | 14 +++++++------- joss_paper/scalability/src/utils.jl | 14 ++++++++++---- joss_paper/scalability/template_gmg.sh | 5 ++--- 7 files changed, 23 insertions(+), 27 deletions(-) mode change 100644 => 100755 joss_paper/scalability/compile/compile.sh diff --git a/joss_paper/scalability/compile/compile.sh b/joss_paper/scalability/compile/compile.sh old mode 100644 new mode 100755 diff --git a/joss_paper/scalability/compile/warmup.jl b/joss_paper/scalability/compile/warmup.jl index f03d9628..d0a23686 100644 --- a/joss_paper/scalability/compile/warmup.jl +++ b/joss_paper/scalability/compile/warmup.jl @@ -16,12 +16,3 @@ stokes_gmg_main(; title="data/compile_gmg", mode = :mpi ) - -stokes_gmg_main(; - nr=0, - np=(2,1), - nc=(4,4), - np_per_level=[(2,1),(1,1)], - title="data/compile_gmg", - mode = :debug -) diff --git a/joss_paper/scalability/preparejobs.jl b/joss_paper/scalability/preparejobs.jl index caebfec4..d16ca9d3 100644 --- a/joss_paper/scalability/preparejobs.jl +++ b/joss_paper/scalability/preparejobs.jl @@ -43,7 +43,7 @@ function jobdict(params) "q" => "normal", "o" => datadir(jobname(fparams,"o")), "e" => datadir(jobname(fparams,"e")), - "walltime" => "01:00:00", + "walltime" => "02:00:00", "ncpus" => prod(np), "nnodes" => prod(np)÷48, "mem" => "$(prod(np)*4)gb", diff --git a/joss_paper/scalability/src/stokes.jl b/joss_paper/scalability/src/stokes.jl index dc80146b..819bd7ed 100644 --- a/joss_paper/scalability/src/stokes.jl +++ b/joss_paper/scalability/src/stokes.jl @@ -10,9 +10,9 @@ function stokes_driver(parts,model) reffe_u = ReferenceFE(lagrangian,VectorValue{2,Float64},fe_order) reffe_p = ReferenceFE(lagrangian,Float64,fe_order-1;space=:P) - V = TestFESpace(model,reffe_u,dirichlet_tags=["bottom","top"]) + V = TestFESpace(model,reffe_u,dirichlet_tags=["walls","top"]) U = TrialFESpace(V,[VectorValue(0.0,0.0),VectorValue(1.0,0.0)]) - Q = TestFESpace(model,reffe_p;conformity=:L2) + Q = TestFESpace(model,reffe_p;conformity=:L2,constraint=:zeromean) mfs = Gridap.MultiField.BlockMultiFieldStyle() X = MultiFieldFESpace([U,Q];style=mfs) @@ -38,7 +38,7 @@ function stokes_driver(parts,model) blocks = [LinearSystemBlock() LinearSystemBlock(); LinearSystemBlock() BiformBlock((p,q) -> ∫(p*q)dΩ,Q,Q)] P = BlockTriangularSolver(blocks,[solver_u,solver_p]) - solver = FGMRESSolver(15,P;rtol=1.e-8,verbose=i_am_main(parts)) + solver = FGMRESSolver(30,P;rtol=1.e-8,verbose=i_am_main(parts)) ns = numerical_setup(symbolic_setup(solver,A),A) toc!(t,"Setup") diff --git a/joss_paper/scalability/src/stokes_gmg.jl b/joss_paper/scalability/src/stokes_gmg.jl index b0a579ae..d0cd7992 100644 --- a/joss_paper/scalability/src/stokes_gmg.jl +++ b/joss_paper/scalability/src/stokes_gmg.jl @@ -11,10 +11,10 @@ function stokes_gmg_driver(parts,mh) reffe_u = ReferenceFE(lagrangian,VectorValue{2,Float64},fe_order) reffe_p = ReferenceFE(lagrangian,Float64,fe_order-1;space=:P) - tests_u = TestFESpace(mh,reffe_u,dirichlet_tags=["bottom","top"]) + tests_u = TestFESpace(mh,reffe_u,dirichlet_tags=["walls","top"]) trials_u = TrialFESpace(tests_u,[VectorValue(0.0,0.0),VectorValue(1.0,0.0)]) U, V = get_fe_space(trials_u,1), get_fe_space(tests_u,1) - Q = TestFESpace(model,reffe_p;conformity=:L2) + Q = TestFESpace(model,reffe_p;conformity=:L2,constraint=:zeromean) mfs = Gridap.MultiField.BlockMultiFieldStyle() X = MultiFieldFESpace([U,Q];style=mfs) @@ -38,7 +38,7 @@ function stokes_gmg_driver(parts,mh) tic!(t;barrier=true) # GMG preconditioner for the velocity block biforms = map(mhl -> get_bilinear_form(mhl,biform_u,qdegree),mh) - smoothers = map(mhl -> RichardsonSmoother(JacobiLinearSolver(),10,2.0/3.0), view(mh,1:num_levels(mh)-1)) + smoothers = map(mhl -> RichardsonSmoother(JacobiLinearSolver(),20,2.0/3.0), view(mh,1:num_levels(mh)-1)) restrictions, prolongations = setup_transfer_operators( trials_u, qdegree; mode=:residual, solver=CGSolver(JacobiLinearSolver(),verbose=false) ) @@ -47,20 +47,20 @@ function stokes_gmg_driver(parts,mh) prolongations,restrictions, pre_smoothers=smoothers, post_smoothers=smoothers, - coarsest_solver=PETScLinearSolver(), - maxiter=3,mode=:preconditioner,verbose=i_am_main(parts) + coarsest_solver=AMGSolver(), + maxiter=3,mode=:solver,verbose=i_am_main(parts) ) solver_u.log.depth = 4 # PCG solver for the pressure block - solver_p = CGSolver(JacobiLinearSolver();maxiter=20,atol=1e-14,rtol=1.e-10,verbose=i_am_main(parts)) + solver_p = CGSolver(JacobiLinearSolver();maxiter=20,atol=1e-14,rtol=1.e-10,verbose=false) solver_p.log.depth = 4 # Block triangular preconditioner blocks = [LinearSystemBlock() LinearSystemBlock(); LinearSystemBlock() BiformBlock((p,q) -> ∫(p*q)dΩ,Q,Q)] P = BlockTriangularSolver(blocks,[solver_u,solver_p]) - solver = FGMRESSolver(15,P;rtol=1.e-8,maxiter=20,verbose=i_am_main(parts)) + solver = FGMRESSolver(30,P;rtol=1.e-8,maxiter=20,verbose=i_am_main(parts)) ns = numerical_setup(symbolic_setup(solver,A),A) toc!(t,"SolverSetup") diff --git a/joss_paper/scalability/src/utils.jl b/joss_paper/scalability/src/utils.jl index 87c4b3fb..0e918a84 100644 --- a/joss_paper/scalability/src/utils.jl +++ b/joss_paper/scalability/src/utils.jl @@ -7,8 +7,8 @@ function get_bilinear_form(mh_lev,biform,qdegree) end function add_labels!(labels) - add_tag_from_tags!(labels,"top",[3,4,6]) - add_tag_from_tags!(labels,"bottom",[1,2,5]) + add_tag_from_tags!(labels,"top",[6]) + add_tag_from_tags!(labels,"walls",[1,2,3,4,5,7,8]) end MUMPSSolver() = PETScLinearSolver(petsc_mumps_setup) @@ -53,10 +53,16 @@ function petsc_amg_setup(ksp) dtol = GridapPETSc.PETSC.PETSC_DEFAULT maxits = GridapPETSc.PETSC.PETSC_DEFAULT - pc = Ref{GridapPETSc.PETSC.PC}() @check_error_code GridapPETSc.PETSC.KSPSetType(ksp[],GridapPETSc.PETSC.KSPCG) + @check_error_code GridapPETSc.PETSC.KSPSetTolerances(ksp[], rtol, atol, dtol, maxits) + + pc = Ref{GridapPETSc.PETSC.PC}() @check_error_code GridapPETSc.PETSC.KSPGetPC(ksp[],pc) @check_error_code GridapPETSc.PETSC.PCSetType(pc[],GridapPETSc.PETSC.PCGAMG) - @check_error_code GridapPETSc.PETSC.KSPSetTolerances(ksp[], rtol, atol, dtol, maxits) + + mat = Ref{GridapPETSc.PETSC.Mat}() + @check_error_code GridapPETSc.PETSC.KSPGetOperators(ksp[],mat,C_NULL) + @check_error_code GridapPETSc.PETSC.MatSetBlockSize(mat[],2) + @check_error_code GridapPETSc.PETSC.KSPView(ksp[],C_NULL) end diff --git a/joss_paper/scalability/template_gmg.sh b/joss_paper/scalability/template_gmg.sh index 32a87427..094063a9 100644 --- a/joss_paper/scalability/template_gmg.sh +++ b/joss_paper/scalability/template_gmg.sh @@ -29,10 +29,10 @@ mpiexec -n {{ncpus}} julia --project={{{projectdir}}} -O3 -J{{{sysimage}}} -e\ petsc_options = """ -ksp_type cg -ksp_rtol 1.0e-5 - -ksp_atol 1.0e-14 + -ksp_atol 1.0e-8 -ksp_converged_reason -pc_type asm - -pc_asm_overlap 10 + -pc_asm_overlap 1 -pc_asm_type restrict -sub_ksp_type preonly -sub_pc_type lu @@ -43,7 +43,6 @@ mpiexec -n {{ncpus}} julia --project={{{projectdir}}} -O3 -J{{{sysimage}}} -e\ np={{np}}, nc={{nc}}, np_per_level={{np_per_level}}, - petsc_options=petsc_options, title="{{{title}}}", ) ' From ed8fff3aeb4b3cb3440e40285c1002aee67be615 Mon Sep 17 00:00:00 2001 From: JordiManyer Date: Tue, 6 Aug 2024 16:19:43 +1000 Subject: [PATCH 18/30] First finished draft on the paper --- joss_paper/demo.jl | 10 +++++----- joss_paper/paper.bib | 3 +-- joss_paper/paper.md | 16 ++++++++-------- joss_paper/scalability/postprocess.jl | 6 +++--- joss_paper/weakScalability.png | Bin 18538 -> 17548 bytes 5 files changed, 17 insertions(+), 18 deletions(-) diff --git a/joss_paper/demo.jl b/joss_paper/demo.jl index cbc14783..398b4f7c 100644 --- a/joss_paper/demo.jl +++ b/joss_paper/demo.jl @@ -13,8 +13,8 @@ function get_bilinear_form(mh_lev,biform,qdegree) end function add_labels!(labels) - add_tag_from_tags!(labels,"top",[3,4,6]) - add_tag_from_tags!(labels,"bottom",[1,2,5]) + add_tag_from_tags!(labels,"top",[6]) + add_tag_from_tags!(labels,"walls",[1,2,3,4,5,7,8]) end np = (2,2) @@ -34,10 +34,10 @@ with_mpi() do distribute reffe_u = ReferenceFE(lagrangian,VectorValue{2,Float64},fe_order) reffe_p = ReferenceFE(lagrangian,Float64,fe_order-1;space=:P) - tests_u = TestFESpace(mh,reffe_u,dirichlet_tags=["bottom","top"]) + tests_u = TestFESpace(mh,reffe_u,dirichlet_tags=["walls","top"]) trials_u = TrialFESpace(tests_u,[VectorValue(0.0,0.0),VectorValue(1.0,0.0)]) U, V = get_fe_space(trials_u,1), get_fe_space(tests_u,1) - Q = TestFESpace(model,reffe_p;conformity=:L2) + Q = TestFESpace(model,reffe_p;conformity=:L2,constraint=:zeromean) mfs = Gridap.MultiField.BlockMultiFieldStyle() X = MultiFieldFESpace([U,Q];style=mfs) @@ -71,7 +71,7 @@ with_mpi() do distribute pre_smoothers=smoothers, post_smoothers=smoothers, coarsest_solver=LUSolver(), - maxiter=2,mode=:preconditioner + maxiter=2,mode=:solver ) # PCG solver for the pressure block diff --git a/joss_paper/paper.bib b/joss_paper/paper.bib index 631c3931..3092d16b 100644 --- a/joss_paper/paper.bib +++ b/joss_paper/paper.bib @@ -187,7 +187,6 @@ @article{Kirby2006 author = {Kirby, Robert C and Logg, Anders}, doi = {10.1145/1163641.1163644}, eprint = {1112.0402}, -file = {:home/fverdugo/.local/share/data/Mendeley Ltd./Mendeley Desktop/Downloaded/Kirby - Unknown - A Compiler for Variational Forms.pdf:pdf}, issn = {00983500}, journal = {ACM Transactions on Mathematical Software}, keywords = {Automation,Compiler,Finite element,Variational form}, @@ -289,7 +288,7 @@ @inproceedings{Arnold1 @article{Arnold2, title={Multigrid in H (div) and H (curl)}, - author={Douglas N. Arnold and Richard S. Falk and Ragnar Winther}, + author={D. N. Arnold and Richard S. Falk and Ragnar Winther}, journal={Numerische Mathematik}, year={2000}, volume={85}, diff --git a/joss_paper/paper.md b/joss_paper/paper.md index 8e824552..63d83965 100644 --- a/joss_paper/paper.md +++ b/joss_paper/paper.md @@ -39,7 +39,7 @@ The implementation of exact factorization-based solvers in parallel environments Hence the use of iterative methods is crucial to maintain scalability of FE codes. Unfortunately, the convergence of iterative methods is not guaranteed and rapidly deteriorates as the size of the linear system increases. To retain performance, the use of highly scalable preconditioners is mandatory. For simple problems, algebraic solvers and preconditioners (i.e based uniquelly on the algebraic system) are enough to obtain robust convergence. Many well-known libraries providing algebraic solvers already exist, such as PETSc [@petsc-user-ref], Trilinos [@trilinos], or Hypre [@hypre]. However, algebraic solvers are not always suited to deal with more challenging problems. -In these cases, solvers that exploit the geometry and physics of the particular problem are required. This is the case of many multiphysics problems involving differential operators with a large kernel such as the divergence [@arnold1] and the curl [@arnold2]. Examples of this are highly relevant problems such as Navier-Stokes, Maxwell or Darcy. Scalable solvers for this type of multiphysics problems rely on exploiting the block structure of such systems to find a spectrally equivalent block-preconditioner, and are often tied to specific discretizations of the underlying equations. +In these cases, solvers that exploit the geometry and physics of the particular problem are required. This is the case of many multiphysics problems involving differential operators with a large kernel such as the divergence [@Arnold1] and the curl [@Arnold2]. Examples of this are highly relevant problems such as Navier-Stokes, Maxwell or Darcy. Scalable solvers for this type of multiphysics problems rely on exploiting the block structure of such systems to find a spectrally equivalent block-preconditioner, and are often tied to specific discretizations of the underlying equations. To this end, GridapSolvers is a registered Julia [@Bezanson2017] software package which provides highly scalable geometric solvers tailored for the FE numerical solution of PDEs on parallel computers. Emphasis is put on the modular design of the library, which easily allows new preconditioners to be designed from the user's specific problem. ## Building blocks and composability @@ -104,8 +104,8 @@ function get_bilinear_form(mh_lev,biform,qdegree) end function add_labels!(labels) - add_tag_from_tags!(labels,"top",[3,4,6]) - add_tag_from_tags!(labels,"bottom",[1,2,5]) + add_tag_from_tags!(labels,"top",[6]) + add_tag_from_tags!(labels,"walls",[1,2,3,4,5,7,8]) end np = (2,2) @@ -125,10 +125,10 @@ with_mpi() do distribute reffe_u = ReferenceFE(lagrangian,VectorValue{2,Float64},fe_order) reffe_p = ReferenceFE(lagrangian,Float64,fe_order-1;space=:P) - tests_u = TestFESpace(mh,reffe_u,dirichlet_tags=["bottom","top"]) + tests_u = TestFESpace(mh,reffe_u,dirichlet_tags=["walls","top"]) trials_u = TrialFESpace(tests_u,[VectorValue(0.0,0.0),VectorValue(1.0,0.0)]) U, V = get_fe_space(trials_u,1), get_fe_space(tests_u,1) - Q = TestFESpace(model,reffe_p;conformity=:L2) + Q = TestFESpace(model,reffe_p;conformity=:L2,constraint=:zeromean) mfs = Gridap.MultiField.BlockMultiFieldStyle() X = MultiFieldFESpace([U,Q];style=mfs) @@ -162,7 +162,7 @@ with_mpi() do distribute pre_smoothers=smoothers, post_smoothers=smoothers, coarsest_solver=LUSolver(), - maxiter=2,mode=:preconditioner + maxiter=2,mode=:solver ) # PCG solver for the pressure block @@ -179,13 +179,13 @@ with_mpi() do distribute fill!(x,0.0) solve!(x,ns,b) uh, ph = FEFunction(X,x) - writevtk(Ω,"joss_paper/demo",cellfields=["uh"=>uh,"ph"=>ph]) + writevtk(Ω,"demo",cellfields=["uh"=>uh,"ph"=>ph]) end ``` ## Parallel scaling benchmark -The following section shows scalability results for the demo problem discussed above. We run our code on the Gadi supercomputer, which is part of the Australian National Computational Infrastructure. We use Intel's Cascade Lake 2x24-core Xeon Platinum 8274 nodes. Scalability is shown for up to 64 nodes, for a fixed local problem size of 48x64 quadrangle cells per processor. This amounts to a maximum size of 37M cells and 415M degrees of freedom distributed amongst 3072 processors. The code used to create these results can be found together with the submited paper (LINK). +The following section shows scalability results for the demo problem discussed above. We run our code on the Gadi supercomputer, which is part of the Australian National Computational Infrastructure. We use Intel's Cascade Lake 2x24-core Xeon Platinum 8274 nodes. Scalability is shown for up to 64 nodes, for a fixed local problem size of 48x64 quadrangle cells per processor. This amounts to a maximum size of approximately 37M cells and 415M degrees of freedom distributed amongst 3072 processors. The code used to create these results can be found together with the submitted paper. ![Weak scalability for a Stokes problem in 2D. Time is given per Conjugate Gradient iteration, as a function of the number of processors. \label{fig:packages}](weakScalability.png){ width=60% } diff --git a/joss_paper/scalability/postprocess.jl b/joss_paper/scalability/postprocess.jl index 4c670e45..a0065ea1 100644 --- a/joss_paper/scalability/postprocess.jl +++ b/joss_paper/scalability/postprocess.jl @@ -16,7 +16,7 @@ end # ╔═╡ 4725bb5d-8973-436b-af96-74cc90e7f354 begin - raw = collect_results(datadir("run_asm")) + raw = collect_results(datadir("run_amg")) dfgr = groupby(raw,[:np]) df = combine(dfgr) do df @@ -42,13 +42,13 @@ end begin plt = plot(xlabel="Number of processors",ylabel="time (s)",legend=false) plot!(df[!,:n_procs],df[!,:t_solver]./df[!,:n_iter],marker=:circ) - #savefig(plt,datadir("weakScalability")) + #savefig(plt,projectdir("../weakScalability")) end # ╔═╡ 8d1b090e-2548-48fe-afb9-92f044442a80 begin plt2 = plot(xlabel="N processors",ylabel="N Iters",legend=false) - plot!(plt2,df[!,:n_procs],df[!,:n_iter]) + plot!(plt2,df[!,:n_procs],df[!,:n_levels]) end # ╔═╡ f63c1451-e3ad-41bb-851b-0366eb71c0cc diff --git a/joss_paper/weakScalability.png b/joss_paper/weakScalability.png index 182ef9a60b9df5c966a4651bba87255dc01970a8..afd65aac76ebee79d1197d76b08aaaeac3aa9272 100644 GIT binary patch literal 17548 zcmb8XbzGF)*DgL5jg*4YNJy820ulpCcS=Y|BOu)&3@9R?(jeU+Al;xyDJk6{C`d~; zXN}Ku-rxKEo%j54=J|j+GxvS(z4qE`UF%xcenM1~r15dcaS#XuzO0Ot8Uk@q2Z6Zo z2J0fcbE+NE3x8mk$V*Eh&e8v+HsnMi5VQzcDG3et_iIyb;v^%NFgM*|QTW(ioev*4 zWLzt(u1f!rTbWu|I5ln{>}NZqYfDuztZO?}SXU9QtD2W_RVvTrLI5poshOmV<;gJi#c2nO?(9#~Lp>>BgF>@U$pmc(QuyavgdgM=09M2t;J?|KsmrxxmdNw`tWp zjqb^~`D)TaymD5N-jzC>RAG_cCpTpVBv^(z#qJIWDf7lF#6(9kM$09ZIhdJc6&Bjc zYh&w%uUa@wl7CUifv|`>E1J z>HEXK+wDoqK9G*{f9Hc0DV!3f>76oM)IK?(HfxH#c52As`#XZ_o_UI2g#w$fG(1e) zrfMzre#NE7#fED4PXcKK><7N}5TOu=zWhHT#l-PMT01FfAzAuklUF*{34G8Wcn&!* zaK>s3cVK6sjMqg|)#6GN270})s&8FJ_zy36^CbBsY>(yZ9}vtmY|wuFO0yzn?csbl zi2bC(kyf7wp2_A|5WL=zx>EGV3};IU;bi@KbS5V6=3V=imK;u7QIEPQ8Kg%$ThpFn z)xm0Dh3>su=qc|g_q?4FT*`J9NR#1jPCU(zwM<={icw)>E66zZJ6tJl3V(l>(PS*G zw~>OBG`moe9_s=^Jf(+hV2I^p53jL~B>soI8y;_&^{s@Xo!1?Y*thxWQ1sg1-$EAs>Jh^B(ETTJ4!2H zmfrGcq$<@9ahVQ*ur#!y8hiWIUSZx|yS)uOb?-$Zv2aM^%Am;KVe3D}A4~ph zFEbzd)6|;Q$+7%yUs1HcbAc=9ITI9y(>v6hWb={xh0@G*b1omI1R3Gsq76;md}vb3 z6E@?RL=eMUBC%R46H@W~cF1#W8U(`mhdKW#OVXKp7#WXgshE?1+^UUDs%~l5TMB3Dci+|V_6^R-}GjZF_%&jS6N>?P})Q*mwf)oot7cX*z!zIzs;{$C$5Ex1yR#iNZ~mfsQi zysJd@hf!W0-#->HIvK?MoIBD?J=gt8;r@jU^oG1~2Oa^VPNgEDXgM3zHnE|#Fxod7(=gqF;4wS%UN3~;p<`auwTD;Pw!Q>!G8J9UpKu_98`D*y&_MZ zHO3Vw%~e@!-hUq-C+<3d7{;O5><(-qLHJ+j%*z(b8Q{;Wzb6l7SO_5&d}4_8a0NZ7 z;2fl^>&z_pmmk@HrI@RaA-(>;^Tm*CAl>Ux5N*~V3;2%q8UG71oEa0m2l!Tla%g)- zg%De)8~l5tLhb*-+xVVa--H>Z4iODEKTc{cAPcy;-umkM<#As0vkN<-)oNLi4Bf)^ zE6vR-pZG1Ym3%>Rb9}fGsFJ0osiieOHWngv%f(niLPAR`=EaK_v9a2nROEsl{tjJCp8x!NYJZLphX{#)HXezIPK%_>K3^z~zH)rNA!rjSyaJFeu;y(J(ct3{V zCa=`N=jrK2c=VMkS2!G;5&l{$|A(K4R(#x56q^y}G^mY=i#u4Wnh~hD(C#1|L0wT< zN$R`1wPih0lvHGmiAXj3yAKSLn*RwHsFZc3?i?RYXk2jjQ*x1xrqbUTQ z-t|7%l;>(NwXqR)TVq|l3GX~y`KIC->kT?HpBMS~U3-EDVNB#Wy zGb<~rxamxYF0;HmYS9DP(U&G^V{I)bFCS!UZEdZ)w1hy^(y?DInTbDQD&Neg zq=P%r#~r2SGm=Kc8_y1jA>IcS9UWa+S;=9*XvvtLpWi2KDWtn*6hGl&YQY3k?#;iY zjKO;+V2qX}L7XO9b_ep)4Zd9@_j1KYY4@^Q45`o25nd$jBVpGto@hsS2)@b4Ui7FD z$I~az0Zmw{+3`=s%Z?b#+uY-2^uKri;c7&X(d(<|4MlfPde3~DV&GuliZe>^Sd;5R zkZl)wPq`&=Wf!O)tljh`uwY_(GbP|((cGR{@~iOdfI;0r)}~g$S4&XeJ^Scep4@uCb}@Ln8>{6U0=Wd5Etfq-h7D` z>%ynmypiVcrq+l}bF$qQB=;sEdR(7S6(6hg7Xz~to)Xq^s+oi-^SjI3HA8&bocxe2 zmNh_A{O4A%!k$IX!w{IDfF%wjD~t@S2YWv^!DN{xLZ2fHj_;BNT;PD=irciSJLdb_ z$t_lyZ5sDXh&dffX;FkV`ZN5ONc_QA4K)VRuI5{??jKmUOlh6GqwdPQ3Ki?) zhecdU@T9x3t2;_x|A>f~aOW~Std<^}m_Lx9^a(`k3Li!R9bAeXeE5KavnpY|D-%$X zW4H*>r}xzKd+kW>K5H;-YF5X+Qo+Q35-olGSh1%u$5YENuy3l*V(9PXYrb9>GhgCE zuuZG*jS4r7FBM+zQAG45tg^yJH;DX}SjHgmQZbp?0U~+dGNi0oRZZrv>lElryJ14U z)?Ty$(eCpbc$B!L2;-lsC9sCyRhqePu>Rq@#*8pVJbW-CwXWmMshH2jO%DrO;g$xJ zoDP+JZb=9}@EZSLtcCQ#ypY*X!dQvdI!^?o2FETarW$R+I5!c^x}yc^?y%FoH0bao zh=0~W&r25Gg$Yj&!&0si_%OQaU}5~-Oyu7R=qBz!R`V`-uYhl=6h?|UnudMtH7N{Y zUFAR0!zTu~(`=9Ks9w?bXEi1m6&XuOo5iI#y!We!?Ef*Yf8K#h6qE{It#;07=WNQh zItS3y?K!YtzQhD8VZ`b#r*yqf@%Kq3o@iG*I7HZ%aW^;aMFr{r!cPZT|AAj@NEkW6 zIfgBv|gnLaN&>|@8^gyewo1`BE?h$8Z6;O=yNu?|1p8I`Qj?o$>}MD2rD=O zz^>RY`)#_T2!a6G%iP>eG_Mm6kGxz&ja`O0NypfPy+&;ftG-Q7C!f?2b_Lti=kNod z;>3=<3m)K)38?VKyQrurt71P55ht+{2L_Y|@z!F{Ot5nDS=W{!Ozuj>!dLNM2h0v{ zuE9{cVWGGgLbcCSL=G;|UAdwkn_JfI=PXzIr%AN&A1|GyOPfJ_;?^kW5Uctf(m4`h z(T+hdcIe_g#qZ=ykxA>%^y~pv-*Oa7$1YC7V~GK_L9xDmA!5t+D0xbm2NxVRi<%l9 znLUq;jBK>VHClqjKhn^Z=4G{d$k?4*ef*`6eA&45VB#&|O_KFy1$?8^wU@>CUw9YX zIovm}ql@`8wCn+hpyGM?|qYNMvmnJuzj#l%`isHrlQ)PByd(%(2e<~z? zsr_hzNNj>>Nrd#K%21U}DyOB3>}(E=9p z-@eJ|=}p8G5tkEdYHA7y2pko)Uc!oZ%Zhr;QAgr6@+~i}A<<*kug~awu_tMwe7kD$ zB>klwRf?Bq0#VAjV~svLX=v-gH#=fOGj? zB{>ag;-FwRd&+oJ#EIb33%zXY;c*y$$Ku_)cTb-_eeAQV=~APXcd|EOANlTGKwzLn zZ?bL6+C8JbYa%s|xVDR;`U|q$Fh63&dLn;S#PVnz@M0Uy>Mo?XuM8L3j1=(;3Kr+( z<x}=E~_s1@O$XtB7x~KT&(__p$46d$Hl~3!CoOuU~U>b2-W>cC1(z zs9PJ*YY{U{yOuWo1r}A3s*7==`~rlk6#XtLDx^ zDgd7{i(WhH`Q>Hn?5|eZMo;q#3Rcc_vyFHQI>M}(YTnDTSlDaa`F&lgg;W|ZH?K34y~>PH$Cd}5 zg~!=T2Ui-8%gah*KFn?i!AUh_@4SVjC@s$yCSXyBJ(UFWd3^8Qy(qQ0We#CsuY=7w zMJ;)DYC${wJI7d$Vp>bd+cLk0;CwY3)R#K3jW?oDwXmp^M!=p3$b3!3#Cn5&&+w=S zYbl^{+HUHKvAcV<4!fxP<}83YHFb5a$Mtuu2bq(osLDCT#QfU1nwTsWob6V`Jbqk` zSIGXXM6RIGkxQ#B_qzgT`Kbgd-eWlgXU+6=#b4yPARzEd*Ov!j)U3;ZrJO8slY&AI zcFyC+)P#h6#fD8ZekZE8-m*U`Wnf}rx^?Ro9o^i3vY3mjt6fM&;MET^^*^V2W3Roq z|3z4o^IpPMQ|svnmm%NNj@e_8tZ@1oez){>>qqk>*O`tY-E7AQZ$1f$;YRdDlg5jx z>gnnE(Y^*~mzbEi35!1cvorS6#fuO|H_9*-QY^bmOG`cRQZ!DM>24iX#a89F?A96Q zSwvvl$3G)WJ66!xtTIEc%3^vw3=F}+C>dt9#Kq{0yOxn4YHVdSrqNSVhkjMHNJv;z zt{!CZoz&E^$Y7N?)K57~E}3CoH)zj^RcBPbxEd!e%+wdY<7}Rd@$tE9%K$mO*M(I6 z=6!Z|%SRqT?>`s>BC`hT@U=J9xKvZA=KB|(89{M@`J`kvE$H%r%&Q`P=5jV!pY$9< z6q_0$0uO3S>hUO9lGWZK`k%YQk{wMzZMqw_8NY!h*4M! zrP*fmhJoW{HW+E;!_5ahEC+NT*p9r4R)hK6hRLqf}6Pd z7V5>v1L72I$tr>tt4oNsODUFp`WB%}v1B!Op0t=(lLJBwh~Tq~;l1K#El(TQ2#Z`Vz?i0mF!chDFFB8)N#QP0Gr0=u1^ z-F>e;N?ubLQ&Wa7sfAr}2Ao+mN%P@h$PrpZ+^h3)AMnS9^Rr{vvfsadhYF7#br}=0 z%4#6HzTW$A>*wk5KEc(iXAq+yJtSM>aun!3V4|8MeGxj|x#@z5P`Dz=a@|+R=~oxz zgQTRSCr_TVN6^qvQtJB&N6Wd)H3t>yRh{h&D68e|w1klK^z?9ObNnB?W2a`Da|`v) zE~$kV?RvWnzY6m6Aw%pK;Pw&K=%XiM!lx2<8AhO%$Ii~~>^)hRqitX?^(~q$BO~Kc zvcNVCWOcz z^YakcgRfG_Vg1?NWkfCf{Q0{)pwDE1GTs@bN656i>R%**;5G=moK{v=286G^J|&&$ z_OI@WyBN9TC!$H$J=oi(?^LU0?X-oeYjzj0>MK`M2mgxVzYj>$hNT7?_wF zqb2jLWQNziMbL=Z+1jqn&u2`KBSS}HKGXE(6zJ_wdLR%~mfo&CCx_dCfq{_Wr~S@P zM@MzixVgAC!UU%Y6BItDW@-0nD?4Dm2wmGWhobNc$}>{E{cZbRZnAx^=u8hS#RHzS z=}aSJgY5Tnl&A4)1t%4`#3r6{f#RgOi)%M7DM|p_Q+I4t?XoiLFyF?=$T($Tle`Gr z%mif6*RNm46jNBvge#mq<1`3_sr&m)**?1gJWZP93#jIWTvu;UQB4(S7gyR&2;IG# ztC0^pL}x75`R<6Juggp zUvcPFYG`Wmj}A^v`Lw;h$<=s}{^EtSyf&^6N>{u6?X;KXK;W1*60c76nh%QU3j91U z^g)FY#Hg}HhlGTXaXkjFj*?-3?P_dnOeN&pH@5bh2&KEf%)(jl3ThD3rN~7hxKhGW zZO2cn`G|p7Rto(>h1*=>FtrzG5=>+YBcbrbxrnf&YP7i7w90xkiM>AZ*#y(cM;n6Y zF3;-vEq7W*<>Z%(znn?${|YDoV?3nA3Y!$RxQFS)TSq@_iSTbrvl@(B>rHE#6o>fm z=@dTKQsPf?(66w0GfD(e4WTF*8t9m&C75#YhM~M2EfE&*a!hx!+zJ4T3hqAHKXGP)-pwFqj5t1uHi>sXsMzm3q3{ z`l!ocsQ2`NQtjjS7Z9Il`H4}$hgI24dH#yGAdjl6sUbq*;NrTda`nA@bzA*x!QAcI zuEUohT7-BRA2G_@+#Fp8Pp68ov4s-$ckPNvaES`_#+NK4DJ?;cuq=8g%0op-i9{l` ziu7e!G>c0WSx$e1a@E(@U%Yh5MNiVQ=s?f;3HfPWQ(WucyYh=hY=&|*-Vkcg)lzK99X7|=60HRJ**>6BD_B%U~o}|b<gsAR3&x$L8J~TF?r;q)t!x!$RN^Vd<;&GIHO5SB86>qB zf>N)h3U>^EL;Eu5PBT|zPzShM%fP^ALsD^RU^K{rk3+}heB)yA;r#u9`H*XXbyA(8G|85`5Rm0536 zq|eb-RaLdJ`VD2pc%^MY>-fp({tRTsuN@uF939p2Gz{uo*MJf{Wt?0ttE{v;%T-0b z#C}Xf-I=j+M{}|5m6=wyOCn>iF~D{q24d`hKIigu)eOMr@G>(q8^7NV-ZMEl`TqTT zEIP<|5S`-VL(ux_iY=Sz7 zk_r4auR=o!VCu7LP7%g6e%f5 zQ7%L+u= zwebqbmABpsh}qKreN|LTk~n9-;cRMVlnQB$yH9{f#Sgn|3_GMhOCDXonSLc+!=MvM z2d?uQ8tC1kq2PN3QB9gY_By=evK+_Dtz)w<24{Xah+h&BcHtUb5CjX;bK$_y}n;fA^~5!m{MhSN{aFD2@pTyg&Wl*qt4 zME(Z706HT0Yw?Tf{2H@J>F(*F<}pFZFsKzfF@c#Yl~^&{qKDt!{!z1W{L z=3$t|3JR~k7U>O-t00r_^@MGkwV?x7>J>g}6pimeTKLKZ#YdhYy@hEL7$zYs(EEBG zF`~s$@{+T@BS``xtaxY*1uB-);0PQA?DOTz7wr!^-QU3-#QdFe z>wT&7NVzP}S^SN;?+u$|f*q0Zosr3MneL61x4qc~YKuV$>>fbzTQ z`HYsB4)l!=dvC9&N(6!vdLcU|AZ#|{a$UZBIhO`1DI9$KwaJ>YA~s9j{3ug3ktzoL zL8V~yc%}}O34{N>h>_QDTB{xA`v}MTl~P1~4n_?{lrcrlMy)y1iz&7>XaV^koN>0X zHg2=A*=%fJ+AR=?+YHDWc$t#@dt)Hhapm zv9CRHdM~)O;-oFrxFh;9%Zofv;#iNWpFSAvU@7bEYrSree0G>JyWUW&Uz1*#)O0=; zkYR2hmu<^rOd=bxv^-kz>P^~)&p}D$9svT;joNq=CNCqSQ*O1f_UI4lIuSBoIP61D z3a{t4f!}03A3u;b-hX;*7U8`Ya>V#;x_4EN@@&Zq1L6N7fcEL+w_QEEeLr=X&%%=d zok`>N!oGCXPe$YzRsUq_KDaKDr%7~AdVv!DxT2h|snHrK$2iBrqD_dD$qOJxe!Gs! zkveTj*F{R5Cq;ezNXhog<1w2AE_zh7Pxbc-z55Fk1Dkm#1Zb$fMn}7pMicVLe`tMu zQ=YxG1u50_phZrBfxSfVi4;3_3}G-~{5?8UN0&}8q41qsCNINKGCC%V=AN8{KbdVp zTDI1wJ)+~Z>8^nJ@gzp8jhY0caR*+?i=QIfOUtJsZ@aDL28oU14rH_6b%q+GSPp2kUUcIM7nU*uP3qW_TjUhnXCK} z=i{rr*Lw;cK9t4%<#&>-<$4un^Wry&;&{DFJ1&iP(Gp*9nNwM>@W$i#M!$(R0bBCQ zSkYGiE7fz9ome?<{U}LIrR7*3due=xlKs4c9hqMIn*^(wX~-e7dpuqFsS@w?pfoCB zKx*80!`|P;E~m+&Gl$O=I;CQ!Q|}}KnW%d?AUfWfP(U%vn+~h_{H~z_?oZQdK~BsIP}v1+ip3cGGQ>B2fZlD9&uQxmWR&(6HwOa$2ws2- z4*Hr_>P&s#(X1F1ZFnLO{#T9PeWH>}{vbQIPteMJ6M+~AU~3k3HQSPK&5XQMi@qst zz-G98(N?kRC&NpKPYbgdDTK%~&NRV-uS)_4HV8x1hmt#?PrYrD0@&?0BBL{(8rGHC#Bzq&b*YvL1 zH{0ql^tE+9Mo%sH#sm!*KcRP&mzMEs^o|84tI&wp=%+4FLH7VG0?djIm(D{_A8sJgo2^ar83%vjiT z$KUitleXI~Hwll_iPc1Q_~btzJFJq72ZH~@hY#%XeiriQ$vywxgABzQ&C0}NK2~a0 zQc}Xb`Siuj#*KX}jz1B+g}-%k6AIkmC@O|@CE52XswZ@20i4$ zM`h+fynu1h_%4P+Z@sJmzIP^CsbOS_bfp7}w%Uoi^!uR^V zc5ra8d^KYuo0bk`Ebn>$c)RHSQNcBk&=G&R+Pt;3rOTERoFxz*RMokoN(HVe5=?~x zKB)8r7aynF?u?3bT#0KJf=%F)QHdKwM01bd_UPYtu38$_%-K#f$rep(lDmNLr{h`W zw2U!}SRE7n{hLfJoQutQX;v+Att#xcW{kHf3Gib@$Wcn{bg*l9(Px|j1;7fmJh5nLT3YhiplOjN% zhJI4o>_3AwW{}8T;L*W=cj2%9jQcf=8~Vckj3^p%wK#eLaVh}&5_tamZT}kMKc0P! zlApvFc^`og;iJb!@`Zt6{tgK(r~GSZ5QD%wboZI*D3x2$MZ<2_#oZPhYjmJb z=`s~^$usRJ_HWJo=hL>dTh2&3jh~3~l1i{KFE8&koCnaEmX!_3$as=L0qvIa?Orig zSJ&oZh=HV}q>5{gOqJMMK7@rSh7MBb($tS+OP+A~XkT`gkokP)g2L7;fKQEbA3z_6uKjO3Z$B^<9p2{J%@c^8}0VuluyUrW1eW zg*IE9>Sa2nK`>P|yrmH(@y`aiaq;qBe?wnz<;7nof5UV6ro(dB2e|O}F-Pdn)#3B%WAJ`U^tIY(r$#>nttawt z!xM)7n9y|rqn6^{eP1@#lb?b!1s{M02?Jd6X=(B5uI`^N!QC6a4Rd=ad%)%#y{FIi za>S1C>NiJ-!1_MgkX-SC(;gG$YvdCwa@C%$D*Yf9$D;cnLqzRW)>+Qgbfz_<34SDL zk}DDS{(YC18=bkV@y1O$dg1~NRv!&axM_46hB0s{ zGFrrQQMoakhc8eNB6Zui_AoeH_H=9>`3p?wU$zX>GxSPuizWT zJhQ498%_5nn!*+_x7)Q?J&1~NR)}ktJ4A2azU?yt6%8J?B+HN+6*=ACe3GeVi- zDCywO{DWHx`GVM(1k`v01oppbE(CM(@Ic?>`^=0XI<0qfbTG59ph>bn4)_&T<&99| z-sj>{o~b}kJ*s|YY+{n$XmdfR)NZO48Y?Sxq^b1i{We}<eZ|0mMIe0 zkBW+ls;a6+wu`lIA|utIO;Tha%(3n?S?$2pu+x89WtAmSZPrO@VXg3u$$-5@L| zSal}Go3+y`b}r(&8VCKFQ>TZxxpaDUu7#nYow!`}gp`yg{qkJD>x5Ekn$C_ofu;WX z^=p5BsqY~u8-YVk_SqX-pQ?+Xy!UR=Lyo0)JL6&4@^Ik<#F&+(r2`0MSQNJz7)B>1 z#7=hx8es5Kbydi;zAtRg&F}i2R4Xu1+%Z=G%1MDquiWyj%Hv0mZjzG=S@xxZa^Tz8 z81&p9G0&8mg+@jWK;)YNGz)z^>>!*Vq$5_H%}eqN39S17-R7uhfp_BgFQWzDtg}!|H?5_Fk4onu;^<=Fpd{ z^}X3A^5DS(;6a;vUl7m;2nv#k-bB-7potHkXU?*(|MO5m0WT|R{;rr<(^Q?Cx}5z? zBS*9xmqD!*%T%SU9%z^DTcs~aC*MtxuR)RlmoPsUP|>b|G5x%saOn~`!& zwo6yhofiTE0+*b(IXR2a+uhM)>??w{;Uhi0f;zoDG*~N2O1^e? zcXxKq-|gw_Eco2k_pLig*ll?*uQ|u~)NL4+zP!8~6gTJ%ri%{vTr17iQ9u=c|u8 z^FiGW{*I8|q=*hg%CrtL}+-rxG!3C(IU-@@}A97zUL zXe&mJ1eA3d6!TQ}-O!vGfca%eNA#qK`3>f(HoL_hop&^!f7o=LIga?CmbaXO(d^ey zxorMP8@0Z^4$ir3h>L>*%e1RQ(}$iA1kHp5@;U8Yz18*g$2vO9*S`G(RNx2(oTJ-i zQB_fyotsOG(3N!ZIs6$5kw(;c>CI7k)0wcIii#2R(jiF0h5Gtk+x4|4(*r>EiZg#l zM=o4^d@51zgXSPyNhDIhep=z|_^OtM#^CU#^z>UC zXMq&Y2wm64H92DqtDixuSx?XFM@tBF0aohNQWZf10Qkwl!2v~>yhuPJ%3j}kdbBI- zeeiMftNT)4n!i5+b}W?1o12?~m;^YzVWFWbBgH9a%U20DpyBh6xt z&^15aTZ2W&QnP}fk2?31`wiwhM+DmyI~G->CQRh4&Kkj#k$(s#*?j1 zXr2S@4b?hfrD3_Hs<`;4cZ~AE1T@q)Z?as2t$uWPs0bJlgb>ig0mc9Kk8KK|MI-`v)16^i%x(OUgfCzmF^Xj#SS8;pZ-z)Pw_s!D~@;P#b z*kjOm*|HJ(wURHc4spfvS-sFbu|dQKEHkYb~)$0 zh{qQ6XPV26cDtoPK0jQj7n_)Pd@vsYKB*td&v$?>*xau{$OW{!i56!fjQam?zwz7) zRS`*JU_nTqcb}zFue^zii%UpgjqObF*?V+f>(ZwKBWk)Eaey3>?|Ts=(Jd{BK#aUj zpeK^vym|JkD?a8fn)ZPtgl?e4jCwdJB#U^!JJZwCAol~Dag>kYuL|;>8#lt|p%8%= zT%~?)qYv$TIAB7}HthH=GO`N7HY8-Anm~XGhi*R8K9G`nFXrb9UKmkLle`E}W@V;N zGf!i>(WfqN4g3@Kqq?AfVNp>}Z*NoYl<~K3-`2~VkTQQ8jaGq3x>;f8rDrL(aom%+ zn3)OJ)dm5mvF;L|q27Qeot$_>YkR4Uvig`;9or zm&i2Bd{4(35izlIQGPN3l^KwKslf%g8wwXUWy-7VSp2VI2~HlKZ1+Ns3bY&$Syj$j zNR+VzpQS~C0mdgLKn)^!`T?AEn9F19>ureI2pdt+lG z<;|N^Ur%UyF)1gX94@Bdh9-S8Ed?#yNbwNpC_#YdF!#%vH~#7RWQ|BKEDdMieoC2;L7KTsP0tzGec45_Y?S2gupWu`; zAB_V;LviU^`uiQhvnkuLBLL7%z_-vLM>iZ+F{&SUBhaS60w7=3)Eo}xsRy~dx#C!J zf<8%E{0DI9F`N*=C4e4$CDZU z*~LY-)zOlN=b7l%%H>NLSDWE-Lges=NAyu61kLZ?QzAQ*#ZEJQwV=2?eFv2Z$S$Cv zwLVt1*cE^0N!x3p5UE~~?XFzaj5GAANr$5V_hDeloYZ9Kt3SCsQ2%^HwGZq0;Vo5-hiFP z2|7-T9?D?jU)_nfMQVMHU6C(A@PJ9<{S0Ih5Q#QL!~jdl%XdO!Z|_HID3}Y%>2-LL zkZ?QL>a7f#t8MxFvTr;zhYi^*3F*4Bs_Go53QogDZvb!*O=o9k$H#Sd3l6qAxj@kk zX{yv|QE8V0-P(qT>;c9`r!fdSfW+mMm2;w4qrX9g1uF`u2Sf)Bz)yppFAIOgA}fZ5 zHY~?$NUx!xP&Fjpbx4nh=!F~#DK8n$SV1DB$C@L8?foe?@Qs=fWF>UE=j0%uK$(;r z09bA=hCX_8#RCI_B$ZN|5l&Quk z4yd<-L^$!8Z*AW$<0~49E4MR8MMZYSbHuwgS4(S683Ln5~Gl0tqoWS)Z8+%`1KVr{?BN z&VmB8`r++uZNq*gs5byQ|5_Rd37_BnS?HK??_LK>T2{+%x4SIWRSK1!%sK}4!pf&;Nx{gi>39-Z3%^zo0XHprB}HM@dD&ZM5Q9k%$86`1FHws-||2X^m6OrfFb!@ zg|?Q~@OiVUh6V$rjDgy@;~T;s&_e@s47erowsKob3p?yH5N6o_GQ>_*#X@pF1vq%e zyi2FlRMywGv9>QXGSvxzxXgn7{{lcV?dn=R+Fe7L;IC)S&UiRD<5l)4wV6dl5K`9D>}rxzQn>is^9wA@%m81GEVVT! z?{CZ$a}pgbL+Q&{^7<8SDwvl$jn58I7WzQP zGI5LG$d_iMq%$=+LEq4f=%5b!z+;IG9EdkLF;lQm9|yrA6d zNxl!PABfvQTr~}95kbF2w31CttsjoLCj$z+D{yYM|V@Z&3QL5M(~>$yH6mR9YH0y ziUm&ixNF?tPl$=pI%ffbsJR>|0ZNzUh!74AjwFEcwfsXs@!eO0B77$=Cs|E-o*o4| za}8?asD8j>xa!nb=;-KfD<@}0P1m{<00>)l1nv=bTi%O~?3|o^=mwrHLCXL-%BUA8 zjK{lpHzjm*;%ZIUJC)+N)4*V7etQ1LUvx8!%I}8P`JQ^(+S;~rBQ_~1$10- zv33fR9>IGd)g4;N%-X9J(f#l?l@2tnrb4H#NBHYXPsTc}=6 z*BiiZA&$P1x+Nwi2KgSN{aaTTJ1grJl(<`4o<_?a+uww@9ZTB@NOgD1vk& zefIeN&ROT)b^p1%Yw?0J^L(G!@!6mKd3dd?C{09godAJA5Xs6sQb8atXd)0;k;n`1 zFQ*!>``~|ghVs&n5a*cx-q&ZxAP}?&*+=54ZpmxYu3}F$W-o5WNRCAY3ctv`JKyv& z<8GC8$}6?BXWu6bLiPsotcTh2MJ6mPb;>6idM)xt@|=H?YgZ{N+t2B5+$4=o+Bvhg z4nvXghW8y^1djpZt4E@AlMNCbkM=>O;U!cx4sws5ML z=;-BYt`<%YHarEa{ZHc=v4*lTWu~#j4n{=H^wMt`?%Wx5G|E-Y8#Qbq)8|O4UtYGz zR54#=7HbUK7Pl$DJoWt*qyz#{V)lQ3_cEe@xhZo51qIzvOz!C7Kw4iVB_a7TXXmv2 zd5Po#LX3WM<>)aT$!Xh}yV0Q5gVo7B>zcjpj*k9Lo(yRiHEVq^Q$FqdbA{P9{;d+m zK(eUH(2W%DwY@P*(}>2CnE&2n)0s#l>1|TV)bT-F$I*R$veNedye82b>A=Jm8Fk@W zXZ{CV2bH_@xZECh@23!2J~R4hfzFDc7PgqFJCtv@>yJVpcHfe!(9#ESKcl6`b$E7{ z-kn>8HY0*+fuZruudbZ1QkPQo8cjzAiFuP2`%*@BiNUGYL*GX;-Y=1m9Gi(`zyF(_ zoxRvK8cd6W@I$L)GPDkV)>M46z)<_fM9rc)frH-bG70``G+X=7yH^GGZqbq8%U%(D zi@w=D^n2^Epm9)4#=|wV*}+zP`cn$Bi|e%7Ge5JsS9%vBo|RkdZST<|u@LVSTOFcr zXSQy)kH2H{UN$LBexW*Oag_w0#&q&$@|9?@i*GxEUIpH$XdOTJ+V)_TV%&d|fcn@CHitL}VZ9ChHSd~;8W#M$-nP3N21 zuVgw5Y&|CJlM4x-Y*Gwm*WEW9YX}ITexbWE!s500quQ&%<`y9wzrMQeZ!zOnJlPQk zG1zg`{5Ohheiy6+H#VM2Zzz2~9J&P)zaG9wozoqwa)0OJLDt(E3PELaI#eJ6(Kfnt zSvBzr4jsaeI$`UjT-i(5JvZ@t%I4IVm-*(zG9|@P7yNMD+g{2!f4|y+B)-VZh7Ysu z#<{H89Ey1@*QnN>SYDhwCNPcg z$#5zG4`P~_SyO(VGx_Gb8`{%*8t9xI>HUT#=ZWi1?^Ry3Zhi73b%w`>Q~0Ca+f$uy zg(aUC+@48a?OA3w;#r+$^gY|#C$i)fXn7^)S3`@$y3Q3Rw@1tKJjI`NBJcbxNRRlz zN7l#~u+`!ZYJ^e9*P(pvt%+5dj#E~z9?AIMxX(WF=+>{ANz`jTdlCn$O>YD_zVBZ| zc)Ut3L*28*@q0H|s25M)31MC)6`*>1MX?p@I|POPS6c8zay#lB>iOcGLiYpzx@kPV zFZUn#d3^ce<0oZx?5*a$@n?kUPUM^B_Q$OW^P;oQxm_!stXwct=%>dqiTI<|&khM(<7wKeu$D;uQBB{cS<_4JP-<8@ z7*O_gk;mLZI1!#dyAaPV*O92I7E%9gENcoI`>%{R!dol?)$~X>@4an`;YT?K6VAqj zE4NT$day)J$;plU2W#ps2R|5Saj|v@g(VO}Dk_f|Qw*XB{=7Up*=K(xhNyoC0|b6G z9vl&jb@7V6N5+KIr-euskK>YcvL51OL{^-A=N78T4yN0Nj#p|=)t`8US;MXL(tNBE zPKb_1Dt)nf1L0I_5l^oor2J@cI1`g@JFVhN`6p~6q=L@Hu)@TIpPoJLj=jjqzl+2o zWXpb2=Pjpow_U*<5#vWq=O=F6SZr}n`5}g7kyi!P#QR<%#XiAOl$Y(1&*Fxqyqr1eY$gG&fgGwcF<_-zB|r#nHhbk>7fv!g_Lo zn2DLW#`nD8$i&XBq`toXj2Va1kNI|jW6*e_lz@Jgke}?JJZh%FyY_jd&G7K>!oM%8 zcqTFeV&aJR@8A3R`ThO-Hzy~jCrQBZo{~4m>TIL$;78fVN&LpPMLM+=@WW9Ks)>{R z*U;D%1uCw^LCywBo&_w0XkGvQp`o3rYR8e0k%oqbWhv?i3K|+3Qc?+IXI~#NA)$hd z%+gG~XLx(V_BWQ}{k0)!HAL{&-q>@lpun6HqqFv1=wc63)3 zE!|g`>dcHTbKHoJm9g={>nnGJgoF|WY>Py)2>Po&#tnh-2#Qn-g>6`ew4@^#d+udO z30RHkTUtJ^6_i2BDk-H2*y>tXX_tiThOP#^b%UW=B)s;eOnRT-6I6Z9#ARoUPs-3;NzX&az1i&`5E{bM%nJ!V!RXZ*k8XEH3&S(qb;?r3XkV+I36wJ8& z43`WgAKnx4@$vCK-VYBCFYF8DCN#3NJT}CAXk@TkUw7KD{FA5q@9(MVn{F41;z(Gy zG=!*$#m)8s6Y!g%bgLbV$4U$=M~l`bD~oHQRG4Ert*opptdgUn=NA@?JGK$J3g&!= zyiu?F(vOdJ1ohw2iWy~~XqU`$EzU z3=E_Qx;D6O=)ym4Z*Obs=-hdn_zS*%{&d`19tKmpDdeV1vw*eKAEj{QBE-`a;%GoQ zdzpm~i&w(q@Q1^_8&NV0q_4@@wPlb7#KxO$b$ueo3XVMcVmbNwE77;*#{!x&q$5Ne ze#ywmy=iRpb#QRd*MBea4Z9!z+QWwGZmXJyzH7anJrp;^{o3Rr0*rSpR>w;BM5Q%# zb=QCVc)t9Mf;-{Rsz*Cf~4m zeUK7KzK4KtBLcz7tzn9J+eRqnx#ShR zodA129mI}5HC>bAiL>7DzOCe|R}jIvNj(A)&W5JnXnH;f9&PH`OXFOKzXJi~vtXj+ zDOl4}!611g-X~YZS&^0}c4qZyR6R&@#x16#82ON|Y;g9;k z6&&3Gg?K6-NKL0%bn`%M?M(f%6C-{=lt1F#C1Ht@ zN5%x)KZP6_Vr({gA~wZvE@<-ZBEQThcV^yn5lLl=Aj3hx@BXlT#czKQS@{&7Taj_VbX4 zKy|;AkR;b$ELM721e3S%_uW9e$3g}6FX?b(*{15BSXGY(U(zlJ*C9gi^8cN^cAZPE zV~|X(&-)wkGY0|Zd?|tpO_v~*UC$4>>NF)Qm3wP3cpLlXy@iD~b|m6D9CG@T2{-9s zbM&?v%{uJ0*u}6=EI*16YC2x-L+))|A;;omYM<~~az`AjADQXW$nefDgx4#cj(<(n zz7Y%ypT>+H;((iQwxMuH9Z!!$)C9nXFTi4seNDg7<6?qOAxI!Af;@&3w~`3|Lw=wLcrtR^x#q`2wb%qt?Pspm7Iyt`_KKObqIbY1MJ zUm4+h;C(t+Bt+LYwugl<3!ueGsdipbFPG<;|B*MIv;1Ix{`JE)W_-ktYElifJ)X3=P z59JkFs`%6k*R+zmPu}HuWaA#p$h`bXHS5c_QL9Jc{QGa912e%_?Zb`dK14_)r%M6? z;XEiGu{l<1RPV9p;Nr4AR+?dQJYHscdV1=)yEM13Q0}h9GlFVSsls-4 zW-RUkLMRk~$iRR-fEGwAgH|7772`SQ@{1d4TsLM$3ZKt8~2av42e>7ov$6n;fRgrC|&9f-giBgv$4cXR^-W0M*o$%{h6o3qSD~d8#4B>~@ zFOPCr9kc(tIj5_un;4E}*TIY_>LWz=QKMKF;E(6Qdf`^ntnaxuOhlVqgO$AT@OM&@ zfbEF1`(|^V+Kb0|dE@-8nHM}fIDL)CIGV5=e__41=b5_Se0;ER5f3j;15}_t%EL`J6$*GSB z39vnECu81j7akTstBJpT;p^)oSU`e}_}y$LuBonWVzl!MnV6EIV{UF9PINbO+1kSP z{h$4{N$Ch`ol-+w&!K^VEqIo?y81JLqMRHz8+XE%y3?)CF|ON}Cg0xjT2Yff;62>0 z75243g|_y27q~xt@<}k@*!okGnv90Cii@KfiNv?ZwkG=C``o-14%NwB3nE6zH{-EV zd?O1B3)7ATjH+mz935Ghn4ZkCva?^lbZKZxjFo)n@FY5#R6sz$1>4p2peD6YEnod< zmcnbmFN^u2u7t}GUA}Kw9CO_kDfUgw$0DgZk454|JkR%MR}M){{>U(FVB)2y81_(&8K(tApx^4$1a%PP8L9Nt*Kdb&rji`uTZ7iC1!&HEvsmN&HGZWxu!F z3ZIqN*v(z=xldoee%;#I3M9tC?rzwtS4bpsX(+qzS7(fAZ*sNy5Mz|g z_k0Zwi4F{)wi^FrJ;^W85klj04_^N9<41RQcj5EO)ccN7Gz;cxPqbvMQY?8LQl>)R zyaCuts2htT z`=8wjJVr)FNqm+SzUMwJci&`WWOyWZ)owIA*VPp^P*-Yf2JOX})Q zsxPJp*qj+{+g_BP)qHp_qVX&Uzp;V6uG)NW8(mn=Ri!1-*zoZ|LdFq8tibKqNlK0c z)wFuT@azBKsYv^vqxQD9hkZS@2j{k0eILjJC|{;5%?uPhzA;*GG(6_~N9Wo76h|#* z&WC&%M79qh;w)*m(lFLysA3Yn)G3p1SrNQvbeF{053t{MpRi z&$lb}C=_gaBq7h>HVe!?t*ftJFrYuX($x8cvX;!7IW|UeG=5e^%_z~KI099biw)#2uR1y|zrYPu6zX@yXr7IDNiA6CSAOgOCKe(ieeSQ{36FX`d4 zyQ(jyt%oZ$R1HWSmehQxO7GV(ArLy^D3ROpazfVlbPNj3)hhi<0m_n{Z!hL0P^e;E ze&MR8dlB7s>H?$}miS+?3TYXn`JFj!rjJI6*K6NXNa7!^G%R<8GE-P?`%b$@r2_Cl ztf2yxI!BgB*c-CAUG8%%$uOR-UT9g2luW4(NQxhDS41F$#o(#W1s+-5KDyU z=X~a`%#PletlWU$BP%P5i;L?eGTYKZFCy^n5GBpNY<0~BN$VEj?Q8WE5 zV50!UuxXgOi|k7g6mZ+r*U%U>AIc)45rI*ECr&cOef#+FBh;6UGj;A=XpV@82v=uk zKDY8CwP%n2-Y`7aD28K9aG5c~mS)Plqm)|b!!tZIT?o^-74^!RBQmPBv)eW|&-Izz zMmX84z&2)}^%R8gC91X0cKSryzkao=e%8yqb$IgjrXVK!tWQ-po@|F|vkT});eAMb z)Is3%D^`8z!ZrI+ey7{S)Ix67zs7Dn=_2jB%pb>ex$L+1!9sR`beOJ7ikACBzNxe% zZiyc+`6RFqPIIy-DJiLuMJ#@P{-@cB!_S4=Vs#|ztkR}_p$65vtDimc^t`tF`EKU! zFo|te!x8qDr4llV^w!$pEb$^ya^j4kGJ53o!$Yjgzr7O45lv0Yf?qjBMfHlEt+TG2~66V4<_aNd^ z?>a(le1zZae$v3(w{II88$%Ab6{xDJ`n9#SCso+og*UWw7t2Hp^K zQD8?OY|N&i<~KH!ke$7~`}_O+X5qonY?-MQN~fl@Kce+b6&*;{R!ypS9JSe9`)F1? zzJ{@wPoTFpzH;=C>P+m8U;GItZ&q?0fnW`JN1dLZpAQg5Tl)sxuY*}%;8?6CJ~PoV zu&{)p=2us(evlF|K9>`+?2$FQ!{6+dki{l(qd|yIpyrft@z(;0I-2;D&wB6}uLuj# zBw_Z4!~qDW7)I}tt#6Kw={Ky&D&@>!067mv5CuC6MH96|%d!snyhhmT)N zN0KhZ*zr$uEgf-l2eZ~>9c9yuC}2aOuqNIyXV(@%BkFVF*3+gWdiou8^_r}DA#8Ys zVeg({o+?Rz&alCuvl@kfVm!_cE*64tuCr25P%2C0Zsy)>RcS{N--}pd4%!))k?CI_9K?O&G;`b5ve-o9<{I^5=|P|sEM++W2fCRUP{AM&-vi-WV^ zpAa-yBfMAWBl{*VLZC97a#|dLz;<^q`)oVg5JF6|4hi1O%uGQ+VX-$QARwUCuv8_N z_3m962v;&PV-4P(Po4~XHXrJ??NF$QbsINt46Wy!i_h{XPXC<9+SvP*xNSHZ=DpbI zT2xzm985%&^*E`*u=NelHEOD=-^acKjkZ=fd-fxMxUR0-?sd#+-UH7-1Jd4uQ9>?V z6MJr+uMt$Mf$QKmbKh6J;87+$iF_;?MRj&_&A>PTm4b8OWk`s0wqgo&9gN%1*?sS# zBOjQ^d=i$Bd!;+;^)jrWCcTD}5VNv?!LxD}6tc#71;DWYwuaH8a&+7i$1=JRq3s<1 zqwy6kg2vi!VVCFU=Sg7)g9lzm4C|Vy#x!s|jCv|LGNbN2LtcxGXzJhWIo4bYC9xnF z+6lrxM}G~o?cK{QFCs!9mZ zNNACe<+9&+j0suNDhs}*=1{oDA-m@zU*uW1*8uj3;FOArtbwUrf*{9a_A;zWbLDR) zP3({kv1)E{(jq|-G@FQpB_vW^;c3lu9}D9h%ViEynu|H|MF{4Ef99~lOw>}u|AGPR zE)6`q6$MOZ3EQmq!891!i=1Y{R4kFnrr~8 zER2o$UUZ-&!*G-K2Ku0fNfHW|6FoDNZ zF;(cWBZ@vz!|=_kR{-0)y1RL`orAZh90LPya%2oq8_019Clnst#lakh(cm^)a#B)K zTpXW4bKvzgYHI4h$htbAhu@7g$JP4oLz2h0|1FgFUS(RxXE`)9uWxdDb>Jorf|Hp| z+|1Zm5*gL`f^ULG#7hQg@8G~KriBN=Q@3x0grJIHEFq>KV~ZP2u+)0tK`Cxc00JSE zwGgVwN*)pzcyhAS7pt>>K5bs*AO%hRr8hy zDlCOyjCRZXoE>u0CZJ#7Oy68h&kZne|Q)_c0P1QV5(mf{ulAsq5^Lis{8BD_Y2 zGnX6dI=rS+ErV{rI%KU>B30Y)iExcsVb-kCtMSZQ6pv2GG{63=f5w7CuatBS9g0&zU<$~- z*JnS@cn0hbb|Nb)iq>wo8pB& zF8BNkNgSeUjf0Bavdk7u`vg}LTqXI6GZM}jq3y`TCfp46bzAYn;qwWL{mql*3lLe2D&4NY@?2vzMHAHhJzM zKRoq%cT)_rqh#jz&7VGT2T8zb-Ai%3Xxbg0u9AzSBM7?12yAZbQ~c{E|62>d#K+&! z+sn?vk|gRY;yE-q`MeVM`#ZJqk}H9Mk%)_mLDA$L%jJrdVRVat8v+4h=sQ+!iPmJL=F}Hkvj)pL+WnI;IqDsJq#a#}cbU%mYw|~l ziT2||;**7}ynsvDWmNyRU9PK>>vBk$&2$aqW$Jr&?`vwF(G=GJ_^9znhGYwE_OAZ) zhv!Et9Nu+KL`2Wjr3zC=jq+5ZH;TDtotg+e;FwznYBUw>>^6ZDf5Ck%qTSKS>3dy% zP7WQmv5<#_RCYkqOb&t5*TcJljZgc|AK7rD)Y=R$s}UwBZ=Buz6OTKR=#y>K5Z$7+=;f z>Bx_r;?|~%w?m8Ws@5(Nqvdp=doTUqoc8(i=c2C0lxwc-ZEfnRs!Jtgk5@*EHzvyS zM_8sSZD&)3sV7T~+5ypn$qio(OixeWXgoj5d6F@fj z=Cv(GllLb7`tU&M$~)DVG+S=Hpt)5DkGYTQk| zr=|XM@V$u%k!$Z&o;=wa)${e<=@Ee7W78>*GnX=FCskrbi0c~?{|4>C}}jk z_1GtFi&eJ-c4xwir)SJO-4Rs3USZ;Jt|zyq$?4IaZT;Q|Bs1_zM`!|0O=%Yrb2i1r zGqbS3@>A=30Kp&Xljum-VeB0P$w%e z=I7+unBQ!A@6GQxkMu=WSvh3yx*W^x5Z_aRw+GO<-Qo){L0ded0~o>}>hh zix7d}e3$KSH=L)I=CzG~TsCsGl!8@L|J4UVDCldd>i@R1Xy3aOlbXy>_gu&c2>c4bj&5wE`o>%#c> zTH+*M0%qNH9CeLOOP}I1(OG40Cw1kWSIoaIO}&l~%h*;$VLk^=6BxO?yu7qz#v_Qz zt8hEa+t9FOCQN(3?XwUPm`|l4c~Woh?3S%a1kVpg7I8X-$82 zM4B*@xabFo+sum9rUn{;wk)e636HTpsXcz+Fpn=aZAG zDP;HngN^J%q_DBEeV_9#23$LpPuz>S@A2BP*dum!vet+PJ?VFjA1Fbm$CN+*jIEsH z*`H792t>aq%!*VZ__D{^e$Z3;i;sv1KTYne33i%B2!JipeS|WzIVl90(QoCXFmZNd z>FuO-7cgtuZ9F=&kWd9^;*8^iC zC9c_f4|EM-BmBlo`e|9Cu26}Te(HB6Mj(_l&B#D*^)Gelmx{-01VwQp5Mr-d@eCTm zYM%WQeZ8X!{MyqtCW%1!;lszjrayrPXqlnr`(e=~to+CUdK?5og7qGf$poH(C>^8i z&c9`YQJwj<;?a>`#|8BqY)x%{&LzAq7{&PC&yl994-kkZ98E)%;W`YEqh<2n4?-rd zF%ywjf>({+Q3n1S2&5`dN+-fc-^F1e{MuQ$k);LTy(e9y0h~}xAP|4ixA6=_*rhq>lEh6=dRmeS7ivkcaU0_h07yQ6W-T2(g%j|M?uzJBmQ=V>l@@NO=nw(m2)n_glA>0DV>b_}(n zgoK2X?!)vOm%=X~ny%ojk+ADEXy>ZFd2}mE=7p)LhG&4V=l<8WHsF0~YHOt*J$hkc zvRvU8ZE@=tspY?QopYN-M54CWYd7#NUL1Aam~RbT1ST2al7x(d!*(K{C8W!Y7LC5* z<|%Lrw>Rm1Bt_cQ4kHCx024mMj#P!Jc$}RaLRV_v1S(tT?l>PW-^17sALVE1duIK# zqN1V+3j3XR(oRkfW}$5OURRluBg2lat*R;!(9zK;D=S;^$b)9ebd|jk1dG+jEG+qH zzGvg#zR9h$L1MjeIRv$x6Uaio$;|ws#$8lj?z`u;?zn$qi3=~A#)AB)|A1V=+PY^^ z^v)l`>w(~8#=evSHrirhO$AzI&pw-_H;lHm1!MFXJu|VLFDfc3oQ30{1gorc1#RJ2 z5&+!Z-d<X3D)+3@cj)r)0h z%yj7&Rh@HVV3E{#Yp`cMU{q&Y!2LxZKQi$0#-bd~&rTnPTrOSX;^Io;w+5<8L&Fv# zJoKV!3bDGrfu9ituaKmQ`ksXg%$N;jg=@1bDJq^FE~ZUSPvg^JIy?a9_Etvm>B>y| z-iC#(f-b|yr=hmiQ&`Yz56wY;Q`_k{7Ob@D8aonFzbAB;YF#%7f|?^EQF~1FPco#4 zC^`L3$7g3n`S|#_xCTZ>DquWre4gF;1@k5DJ=E65 z$jz;ygsXo9j`>|*YdXX0n)(zT!Y^A>6Qyo|i2!bIaB&UH`AD`){`py}zdvX3o0|V# zO2oyblftaen*8d2p2er7iZbi~hy4CR9OK*=k5Z0-w_OIK7qf1wJ_>w!m7Kh2vgA+N z71sxqgdHyK(?df;V^?8@xt!{&aRA8m2>rK*SU9 zY><#&z4|!$%y{Nw%h$HoD2b$ljtwD?w)3O?w_!Q;%fQlW(!#r_@41x}Os`lGh%lBg z;ir{&FWUUP3IU4oYj*%A&mE=T$~)W#a;C*0TGdjy7ZC`B@Pvy7J7T^e&CmI+2UX`5 zCgM_NQbQ=_GocPVJw9-)agO{o>+=AQRL8`Y%E#!<;rG`l?TLZzV)h@Kjp ziOV48px-6v?A%KPN~K#f9I%UN32F4IFiDA-4~ccg(1pgTc`QOV;= z+`t6hLR26;Xn^$&lBpLX1GdJzC0KLEPE8j=Zl}us#55Mrgd#L~FA+#2LooN7V7-mx zihzjw`BUtF0l)v7iobHNGMyCWyfDxC7nRk?$pR@^A?BBB$)OBOARe0;Gl{?Y7cZ~= z7qQ`t|E~ud4p_rud;VXK4aJwZjd_zFIc9cOK}~~rAC!sn&p7eQa~9RUxR;Yy9JX(iGotU~JUPm_%H&uvs57V()t+8DIz$ zlkY{L(G*ZDT)o;l_g%;q4Fb#P=noC!xlixhxs#3d)xm7HUkxQY@+v_PM5ITL9zoy& zUiJbu_QC3S%c7`SQxv^)mSW08k&fX0Z%|u1|IcZ?zZ`r?f;s}7n7F*Q_87^;#(0CF^0%^Q=NznPiqb&K?Sl-H|EB{%)>DT$>E*V6XF*QGGCbBm*B0u%{ozBFe z4cm8G#wCGThhzrn!`|LrBE6(d=lrnG7i4Wjn4ax*K8x=Vy+QW)aZi?vKoV$sdwVX= zr+3Mi&cN!c?xIx7S4o(gz|G-83l#igM-n|m2X_$ z;CsvZRn!LV?uSsWLGmdz?s^CNhLUItyEX)MIW+$TEJvWs>Kh(*o~p8kYy`*;GzG$5 ze>Vp*q@A}1<1xnR%%zQbrlkdqV#Ttcn~#>)*Bk7AN=ZvgOLPDs44k8uvT{Gv&2cOm zHPzLBer4vXeeEU4zyOU)Ex75*cfFuuw5$}ocP~e2uiSh{YNfgb&0zEjqdUbIeA_Lf ze?Ao|i9<&|4I#QF-W|o5EHa%tYU$gs^KErbNA%3eMN%@Tlm!+wWSAUhOlY(>K~$!0 z!*sngtO`#(H`cQE+=upqt4i||Pft%HG#)Ut%~qa)1VQupL84^!Whqs7pbkarO*?x@ zC2@(e>KU&N0?Y}Fw?^qU`lRriE8*}bhHvvfJh&-rTo8?cbqUreZo-FE<(E$!+zB&y zF=7|}58eF8hY}JU0hb=_sjhC>-dNw0L)TElw8%TG0{Yw9e*biByKn7-mZ8y0_xGgl z{n@>pkmhAI)eIwkjNi!_;1E4!=&+qcUkZN5qAC>k$?>>ClxCcTJTzPfEkf`z)l=%K^D!9xT@K2Rwj$)N1an-fkozDQu}>zjc$t~kY6<#_joROWp-}ym z>=8n0ou6*_IyyRr>)2$anXPs-Wnp3Q*1&C3QCEMLn0TF< z+GbTkECMJRdKB{H^fX)txikPIG|)JvmX`VXccArgw%J0QR{t1oh6Dx%e)#ag@fs7)uQeA&^giGmyUi+HZ?U&3^y> z2Y4?9hb|csr>LlC*`->$0h~3If&5(FN{K=ngx{d;_zA3_0v0WeAm$SQ6{ z>A%R$%`GZwaNC*(KE-CH?$6RvPIbViPoIELi%U(_{mJ9G{i_rBipjCDs;a6S0pMPN zpFRMj1hOR%>-H{Zn5+T+;_T|0cF*1rq(&lVe~paL*hJ(vZ(75bt*t1?cQ8=^(m8aq ze?sZYMji@?@>|6n28LIUKHy(rbXuF>14NXrT(V|cwQEa9H!ZP&lYdF1p0c4DG%T_gw3yxsJCz5LhLh~ymIAAtKk+< zH1xM_K|R5U?i$~(obg~`V0bbA^)>A9^9m~is;5t%>S$|YLKTD-=og{m;{N0bIbMbq z_4@jHuaCTmsp;gz#6KsvYZiZzm>R$-G_K9?=D>JPPfk9nFl9OcD*|}cXcAz3J!o&_ za<$oQ2R^hyo?*%azKb*z#51k2gN=CwX`A5`H!Um)}q z!GX$UMynhRYv=(FoIK6Qz<_ClX2-sZi>t1#26^#Y6^!l+lRwYPc09FnY%o*qcoqWtLY!y9w0p%ig)#(k;6_;k>&fu%v-N-GWZ4+cUt-y87)ED`(S z<x%(QGl*!^dDhK4fIz(?Picq~6nw1sFu@ z`SUWW_V#x4yLZ>u*uL_wN3_iLZ;jJqbY&_$suB zV^y4M;TjJ2f^5yX>_+`(GLTsXyNeU7c2oOpq7bmV|k(6zAb@9$qwz*?$VpsOP1 z$dNCwzq$Ei=WwhmD+}BW99gH>9h@X#OO9yjGtn{-gG^5g^YA=ni~(Ff zGNXB?`#$no@PKx9Nc+#yb8`u!c1}X{@h>IxblK{<`iA}9deN3?)8fvSseU4ce}$eO zsGLd#_zt)yIaD`{_a5>deEOY+6VmcUm!A)qROs=H$1**|x$p&`6h0kc3rb@7Frm=W zD1~#8c6WET8-#bW-90^fySu!G!tYrUJuB+!w3*{P+}+*X+?3^Zc6P>H_SQ$HG&5Be z78gC;-LGX@xdp>R3pGpUq2{x(v9SvM{kE&8XJO|MegTsKb0-LrnAY{=D5HYgdDp=p zM7)o}0Oaia?g1r9aZb)A?2p~O#566JV)OOKa^||ascI_35C?lLVW>VUJ@>x%O zpPoL2un~yLRB?yS*Pia%(9n$7PgIpQ)6X2ch2vvSPQ22U&8)52$Ovkk{x~f4(^Uww zv$HcZzd?~+z3ROEOSW2D5qaPDOen1n>R7%>T~DLw{^1;|q1{#rz0X$T4j|+Js^SK( zS!8L~OT0;FQ$duZK%x3`pJpG-cyfq{q^dBZyPlU@@HzbI0B`hkb1N}IGG#)Pw6nKA zJUia_PyGg$J!F~Ga#g`asqWeRh!8zn+SoWBQWCwkb1KFLsy(>S0k*esx&pQzJNV(l zEgG6M*I94ZFaEe7(doG__vc$2i?_G8@87u)aG~Ikkg}inzz_qFQyC=zG3CT}!pi*5nALWC!d7 zt%wN-5=DI*M0B|X1wA3hfqdi0ZTqOBqXT*QGBuas*OS8?s44u=Ujz(o38Ct1VP`K;b9QzHuoa%lPWKBe5Y$V1vf>E} z?~;sJoP*?U>I8|92v$t~Dow#dxPgyVlU zTFK1B)Y;YL**2&F!luN==eeqhNlBsk9A7Spq5Ko#p^pq@2#|W^Gqs>@f>?ZbS~Pwu zrW1%|NR%_6U4|M|K%i2+@Y!^oI}b!9Fj7b_Fh#h#10`2tpO&gBuhWt;)Eyw!g&Sya zXGn5qM$hLk6NE$%+(=&oXQjHMpqnP*ZD?eqQJ^^oegJo%0HKVqSLNj`XGDlV!cR?2 z1tbEaLK+D~<*G81heDIM4{oWw27QgD7u+Yh9-(%Z%&_z?2xZ-l*DB#Im9UTyq>=9$ z4zz(`R02g%B*FXl;A?X;p2zeu9Rp-nA)zTa6rg;%x-7>(!HnI3&;m<;@Zf=mhsU$$ z&kvw&{PAO8{R${NAc7iiM9VPnnD)NlxG9afw+0e}D!4lZC`Al zSXo&CUDNO1CYF}a8q4=71XKHLJ*mz>S`Dh6qoYq0_Qqfs!QNe9aAdC^t$r(&etW}w zI4=%@HVju@?hGNa#DRe~tVn$WkMmuc&(UiX^eFb{LTDz&$8$$N-U(Dlc@TGcc)Zbg zC$L0=IcK`rMSW^|x;>l<9T{0-+DC0`y8-#%_0OO&1PX{>`5MI)pFSntcVs-)bh&Bw z`q4HF#(4WSBpBR_7i(*3J_%FqcLG>jO!M{Gnr{UtNObeTcc_qH^{e%Bb8~QJbaZr+ z71n}|zZJlz)&wA?%DfaSF&p>*ql2*F2?J?RaPYSm+z{|F`uBQ|!{cMvCiuApp2J7| z9Ej9+aQ&_XRF_a;fc`mM+3obm*(D%8p0ZSPbz`H^pd}dg7{Kix2p!Nm2CfR!H6X;E z55&KMq=S;!s8UHu3FMR3gC9GgPAUfRCo~Y5o@SNU?gLl?yAl#Q0SO4G8{pk#rKN|v zLrTLNAHeJ|uoCbP)EAKLTp+*5{Ta^JfJK8c1su(Grmh<7xVZ5w(Ypg)2$C>xbNdUu z<=!+=QIB0Si4M?Xjd_k=syRA5WYaA90eFqbsxbv;mCU*9U%1RoD5s+*k zVR*x(F_;>334_mXzFd$%Nn~VSZU%fKC|C;=rOBrJ?kx3UBzij;L5N6Xvr>}pX=!=5lLc7+SQ2z#@<^=`{Vy^^F!!Mw_Hc3aEKClX zza1j!H9>ugwY9YZW+*G21#`9^zzrpGNF%qlw{HcO6vwBfZT$S1v8U$X=*Xc{^YKve z{Gjn1)Ij_*5Y4Y%xw6y6(de}N9U`M6On;Swj+J#4@X%d0HcY_|jRL3x;U4SCQ`xoo z+oYW!TS!SxW>iXDU0IogbwQCie2m5sL;9n-!^+BbveSHOpjNvcb28Qt^*gc|G>B-g z!s|K-fd2NON8kNVO`5G!<0M@=7V*>-CY*|hERAL(%b{s2l_|oxfJO& zHhliP3-zF}G`4UpX<=*G{lv7ib0`Ys8KPm2%G%|zmf>%0{5FC59vB=w@4Yw^dQUFP zo`+c+jkH}5TZTnNA(%lx{3is45?zfJg%P#|rB%zL*#l8M3EMzPawil3INtM(h8lFh z_o9+q<{(HtATo;lybAj8?D3U}TGwJg=Q1)#ItCEBL0AP3-p8h=r-!-kuNLKr(PA(H zm=?H(4*iF^x;jjXEYb!INdi=1>JBG=?R#c+z(nio>UfDKEkMr&B_0rrQcS|WXD4L> zc>O>TIRd!l0U0Y`WHWxut>cjq7YGQT|E_{KcLxLr+bhjCfI)#g!okK?4_PkBwgIku zgY;2H;uT04AuvMnJcV5ocH4wXc^P~i$_`WMSKp-}E`#&(LDYl|kBy6KZEtTpKRbpi zPOGSI9v&dWgKBAeXNQNI8-%-hfCt>Sf7S4(avRGzJMTeq#T0YO%A7C&JQ!0y3qLv; zH%%jv2qwJd@Von3uIdqFj!b#zM zmG1(#>HcRh^YYCHC<)B%8K*ykz(E83JT)~%M8%H`lGvYp1-ksJ6ciEHco^B(*x*tQ z=;xJWWa7Duf{%8si69~+gD?{!iUK1G8{6puS`SHwx#WD#L4e4RN*{O{ab2zQF4>zO zKlDFvS;VT;7_{I)mf!CadGH`nI3nl11x8)I98OFlqJWfDP*AP(0LS6_?t;H4B#g) z9Ny(%*i5fOAK_|^fW!9}^)t_5(eM7@zcporq4qUfSzz-^b$??SdU zHZF%m4<`gb^(q-z$xct&u~@JHWHqKZx#!T2O_;ZEU7vy^3*-VmAz^|2C=lZSc%dzU zec=MH?^S@afByV|&cW=g9xHkNq#E-u=I*I{a0HQiD2p5xCzaG@@70e8?mW<57XKw nC&KjRn?Ar_9{E4tPer3d8*T*Jw1CnCfsmC{d{iW%@BhC5ib Date: Wed, 7 Aug 2024 12:28:50 +1000 Subject: [PATCH 19/30] Update paper.md --- joss_paper/paper.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/joss_paper/paper.md b/joss_paper/paper.md index 63d83965..54b02c20 100644 --- a/joss_paper/paper.md +++ b/joss_paper/paper.md @@ -22,7 +22,7 @@ authors: affiliations: - name: School of Mathematics, Monash University, Clayton, Victoria, 3800, Australia. index: 1 - - name: School of Computing, Australian National University, Autonomous territories of Canberra, Australia + - name: School of Computing, Australian National University, Canberra, ACT, 2600, Australia index: 2 date: 1 August 2024 bibliography: paper.bib From a93c2d33b4097213a1befde27d92b3d996a0a8e7 Mon Sep 17 00:00:00 2001 From: JordiManyer Date: Fri, 9 Aug 2024 12:58:42 +1000 Subject: [PATCH 20/30] Alberto comments --- joss_paper/packages.dot | 7 ++- joss_paper/packages.png | Bin 25269 -> 15952 bytes joss_paper/paper.bib | 14 +++++- joss_paper/paper.md | 59 +++++++++++++++----------- joss_paper/scalability/postprocess.jl | 39 ++++++++++++++--- joss_paper/weakScalability.png | Bin 17548 -> 30779 bytes 6 files changed, 86 insertions(+), 33 deletions(-) diff --git a/joss_paper/packages.dot b/joss_paper/packages.dot index 799f3141..cb8f560b 100644 --- a/joss_paper/packages.dot +++ b/joss_paper/packages.dot @@ -16,7 +16,10 @@ digraph G { "GridapPETSC.jl" -> "GridapSolvers.jl" [dir=back,style=dashed] "GridapP4est.jl" -> "GridapSolvers.jl" [dir=back,style=dashed] - "Gridap.jl" [shape=rectangle]; + "Gridap.jl" [shape=rectangle] "PartitionedArrays.jl" [shape=rectangle] - "GridapSolvers.jl" [style=filled;] + "GridapDistributed.jl" [shape=rectangle] + "GridapPETSC.jl" [shape=rectangle] + "GridapP4est.jl" [shape=rectangle] + "GridapSolvers.jl" [style=filled,shape=rectangle] } \ No newline at end of file diff --git a/joss_paper/packages.png b/joss_paper/packages.png index 56721ab864ade953442df62235ebe5b47e78bef3..b2987c625411bb6b9db78f596d70250058ed24db 100644 GIT binary patch literal 15952 zcmb7r1yqz@*e5WAbSe!xlr+*XgaJrMNDfj%Bi-F79RdO(<)4xmU!}Lf_08Xh&X1XRcz5wMR2~@y=5H;8btL;;4erg#tb5@qOux-*fZJc-iRm$5}LK>wZZBNt_XGjOX z%5^I%&D#GOZFLwMm7A5Co$n}08&vp}pX=hFBr*GS<80CLF8Duv!Ub=hh4~wN&{8iF&KMHF*vi&_p{aNPC0~(mUf6*Qw$8IyT!vqOH2BSE;&A)@G+^HiVDF$gj`^7lwLF~ zZTiFeJ@N7Cci+NPRJdvW%kK2>ZB%!pXrwUeI#M@^8MPD1%im%0)OFx{x`v4I7t^Z? z1|_armR=vvEb1*>aOY}Enux8H=QtLLKW{g8ifXVfG+8^Kh5~zsGOiC48QGWmq`XY3 zC<9a;N;E3_Av;WTS!{U&b%|Pgvy!W&P_r%(|mqWrasg@seA z&{Av<;p)9*x@Ep)_hlZOYa@eTqa?}L0F~I#3l)FM7;dvws?S)|%dw@yNcza9kx!zq zk=8F0xu*Rxs{=&?&K05BsE9mjiUFK_E7dv?>}Th2|?kneL}dtdTBWD z4>U!&M81R`zADwl_or*&XyItr|HA`4)GIfi62Vp`jz;9jA0Iz|(8~Rt9D|6n?mwJE zIpCgwR%X+-QwpMLh??X_in5k{+VID1G-?S?r~-*jOrryXVY8UU=&d)`ct^VXA@(kp zVbH4jaGU0wyePRb?`DIpEfRs+Ov9Xj>7KC@W#T4$Wt*~3R^0K2D)7w{zU2_OK#*11 z(m!~R=OLI=OS7mllt*RYuYye_JWl~nv0V?oBgV)(D1MF+-m@A(jK{rJzaMi!MIKso8u57I+UlO*d&}A~Znh=g1o_QjG_5E<5^u$K#T76?Lu9)L| zG$_pK1HQ4d9LtF^+^tf)waLn%#+UEiX`oD3P|d})jia-hVs=r&@z^vz1alU+)eS3owjP`aKS*2JRJg*v8>GLS$*Lhy4=d}NwX-FM(aR}@?AZqgB%&^P1vE-}2T5uokzUTd>a`;TcK4j0p1x~r% z%`vW>@PhY89>+-d&-tf0EI(|@B8{NMbFX>Q^}(3ngNku=d(Ox&%vTK$(ym8ZyBRzH zdIpY{)$26dc9Nxm_5F;g1t-3KxFB8d#GP~!X#F<99~?*!medrx%G;SQGU%U;*fF`w z;r^M*wItrqfWd~O1HDP0(Om^Ey4##HT>2i}#2c9QyCLGZ#jv-vwfoUu@kad%*I8@y zBhF=?;T&G20FGcG!#%fk3698Nr;G*Xj0h&IZ3?;><|fKx7*?J*h`hAtossQQHYSs0lBLHiE^I-v z!alrFHyfV_viloJy~~FDII(-6&+4DD>mi*BD(nyOH#&s@V7^_>6hLdVocSA zy4gn~Z(fHDuw@UAjb`{&9Vp2dL%{S$XBC>HYTQQ0Hhe{o4yT@tgjcqH;Z_f^=8;1b zihwhqlyEy)LKhTuF_-p_BQ2wJ4LuQ5d1uR1YlQH;r;7NZI=p7h2$#VbK)lQRazQi; z#;tCKhze9h2K&-h&a=1*aMaZfg?|gree}8ZAZP!xC8u)?ZKZnW7r?a$`)?eK|5ty6 zkxffy0|BatE{V)SOL5juTcWPeJfS1%+xlI9GK3{uccLuJ`WpGt=|@=5Qd=had+)dA zN`ZNgqU0<`x?5JsP^H>}^{MzV+!nfJB(QKFM3kIF^$qU80@!!)Bz zAAbGxbgQlnozN%p!Qu-teyzh2-BUtwA~3HP>LYIanttcuAEjox6No3V@$I)BsYLGh zi#+@7I27bmgye{d@0pcFWY{b_YV?gQ?a)ig$i}zIOfu9`;A``Qfap2WWCAKu(ND>~ zG-B*KOIVT*_JRMLIt#2fA6va*O;z6xe!uSsheT;TPWfU*{T#{NaNFgA6C_Cmyz-pz zf*AUUJJ?vcv`Yor{SDLX(CSJdLWyy|j1YWW;m*}lgPlqX=#>jin4E~<6HM3Hgj!#Y$$%Ns3 zR25vvHC8u{^nokKFE!SrB($d26J?CeYgZq4AJY%9hw)o=Z*TiybT@Mv0FhZRo~z~5$TtrppYMY&B*P@alf7g(5UBb+DDI(NbP2?^*=Oq8}99*OKu zCDd;nZ6I?~e_!2sZyV)Uy({1RX_qQJCV22w-L94Ja5H^pw2_x_|9BK8EK!i&k6T|n zK?JEp(81B*$_20a^+fVSpPL^P)xKLxwV9N%oLYeLB#?HCsNJ#aL`uw7$x}k(*-@4t zon5X9!lTbDSQA@|s-p5UH{NdU=DGt(Vphy;h)b^@$~A5Mc%W*$mO@q}9d5!2xi#R! zI)i4$K7>E`WI*%C{#|EiVA^ElHLzK}v)#gpqro=D>qwrf^SwGPx2EybeD!DjmL0yZ2O#(T0aj!@>mqbwpcIQ9m@yMF{EI>%i0! zd(n~q(9c5Ug3z0+DV(KDlr8<@`725|+Z$MJ_L~1?=(qU?{1X;&^m4IZmcOi{*OFV! zU*G=pg0#@bQAAci8-S{;5+pm^4IQ->RYa>Hc~J4rshEBugj+^dzsfNJcUQGWFW_ zltQj6`%K<5l~Cz~eXd*{s48@N)*IX)b=(uYpJfBOIQvH>Q)j2-<(VqMo?L6dqsnJ% zRH?poY^RblT0$=%;FR+bJ=SM!qkU?Blj74hZ48qCl0x3F&D>&bu1^QH1ETN5!+QSY zeNU}dO!~v8kCTKKY~)K|e9alcD?B|*pv7y3NMGSX{&}kJ#hvZs4>LG5f3&TOp^rUB zjNJyWl#C!IezL5t6qUqtzZOM0Bju+<$Bj`XVO~ zD5vY?9~p+d31`YJrj5ZV6urblR;!yCUwK44Ac7tCzA37fWC;>xhe znPjybOMh*mD`tNhaO;N$ZC~hA1`sU`Oe7Cv#p$(WB;|gnF2Ur@NR0Vv z;Z|`Sg3>YWf0?pFaX39{!I~P7bY`cUc6bXtl;LI|7?ZJTw(fjS`VNhTkmBQ%`#voR zylu_q_7X}P@39E?-(Y?1YdlM8NW5g_YooBR$9A4}gL+)U=G%cq9`W4odH+#+tCRzi z0LQCKeI2CYXdP7)v?6*7OF3}^1`A=ZmXNL+_yq}Jki9<@A3aMhIOp8)MqP>lW(%pk z1$h(MVyoQ6_r3O;{C+J?0P^86KHq^7K#g^w{h?LD1F<+T7=X!ZVjm?iIA_A_4W|@u>1=VLo?)%G}I-RLxy-D_7|l{3Wrh7|TmsOW#AAM(7O_ zgnFb8pIEkk<8+$>4+W+#kK;yeePrM|{hFho_xs6BYV7v^Q*!FR^&M4cNXsF+qRZqk z@?B*dnU6%|627*6U|lLYxsWTOUd2C!_37YYD z=ua1N@YC@5Xd}ojCi9P>q^_VYl57H90>z4pzydE!QxR$Tt@c|p|11B6lV4IE6v306 z8yp{1IgE5b{Xtz@&*om2rR^-;#=z>=GZ1rkJVd^AIXPP|U6SFz0owGU+ev3=qTgx+ zTk#j$Pw{g=9{6symIj0hpblaRR4S(`s_S8hkm)!#xhP?4T-wedjlacVQDl2m4x%AW zANQ5)NpmvjRHA0H1*VNDBZyY*~U?H4IV5mT{) zfAyq4MXOKu2P0P^awmw=*nZ&wWzpPR>osLlce|3myY7I{5-k&jYk!sGQcAEptG`Vf zh2T#+_U@`LB7WODB`d@Q>b{o|0XR5JB0J$;_~p+<=!{iT%{x_;DXG#&)?J|{+bp~# z{5m{kEgD_wWj8pLpW-fc2gX3ahaN#y{a$t;XcQYVjaNG$jYvVqHr|4e|CluWwYMJS?pHB&s&$m)$yh8u2{Z{&Jtp^PaKr zF$tVq=y|ex`><9|>}{%UO|g`M^E!o_v|6WdYNBa5=Zww-F$R&QwQFi(%lh{t^zx78 zL&0Thc$APa^Gu+>v~iIsxeM2yH1-RR{XlJH94g?xaTg!hbHh5N3UCuTWc<*sZ1q;0#n8XIb1NZMh9h8g7$L!I~^t$dy_S4D( z(l5W!^?J~Uk6zBDfMMfMg5OHCNZnKT$PegSUDxY z{GrkiA9Ess953-e9b}u7jA55Ybp)w3pB*1LcYhq&*~w3Cf=&+#!1(+h3i`QkpSf+% ze{5SkFzVV#s-r(mb(dLRf?=S=_O>3YirKpTnsAGC0{CSkkNRr0Kb5GAHS3T2u5lGHVLJqHC(cjc4<8Q#iq$&GpvY@FoX{?}E8iu} zjzqgk^o8PrrKcq(brv5?d#vazD-+PEl5^r*c2an%b8h2sMpjKH(q9#0;Q=|`D2IIT z1twBhpQj#Vaf&9yC=G*R;B2h|l8Ko7q;!T&+ zK~T;MYxHqvYM(7^k+`;xg#;_>1whuO^nMso)ig0Z@Jmrs(qvr6HI*a4w#t=5s4je=G3r=h zv7I4rkgthEeMc54G!R_k8vkM@WkZ#320a^^yroiW_e=6eved58QD;+sP#Cju4z?;v z%6LG8o;^aK+p%hi85c>{K2HTde*RbX-SCPqYHXSAGJHGNvkJ?OlUc~^^U#~#1BuEv z-_5co;t62MA&ka2!&!v|cT;-EJ3a*!WH^n+x?(kV#ST79{N7wX^ zi3P(fxH?CB#~$91ubYc3Q1vU>-@9vu0B!m%6XS$_RlcedD?mterxh#taLklA22Odm zBMSUQW~v87D?>M|K)SHQLkuGLda^<`_MiH|5k#RK2m$I@jq!&dX{63pDQhBPS)_3C^M*D38R7iaEt zJI`pXHxwBa5=tmYtXZu}-Ac%=%agnR);apm=Yr|KgT>zWNHzHOi*V+>l(3jZiI89NAh_0`-)s2&gK!`Gbb1}_ zl44O_sAL3a$g9iMhlnHXaIltK7@WmQ{g-_AQex}X0UC1LR5^e%r=8 zgI%PrDd>2ubm|mS!;8VEpc0l(XY&$U#}X)j)aJ`zQYLi9w?)MCOFZn{ zeVj>Qgg38KDiPm)$w03mw5{anrT!psEjc6}XE%{Buz=rv&j<4@wKPdtRZi(^?hdjitaT8h?f}UrQMm}+yBRReDDux_Vx-h` z35Sl)(`Hc_qt}sLf2g&@M<1cu(f<2HVg%^%pl!m9y&{lcJoh`F@0{i8+bxCaL-tg9 zYZpsEe#|G})A7w1=QbLp6ijyx<~dMKLij0aJ}c|ht5J1`rU4E^HYp|2le&XrCh)YK z;v_?n`dDRx--0km>7F>vgti!rD|dl=Qg{Ks97cG@wgU+PAv1nRN-m2p`Qp3xw{eBY zu|V+UMozCK!I;ovR0;jia}ct?Nq9Y=>o=hRp2E6Y4i_# z4MsC%#(`3++bnxooay;Lk3j?QOo%zs?dl_}6$foFziLr*qqHy6)VWEX5uf!1i8K+e zI^Pn)r@AF@pOY;`JIH`Jz@}GxIehA*D2=K|N1QlVVU_|T{i0k7oX+-*nCb2EX&@=F zYh3&1oybssf3gv(&%g45@hjNXB{M1qY%DTXQ}%*6Esh>>+mjgWhZYlNj$(U_1^d@^ zdw9R(r+l3fYOQ1Tx7ouIj?5?cMKRKH8fC-0Cuq^Q5zaTKq=r>Hk6gp}D2ieKJSy2o%KvMGAC!(Uld%(jL=y3@x%#Ug)2~hrt=dnqrI2 zu-Ux1WDES~^}U6tU;)cy9hJu)2(sFS>M9_#a zNcc0Qs=)JCH6KGhQ>myV(>+_?1nm%vAd}u{IGhP1D8GvGWjuHb%a~-+5=%l(TxXa% z+c1uYhyjCeKPKf35=I7%@Y)vOar<1bpi`?JsEQ}jJ#&_~N%;~G#~zO}#tS7=;P40X znT2%EHqY>Zb9rDMe+XMU;$6HQ4bQtO=GR6;`7A&WaIPk4xieWLW8_ZM%)f+6e+#Rs zr9#j^H!p<2FF#ZdYAB(}^5at4Ta?$JDk|csnqt!dcV>P^XrQCw%Mns!u|1ts|%XymOia1lr03?S=qF-voDClH{Y_cw;+ZflZz;GyTy zqyt1yBhzsJ_U{eCx+__Z56mCM7cDUSrd!!?%wvo^k36*|$>1?gzwnUa_>K38M4IHz z^D8E5HUhS|0;YsgiwM`%N$uvl?xGnT11M_LZw5=QMY2dpWJ52N%6?uROy%MTT*<=} zSsg{Utasf@JEZ$Q=~N*+GfO%4> zFQiLW58^_QXHrr@XwqFj5_0J;I-=R^Wk8L9L~vL0_rx3iHGy$1CjvX5bF^sy$G)BXGaI^&johu(Efo&-r-^ z!=mn?jUG3@VoS2S%fpm@vQ4mQL-M@Ri3tH>Efvs@0@f3{1f&3FVL30dH}s|6g*A?tw19RHmS8A3}(8PAnM?hdW)bb)s3 zzqt!|)LsPYBcO!=4>K7St4rudzHsMot_c;@HvE_|%unqiJl{^UnVr#CtE`xvCKnlX z-g`k(S<7f*NSUIY_~hU6 z8?Brd=zwm5z?>6}auUx6f40^qZ$s`bGPEpkFE|ttv4hh$83j06`TO`o(60O)NBpCL5WJBnHIyI6$m~Gd52Y=koFVZr z_IdJr&?mARZl$|P3~*4J$MWRVqB7oXn8-HEd}lLZ`Zw|yqpa3R^kbdDY`oZCl_C8; zf?`?rc74?yZwSU{iv8~*oa%b!*v5s?fi`0i@4_^b*>0JcH?4Qjn?e!B(c^!J6_R*b z>EL--LDV>1-z2Cy9Ekk})l_zhJ_jUyWSAC-KGb@k&^ij6&9FanTTg+y?P{z-y?_aJ94_N9N4prts zXR@xacOfV%Jz6lGo}>xnPx z>u(g)F~J_ij0)EgT0rWLu0M2ioKu1xt_4B=)QTLjE@+qb9FMC6n=L#p{d&KR`)!S} z^%m?ju;yJc0z{*sYNRRVQZO5+O?kz;N1tqF^DgZ;Y7G1Av>W=etY&auVsE>A9$zpk z^vP+4KC{24ck7&+NUyabuFo3W$AVsan4DN zHSo`LUqytF*>tDd+daOrO<&n~}rhHt2wY~74=Ua_mcXTXa17VA}Hju z!m_dJ>JmNnaGK5KlYgeJUwLtEB)d5u-beYOIl4d^7uU2#I@VoT2B+ouG1Nds-+6&9 zifXh5NqkU?4orHFIuYpWh&s`*scaav9J_`TTyqYFcjrPGahFKCksfrJCm+pemj{q* zc9ZbS-$9dOnP?R8CbsT`WgM8-n!chm@4Sn9_t`nb2+x$cNssbpx{&XZx)NrB?+Q=ol5d59Qcy1*p( z`>^F5DeBQa_F7!sSbunm2uA0-zL)&bIVY$8-Hd{IuMKo;mVPiShNat^0~5{u?mD)S zJ{hcUWH3E0N0j&e*MAg(tQEcTI}YPyt!TO9a%(5b&)`U$8k4|y6QpL{Z%gkADjY`i z#DEZlO|&kKCqRKmn3W9D@3W>Sr)och8r#IGs5>hnbUN1GCiRg;S32+xc9a$pFP(FN zkC|B_1whl#C_sAM)z~yafU_X+lGk!ILO&~&jsiW@7xVHYACdoL1!)5&wYMcz^B%W^ zRx-c?UNoJ^80(KrUpk4W&xxq$XIDdF?b65*E~n<3^WAscW%+aC{cy4m0GCHRoi5of zS{OY_syI|D$Q}ObkQV|0+Dkh^-DA>WW!03#Fp6v^74BHTR!$#~Nv8~3-!R}MVBM-5 zsicY0tAv8WHxLa23^J)zru&bM9X(^l6LFg#M=Yq>%oO9_-yyzVd6tgN@_KMUL zq_`8#;Elg#k6*h3l(|NX_SLdaN#Y*rPcgL1Z&)ElLX%F^0LOvT!&<_<2ynmn+wf0P zzw}?IgaX!oC9MWcG*>U&BT29gzT)9gsS+{~eJMy6D~IX_&hd&vTLeD7QUS)j9_Z#8BTd^C4&|>$XhP)AQCWBM`^obzI#^b|ZeS8v{ zo}58S`s+0U^d?_YsdT*H$&nOrj#)a*jSE(UCl6glU?2kJ1%T=bO074O-@XV4ygkN^Vy@eh@sU$(0u$*+N z;dWF5lohBlUyEvjHvja$0R(peog_l6k5JEV=ZuU1K+mHaDNm>@cD45301onQ7d{mk zg{y~6_rd*PV< z@id$xdEQx_>{>PxA+QcEQw(oQ~xhrP6sXmfE_=j-Q`Bm zw>(&bhEv0D$q@rhbGK1c%W`l=W9Wm6o9hAb>XABw#P_Kv{~U}D{?I$R_2)oUh5Ml} zvUT*cCz_Udz(W+wti{8mQQt-6)|9G~s>Qoo$7Wv5pBtEWGr`(84^f9pq?U(-U3S@S;d+hnuH(!$d5ywAA( zRK;^4%Z_*T_4NXs9#u<brCAX!ng6Ab_4}$jBywh^tzTKVFRo*+C zwUUUq3gy);93>a+pwWMOB_Ja5D?>}f`gDK2ol}sX-?~5XHXMqK-di0!*dGy{Z1cm? zlrU(B@-Xc!6kT690yg9_qHu!J7a2PNAx;b5)89v_R zhN*X*tH&#;Ep#jj%U&PJnQZa0t?j-FmiNc0FSPlc{>t+RyuLinm3&D@L_}mA$K3YJ zDsAOvo@o-!P}GQM_su+kXMaA*UF`ObC@ysb`Mj6$(D|rS@HkP7Nh@0je;G&;l(@{+ zwsU$+SKS7#xD@LaDOtbfrDJLAiLc|;$-jl_XS6cE*UBv(uxQ~Ice+MWWEQ_LJLdOz4YEIN zc3)tU_r1TOScg6%gvS?j78Y>uCO;gs1qhc56^EqA!FnIBz?OcNI z**nxqf70o3_x0DpkZsb3$J-0J(rzpfXOl%OzuvpZUY)E4C~w#grm~xdpRTXdVUD&i zrU)25IituL!29xu>#M@LEr56cx5W&~pZ)ydQdeeN^J2X);!j!5eIcQ>e4jCVoXo}L zPQ>_BrNM~s@tk83z=eR_n{SL;v(d?-CH806`S$7%Uy)EV?kQ(jAMmJreJ|Cz`(sYK1o>9D$Li2O9?fb-bpHOrGoM zp)3Lx8qb^mesA`$3cuKUx)`{c;+3!L$Dt<*pX6J`b1ogZbp`;;o^JhU zd^uT8$)%S3Az)|GYc=blR6u;s2w2e^!W!f<1Gg>;JK{e!_njWz2|tE7e0^{9^RNIC zpwreoJ)9-@a;hTTadrvVM?I`3+B;3ywEkb^V~6+$c3Ej?XuKY5^ZlcLudrM@?>iuK zU1yusWFwq`%TW|;vTve5Bm@YDuPgvee1)4fv;f5RH&F$kE)kHsJb*JMfoa?#GYq{n&EZn9^;OXODKxmE@QP611~H$Cx`lX5v% zA!FjhBam`0|Hj@Rrz4=^6wKnU03sosCcujk*J0VhW@<@ygb7YZd9~l~&URmAeg6D- zF&JCpJ#N&D*#}7745`HH0ALLc=uE($fpy;W@87>KGy|s0d&>-|A+s3l1kk*$W1lQw z-iri~So^(|e)FL9tiOLI2O9|QpK6n;g|15rDIbhVgdWW~Mui&#uxi>J9`@Rw4Y8<) zG_JG!8h*YtXk=oc+Tif@zJNe6AhSl*#{I&=9hXO#+wv@M%g*}fn#uYWf7$X4%{PxcilJHBcmCvk2z=Nsi#Q%4KS(UCqGFFBd&sg3HU`s9LDnH0A7?n=IU^ntGAyANDen|%>Wo*AGl{z z+xm;ee|uhIyihSfRK|NvT`h&Bf4;_7t$YUkTO zAYf<7?Hy}MS-?(JRaKKY6yGCl2fcsgI({$JpifLptgvX82FSdU(|y5nqXkc5DJnmG zgxVJ!^wH~j_7E{5mcq}U8yW)JV(Ym)Ub1qR?8Lq`(D^wA*;zyMQk7Ix!~@Mm;g@F* z{BdY9TMhF;HXn*Kh5cKA#NPZGb~G2)6?VQqWuTjd&4vH~ua?TD0a)>uiyd-}&eO#} zA3%@2f4*iExBg7{qsgs3S9Cr#&z&nN#N55bkunUt}$7zEeAL!t=CuQe*jkL7u~qD zUchYLU+%@>t`t)hkDxd5Nhve`W-6rRfyVaai)b|^^BuWLz&qu;~2#BdUMt6Vqj%49g+w{fLj=pt1xcED%3s`)j z$QUa5`)f|z>{G;C;HdNQh^vNnvYH#m&16W!-vGFC{Jj`(m_B(d6#`y`@A2A)42)qk z?cTRW7ZZ!-zNa6H!sUS{7;UZozViR~Zu0*vcKBK@L<(qZ>YvGvaqO|`MA(@catI-9 rWh`Tq_=4}qGA1eh|M;jMUMLUH{*KdE?XCc>h~sIf>ZsH{hDHAm?DOF0 literal 25269 zcmb@uby(AH^fyij$S8%;4bmVjU851Ckq$?Lq*4N-JEcQ%QqmxuB4g4epwyJ^hLQWt z&-eFS&-48MT)W=8w|non&mHGJ=e%BLu{y6*i2yVJ3=9k+bu}eD3=B*p`Y?Npi>{F< z%9leIxNlynDq*1hy^6Za(=afYFw~V44E*x(*=`{+*SEi>TzQL12fJHGgHblpKCg}Crr z=21w+?3Qk&7p*?`bDW3NxgUEok`4c!r4dcug)a3Yjr(cQ4>pYXfBKc0yAHb+g93|o zD(=nM6lrWBMhg$ZxGB^P(S^AwCj&<%VU6noKCVAMH+4?U*=87=p*VkTfu;GpNjg%t zNh1hWJ_BiQmw}(vc|A^NuBzfS>(A?-CcL9)Bl2UcXBEJz!RrzBAoLLQ2+}ys8MdVg z$Qs9DfFd;>bPP-&UzM)Egzog&^l#1MtZLy*XSZOKy}8EspsB}fdx8|+^?zS%d24`B zGB6=Ymyoyc9ONa);Py2_a(KKuZpRVFKvLI}c@Q603jNUC_O71W3YP4sWyEyAKEmrM zD3iB?Ruxo|cTp4GVl&DQ-|HY!^Y015-nry#2xUwJoUa6uwqfR|YF$?-5?$H#ZH4&a zu1+Xs66z}~D7oVfQ1@T51n^w-h%*?H7J|xp7en7y{3N^{>VOrmd?&TmPeEw~?n(E{b0}X>h&Cf zf7w;3FHa@!N>sJ%TG-L*&RUxT$DLf(o2-qOp%B8q5aoUuKVd3%+{G=j%Vm|Lu&=7G z3et2V8whCF1GE^oZp7a z;EWqx2+1C8gJ94K!5vLEgmtQZo?u&MNhBm|x8enA=9dN=3lCk9MTEa-c zT)~wll74dM13A7w=okFB{Y83f@ikodUcuu~-+Nq&r|oh?N1Gl|0|%V?K+jciuKs|| z-Kh1wfxpj8!vNba@NeB%d&5Y~F0N9{)!DzMA1QYyEt8$Dg1oeCMUa zcLWT!M#rvE$zffcgTbBTqH4H%R+|i6mHw9aEmF_+rNSz*ye$e)(d47@*|FRq9Hf`` zGJjgbr;zJE;CXVbMagQ&2`&6nUM2Fu1k7P4+zGkZl^l=UrRZ4h>wf7O&`QG_A+p5c z)x7$nx&~=ax0movfd$xyvCdM#P0G}gBPETl_=-Ty#80IuUf8Dvctq6S+gTP06oi77 z_X2ML{$+0+jRqN%%5EuUOM7Dr5|6c6yIU|sD`Wxtd8#c{Lx2^+s1IQWd3e_-@ z^=gf3#F#bmZT-DS{p;9?uNVsjw)$%)_`OH)_sIm^i7UTVuKD~jxB--p$pqD z6DQov*#z6w%MeObl1rf&!wwA9V1k@9tL={aK?LE+hkjh#;MSrYw>4dUKN=4#ZdRKd zPUzAsS>nZ!me&CDd8tI=-|a;7pvUXx+>uwg2df=*Y`BT&+fXzD5|oK@Ydn=?^-Bq1 zl`i5z+P`*G_;%{Z$QO`H3(w*NwGR3GZT%lh>~ztL&<>yB;mT^cd8f3>wIA5d%c;+- zf20}}9=l2s#fT?M!uc^#J$W~vQz~Yaqn68_^~|M>U-s)4V*kY}!_99i{>&SjZHNzt z&0XdXy~eILI7fUMg~@-|&%TrcNNh+m-nBgC zf-x2~)StzU=7-Bn?CCl#eeS=C4oy?@q39siVhbQleEVEs{Do@qiax0?rn|g;*o)%f z4ZZbAefVsyNPdTNC|dZycPxWG9J#Ly_wU|1n*P3ahkPJEnV!? zDRngTxQp!jTA>+{_!%^;wJn{+pREZe@jfMq?3bN6ez*rL&zoDuR|C&thEFXj{#@^1 zt$UliFjV|}NfoUD=}rjsvh~P+GnP~LpH-1J{O7h}K&R>#-{A9+%1Z=&{7QDQAdQKT zyk#dW`|nH$FDs)0@w?9+!8IL?u4LVU;akfU^8qUL-4Yg_36&b-ir@{1>WBf#9e-Rm z!|oRVeR;!E98n31#8?`z#=wZ-kR zI(V+?W>3J}_&k1OLbNrwnacLq-qRE%nh7d#x0=l_-oJLW@=XElrkutf^)e!5j#aHa z)!DSlfJeI6^>sKxYq)JBTLJ_eaUU&8*k-Iwr#bRblYS^3g+ zxDnd#*5^EHc98VB#rpQ~vA=b6`uYe~C-y2%=G58Bv(L|G(6{mIqotvsvT% z$#viz^p(nccl@OSJEzj?Cd;hQevmYbo!$_K)_25=p3avxT)M62dC+{dupz-iZ+h|D zvXP6&1>HQNJB|G$sVId7`8&`5E>E1t;@NrY{Hd@^hZ@m+ZeV_viW*gb7rU_@KScyu zztIpX>lZ=15qVie-$v1vTR@Tg9FgAqX!MEqwv6jZj{%zyY?jbnpFQo5nGm!SGBu&B=mlS&pL;SIG=Eda0z!5r{|F-NEl#Q;sD-GFo2FyBv!a? z5SzI^@_D8E9IQ#?@nZM=;8@j(2tQiywzPO`L2zV?+6wV%euKk}hY3 z-@scLJ*XRFcrnMZVOgeaEW#Ax1b$B}j`aH|)q0ZZuNb(mnns?1**qju*K8b__Y#77 zMMQs~t@R%s>lbvd*GaIgRLIY9S?uJ**r=P-u&2)ZU2sz%QqUzUWtr6xoylz8CdS>G zI@PAfwYW)TM{V?s>;cI=I?k9p35?T=b#^bu-OJ0^KM9sqEzm^j9;nE~%RRYJ;m4vV z){xwPEB!Gn!Ngwt*nO5@u`&deA4T)h^DM}y#H6KUrjX|bTa4jsl)*xzaniOWFh!N(*#IUKr!XVxarFVe8mBDU~PG-B9o7+P9O z5x@h6!KXFa?8PhBO-geSO*)P%AZlFcWV$C>?w2gioGAC>#j72YG9P85MmI!UqlQC5 zB|rrm@3NtndeHU3XTS9<78s&#FPV!i2UKCjks?_>l1dc|9(a;W%duZ$l^A7dRJ?Ly zsVIFI9g3p;D_E5yJSl7unNNYa5?Y|}lGlo?vEqFZZkr$sm?*5fe}T;2>rvu?=GYq|F%P@>Eg07!U=SZ&_11eF-ch z;xu(JzZgl1ic^-1A4BkZa<}Bbq24EcU4mtgyKpY3(z^Wku?{)w#Mxm~h_pWM3cO%p z(^Nk7hQl!6b))f0T4j>FRE8R>mu29~Z#_JzltESrTpdcV?A*0i%^@13szu65njN#O zgTcbi2{MU5)SOZ=xO4K|SAvw*7WJ%dUGddtvQhSYz?q3}RNw#)3>sEx-iZg?W|68N z#J5Zi?JuCSQXvq%z#(a2k#TwE;Isw2j82&mjwHddIi&Et)Rezyy{iDO1bjI*eE*2li?>Wi5R zb7&m$hXc@Vva%i4=`p=?3WdHo2&Wr8Czp!D#m8GJ6zZu=oIq#YKZu7)V2jqQ%rsz4 z7F42>zJ!mtIAk2V-HFTqI zeue@X6&s!OmK8v6HvHLkKy-I8-@!B%?WOseZ;IL{Q7?rRsO$rxs$caEtkx{lPSh*KZa9+WA}R2->?_O{Ys_R|&8L1GdO}V-S*F$wP}^zh$nbUo>)eJg_L8+{iDL&S>u4I?#C>G&d<4V7p5Z&du2xuH-40ZWWwqJKd zZ8GTl#!NYmN+vK}iFg$)zR)U;)1;S~uYgcktQk204DAAOC8Q<4*P|7{ejF@xHNAz& zP+t1ozI{M+QjL*43(p|;yMru@&T?+u_a02nOo!+bVA;d#^L_e`4_~?R8{oG00#ylH z#RaZ>k|mMxE7Gw67eO_;?lQCqm%z+Z|X7-6`IE`{=8Mm&T zC#IL92rL^@m?kkNSn#ca&?~p?g`aBb2YF>W)VQk|M{XVhjfAfbrKMt79qu)RV=)b{|Q5+(E zsat0H2CFf%E9S6wzT)UdU5MF#qESl|ny0or;Yy>GykuLSYzF6oeu|@KgfH{iy`rPB ztx_(?E<`x=!q&S63#=I>PHb54b@y2dkKX!7WzuBBPmRFDo6p-hTCWUC&KHO>e*JPH zc$4tsMJonM%`8SpI3L~CM9mXQ4E-74FkwurBhr?7isk~ZoJ$8VTa3HI@o%Q3SzM}Y zZktlCE|EyM)j(T7Lq@FVCM4_Rxy+)-kwO2aSmOu^o8fZ@xjB{OB{TGKffBL#8v2&N zjLE=NVqOKAJ7-9bmC8#(TCgU+k39L`5Biy<1`);J`5EDNuA*Z0&YuoaydW8li8tLd zz>0`2Vo_t6YTJaJ!9hnFJ<%KyGhO$4L4#|V&@A2>CxfiD^~dYs$A|yc zuWh_hbHY|RdbT`9-sUWpdhkyC2H25G@(3Gvu#I%!K)#gRK7(kK|QE)bs3Lmdg~m;Cl_Br8CjdU z1n&iD{74!t52C`h%?a;vT)X#bwu2a7P6XIInsQE#E`{Bgg&(PZk@qQu1zPpCmNquy znDoEEE!4`X57SQmQ&Io5q+w~QqCQapJ${6jk%dE#{8d;6Wr$^Z8!GyU%THN6BXQ&` zxvJT#2Ch!#)9$PGY>3N zrEN;bnhQfk@k2GBvoA{(B@EZeJO#!M)m2Ko7IY+v;H_uiLWx$K^s=jcc~j^;!!4pJ zr)iI2+)o)@qx*+8dG2(Dg6qI2fcpe_6*%lir1zXQ7C6Tqz|&bzNk@_S0(vYzTqlKG z-DFz*y&wS^%uRk@G4x4k;bubBO%W`zyG+6=0TiCq-}+Uu`CP_-xKHYgDl z1R6#0!Oc6kwM}CWQF*%Q-JknDD+JUg?+KOZ_+X9W%EtT!-4VAv)~bAz@i`U_4*D<) zC)`UuQOKoo^^T5_)csU1sn!kx)UXFK$TP(f6j}ywI(|)WbbL*%Vz6CS((p@N35Wkd z!tL--h5t9;lqVayI{(o;MKc&QqGClsgfmJML{R7-gXX-CxUq5x}vINq4z&eit$D(KYkzHcTXZ zc2d&c&MfwY5=gkoYR`_CC(wL+YnR@cOP#!X&ZQ6sDM%=IxlnJa0NpRdZ_&m+i{ULH ze<{j?WIvZL=YW3FeJN>6?);Na0f+DEjHSBSCC6x%DL-PRaA`t6cJ$+0rS6C&k;31* z{)V(QeTRM3)}4$1KQh_ddWP6?f5(-(TrUEb8(XP!I`8)byd}&qHqOmKX^}68SSG^+ z@;Vyeo#Xn~4prAq9=*e2{T~341FixJ+(ovwA-kyYXR~Q}scB`bE>GZuom5;m^?x6? z8ouSOmWN^EeV-0ivT@F0{e172vRUHL)^cT|895abWq1kduqCn9woP?XB0_TLREG%2 zsH;eYWIz$o#kAdGGUPYS)+qLOr2`tmwWG@cFPx@9+MgNSW1=#lSv!@-jEX0B2V+(N zG^to}kGam~OQNXZwjbn#zDy8JPXld5cBT^ou0AkrEn(|Set8O)@FWh`I>B+*?f=vRIt8D~IM zW;FZMkFJeu4BShp{X^*5Nj~vhM~*&>j+-_m5}9eNF$l^c$C0Dv^YY-Vm)or*QC!O) znQ02kOV-*g>!?(exnIIPG+sZEBe|e@AX*CLCYuo>o1BG1+c-It0LIZm{vG$Ac{J-b@2c;z^C82x$kzK z)UrLPy=%on@&Od~C1KF9mA?5&9WybZWfX-dR^uC8#wRl&$(Rt;8Gz(u+gy8vo<^Lz zYXY@&Vqa+Amag_!q6=a-9NG9yr`N1nYN03`iz5I+wPM~t>t%cH?i(6_Q=pvQCBY;O zM%oOpik}^`5Su{(s}AP@qpQ#{E>^HiIstnH$2>IMQcP$S^2za1ovn$y5#zZsaV$!H zesO8OFyRN&hZ4iP%+m4Wi@yF!4WIyF&g#J2nFgY~Pq1GWta&5C1j#}v>QYHf8J}|* ztgtAPO_J8Qq?RsSb1tL^W{e9U=pETr#Tt9)bE25iLXV1`B-rd-%PgF`Y0f!;7&hiv zgdd!nMGi+rB55#6Lr}`hN>&PU-=H|!36;1>@`=Z8B3<1OiB}!i$1csz^DiAWgES;w zEs@kImcb((3yw|KBOzqR_#&%*MWZTW7yla%5(L`gZpD|s=d8UT>XUdz5(^Bl7-ZS$ z8p7)hsv2zsrp-_zy|p{=oMe*4$wDTIPI`$E&LWI*E_BIn?(3A+GM$X+ox78Mh1K4Z z1|*SjY>a93Dk!SlZ^W0c;terIMrAOtb{u(6k##X zT*#CFk>|aS6Wt4QOL#kr7*5~BT``d0&I^{{SnSR8_r%}8&9?pBMw(Dd)+mDQ-yBSP zV{U!a{U|TKSG`ODmRq>~(WSVsg6&T|)E5Z^YzIP}e2tCoF$x)W`)MY#UsF2wyUEOA zQ{>#oGOWvT4SP!NQ1+PjFs|olqsl5mt#MZKW-4WI7^kt&nmjg3FL*U~c~bmV#?j}M zDY`4hXv2bhiJ6Y%C%nxH?!_M6t_t+6McO&D0v;Q>HtN~SMGtSz-ww~d9$XC>};Iwq1@Gonb7!%C;`UHqPy^O$kK19wj(Cx#K>k%N2U4BN>(b0Y?Ce2Y|f;%vY{2`_L@-`kHB|$9_y&qw@K9JhFUCViAU(5?=Sv*hOs;OYU z@|pcW6&x;U55A^7#Z^Pq{u)%+3S=kx!#a}jYImwvB3=Zdg*)EA5lnJ8Kzzo>@WT(F+GXx!Gub+^7IXRLB?Q>&={2?%Fx=NA8TTDCo zUtEAemdXqterNTgf)u&@FWI-GN71IE{8LTjtBG$;qa#k2vX~~`iM*ERnf@tnHj1KO zzx86Af6wEUyTcE4umb-MZ!e6r^k$YpKCE|>U%dGU^KrmIbuuAkv$qPA*v@fp)-;-_ z47jc_^~6nz=k~JwoE78FNCRFaGQ>~@^$1N{k;vL|(+J(jq^4Q<$*jG(z6r35)Z?cp_1az+St*4jc&PRY|OOkpl zdCag#eJw|f4c_#mdr}ye(;?rfzokyJMlNwB9H#;i3|#PivK992&_S)C&Yx*NDUifc zDXxdiu-ZatLM$P{XxqEa{(-Ryp^|k>*Q85#U#+wEepD{D)|1MV=wRU7Rr(oP_d+z` zR*(6MJTH7%drRnb#pkgn1;A>^E+4wT*W9rX7D%Nf(ZDktIYM%T9UDaz)e@QOsx%yx zHT?*w{EmM@7035c$Lf3dqu0*4q2Ta|Ac;KbxKZ!&;<;;^3L$?iEiSGIy_E@%UkIpg zi_TWvF%^*lpzvjvLs}#ii3w{pS=^M=@Q^XfP!%sil08p4IqG-Eq<|TGx&IRCMUc+& zCG4PWiC#U!jyBgfS1L1IIhfW}E^)Ig^CM2*<+HqrByHqd$v9dMhGX71=xn0tk87g~ z$xPd`U*r9PfInCw3>PBzA5cC;@4xfQzv7cB!63_mokjp!7QT2Bo#!fUe4G{1j{CWk z)uqi$^4D|y$TNa{5LEaIXjD2ukcQp8z;-|bWyY4y__U0m(=t5GJC3&{8B7>4*6f`#E`49ns6{dik6v(HN zQ@z!n^A{BnI+Rj+@;W=M$Uoc$W9U^0Pj$-y+i-Cxb|=37On?opB*C)fB5SYcSnRUC zFZV%;8@8T?<`1zBHMKcx4qJbq8oY30*cP22XhD+`rk=uGZ)5_Jx=EB)UQS7Q5uA?F zCkKFLbmTMSp32Oi#rZlNv$7#q+OVSMNyg1#V$!*cR(y9@MD#tFqUKShr-5`rQ{kCr zL^0{du`=MCNOWzS#%eJX$Qz<{v?(GP3B1(+!w((g+d zcIRU)Zyv&`fndOh*(v<=i5E+NXr@^D776f0u> z*L1Tt>dw`LZ4&sO&H$tW}Pq)8$s;&@@2X#Nn5ZkFB&g-~A}7`izQ; zDmNX+-76COzr(#Z8_!ja5uF#n~1nVxF1K zg8U+<@!Bg4&5>1kQ}EMCt&Q~JwaMzZ*WnWz_Ea?BW%+|gKyJ2LFo1Q$&WE(ovKAyu zu4vCY^^NRmQut8`wdC`_pCg^Iu)8^onw_{A{-%_)w_H~&wC~RygYRY3D*17HLeqnkrxRcgJ@NSg-@e8dsx*Xa zvzo5VQq71h7OR|U>y)+=aJ59Iew0hy|L)B%CRIYUPEvH+NTDq9fdroi&1AvdO;vqp z)jc0@6G;!^a*lYB{uk;LFp>mXbBr$9@rf>J`1aGWavBfJIPOij3d<3ypi zom}*%UP|}qyZYj4@Z|9V zLAvrmcIy2cf)sv4soz#f_)7Gm^REg(n@L09#Yp3mJXd{fDBPE%rAf!O$|H7DSt;?i zM8A)6J5G&Piz4|+s@l#`h~~}N)EnL_)=1Q2wnL|!I51#0CCW(-_{D~`k#IRK?3;|* zk>T#J${cwhDPIN4l1$5|NPq*qsk)=AssFYK^tg~{ngHt8q@|^s%pFP%HxOf06V{y& zq+=wR_avsR)Go@npG!BfEM<6YY|~5eo)=akxgm(qJrCT(buy8BlPff>Is0T)SnamR z_cK|>uUoLwbLwfqj0(h=Ab{(xomU3R0 z^sJ&CLm7SSW?gOhmXuhbTJa)XzQwLCcpt-7eF=MkAn|ipW=jyE%!q%5{t2OOH70uL zDI^}%WA<I=>^_) zQGH#=LU>L%I{*Q>jeL45X|cd-Ti}EZi|%)r+xd#Lsd~C((s&LD42B$32 z{(PxWK|@ZHzleBnQ)t8my7vwD`W=g1kAD59C+mAkpZeM>{!q9TmSIIZIV<;HDYFW- zYAR%0T~=TvLz2X>Zhw^)3tkU!U@zeGLoC#}6Tq3VXO z+4}3JbE|1PCRx!GUnf6yGBL}jl4scQAVa6rW=r#8EX(jE3OR5Jhi0U(bfr=oC$i@7 z785Hl8`zO=(ZShpx)A3?BdCWei}3Gl#2Kf7Qxc2b%C(+bQJR~rsaYiV%%GY{FDY~b zEnje5TW04Ey2YYb-+=+#MNa)bQUOuV-g4{;bdlb65lWcV5>MwY_z+1nT{ zCc`v;55!0h4(!oCy=&!yi%}SYPQYvpG!`EqiRE88`@ZwOgGgfi`fHR^i}3sNL~B79 zf^$)y`IKnJK#O42Hfl10j=Kh5tvE6HzK4iA*(_DgdV+%Gb1(5v>b#iU2qRlVq%b$$ zNs?l=SQd0#*|=cs`-qg%`$5Oq9ctQjr^F3<6)60yCcd7lt)M=%3CMNgx3=}udiAA$ zZ83X8*aSn+h0#C|I2K@BVUU*YR3X63ZN9#)&SKD}{+$-sjy{-+D+Md38eSk7UcC4g zv=??n8dYb*dvJ z8RXKe0#LG@-d_dvL>wovl19IB7Tci|*K?uoL9dFG8Z6XLnflEIJ2=0Qcku7L(y(-a zo1zRIzvi2&$WJ=&dR_(e?73^{bN-7b6`$19>n$Xd(HcpXrnI_NthW{kcD(}Xor>_6t61eUJ(j)=U4AJ&TEwC%5L1sUaik6AKTzDpIq9*=DqsRDw?| z1H?{26MVh%&}XHB?5s;5+rKXHL5E5_NW+oy;S)Ibl-8+G(gzltRbq2Z5y)ybuK@m- z0|`0O6xt^j$E(+?6xSOMPu|R{RC7-F@#w=7!^bC{U2T;|ZL?4w*&$L3_i2MZNd)gK zt!YR#EexuTeDz!*@r0j+@uMH2taR$by$_4SuUClKMh*1!z+RQ%E^HiV8&OT}IcWdN zY=*Q}$vlR;kN-T?LCAx?N(7?*P+q5LvWtrl{zZ~KZ88(LB&^i%688F6a2@s zl7budX3b6h(B9IO0{|4!G1-{&^P2$TDIyeeJqQmFdrjw|gl{KgUvHiOYK;56%+t>U@6WiwN*Uij3J z_|D2JSdM}V=}c=X3?ocQPL(G9^aY_H_`7ow@b)L{7tUrO;Eqf*py$=m=deUB%2mb0 zb1sQqZDqULzl|?#6VyN9BB&815kaep3RYecco&vtq?d+5r8_%`(oaBjn-`8>+v(%^ z(gp92O>GUOgl(+oO&RUgqE| z@fc^)fZPwp(!VzQKxa*@Q?1kJqJx+H^O##UP4Dub!hyfZU86tqBvoSega&L2-@wnL ztl#*5M+1M6Q|B_}o%}c?de&?4|L|t+dg8o&EvH|!*O0)j!oWiAa9&t8pi~aW{j`Zz zfO3#TT+q%$^Suy*gzsf5=QQzWN;G4^$+({+@#!{$59XGhjOQJi)n*`WbUgTn*O~yp zxU1}vAjmf4VnM6V5S;Wknvr-ZQL7A=O$x?w*Su~cQ|@w6^*l;eGepx44FA>n`#KG$liCC?S`?MZJ|TpqbzZui;IB1YYuUIj z)@W`-KcFXYAD0)apy=vjVVREv{k1b)W+NS%o;LvK>C#l;3FpJ|(J-M%lO>yK>zwxz zg8}Jzkf%-L9*oCW0y5}6>A1u9BX&d+>-7Eo7`g<1p_xy-#{dsnqtma$bA1eb(IFRH zBj`4Omcq4@Jvsj_vvjSy*8Ios_wpoJ$=+|I5y$l-M^Xy#S$Jv!-a+|7D*sUgf4uk- zx)%DLv-!YhCl1XM1pcd&9nlrYCq_`LM>r%12E9S|nq57R>pHbh^nb`~jY;T~h>U@e zKHAW3o&5lQ9dN>r_YqooplMht4$)Yd{12Q}Z?;1pWyhVV=obS$I0o-oN?WZz5i zcNm&PN5X@2ogrKbPmfs3c+IeiGl%$zohZm8V3^;FL^9>u zY61<0%j9fr&$Na$wHJe#Rd6a#Zd!;OHV0zoHEfpG=y+#3Yz|pK165ax)y9J16>^YX zv?#VrwtAT>Lp8;UtT|nDv_FVWzJiJj1hz-W9E)GTu~J?xWQSBJ2;w@*Cnl=c3K5?r zem=q-j>%th2Mq_S*INL(NTHM=%6;+uU#U+%0OjpS&Uq6BMWe_Aot$v(u{s|ua-AD& z*EDpwgk()l7xA+yXF_Lkr7Nqp(f z8-M=GniioBd^}We0h?t)FH;*dmU)nnem$;`8yC=E(f0=GQJ^}^VIXgF09O+(c7cLA7wr@6f9duL)$8>dWfcw z<u5w3+^fHBp}vlOhHZ=qv;;G`b>*0j0Hl zzeGaxGy7DS-j5K;h=8D3owHAjj)R1;6TwkVz9KzcI9%bXy8S`C$SxLo!7(Oky#3g@ z*Q-)$ADHAqi4j}~%h}s*G33gpIB2+oqyQz7{kSIGu2Ok71cH&K8yGf*UO}2~rytV+ z0x@ewx`2k8W;!@1NZ*H4tmE@2@ZHOG!5j*Wj+Z-$ZVQ(3qTA~My&=_L@dYv1vMq_4`I&pn+w+{~FBJ z6T-|na|bCzzF@Y4(Cy@&B+F+;f%Vdhjr?d&bI~d;)(DZN1cqL*&UAi?*EDq{3dCq5 zGD$AL>50I)fcCHIXs-yPS(4=a2J7<8PyGq3!m>t1et};%KOO~$aNG|gTD=-coE^K0 zN4G>_*d;M2chDb8 zRQ&mCP2n<~U0v76Yscyb2TtUxN{J^#Hr=&^TS90ylD9zBlh>@}?VCb{sP}MI2r~_- zF)iF%pJ~DehtB0imE#1h^}|f~-4Q2(PSM8RqsQ;}KYZ=R*ZA)3KE&ksnKsJ~HSoqf zN&|a$ieF(yY{W*km#Di5MwKS1)#sb4ex@>fM6{}@9ht#D_76O(K&&}&T^A1MI#!D% zV7(^BfjD~9)=?Z6@^T_mYNpx(#o~SnG#>3M{>?^6<7FJveGRLLh7pDeRHS*d5+u3ctNRTVRk~pU!IlQKp#5Ga zG!~YobDV$6RnvMO)K~b@@6Pnk*0u$Dbl{IgqVMgd&k|M()!L8JJ`$!7;V_;?S+9P= zzwJp3xH7ZBku$b0QJDpwo1gV$cM+I;_u-s6aS5Ggefd*r20aNi*1ZshKmuvL;4tnx zcObHNEgTgDNL*e`Dm#g^zYC1BrOTzSeAmHWEk9T%dib#q~YOIK4vyE)=`AzAthOpP={(!{kJf_ky-W z3*0Fi#|sb;9@9Ds| znDmUUGRk(p^%>4;R2TrAvKYIu%)qyKHkH4&sejyB0`SuN_I@1j;H|)WG;`i7p!1a* zdBFAdy#0#@A`j%Kq7diS&kWIu`muZS6XUjo%P3M)iiiCFM{Pm@{`4OI(w^e{nMRf3 zZek;XAlk?SYmulRq1^DoFOSkPtGc)p3L@2F$JO8{?GhKdC|&?#nZw#2)jo4XIX+7J zjj!#P-&3xeSJ3YrF||R?=tQA_*OWz3gQ`=vx;$MG_H6sFOV&_S7i#n?{?>HKG!~53 zh>hRbjcIl`a=1Aknsh?Gt&II2c}<@`$U4&{$WMGF#zAo5nH977lPCZrSJqag;7!lb zZmaw^-#TWs;?WZZUGZL|F0kod#8yU}P}HYD{#DS|J{l2eUBIFIQQX)V_L`(M2T*w? zd-S6nPVb|f?=PwM!9{m37J_NUApC3uI!8=DlnU2f#px3x3!{y=7ZZRxga-I*p1eQU zZ$5>Cfy34-rR6DjC~=m-g1o-INO%1l;Zm_BPU_F7t^7)(_>?j3=O66~ z5Q+@IN#@%q#m$T&fL9*6&JNI`fk%n?jSoF|OoLRCzvuzx;oMhp#GuXBKkA^2M5hVF zJ-&}t2h`rS7CA^WTr8Ahaun^6)LpqXuk`WS7E{I3NST#Jsl(CWigA6NsW_F)8YJ_W zE~^yx!jnxpS`c;@{UtT)Jw*Wdz|IHL*J65>RvdcOC;nUNKydETf5-<78RUPE55nMF z^Wt++t;2bLTM~W8hJ_{7f_U(dtNdBVnO54O=4{{NA025H^m_a1n2~W~^zc99RQfo! z?RX@|#~!=!hNCMkFU{s zH2%j2N)L9L7x+6$iG@6JQ~^;kzgFg6Soq<8&P?ej=y)&MhKx2 z+_&xi{M-(oO%Y`uD^xGWSZxPL;MrC7?gk5U7!H3{0*j6NRl8^tXI8^u$fiDg&Wf=S zsMrTv-qU4q(a&cNvD!uOl+OSG+k_uB;--O6Yzck(oLX@hVb1+jGwgYD)oS*;!9OZp zukJ*E`8$xC5yu2B*;}$m^LxtNTUSlC27XiLW&#t^#%W_2f-z!hw7t%yi+6`TmUP1X z0S1NJsP+4no-|(RLHO@%IW zH142J$QITrT<`;smO8Wa1}goK1pC#rt9t4V>~5ILj+pa9S^jep0-ESOp4UIVNI8@r z7fQRmNfTtV#uD;fjR^Q&hp}4b;I~|TN)8m^`a+bVHx*OHX>UVB2-akvNqK#`QK7sN z`6d^Vxo&5b$c4MejPtbCw-+-KHyCef*Ds^*!Exb>ki4iGyVNYEREcBk2kZ~UkL~qSf8FD+4|uw#Y?iayoBU$%d#aXT z;rPF}05#Iv1zm#VXh#B-Aio#R__zz#@J@==eUYedP3@N)t~5QAvf=N9P}-F6628nt zcdGC&s`I4+ zAI8+CKgF^%*C-G*55-gqd2aiomw<7s%v)3dwv;S(q;3PAqdv_=@c6#7K7xj3Dr1IcW*Dc!3}b?; z#5t4YMa3DF4M%^{oB1kKWeY2cFWwyonMg#|>bz3a2&QvxGa^7|DI7jucK~btw861r z*q!v-Y-ZA8)>Dysw1~uF{iGBZjGz0PGT1V9^P&#W@&&xgliJ`IJL@Uo!~+~EV$O62 zrTlgAc9+jgRj!x9)_$ZxbnJ>xXT{4pl%NG?ac?wZ* zAsXNk4Q$AXW5iiK!c_5d!8x(pO$q!i ztoKA}(7h)S8!kqK5dQ^LhVweWHFO zH)IDfs!W#@O?O}u3Hgz!Z<$1{@1Jcn-bshw3G2ai`TUGtx5)K(K)eAD4u5ogd>+GZ6B7P*I*|Oy7j1B56YMbM8ic|Vg`K_Rq2tfMVe`4;5fACol zcJDz?q8qxB3ssNd4&1my_m$NSML%-qskJ@-zaGgwf7o9roy6~x)ExAWe#HNT>2-e& zURuqas(JYRzf-scT5)xF{k(`&wW$>)Nn<&0jOAlJ7F|ftM~3FWa+ZVN z`Iy)VZq1a8@Ec=XE!*njlj(Sx@!wec4a^j@{RpZ_S)W=)^l_Y;;~U6JYksdoqgco0 z*$`HULUWPT#DQ={f|^%>a^@>t?1gaB)=QmmF7kMHP(Wi!_uwJbEn#_~`wF(7i}pTC zNd*TCP3^ZPC=UMTtA^;fy(y`KOx`rJ<{yJ$|1^^R%PpW-_GYyOpLI-oukip9bGmqL zELskRPPrnmNT27VmsQ2Gw(4wFJX>chy04`2dC4Jb0w9f$i?-XOJbzID_5!<2gJS4X z&?$OByT=g)vvl-QbWt!V;~!6$E+BBXK@tD`ghMZ&kAB}|JOZ$93RfkC0I&7)2+m7$ zd2KXEacGS&9AQ8~i_g!1Kvfz|I9R&D;$!*$>Z#ag$0XC91UGA3@lPp*;!L8*p*iLf1R){!`o&-^Fm0;?iV zviBD6fN?E40hAeg2*U2S^D^4OyHPbpf^ZHU>&h}e7)F|(W4M%r%5?<&+O zd$SGrbHBOudn_t0iq3lS17lc{$1TWX?Rq9a&UCR0njHOD+i9hQ3xJ_O@C=ml7G0VD z9=c$SV{VXcxqm?u-+(_|o2aIbe7 zJ7p~zb2{8k7z}^`VHN06t4R=r2&JU81^r1s*%IhUM#F95WPsJ66K5|9w4rJJeNu=^ zVqqkGUPXZB9Z*o)1%;uDM)A~m{Un*rM-f;_f0tX+ZbFY*93zl zz36VEc6-PgQ%!zJ>_oJ=MXm~hT`EFmtX5+< z={QsuzCBi=+2}yetX^txx;<8^Rl6egKGocXfj=l~;joVc^sOAd&?>o)94G8l`lB zO3S}_vI>u1DVAn&7-G!Wz*tv$qfM8)!mO$X_#&t@w%}c+Np*F6iR}7=fAi%uo1B>7 zN$eZnHS`QGkJg&6kKopFNo>15r(+sv@8IG9B*8sBJyO=z*2gdyjOT9>vo3c=5#gm^%q18e|@d{sanVHorV}PeXm$>RCQ?6x?SOl0J22X=WzyqqWt?#nbc3&2N2ueb5|H;JjA_ zXX`ZSPwqI`LZom)aAs`Vyq6nlf6}U#>ka%0!q;rDdp;z3#WEy(EZ@sLB#fwaSo>p$ z2FyU4l8P$J!qO6kn~W(On&9i1>X0^#+D$=$ScrVG4GOh!0pmdjFj_%pP)MyY-)}j$ z$>gYfrydee0B%1>ttA#ul;GBd^Uu2&njWzMh?(%>lV1wY0ToP#a_Q z%(*VK*o>EI?zEiG*VNUWxw1eWZEbHGE&U4WPvehvKVCnaghba4WQnz914w3PXB%SL z=>ywWXxDA`Zb-5Xk9gAj6I&Xy7tPkp!a}gPrNz4!P{3Y>>wM#`A10Amridrs)gG+; zdn41EjEoH7%hh<{lVJ(rgLce0V$lnOjV5O#5E2#t_}JLo-2D8l>tgxu*rDX-hS;{n z-&5M}M@B~4f*}bKm?ZgBQ^j*| z@2N&ZfHZKl43Cey5Pm#!T{QDGdGNU-&BDlZF#$AW{2d>!TlpPJCqyK6eekc7SgiSW zSc2G@cZB8g18x0 zS$TK^6nIrzS67$ewj9Kd0vSjWP4}04+*WK-TTjOC5l@P7RW_B=m)q?Xi z&@|A_3s|%I_&`S=-_(F^jzRCXRyHEB* zc_(xbt!5)R1>hwGfV_aqh8?$pm6qCrH~;C;X}fk>YHI7~{Jf}VdCQrF@50smgwty5 zdoHhw1D`u9`PzfkcqNAc-id<^vAf3Vk`hY@z&f%2Q;hOF*0eq%)L`5+R8k^If_2Lrf@)4YU|8ubp=}w#3Vj@bH%O5zPr**SJ-3t>n@C7-%@0{UFANf%+QuIEgRS_ zABx$)S-3mj@46l<*^R2aN3XcMy*f|3YeGU-z_?A;`%?hneUkKVwl%JIZG9oP*Ow6#+`(awUlF~YRLY*nG>(Jj&ra}rUD%lnWe!};pSS+I z|CnKPC_~6~ADBe6i?eeX25nD=lSOI|*c6U|c%a>PVq(JA1>8O3vKcwe1ii`QICfjqQzBK@ZG=Mi7w~I^uuq5dS?1yzRbhVe>~{g8vj^PAbYD&?8Y4NL__B=9yp3S3tL+; z8aE#6OL;*6TNbXepFg*zfow%Y&l`t$o58GYW#IOER0|h46Z*a)bkY%vfEPA4bE8d; zR{MY#thSosLV;$teqnIA`x~xTKd|cdpqZ#!$b| zk}s^=c;Oc-eswgi6LZ7ElG?(JCO4$j{j*t_A~S&}eg$U@DbvsEaid)X%@W!zQob6o zEn`ZjAXTBy7)uDdUiQy6y88GI_X}beYYPVY=x@w&cb@y~&kkkad+l|eU$waYZftB6 z6LULQr0wLmC3BlwsJyR{C3EoZM;L#KyfFwC_*+l@$`#z+TwSro03wcXUGOF=8T*rF z+uEnVKex&)#=sDqoJ<}UAAdu#^j$?w4ZD3MT1?TxqPSj|jV-F7K>(2NYWeV2IFH?; zxSsEgGeZCFpT^Pg@dSpqKmBc#Ttkzj?~CU#U(@6ZqC*HM9yQs_365p;(fp{Wc;H9D zZBiq*w~*i5oc{GI*1QLj^=xkr4j5ckQIUa}8Q%=1lok`y&RpKST&?5i$nEa#j&kmK zc`vJ~su-D>kxn94XS)#KtK^e8{2$Uu$0R29%r-i{0fQ+#{PgKlOh$&n$B&Q9&COZ# zn#t1!)>Yd~hceNyaB=m|_=Kp5)XTJ_q@{7f!ou?M@;HDhU3RBWGvK>(up}^`L^j>& zrEBY%y4aW)aUY-7`u<&)<2~4Z9O%`-flF3a7UfAn?6c@{w~=^!zKh$*fkLoqp`tOc zHqd3v_wU_6n6j3Z$a5P<(<@Hp-&Ot1?#FH?Bk$>D<1%s9FN7GDwjY@uyldx3k+%&2 zFQ{J2G|9aoMKQOE-SM@u`uTtQtomwd-GHgTLF+|mua{RHnL;`{N;Rtmye=y0cZRe7 z^9tA78f_LDcVPPc>2DB){U+5a>gv+&?r#8{QGyxnE^AgAVFw$!O*GoCNT2O53;}nn zP;WDbCZnWOn3wnURi;}2a9o@Yt1|n|QD&(yCf0oL3m?Td$3Kzo91+FE@>!xS2C5m#He_;thho3xgS!NOKYjiYc49M4jVF?K&dOnxN zb=EUI{+PG`RV!1bS@evIzc(Vzm)&gIZd}5@eVeK_CAMuo#0kV9T!yo^ec7F^o%eXm zAa4Nph+51i9k4@OT-+}}!tTIlTmUj@#=S>N#b>qH?vHu0JCl^8?~^hIb3I-PdCFxv zNc~dQ2+U@Fzlts;C54BVx6=E{CF>|S7zY4QN=u8<``-yVY8uPSMmU6&UBzl8jO^?H zx+2YhExUSqRh5+&ML4cnZz{lrrRvN>Vt2wY*ZqR3sF+IFYh^7MGLzriAYn z4U8t!>qyqf$f!cAEP|XfAW6^TetcgS?o+QtRad~{R#R0$Qr}29eXiY{bTZPCGdUyzSbaY5cO2)*;OPH825QPK+^fVx>vjps4gy&S4BN(!d ze*Jn!(D{s;8&N2qXbeIz2@8wY{IhnL>LuU1x+F8**D3s5aPcyBLkZCs(kSD=}-VRezJhQQQ@JBRNPDZyPNYg9tAl}qZ^uNJLgcjlXv1V#_sZ3T3X^a zqQOBy_ItB$wzs#ZMT*v!UuQp(#HRbf z+?;7hnT`)YFyPK0-FQer5jL}1P*D+6 zQNi)|@84$UXd0nlL-JXd8S8njA$r##WD0j{V54*0Q&Kja-`AJNDIgC4x!-e9(M%A< zaenz7qX^mk^YPePqz9MX--W|_8kqutWEcTZqzJ~^-{#&5!%j3o`T3-kU@Dsn zrq08|Y0L6#bILbw-XtU>RJiWziQb&37MGNSMMk!~Kt@DFeCdm$2MiZQBjmMyGil=( z|4QkM!P)Q6pAQxm7R{#Tw3#n$$b(%SU~{6oRBR%fTUKHK%+>D4CV(VD!^4g4Zf{Vo z{tL*SX!Ovxwn?Nqj4C(9*-_x7L4mj93s=U^pFa;>$wh?;VJMdoMg7J>YU0V`#}eQL zySf-2r@a1HTU+}zI2dJVIk~x z0nPJOo111+<5E*UxVbf)S6%l+QlU4jL@{w%S!(}a-J)k;Z~z*t0O3QO&0P4L%|>M}Eo^6K^EHUm8|{_?@UX_8rrLFf zVzFak6(Tf0fBKZ?t|f>YkS!6y1;Pn7orZQ0e|>K8;05NHUt9Ymr1Pz}w+LWaG1;TG z-MSe&*j#^>SXOvA9_rjJM4|h?t-=}IOx)1W(8ZGSR-?{f!pU8i9Ty=X*S(joUKQ%n z2}EyiThMy$a4Kd9O3BILU!Cup&NsU^=&uAzD65^8KC3AOc8i0uO%ZR<1uSMHrZLYe*XOeYt+OwYWDO}wf8-S&B*fa`HF*62Cz$fbTMx1o@vAtII-M*Gc zq<7%^FBch-D?vY3{+ev_ZEJ%Za-|Kh=r$!>Uwipn%`7|u3@~MdqBAnG-hJOYPd?ju zc}2y+Xb+;6DJv7;pybo|B8P{S`%Sn}Q|%_xKcP?VBgW#H*fL55idP;tOzNHbU z0(eH-iKJP`#lcdkUP~&Vad6f1=g+^Gm>l&`*UU{ale zj)}togu-Vt`y1tC%F2shUtV6WuC6|yqWV@`?6g0hp^__IX)#u;c-5}|nu{xLB=&^lMe)^`NX&pp-QT z+-S=}FT2knK7cEz8!Q=n2fr#1Uw0Br5%X;=sjWR_i9w=7_o~8|I|vzB%J@fGP0hC+ zU<6U5l6y@v#G4J2pPo*Q;zM9&*EEB?xgZ(v({l(Ax)g-DgaT=1ut`A)RUKwgx36C3gzPZ`=H3gF= zAtil8P3<*Rj|C-FK%lZi)dG3@vGH+Qhen;S8WAeJhCvi2`F(jueDJdaFVd!(! zm#`U|`YR!}w`lD~l^jl+$|y(pEOJ3vMWq9nWz8BqNhn=$1HqPLNn)g&hD-#z=VA#` zH1vMqVglb`r!l1zUbnKPx0}KTT2~1${K!g^+ zjx_=U0u+TZ25Z|(+H3iDG7sWGCk>>MXoyd08TsZ~@661K&fxmD9~I=}=-#}c_}0C= ze}T1xE9A064h{p5PiQ$i*UBau!wku(2s+*HC#AF@pWbdJ$Q=BtT_;^sR8xCpk;*5p zrUn@;QrQC}mgKdRNLOq>n`8WxDl6Ft2YrKpzxCN2@dMYeR1r79q7eXm1o zH8r(_q$Er5tE#{}ytw#1ufhzLRc={FI=n3wPxt%95Q7eTfZwNQb6>b3xd|W%hUDup zF%L`~9IBW&IaL&8WDE`#JBo_8_df?&4ySON3OH>(05i?4s`_{Z${FTd1QLK<3V9S} zOvt4v4j$%srX4(f2n45xjZSAScLABRaV>@-)umJ-K+ZP-;(`F1)=LxA5qlE!w6wJ~ zQzC@OeslP_=-strOH1_F^*{62~LcTb{($twjcmw=mHT0!5SN<;`^1FbfCWE z!Ifl_?`L8v^{?JoEeSq<)cnjz5eEk|tldsF?$IZ;n?}S|1lG&=zAs%}T>`kov{tk{ zW-%mywmy9#QBhGj*&YL8!xy7aKopgf0UWrVDfcUFrbO>kJCaKhYTCw*qn) z-_sEFiY{{B#`mY)P{8cGAK%w|;L~8R$S|$7vj8||=!q0Xfw|V#nO$|XQuUG&{Wc#D zE2~mU8X8v5N4}DhSPvgQ6dlqhyOx`99?TL`0`9y}DXXBaE^&HV>-MzV*BnHT4q0Np zdd{%9M}B=@EE0_`D-c`4_oy*MafvVsuNw&ih9AZ*VlNfuVsMJ5o5sB~|503g*aYp} zZlc}!X;%4J2P6w4bT3}S!ewXB&@iM?-w}||>V1{^tFv=!4#vNLp|PGk@18g8c_fI!&tWZ?D|}O67nJ5fT!D8dDDti6bu$&+z2rrov*}%dNL>-#UYB z=A9dL-utopKPFR%^qGD6_Tnu7F(6y4P>dXZnIBIdH9+Y1L^SoghLp4bL}RHQr`uBHdM)?v-_HhO21z)g#nUw)FtA1SnaI*kuZ6XBw@p1D z)CGh38T0=hKTKVr%rEX?dl$Fj{b6lQ56;m1MB&dYu zEIv?QUdZlIuD@$d@|F4ROSiZ8Sk$Mb*O8cCb$L#4Jeb4(zE?@OCmZL1Z1Ty4BIf|6 zLs)NENWYbUEJxbTu42Zh(x6t0zIuv9vrN6)iVv<{@iRcx}quU2s}gKQ8>Px9Ua9I2h|Ci`kAZg(a}{9-R|}DNOy7&C>`8#-P1 zeo#pu)l5t?))XyAdbGgT1ToWLW)=;rH*G@ysNXiIhheeb)75wVu_1*Z6fc+i(=_68 zU~qZXqN$QVTg9SHFN>G>vQJ)n2IEQlt_yRpA(npAi~5O*xFLGzmh+ znyn4Z_c;hUFYxRvQ_pOtgLP`Xg%)tg7sf5~%DOMRRS{ChUCLzv&+h#{Kfj32@80cU W8=iAq`~}}bfhH@ZC<%RM5b!^OMncs9 diff --git a/joss_paper/paper.bib b/joss_paper/paper.bib index 3092d16b..a0b83daf 100644 --- a/joss_paper/paper.bib +++ b/joss_paper/paper.bib @@ -218,9 +218,21 @@ @book{Elman2014 url = {https://doi.org/10.1093/acprof:oso/9780199678792.001.0001}, } +@book{gmg-book, +author = {Briggs, William L. and Henson, Van Emden and McCormick, Steve F.}, +title = {A Multigrid Tutorial, Second Edition}, +publisher = {Society for Industrial and Applied Mathematics}, +year = {2000}, +doi = {10.1137/1.9780898719505}, +address = {}, +edition = {Second}, +URL = {https://epubs.siam.org/doi/abs/10.1137/1.9780898719505}, +eprint = {https://epubs.siam.org/doi/pdf/10.1137/1.9780898719505} +} + @Manual{trilinos, title = {The {T}rilinos {P}roject {W}ebsite}, - author = {The {T}rilinos {P}roject {T}eam}, + author = {, The {T}rilinos {P}roject {T}eam}, year = {2020}, url = {https://trilinos.github.io} } diff --git a/joss_paper/paper.md b/joss_paper/paper.md index 63d83965..624d9321 100644 --- a/joss_paper/paper.md +++ b/joss_paper/paper.md @@ -39,8 +39,9 @@ The implementation of exact factorization-based solvers in parallel environments Hence the use of iterative methods is crucial to maintain scalability of FE codes. Unfortunately, the convergence of iterative methods is not guaranteed and rapidly deteriorates as the size of the linear system increases. To retain performance, the use of highly scalable preconditioners is mandatory. For simple problems, algebraic solvers and preconditioners (i.e based uniquelly on the algebraic system) are enough to obtain robust convergence. Many well-known libraries providing algebraic solvers already exist, such as PETSc [@petsc-user-ref], Trilinos [@trilinos], or Hypre [@hypre]. However, algebraic solvers are not always suited to deal with more challenging problems. -In these cases, solvers that exploit the geometry and physics of the particular problem are required. This is the case of many multiphysics problems involving differential operators with a large kernel such as the divergence [@Arnold1] and the curl [@Arnold2]. Examples of this are highly relevant problems such as Navier-Stokes, Maxwell or Darcy. Scalable solvers for this type of multiphysics problems rely on exploiting the block structure of such systems to find a spectrally equivalent block-preconditioner, and are often tied to specific discretizations of the underlying equations. -To this end, GridapSolvers is a registered Julia [@Bezanson2017] software package which provides highly scalable geometric solvers tailored for the FE numerical solution of PDEs on parallel computers. Emphasis is put on the modular design of the library, which easily allows new preconditioners to be designed from the user's specific problem. +In these cases, solvers that exploit the physics and mathematical discretization of the particular problem are required. This is the case of many multiphysics problems involving differential operators with a large kernel such as the divergence [@Arnold1] and the curl [@Arnold2]. Examples can be found amongst highly relevant problems such as Navier-Stokes, Maxwell or Darcy. Scalable solvers for this type of multiphysics problems rely on exploiting the block structure of such systems to find a spectrally equivalent block-preconditioner, and are often tied to a specific discretization of the underlying equations. + +To this end, GridapSolvers is a registered Julia [@Bezanson2017] software package which provides highly scalable physics-informed solvers tailored for the FE numerical solution of PDEs on parallel computers. Emphasis is put on the modular design of the library, which easily allows new preconditioners to be designed from the user's specific problem. ## Building blocks and composability @@ -49,27 +50,27 @@ To this end, GridapSolvers is a registered Julia [@Bezanson2017] software packag The core library Gridap [@Badia2020] provides all necessary abstraction and interfaces needed for the FE solution of PDEs [@Verdugo2021] for serial computing. GridapDistributed [@gridapdistributed] provides distributed-memory counterparts for these abstractions, while leveraging the serial implementations in Gridap to handle the local portion on each parallel task. GridapDistributed relies on PartitionedArrays [@parrays] in order to handle the parallel execution model (e.g., message-passing via the Message Passing Interface (MPI) [@mpi40]), global data distribution layout, and communication among tasks. PartitionedArrays also provides a parallel implementation of partitioned global linear systems (i.e., linear algebra vectors and sparse matrices) as needed in grid-based numerical simulations. This parallel framework does however not include any performant solver for the resulting linear systems. This was delegated to GridapPETSc, which provides a plethora of highly-scalable and efficient algebraic solvers through a high-level interface to the Portable, Extensible Toolkit for Scientific Computation (PETSc) [@petsc-user-ref]. -GridapSolvers complements GridapPETSc with a modular and extensible interface for the design of geometric solvers. Some of the highlights of the library are: +GridapSolvers complements GridapPETSc with a modular and extensible interface for the design of physics-informed solvers. Some of the highlights of the library are: - A set of HPC-first implementations for popular Krylov-based iterative solvers. These solvers extend Gridap's API and are fully compatible with PartitionedArrays. - A modular, high-level interface for designing block-based preconditioners for multiphysics problems. These preconditioners can be used together with any solver compliant with Gridap's API, including those provided by GridapPETSc. - A generic interface to handle multi-level distributed meshes, with full support for Adaptative Mesh Refinement (AMR) using p4est [@p4est] through GridapP4est. -- A modular implementation of geometric multigrid (GMG) solvers, allowing different types of smoothers and restriction/prolongation operators. -- A generic interface for patch-based subdomain decomposition methods, and an implementation of patch-based smoothers for geometric multigrid solvers. +- A modular implementation of Geometric MultiGrid (GMG) solvers [@gmg-book], allowing different types of smoothers and restriction/prolongation operators. +- A generic interface for patch-based subdomain decomposition methods, and an implementation of patch-based smoothers for GMG solvers. -![GridapSolvers and its relation to other packages in the Julia package ecosystem. In this diagram, each node represents a Julia package, while the (directed) arrows represent relations (dependencies) among packages. Dashed arrows mean the package can be used, but is not required. \label{fig:packages}](packages.png){ width=60% } +![GridapSolvers and its relation to other packages in the Julia package ecosystem. In this diagram, each node represents a Julia package, while the (directed) arrows represent relations (dependencies) among packages. Dashed arrows mean the package can be used, but is not required. \label{fig:packages}](packages.png){ width=50% } ## Demo -The following code snippet shows how to solve a 2D incompressible Stokes cavity problem in a cartesian domain $\Omega = [0,1]^2$. We discretize the velocity and pressure in $H^1(\Omega)$ and $L^2(\Omega)$ respectively, and use the well known stable element pair $Q_k \times P_{k-1}^{-}$ with $k=2$. For the cavity problem, we fix the velocity to $u_t = \hat{x}$ on the top boundary, and homogeneous Dirichlet boundary conditions elsewhere. We impose a zero-mean pressure constraint to have a solvable system of equations. Given discrete spaces $V \times Q$, we find $(u,p) \in V \times Q$ such that +The following code snippet shows how to solve a 2D incompressible Stokes cavity problem in a cartesian domain $\Omega = [0,1]^2$. We discretize the velocity and pressure in $H^1(\Omega)$ and $L^2(\Omega)$ respectively, and use the well known stable element pair $Q_k \times P_{k-1}^{-}$ with $k=2$. For the cavity problem, we fix the velocity to $u_t = \hat{x}$ on the top boundary $\Gamma_t = (0,1)\times\{1\}$, and homogeneous Dirichlet boundary conditions elsewhere. We impose a zero-mean pressure constraint to have a solvable system of equations. Given discrete spaces $V \times Q_0$, we find $(u,p) \in V \times Q_0$ such that $$ - \int_{\Omega} \nabla v : \nabla u - (\nabla \cdot v) p - (\nabla \cdot u) q = \int_{\Omega} v \cdot f \quad \forall v \in V_0, q \in Q + \int_{\Omega} \nabla v : \nabla u - (\nabla \cdot v) p - (\nabla \cdot u) q = \int_{\Omega} v \cdot f \quad \forall v \in V_0, q \in Q_0 $$ where $V_0$ is the space of velocity functions with homogeneous boundary conditions everywhere. -The system is block-assembled and solved using a GMRES solver, together with a block-triangular Shur-complement-based preconditioner. We eliminate the velocity block and approximate the resulting Shur complement by a pressure mass matrix. A more detailed overview of this preconditioner as well as it's spectral analysis can be found in [@Elman2014]. The resulting block structure for the system matrix $\mathcal{A}$ and our preconditioner $\mathcal{P}$ is +The system is block-assembled and solved using a flexible Generalised Minimum Residual (F-GMRES) solver, together with a block-triangular Shur-complement-based preconditioner. We eliminate the velocity block and approximate the resulting Shur complement by a pressure mass matrix. A more detailed overview of this preconditioner as well as it's spectral analysis can be found in [@Elman2014]. The resulting block structure for the system matrix $\mathcal{A}$ and our preconditioner $\mathcal{P}$ is $$ \mathcal{A} = \begin{bmatrix} @@ -85,7 +86,7 @@ $$ with $A$ the velocity laplacian block, and $M$ a pressure mass matrix. -The mass matrix is approximated by a CG solver with Jacobi preconditioner. The eliminated velocity block is approximated by a 2-level V-cycle Geometric Multigrid solver. The coarsest-level system is solved exactly using a Conjugate Gradient solver preconditioned by an Additive Schwarz solver, provided by PETSc [@petsc-user-ref] through the package GridapPETSc.jl. +The mass matrix is approximated by a Conjugate Gradient (CG) solver with Jacobi preconditioner. The eliminated velocity block is approximated by a 2-level V-cycle GMG solver, where the coarsest level is solved exactly in a single processor. The code is setup to run in parallel with 4 MPI tasks and can be executed with the following command: `mpiexec -n 4 julia --project=. demo.jl`. ```julia @@ -108,19 +109,16 @@ function add_labels!(labels) add_tag_from_tags!(labels,"walls",[1,2,3,4,5,7,8]) end -np = (2,2) -np_per_level = [np,(1,1)] -nc = (10,10) -fe_order = 2 - with_mpi() do distribute - parts = distribute(LinearIndices((prod(np),))) + np_per_level = [(2,2),(1,1)] # Number of processors per GMG level + parts = distribute(LinearIndices((prod(np_per_level[1]),))) - # Geometry - mh = CartesianModelHierarchy(parts,np_per_level,(0,1,0,1),nc;add_labels!) - model = get_model(mh,1) + # Create multi-level mesh + mh = CartesianModelHierarchy(parts,np_per_level,(0,1,0,1),(10,10);add_labels!) + model = get_model(mh,1) # Finest mesh - # FE spaces + # Create FESpaces + fe_order = 2 qdegree = 2*(fe_order+1) reffe_u = ReferenceFE(lagrangian,VectorValue{2,Float64},fe_order) reffe_p = ReferenceFE(lagrangian,Float64,fe_order-1;space=:P) @@ -140,7 +138,7 @@ with_mpi() do distribute biform((u,p),(v,q),dΩ) = biform_u(u,v,dΩ) - ∫((∇⋅v)*p)dΩ - ∫((∇⋅u)*q)dΩ liform((v,q),dΩ) = ∫(v⋅f)dΩ - # Finest level + # Assemble linear system Ω = Triangulation(model) dΩ = Measure(Ω,qdegree) op = AffineFEOperator((u,v)->biform(u,v,dΩ),v->liform(v,dΩ),X,Y) @@ -172,12 +170,17 @@ with_mpi() do distribute blocks = [LinearSystemBlock() LinearSystemBlock(); LinearSystemBlock() BiformBlock((p,q) -> ∫(p*q)dΩ,Q,Q)] P = BlockTriangularSolver(blocks,[solver_u,solver_p]) - solver = GMRESSolver(10;Pr=P,rtol=1.e-8,verbose=i_am_main(parts)) + + # Global solver + solver = FGMRESSolver(10,P;rtol=1.e-8,verbose=i_am_main(parts)) ns = numerical_setup(symbolic_setup(solver,A),A) + # Solve x = allocate_in_domain(A) fill!(x,0.0) solve!(x,ns,b) + + # Postprocess uh, ph = FEFunction(X,x) writevtk(Ω,"demo",cellfields=["uh"=>uh,"ph"=>ph]) end @@ -185,9 +188,17 @@ end ## Parallel scaling benchmark -The following section shows scalability results for the demo problem discussed above. We run our code on the Gadi supercomputer, which is part of the Australian National Computational Infrastructure. We use Intel's Cascade Lake 2x24-core Xeon Platinum 8274 nodes. Scalability is shown for up to 64 nodes, for a fixed local problem size of 48x64 quadrangle cells per processor. This amounts to a maximum size of approximately 37M cells and 415M degrees of freedom distributed amongst 3072 processors. The code used to create these results can be found together with the submitted paper. +The following section shows scalability results for the demo problem discussed above. We run our code on the Gadi supercomputer, which is part of the Australian National Computational Infrastructure (NCI). We use Intel's Cascade Lake 2x24-core Xeon Platinum 8274 nodes. Scalability is shown for up to 64 nodes, for a fixed local problem size of 48x64 quadrangle cells per processor. This amounts to a maximum size of approximately 37M cells and 415M degrees of freedom distributed amongst 3072 processors. Within the GMG solver, the number of coarsening levels is progressively increased to keep the global size of the coarsest solve (approximately) constant. The coarsest solve is then performed by a CG solver preconditioned by an Algebraic MultiGrid (AMG) solver, provided by PETSc [@petsc-user-ref] through the package GridapPETSc.jl. + +The code used to create these results can be found [here](https://github.com/gridap/GridapSolvers.jl/tree/joss-paper/joss_paper/scalability). The exact releases for the packages used are provided by Julia's `Manifest.toml` file. + +![**Top**: Weak scalability for a Stokes problem in 2D. Time is given per F-GMRES iteration, as a function of the number of processors. **Middle**: Number of coarsening levels for the GMG solver, as a function of the number of processors. **Bottom**: Number of F-GMRES iterations required for convergence. \label{fig:packages}](weakScalability.png){ width=80% } -![Weak scalability for a Stokes problem in 2D. Time is given per Conjugate Gradient iteration, as a function of the number of processors. \label{fig:packages}](weakScalability.png){ width=60% } +|Num Procs | 1 | 12 | 48 | 96 | 192 | 384 | 768 | 1536 | 3072 | +|:------------ |:----:|:----:|:----:|:----:|:----:|:----:|:----:|:----:|:----:| +|Time/Iter(s) | 3.2 | 5.7 | 6.3 | 6.0 | 7.1 | 7.2 | 9.8 | 10.3 | 13.3 | +|Num Levels | 2 | 2 | 2 | 2 | 3 | 3 | 4 | 4 | 5 | +|Num Iters | 20 | 12 | 10 | 11 | 9 | 10 | 8 | 9 | 7 | ## Acknowledgements diff --git a/joss_paper/scalability/postprocess.jl b/joss_paper/scalability/postprocess.jl index a0065ea1..e5ff05e7 100644 --- a/joss_paper/scalability/postprocess.jl +++ b/joss_paper/scalability/postprocess.jl @@ -12,6 +12,7 @@ begin using Plots, DrWatson using DataFrames, BSON, CSV + using Printf end # ╔═╡ 4725bb5d-8973-436b-af96-74cc90e7f354 @@ -40,21 +41,45 @@ end # ╔═╡ a10880e9-680b-46f0-9bda-44e53a7196ce begin - plt = plot(xlabel="Number of processors",ylabel="time (s)",legend=false) - plot!(df[!,:n_procs],df[!,:t_solver]./df[!,:n_iter],marker=:circ) + plt1 = plot(xlabel="",ylabel="Time (s)",legend=false) + plot!(plt1,df[!,:n_procs],df[!,:t_solver]./df[!,:n_iter],marker=:circ) #savefig(plt,projectdir("../weakScalability")) end # ╔═╡ 8d1b090e-2548-48fe-afb9-92f044442a80 begin - plt2 = plot(xlabel="N processors",ylabel="N Iters",legend=false) - plot!(plt2,df[!,:n_procs],df[!,:n_levels]) + plt2 = plot(xlabel="",ylabel="Num Levels",legend=false) + plot!(plt2,df[!,:n_procs],df[!,:n_levels],marker=:circ) end # ╔═╡ f63c1451-e3ad-41bb-851b-0366eb71c0cc begin - plt3 = plot(xlabel="N processors",ylabel="N Cell per proc",legend=false) - plot!(plt3,df[!,:n_procs],df[!,:n_cells]./df[!,:n_procs]) + plt3 = plot(xlabel="Number of processors",ylabel="Num Iter",legend=false) + plot!(plt3,df[!,:n_procs],df[!,:n_iter],marker=:circ) +end + +# ╔═╡ 7ed39f44-3648-4af6-9e7a-dc16d1f66a12 +begin + plt_agg = plot(plt1,plt2,plt3,layout=(3,1),link=:x) + savefig(plt_agg,projectdir("../weakScalability")) + plt_agg +end + +# ╔═╡ 15178e9a-4da9-488c-be03-74abba5824a6 +begin + hline(n,w) = "|:"*repeat('-',12)*":|" * repeat(":"*repeat('-',w)*":|",n) * "\n" + format(x::Real) = @sprintf("%4.1f",x) + format(x::Integer) = @sprintf("%4i",x) + row(name,data) = "|" * name * "| " * join(format.(data)," | ") * " |\n" + + ncols = length(df[!,:n_procs]) + table = "" + table = table * row("Num Procs ",df[!,:n_procs]) + table = table * hline(ncols,4) + table = table * row("Time/Iter (s) ",df[!,:t_solver]./df[!,:n_iter]) + table = table * row("Num Levels ",df[!,:n_levels]) + table = table * row("Num Iters ",df[!,:n_iter]) + print(table) end # ╔═╡ Cell order: @@ -63,3 +88,5 @@ end # ╠═a10880e9-680b-46f0-9bda-44e53a7196ce # ╠═8d1b090e-2548-48fe-afb9-92f044442a80 # ╠═f63c1451-e3ad-41bb-851b-0366eb71c0cc +# ╠═7ed39f44-3648-4af6-9e7a-dc16d1f66a12 +# ╠═15178e9a-4da9-488c-be03-74abba5824a6 diff --git a/joss_paper/weakScalability.png b/joss_paper/weakScalability.png index afd65aac76ebee79d1197d76b08aaaeac3aa9272..51e333b8d7ecabc0fc4d9da567ca82bbd3785564 100644 GIT binary patch literal 30779 zcmcfpbySsq)IExVARvN*0@5ulARUsD(%m3PcXue=DIy>O(hbrj0@6}af`oK;NuR~{ zJ@2`H+&ji^jPu9cjv=}?`-xAiHP@VTJzW$_NN|G!YOGW6|!w zJ69TEgYW~zNJdf&;pX;FMsr>Q0s=XLw3vvh`=_lLH{HjlH>i7qCUJqy(qeHGlwXuv z-=RIgLKN?qDRil-wzR5NFUiv0v>ZFHt{%TG*2dlZfw=G;(Vy-arMi+sM*db*dhB@0|FYRdcK`NE;$ zV#eRFaPKeY@D#b)MAArp)IPcY-yOdc<6*#xJM*8oQJ3nuL(%)>+jXs||4&Vh`$)WV zSo=Kqb675%9p`_@1(SHAp0d10$Bwvg)!{Uu(T&#DzWd)H1gITVX*^Obt3{c0gcpz( zoSu`O9=EKZboZeX^>pNisLS$FBvnrCx?E$OyIybYDliwOl#@KyM5Ca7oGjs6?YZw= z%|Yakg@Ew3(9&hy(q)T*x_kPsTz3WQHws6#PLwGM(hyVIx5SaGRHiS9x*Wtt4@cWr zR?x&34E*+~_)c4~q`l%A4mMeGeR-sx*`Z5B`E#L2L~Y>Wa~2vJ@iHWh3hEWOf21N` z|IM$*C*Bo({;sP-+i|S)ol+LgJK@;4IFYR3>6sZ`;^R+G#D}PmVLd2Qcs#aqaoKv` z=semI&P)Qs!13dwcADjDbMxPM3S`xGHERB`I* z<*VNlEl_QE)Ahj4Zujcqbnvr)4tAj5&6V5h*MIZUeLW9X`rp2N`~LlVQUT8@2_!gT z^Opu^yG6MInIsnKj)N&ZK1&XCmyXf|iq5x_i@O$y7l?UxJdkDnX(4)+L^+^_^akLZ}0 zcXnnPpFVxcgxA;EnQJ%q?cKX~FGv-hc_sE3E2Imiv71F}*T4p|33@M_v%r^&r#PR) zNS-5hd$)Rgs8o@qA*;+fll&r5SU)mfhPZD1b~s#-3yiJ5eep&wE-x?hyKZHStW!`> zMBqImWHXk!f7)O@$tIXiMoOx0a5qzNZFg58Z=@gu3H8Brox>ZS#JhJ8b7S(VkKYTE z54k>myHeGmltoF5D_wqvLY+?HuIN@G25xxy9c+e2Bj{rJnvuz@()?Tt4)E94MJ$rv z`Hk2;i{$(dCf`NKe#DGgdQ!a^BXk{1SleS-NpyADuChveIm#%s|L>OH)x%(cXR!?nqWvHf5r?Kb1S3-)(2A+KNNFd9`rT`s8H$ z4{p{H+}rQ(p&s$nhA*1+-?f(r;oF%vQrIauby~Ts4Y>PN);_f19m*-v z$Qq&5?)ej=VQRYa@%bA(W?jwIEdjIJ*XsI6UW@mcBiO;N%V>RfwSVKd_FfD+bdbwql1Gl zq&sjoaIky#?!gkV{TIZVNji#3GrE~L&cj*w!tC-YS})6~bDIYrv%$IKrH(+`IJziY zMsYDyNBgHw#9m%rGBSM!2ae+N69#@)EjL$X14%Ej{Lh84HWF9vW^OojCi1(EwrHob zmkw6>S*mIo8&ik06Yx4xi{@u%8;=)j6%-Wc>+5@+Y$-YXU8^)3oMmBRWVDjjfgm=< zX{KBM2GyxkL}d&+kXm{mUFZfj!WU8q!yJNwz8j+jFfpHDx8=wrar_ZQcsr}~lfX|a zOl>}2ic(|ecbI*eJ_<$^uM4uFnwpxD61Iz>0%hE%7aoos;kb+%rLrA&FX7gc5WSqDuo~PmaN+v7xVV_`HepFb zM@M%>lT=n$AMfo6+m-G))K8xp+(8&(#GEs08fSYwij3q`G@-}m_3|gq2SlWX>}c}r zk##cPfAePPKCXYizx)0BH|$_#_Bj<56)mO@Mt5xx-qujx94d$zJNqh0`wi|K3Z|Nv zn&!@KbH5p=JMif=9N;n6*&yX?a9WioB1(mT1qUeAstjgma$f({9Y>WZiT2jD|6}dm zR@&Cr>-0?Knu>?E*W-TO1y_Fmj&|p=qrQJ~K)}&<5cM%M%+zJx9m))FG*nU=ZuUBD za9&^d`*-NOl_4obe$3}Vsz_e1oyC$a1G&k=ntaN##cKzaog9WTt>1f}_H_}Z+c!Qx zE46a!{Ij5)l`-jTMQV6IjmN9v)^BIXk=4OoPju-%WF6rIU|~=jjd! z8CiNtN`udZt2jmU;{7MgFYNz*55=ZpV`Do%T0h+HBGb{*Dp4zeno9<-e*d*h5f~@o}M1bXySU$V@r2;_jbRFPoF+viCS1#^ajB~ zjgh6ffA3!A$@2WX!^zep0lR7b)iXp)Iu@4nM)!^}B6<#XQl$^8_LgnZDvURmXSiW~1n`prP3K3nuv__7rFOXWKi>!Iz!)_e;?6}3X_SgFxJwKPjY zUg5HK$o#~dwQY?JTbXm4NlTU2c2>2mmsUEjZC`QQKyH44kg{ll)zHw8o1HBtAz?fB z&1d097#an0l9i#M$V`kPLn0e~q*9iIUZZRO#6&HDS8sp+_V0?U1-9UTfS2;}@;-?U z#6mvjRYtF)i;AouL2eewVj+uQ^_xF3CnfIk320-xcoxfYhDi}9Wwd)N;MPt{+aY1~ zdUtfbl>HqwaxR?*iQaMG2hC{u5WKO6MeJ3-oe!>L+1?nrS}&`%^Em%}EAcIw11a)l zR~?42u-&}Z(>cF7^Bvl}PT6NJTLF;cQl3MQmAeyp88wwofnZ}&n zRhUn+3d9eV`^4h__XUYjPD)BjL4k#y-jEgF)zuY10l^D*dl`BEunr@<&Fw7;BqXFb zDOCfLWLejPC25+&<1-Q#7M3SZlFk_-#3(zSdA~yu6Bl1S@L48nsxM=cNI38Jb>8c& zwwh>aZk{VE3#GgFPRT-SIrgJ_nnGFjN6L16A@RFosY!}A-{L7Mi0?VnypLt1443mt zlTdv?FeJrL^;45ttu-`Si!e4EJv>QqFr`B30m=~paJD)^e;sMxGB@6jVbXaejFD`ofb!M6`(qeBhjDM)8orU# zvF9kAsQG9&Hbr{?pc4&U-L3iO8V>ie^nSGx9r}-tluAU67r5=tPEJk&$~ZYWc@(w; zSEN^Ko7Q>qU{w57hBv`|>5l4N3%2r9|g zptbVnU-Dl@8XKAWKjxYYTQSWgva)GL6v@{hM>~FXLB@pStkubQ;Ew0O6oE5uUWIg$ zMCOx`X1h#=fKV0h8L-zHfC^y9eeW+~^df{Tv`hi7laDDW@*i160{W3kHcc=Ij{PIO z<(rB{-tfO^ME&Kn&59ePiF)aLn&rEyJv?({!S2KYjuYKGOj>=+np zl%^2$^qUeJZUpi)9a%V1tyVYT=1CX!bl_8_t5Yrq$x>Q%QTYv~v3@)vgZjTXx-@^?_XMwlnQuZ*Q)3 z(Q5UGx8c2{m}%VeNijF4twrXlITThot*91Q&_}v61zhl}DJdxsu#}6kFiH5{cNjg> z`nvSTuEpl$^Vaq@oTlFEbSIhJY_`rJtFW-p@8-IHmN27R`bYuWV~-vN)}FhahUb^> zPFK~ud7~CowyqpiUuD(Q#$fW%(a~>b2MR1lcF|sVqphtC%^xdF%7BWU=$yId)^cpB=YVyEsQ-+B4+ zrA_S}bF^ZnEDh~9CIo6blKJ&d#}5t;Uc7iApT@^&+#PE+`1$PooRrrQ{!QVc&h`9x zSXh|S4;=t@M4^Vw+D6Yx-bZ#*Q>H&&VX}tZ(jT;9{l!3q>}by`MlgVUl##FY2VwN$ z>F(TMPY-s)T>YVLCGF_V#coSYO^xaSFF*f7(ic2}g5SQ8XAheLC}Jq(EPR;%RQSgC z72(hjGA)xKa{HKS%fUkg1a0XVk&p55Y$m<$;j)@t74 ztdTL2zyumOHV%##9tXw~rFu{yV>GXg6-@wsK0n@^0N`C*Ts$;16eD;p^-Rb1DQvBD zuG7t*)097d1k?4LJNGV;sWbcdd72!?QrEwpEX-OhLvOfC6=_%16+vqzftB2>K zBqz~AP4)TFQ%am!fbo|`ddls5fARQmPI{tIR zV(Ep0){@hLI40o)0qnDYr{!~~6KKcix>Tc#Qw={Mcb(!X2Tk;oaPH@Kn$yp}j-cAfw$)gg07NKpFsEmIs zb&tRWWzFTcMhWT#LVWN2o~p8y0rleX@way<&QEeLEfMQmQ%@Y85Q<&Ly#z3&GRD0r744Ge2Lq0cW_D)X-wKJ=Y?{&;O^tzrUxaV0^h& zr`DF!%M9=fDMU%w+1R3N$;Fd@EQF1I9kbP9SEKjMv@`3=HTc{N2MaxT|KMNz7BSA% z!aP}fXR&?_b^T)95~ZV#pSHUhe(61+T=RFxo$c*e+1XWgKLb8Xii?Z;`0-w{i_dKA>?Os;caV@QIHusz+uGZ2{w+Q%{#>nMe_JPT zB=zNxLcKB0t++fi+$~O2lZdG=>**KTTPgQuoR#OH@weUS_1METBa{&RaW|So|1tu6 zwsZM^8GCH0C)i38TkaXDX#V7X->zdWVA^man~Pcz+p5u+S(P%X>gin_Ece;`sR;@R z(NI;bYix|ce@@B7R6Nkj6P?cI@~Wb(8X{~_(dNY%kiy6P4^v;Yg!?tWim*v~RZH=k z;bFjiqNnY6R4Dhc>qHjFNPqMR+BUl_RCldNMQu4Apj*zEp|9~+9tGCuAaWq1^sV7q zmPdNlLPjIJC8+plX;%R$LIiMfaHw})j~AmXIJ|o-u8{>u+?<2(*wM-9P$U2airw*A zPOem+y}zvzl2Q6~_9AQ8Cj;6h_)!=JtH)crTXQJV(u9wxawCbFO%+Dt-gHeq*@+PF zPc*Pq_8xT7efwqZ-V`#7LhSlO!YCAxe4#2K%cI^z#!A!vsXB*67==8!RCQfl-PQXk zK0=~U!Qe%xX=+N%{DFTCLLujos+?b3jN?#ZnUwOoq0YJ9VVy|?D_nj*(BX~7)l&ML z6UW!>g9hyrzYi=4caZ$74T)f$t1RpS?6-bb06mqpast#qL9?HwiOKP#S-M;*m(6@z z(9QLQ9dKkJWWK-KY+#;KNG3I|Z7CQ}h3w`^*Jf1OJ3Bj{P*TzS{G?~QKwaa!fFrap z-;pOzHvC7e0|1)hY_g&dB=DXD+FQU-QpHZ(b-Fc~@afZZfl8jo;mZEm<+)`cPv&Rx z(9eP|*-)6N8t;B%?)B^De{p5_N;IwPDH^4KfB@_xC^7)YJacChwt6iJET2`Nr#P+p z2Y^W*+06!Eh&ZQ1PYb?%l{GY+|B>uOiW9+5(x`*}wZu55>-72o#YD6S)&?$`X!DmV zt3aFgX=yX{i>O&-%~QUdhuu7=w6uBWxyCRk2)kB?B2sawF7}&!Zmzc8CN{S_LTj5m zGBOg7pCJ-UXlST;{|Sdm1=^&2)eOiT2sVn_pU&e=#T z3bD4=1aQp$V|gr3clY0ulhv~W%F4>|2??u%>1DJ1>aU)2kyIEypaA@yJ2D;U876jc zwwk{Cr#433?!{L)UrS4i(U`S?!5o=i@hKNJ3!|7iO7X4IQjNRD`c8OgODE$k7wQMi z5ysI+RvT%bznfO#*4C87tO)88aarR(H~N7}=AH4%ge$cfNS==Pc&H9xsT`}}y&u}D zt3qLTdxG4~s%fR_$C@mQCVDk|JwYg^pwQyFJp~~99Two!6d)a;(B-!Nt!QRK7`3v# zUeNpH4_fA<=U;>BAKwm{xRVGzE^f}h$(b1+*rYIeR`Xm2P3{$r?SmT1JFg`^w3tMH zUHqiK=kYnR6dUY0&bRI1K!sx{p_ksE^dj=Z1J?s3`nhc4+$z-^s3HmhT=l>@b zIc`r?M@C}Y;#4UK2?-;kFDs`-Y2-hb~3Bs==0 z4pJXNmH0>70XZ&K*0tH@Mk7(5v;#(QPQi3mKIgLHVh%2@Ttj@)#%gw@A4U}%FJJYn{! z4bK1m@r)b|Zg7ng?e>43nExqJ`dMXKl@fn!%;SJ=eHrp#d1;A*Z|2TgT~kw2U|=9{ zaia|y8jg+!zfj6C-1pmNX7nm*e8t5( zzWH1jy?G-iE9?H^bM73Hq86MNLKL%p)5o;5?Tw8>d=wBBo1uDgS{-oUx#i`>ui!jT zUmF`6*EKXS?DSIOMNm;v4t(Mbz}lFuuOv*Js7YAtf(5-0!J^4D4q%N&{!{+;%G;HQ zhzKVqr;DA&-EWr%GRp^4lF@-7AtUQY5Xvw}UwlkarAq1x!60}3i3LG*rq1WFLDl}yi zLjnVd1-(D^7|TQcfMRCDnrzY(3O-1bqob?{dFuXh^Zp8@BTN{5-YvE9yu5 z{EdNu$nn(TlyOl}a*mE{((wT6*8kM9xUAhm8zCW~lBOp^)NIK=$Hz~8<;YM=!|r$8 z9FIDuhFhzXj3HG`gT)gs;V}7+5A9M*EOJ3VQK_g8~D;uZoxv5D?JI+x?moM(gP4P?E%=$ma2fVm7~^ zVE^#&C0fPg7RAebiuwEh>jem=6GQ>wZ}b;B{0+*QhBL|Af!=`uyfCrRuzuJKfWuWn z1gWW)V3&xCi^G10<$`hVo(P&DJ{~b~G0!w2A|e!OXqgl*gWo463iw`X>Xlfwi-17) z`c|?tH#190PcQ2@L8xYAV&VWEpMxWtbZT-E7YBzt5J&`|SMo)HhBh*aPfs@h=n|j3 zK$V$~;Y=WcHRpTr%;g({5vG+G$!@U|@{Vj??nDH$+ECDsKcZ>*zn!7EDx%CKNko)86_a@{rf;H1{RhU z-z!gh`|=v1yQ5)Uiyoey%5KFUq^K(@Dn=6nfNc2vTXVx-(b?I#OuzY`w_rpEIvQFA zL*Mej$B+0OV>B57FcrR#aUmGe&q-0ki;pkm+?5CySDeK5;lqcFj78wZpbCV10R;-i61_|nmyT^h>^&;3oJu2TM$D`SXf0>)#XM}nYWJ*(w#f`b9DaXP~MG{(W}!DH7q) z-uz1qjR`<*1F77DP$5oCBoR_aF8<393AntxTp3Kav#{`(Z@XLQ3puO4{w&GBm)Ci1 z2o4ToROtQl<8x4Sf!QptsNnVfS5Bz&du3NJ4wgVl%EsCnJ`N6Y?TSSs^`m=Z^?Bu! zTknuim4$?SFL^-49v^r85sDq|Q8%j50R%YIFF@YY>;GHmh=2{1oSGUE6hyx=e@9P6 zSNGHB&!xIBe=noL3kwQ>f5S#Y&dkj0UHt~daza7^P|f&{9(}>*p~8!JV`Rj@z~Jxi zKg!=un2#+T53D}y)lyXkh`vSX>FEo0)v`1ZVPW+2^aOm)be#Ud@rj8p%e}ZTbmir& zmf3WoQp`yPb@mk8+-^W7l9G~IS{tT5yhRKdPp9_&0}>g_|G$7lz^&0DW8(iWAHB)f zKL)al_(9kA?+6ssaZ;3&l&@rDVBKx+?95vBgKGpq`-wz+Hp4OSnZUrpI01i(B?=ix zNl6Kf;^oWsA3wyPa8u)AgJcbeX~PU8{+~a8u5jH}Wr`2g6r!$yr-bQLMHhrwV_?_- z?H@Nx3<(9}ca=p#I1$W)(DlDJ9{;`C>FW&JQ2>7}m^u(^RM42C{Z zYOOf~-%LgK3kzFyPVh{*nnr6fsrUJ+VYdDHbsI3QykVs=CtW}ZP&v5;CJi0|0ljB( zN;M{Zu$~BU89^zl=0;d!a`8D5NhMF{QN5AJnBuF~WlreX@U}dk4)XCkn>fQS{WrLp z_xo<2dXY&>AxdUgn6#D_!LgIGa}~rWmx@^Vy!*&^#!hR^hbg&I0aAH_UVu^x{TwX2W3gdbh1Wz;^X7j0hMsh#fXK5 z!zt1@kzpP zm9U^;Vq(H}u(Gr)>@}5+hXkoyftf+__J6G^!u{FE47dq*Mc=E*t=w097y9!_hBrl= zT99KPNOGSdpOZe>ug8yxXwWeXb8f;(A`ez34lj)4wXVM8G;izL1RQw*ATvB7LQhwB z&6!VBiXrj(ppV7Y*0%q*gNqAiNV`SneL&z<_l$VKfYx7s2Rm6U=pe}W9oF)FX{Ue* z-mcYKsjQLUE9ZAUhka%|>rO9??Qo&A8v2#Sl;^2f5--sx?r`%qyYDBZ7v!YLU3@ly zqeDUJ-fdI?5me2)M3rHt%0lJ!>x|6Ic_?IUXPfGQE~nWhfmtoZp09qq<07OWr7krq zL2|r{O#5ST=wEPmD_T~IoA~{%d&H8vdE}zz%kgE53=@}EedJ-AuOrDdlQmX-URVs! zvX33$5hhCl4G#!nZjZ+Oh5pU;717M&=tZz?L9qQSM(N<@X7JTm8gkY~k+rGm37}Y9 zg(!6m4I(0?b>6P0imBy+_B$(UB?gAN~)2Ni9A_ z?-7L6CGWCX{oLIW5APz4~f;V|DSKFHlj;! zf69`#UO|(hG9}^dTKpm}@$tFJW1A6mEUtt7g?)E`5bSa=e?yLsco`B#2Y4CI^v7dq zirpLsW-#{aIcOh`4r9I9*wn*MIT4UyNCdwOs9=0NJOC=^c|ND5^)K$fzNCd>GXCJL ztoQ(ZaK0iE8}7S2{4c+Hu!@}M!arMyV&&%M0st4}=O?1z09`2~gB+tB1(UGKeSZOJ ziYjebLBa}Sgs}xYIa*f2xe53Fvko@PHx5YPV}8K5KvxSW&&{P_V)_ZT0#Jvb49-%& z$r@P)>QK<{`o*melw0|7jj78de>{{V6x zNaQ%1x{Z*Wc8YJqyS&Q-rJg4lNu1OX*8^C%ER5FFPZph?KY9G~ea^P)CMujeKfil+ z_H#;YW>dMD%99xs2qpysobCCKQz!3y9ZxYt+GeGFLl_frYjcdk=tjm=j*h2rA| zbhc`3;7o-_M(a%R%^wpcE0%2m83q244u|RP?$93NN61R5syL9%2neK0T1zDUKL)T= z#j<41cHxAzwLnjk_F&9Ls4j1==5Bycgf;87F-rg91qe(%9E3+_hzJc*$DHjrFI7hy zZ%>pmL^Vpih7CTj7{#*uUsR*Z{~gIQhLwtH5tKM~)1PbOB|1f1Kq6VbgU^bTt0n*a zZ4H3h1*$&Zt8+PkS#WU6rn{XN8IoulNCIsqg)my5)qC%E{JyXXrkA&9d;9Z5OmbH+ zQ|kc*+o zE=yMRj$F07w>KPEv(QlY>r3AB!5gT1R|e9GW=%!@$H*o)Hi2Q~EKD;UEL~QWL!J#f z*6jdBJV8Z8E%HeO)Z~m4n1qHa@{n-#Uwe$mY zOoi3E;ZLY32$6ck=DMywzCX|!&GebF)xOYBQCZ|PcaPqk`1hp*Eps8|>tLd;9wqCj)>&U;(Gc#JI&^ z8#JY($+B=V1lU@T(O|GF=Lf$1TZjOL2q+s4Z54(@kga>a|2=*sN#?!RTIX>5`7%z5 znh9^ZM#jnLv2BwhW}%wqxXC~6ka*9i^{FYZq*vGxPi?lF*NWemB|H_`oTdLV+{bsO zWz%G`we#m)#3WI?_k#r|B?Flsm6G1flW`G*b>_5*Rjvs-Vw7>+%Z?lJe=}yXQ2Jh5 zUfFiu@=vH$7EfG&TfrkBI9uvTfC`mKr$&|u@8*273|7%L)bLqlQDI>P#l@faTnfW4 z*f=@WB1!x|jnLx8oB%t_VS_x&Fin@9Y5ZMw2u(4c$e{on43sltCpM}i_={{E> zUEj(}1`dv=ZGx|Q-RX^Qt9W%{iEni1R zrJ^E_P9x%4=~~pd2g5>`-F3wN78h?K`S@CcC5l?Q?daZ>NZ!c0bHj>IPvL`0PcY4Z zTmXbiYH~7Ac$(s8;5aPPYs~G%pvQ5&T4FItVx}`?oYi6^Xa^&-YX*3k5J0Q@khvJ zev5#$_R(8Ym18e;;6}lF4;SzpLq+i2=HI`uzyuD(o#7~#-Z>k z<%5ZfAN9mWO;pbuklXt)UZMqk5~#QMW9Q}e0C&;xCDQl;i(s07Rd9_?;gdSuo$c(t zIo}qFl_W%-FeceZU+roRYq9*AKX-+RaxJld(4nHrIxRE&kd(x|Slm%#^QQsoj!_Ln zx%h-N?*|p@%t~~k=`J}oWZyYd_{e&U*;;fO)*5pnswRZ5OsuV&4*i6-M_t(w%(+$j zSsy-(0cq#x=m-Xj0bM%&h2v7FqlSVwL>RjbnYW#)uZFK_e~f#H(Na}?58$d{$D~O> zM3{Q67BJLx`=VuH+5*0hiG_t0jgp?;0)PorI}nXZro=LyD=_rTv4;jIgvF6qn1zgZ zX$vFVL6{O!3&3hGhXe`k(-4(h+00kKZ@xA*u7ZsU1iZQx=@7EN*#6m4*~2A^oF!}S zsf2s7jB;fjK<{Xe1onA>odF!^K<&v(OM{bheY9Yt+_0UAiAlT6A)|V%duZP52g~U_ zW+brAxH80|xE-M&tfkV#IIYh#)*gtr+O=G%Q$Mnq`=(b?a;Id+gR_1UyW;fu@EppY zotF={5X?jn5j+cXMpVJ=K#zky_6sM2VB&ty_QTLRZ==Hd&ulQV60pU{q3NU56Z2g3 zRmNJGi!GNf6C|t=@6wFTh;FKifuu|4haL7Jf9`mH{YEh6`Ctj ziJ05i2>4xJ25)|s>v3rvVK}Ho{n45yMxO5KOxN6k`ROiHA&DH10=)&vR?s%;HiMRO zojok?@wO+r+A(DXP{6$+a3ErQuqgWB==>t$_a}K)3v*lUsr^ij2gI^)4!(bXCQoqC zhD#+6(g?(ah_;fh%jqxxe4`;7>U^n_+);fp)ju3L(24a95S=n)8-!}QpEX`$Vfa4h zvjUE{I2{=|Q=Sn!tA+G2KN}h<)sf4?I!R^(g#2`g0Q=*OF_>{&TwHLJ>o>XY$5OoH zs6RCYR}QQ;knw;SI^5e+yg0dk|Nh9YUrUblwYA*ogBrdYw!HQ10ioXVky}!YShAZDSAufNX4bynnI!#s@KQ4bI1Dol{t z%~6d^7N$;>E;+xoBN)pS4Tzi>_)Oi4~|_CEUs z%yFWe+J7LPyBZPG3oJLgA1!j6jIbs?F|?9)dgR(R`)q&Ju4%F9BAe@)E+TEgG~@`x zqLglDXAEi3HallxGtMY$7!h<1_F5V*`TuGNLVV(IbO*YDqAMp;uFuOhx;BAm0a1Ze z(EA_og12gKRjprkKNOpD1`ZAm($YQP=|c10J3Y12e#5yop?C2L=>xqoC~vO!yX7XY z)`qFLxobbz-PF#Y7MdBjr}?8%U_6f0bTWyUTh5X3wwY(5gLM*JDB=!n8W6$q^XX_3 zCgZQJPX5>p4-d2GH$gBU0~EbkM1zIqdAt#1SVl!bPW~5&Ad8HzUve9!{SGkBWXsev zP3HyO=N?Dr5z4QwsP@FRHqH>D^u>zrwic8Yy!-t6CGwqjHK_<3ATxU2^?Moh2O3tQ ze0^1~+_iLc*zqIra-rse1+zI`(l~7p=dxo~Ru(Sp3+SXGFDT?fkSty%Hr7^WBTCf@o_U4=6ys$4K+b)S8I~^O}HYP>TcLG zh^vP}DQ-~7^rwzhz&hiJ9tk^sXYhl$)FuVCJ$CVXhvOB70WPPy&tb1<6rxZ>bTmNM z23TT-vm`+U)%B;gpNeUxIE`Q~VTp$I-1 z85tH5;#-rwI8;X`eoiTyIFH?*qLMb5+uf>?GeGl@lie? zA^=8{OchfUc1&N=SmkUL^TmGY^xy*`bJXV$vCr>x1~erylDnB=)wa*yfb(SyTkU2| zLtP!J)$z0U8>dvG4iPG$XEW<*ks18(4NtK#a&O?4APa7I7ka?woHsZLJ>pCHe#gZb z&_$I|0(GCjix;*t4LQ2X(6oY$4cuvsVSBK;`9}$CYVExkn70&)Pk4|C1hO{Be8I5D z!cwLk1Z`Qs>cH;<^RDjJ&aoj()B1hL_#+#H?DM=Fzw>~A09b1!A9BrQqA*HcQTgsG zbFHT71mG*Qes=-|n1w;kM}hz220RwH0QLYn=%Cjkzpj$kX$2hr{yoM}R6*|B z1ey(cbqpU9UX)r%Ptbaws1-CnRFcmq$IO3uTKsh@&_qXl|aA;}mtFH?! zm-8b-h(aY=_0GprEzG$eF!jPcd1e{he+bsmy9ys+zKVK?L4eiW$ z)%!_)`|M5NAS$U}TBAPKDpMP4q3#yj>q9$4vS^|iU_(r~k<4ddqd_km@NLyISQ+=A zCB?zX398}8Nz8h^-Q7d=|** zwVxWzG`bDW9xFcyDsJ&B6L`2;2YRqj1dYK`vToB#X7YE&@D&M$i&Qi>vl`w zv?BcXfj03KD7=N5mARm^0iaMVRa8|Kv;P@eM@W#;a`;Lbldah2SLD)1foG)@jmhH- zNfrI~BV;TsEmQWH-_}6w7bk@*DW?FggD|l>Avy|D)Q^&7b7$N8ftA&}mdzo5|0?I= zbQcWPE|8+ZI0Z`fbiLE+N_7EPTIuM_p=kuX1TIKpJjC*{vXKQA+iS{ela;g|yqJJ zWDLP1T3KDKg{&KF^>uO!5ebFkMedQb{yO6v?-wrz_ina^YNTK+txp|^YUosWgz}TJ z+%;bc2ka;jq8x^xe+)sD(6ciL_5_qk~ivitDXmz!=8iE^KVZ^ju-Ltku=ikozzyin%UWJ^Lz|xgwW3lG(zF-Bs3uv;0Ho2`WQV3RJYNW zpm7?=;c`R#MyOJx5`aeCl+@Hedo*>rrDt(ZJ9DZ7Cm~ zDAvl_=Bnxzy18_Mo<-n@p@H$cSSVLXLjxZm;%CoFz<$}g3~q2C$0aGiu&`fn0>pEa z*TLjbs#|YlY@FRx{t~=y08(!4+6xQ6a_v4$?g)U%5zH^Vyaes6%MMQ9AAvEIlb5fl zu2v%za0l1~P7^3P;Cw^oWi74ugW4b|f};ZB@%MwuhAjw_ARH>l%EDf7b#zp|q?DnK zhjOp8qa*p_M@=0a#tl&B{z1JC!_eE(%@ZvtDQVE)3>{2B`*L!X*_@*J0ZzX6m}Y5p z6{u~`&PakO+Xv>~Kt%$X6}Fpf8lPJI3kC*DFd*{s^0wSudnqbniD02o_>+Sm=lTz~ zIpu%709(J~5)#FRH3_KY(X^DwraBQoI zQj67o1Y9P~a&o9m`1!p68Gwofr3xBFy{;f=n$TlmVrF)6aRHsZ(S^iMo~$k|9zs($ zh~n_(t=tJc0p26Ia-em=vIX`|)$F31ONI##uq5r%r*YI;`ueuU#_!|eGH;y~kI<08 zFZVyDlKLi~9w1x)=lqbA>;jEBftrAb2q+3zo>1RAuPFiMYa63vaC=ZD1pdCG)+JSg? zII9b=t8~09EUucG(t-l`3Jp!?@URjycwE(N>Lg*cJ@Y3A;Tl^Mz?QGCFYLxq*DPW7 z+x}Ksxz*Lxyu3WPA3_T7d-(?hym|8mTNEVy*4Ebl%p0)hASHwHsKAie+0~_z2&P_G zcLlw|sNgxMn6%z7TRu#OgA#LE;v64BlS%=VJBTdd;piK_RKQ-sI?3sU;shVOLutlD zoR+E}Pm56w?-3!&LGv#~oK$|oJbWLd`Nk zeITGFr3PUMI*Z7ql^v?3l?c|@;;A)Cw2yme(-rGK0`26D41g4fH~1>lEjSv9I4$ci z)ZsCR>4UV=U(3rCiR|v=8$;lMWoYl_c5^&oP~d<78xQ1GYc789{|E_@wX=%XFFfIe zk=mbp&6YeqJ`OGvuu&w~wP&kHKGcYaLM+J~f1X1+&Nm)@NVL`*6PTNFFU6uK z{wbJ1!Z!tuuka>LQ}tK7kk)b^3SHW-4P`@OA{E;1wAO0-1$HfiAPgdlktKU5oUBC_zC%y)$ST0nob&?lEw1 z>}+rQEOlca?XY}ZcuybGMFquwsdpW2Iwo;fUvDp=W_D%4ls%TGhX*!tTztGgR$*zW z{Hs@lT-G0Nog)RX9(HziMm5=pyI@sFkt}@w{v8k$uxx=tP^!#KPeV`DYbXlv-2no@ zJM)JbeYZTH53eaP%+z zf5*GJx+>`F6rP}Sn*sO&(qm#Gt|%QVD|`d5%i+IY@jBixfn^WR{%G;Lf$dHeh~{5s zKkvU_-vw5wm#xs_2Fn6$8-)WjHF`QV*6Y*tho^IXb7clC5Mj+5c{9#>`}$~QKeAf? zX5~v0G;9k5c2{Znwx9g#SM6#`HFzcg?1=&jf1YSaCw_2!Md~GAYie>)62YW)c3{HJ zUI~u8kdP2yPeJi6X(k)W_|FUj)gi}x)bzA&JhhCB48O|;hs(w&s2r&&DNxHOAKyNS z0%8id3og#if`WsUgYLZh4(3*{d%*X>`vwmt_&PrO(0m98NkGVluy&ybsR=3$u)o9H z&`2xOB-}bwP*MF(XWZ1))$`=jzHFom`OUVq89_&W7Cf>8@D8WoC+Z$>w)pwoz{w8J zE+H%wRRr1imJEaO^8I@ph?NeDovkGvQ_OKv&VYM6paNWq`Z!NXfwSoo4}5B_toqXM!8@lpSII~b+Tp%j8t3xNx+Y+^|Vp|IWIvNMVZM9H|m zWxWPx*hos#Z2ELKr2MXsz?gs<33w=|*mUb)gVYXf3#l9y@e6{@$_2M>G$A2BaFiU+ zw3T^GdEY*o2I1LoaM^#4cX@{Te|#*>|GS6T{J;AaJOT%txB=v$!JDBS3zz63LCA>_ zJ?Qrs9%BdP;HFprwHZ%cJ%gOV3s1(0Qjg}zO-*Nq6$jEOv;lG{c4giCMQyMB5-ouH zG}q7XV4}_8O;@Dt70R6wAcvo`vHfyod>b6vVFb2^*Q`~yMa1oca0W~(Ih$^uc*Dg2 z4oPGJdXVDt9oH7pu)C_psA0TplyorL78Z=!A`Kcxn_e1@>Awe?7yukEJ4YHIOK(T|9pnsF2a~H;{Kf9#d6-170&I)d6)+Nf-d%h zYxI?*S8jDbec!&Tn_e0YI_`Udxwk*;+IS1|p4#xB0eCD+6Y>GTaktCXW%IX9`}I>$ zF{W&955tuU7QY-oWTa)fe8~hKldH!@nr^?Dzq1FAjlqiOAhdl|>3}a4Xw`@f?@#4I zUsF}pXj|I?$Q3Dea);#>Qh}QbU*R(ZX9JQv39zSXGT|lkSkLDk9vv^)3TvB2KweO~vp^5{3{;S;5D5&~}SX_?Kw3x=Tr4zsil+t|rIFmn9Q=q&D1_v`F zv1olYhLz`cbvyyv1=?%hEpG5;_GBj5Tf+4VAC3xW(r|DbZI-o=zHr}zXOTc~4>kFM%j`rp(1;4nF(cN zRfNd%x$ft9-#^|z-s8BBL$}Mde!uhluFp8j_D236txoq}5tRsBCN#M4Trhj<#HE1Z zMcG)e|vk%6)!=EFxwAq&M57Y|RDwFp8tFM(7gK6&zc;r*7F!zD2M8mxFv zwSq@>;?*t zOG%Z%ss%MJvWy}DrG`1<3M!gUpFU+96nw+8fMe=3;SQT${ZH7TtqkNV0tM+m9IImwzYclPHMb)lkn|tF}LI5z$JXCIgK_84jx^#1Cjd9?q0Y8_n(T0 zgh12q)_4RQ9g0aS?l2_|kMVPJb8~vx`W>Z1SYPM~40{SB`I(&aFrAZAWqM8UkV(kP zTf(a`J>lWuv1IJU0&&gI=lif@84J8Hb?hPNWN9UfKjD_q9Ovu zU`c=j1$8&#Q~UF8*&X10p~XPu>03S4-Peb{A^PdlmHFYn;7QCGzkix&ex6#9lHTG& zAQ3NasB)Wa!chY43YrHPOu(J#+mbsv1n|9(N^X2#FQN$!n28Gf8}ND*e}6yVxtCg$ z9UDq6yS4%A5H%M2j`2UF2bfwyf@D2^qusoDnKbC9s-AFKGs@MT&6{H648_H1SKyn^ z1ntaRUsdX)y?4N435@6D_;Lboz*z((etbdK83tCVXvCSskiO&rtm$Y0qW~K8IQzNP zE{Azw44k2TZ>i?soXN+hK62=d<)m|WlB`Tnw6GJ=1!4LkvFro3RKkM?$`B~4ilOR~ z02GXV2gH16IIa~mkdzL!pFTYvUqTK6BUfAR!M3*lE?=vy52%4{irG_jY0)lslSy*cc`c zpkDSY(BdjeNlA5D7orV8NyEj>y}q_)W2^^gwWTHGTuS(f2h>L@@kpvGCtXnQ>922Z z2VFAxlDRm{!w@WOPpvHX{iWvdN0&$MmWR;J;G2)Eu6fw6;DU`vG)Alw{BtIM{7}0{ zPC*fRNH{e{4wo2y3yqV->A8%)Y!dtd4@2`l$1fdHPQ2ZnLGS*(!)#{i;r*3GvF+xn zB2j6#GDi9*Uo~C4X{@3zw3|pS&&kcrLHyPd8=a8QY9+*TDM2lz<0RE! zk!`d#FC3BTo15)UZ@tpdch)}M(^HG*IUjZ4qQNyn57RuR9VYN6;f}m%`2o*(z{eK-^7&bc#DNBN~o4ZSjjShZ*)} z>19flG2pMt4!5w+q1Qx7-UsiF!a~kn!HMaAl)60LOtp)URy{M*c6NCh()@ML9%&37 z!s=|dq-=c7`lXU&bOY^m4 zKnS_-DsnvpDwDrvTk8M{dI$TtLTCRCgq*GBbiZEifs33N*~LOYw4Gf5*={M1bK&AZ_Z7G`8JOB9|24wnr zu^3IYvs$+@ZGwUsdRTS-zB}65peCiBMy4>%@_*&b8?9 zyqaK+SIu?{f4vjVjA6W0T;_KX{E3h5#Nl03&4W44-R?xD=jSG!;1>s-h=W5eNO34! zqTj#w*4jfuu7SAuD?&Td!y5huoVQ+_%1vfd2yD$}D*h7Pr6H9ojF0FR5%7zD--Gel zSrQQiQ>KzU)fhDrGY4?yK31v1jH{n4ZjsujYyD6ez-vy#G#fp+o!wyK<1d4!@2;es z=j=3(m;cU=-B(7LxcPzJyd>F6<-s6ktnTCAm4E%f_d-L2>a`r|)V&GZ6TjB%Ga?m| zo`Kv0`opC&o88S8c4F_i)KkDZiF+1YlzXtVDZzc&#izHxjj{ag@UG2Qp|1(n)T})s znQP;sXu>eS1GTMuiJufda&w1U9a&yT-cD6WlDPeiD&$<(&7(8hw{I^E*6Tg(ecx`# z?7;4(HShTs%ZhedSt!%1sua*Mt!$Yg<4h4nc2&WsPQ0hEze=kB6EGD{I+i*T8wtS` z(?#K!W`lxj!x&Ek=teiCNPrzkbx)7ZZ01oSBUNqGZ5|%ASgorgdRIoGO|?F_$2rfm zU&N%Zc2D$^C(VY7%foToo10a1)+Ii;eMmAf0v<4+iS0>d5+9JQ5&ul`_B9oi-OE4W zn1ezYYAuDN$GlY3)LTCa+C*NDRAp;K$gzQv*VxctZq)CsD>c>O9O{u0Ec@1Y{O*Uf?_w z;orL^V&^7%``eEpEjOj4Tv-|mr)%=vGtjZ?lhtw7BPN)DGK)Z+@ZJqc^y>V)KtI0WqdM-e#Z8^z`)1-dLfS z9~dL3ad!b;xNR!0>3QpohM1@F;VgzfcjstTUgy#39=E&f5|AC*^sVqZsa9|nd;XWC zEp5}$($;Z{nlyn0)=1Ft*BPZqUe#8JG^VN<2{#{{#JJ6Y$?uY82bw^#4*cC zmBwe0kY24076S9^=YsEOE#I%t@cGCs>hg8FZmOSbN)}mYw-Fq<=d=IJqY65nnvT|i zN{W36<@>udI%GL>mA8f_SmT<1#+T-OD3-6c6IV!boGzJNU)%L(d5>}7SM&)3;<;Ee za(#wO1X=|k*04}Ql6S}zlgj%#I{PAS@V0k-ew@c{@6~(EZ>P`5Ry{qYJ@q+8Kx6h` zxwp68!uWXI_ai}r?*$j?^Di8ye5CR8!oa%-Xt)k5p8KbL?BGa=;}jpW~abFF(nh$Z#MtNExQI3UTF& zm$KF72#2Q$=h34_B_v8&ldl+Ya+>BWQR#(&4H4A#Ao?|@H5HLjfp2|wc?^@W#vDKE z0%OBlK|>5NZvVJ__r2Gh5Si+bEad=Tz0#YUt$|U|u3ZmUrCmjqH|~6A&y4=EzIVU; z?j5cy)Rmdv)%D#osGTn>l9{kB9zy&2dwgZo7ZL-R`m9vVWfJ5U@5) zea0TQ4elrJtc7TbZ_7H@)MXtxv$Ay~;NDA9>y5S|^CN#n-zPIuzmp6)EqsW2!^A@@ z@Mc4&`rQM(zm?>MGQF1EN*+31u{?c^G1`(p<8G+CSS|y-&MhkIw;Vs3&$#P~nmZ8J zo>i>=67wrXN#>yXZiW9giXojWU6h0MEiI%r5?*CCH&k*KXz9quWG$SF>*^YPv>NCt zvKgao2lItB4V_|r%CD9#B+VGpwDhrL<>X+Z0zH?Tn;RNaTb1i}dpSl&dz7e)9*ecJ z2q&@=+oMWXZL*4cnkAMjPM*Cby5N1AIwww=%T7q)e1<|;?8tfXuwch98g^|ht;mQ7 zO#=h?)r~YeURxk5RqoGmekj_z*Cgd{BBw&?6KPXWcY{OHV|FOp0TE4w;~ii0W>TWHlV? zf7biXokyNcW~FH>XQa@i+pU~`6VKjYB+*&R?YhwF;__EPpt#Od#(WqMs*XRom z2TTJ!K)Y~nb&-9ov$b^|Wrt<$qmEKXqwVY}NtZEpf-uoWZ?H*xUj6y9C$U(Sf}>u76#k$8Qp^iitwuhRaO9b@QweXk3={5+ztMJ|H%8M33{rsaIm~#hO-)Vp_jB)Bv)#>7*OR*iG9fzqc5OC0X$rrNg6o%Z z^xprTJ>gzY$xF#W3{3F*6`3RJ-PkDXI$PsG>y#=vyDWaUV<015*CaDy%Cp`t%wsVx zMi556)f&3X!HMo)GiHMBPgPfYHNWw>_AKnsHyhvS5-o0iYieogg!Fq3d+FhyLk#v- zfqex}8hRsE{ZnLRulu%-L6YC@9z+@+0MWrPsIs#1%*LW$Z_3a0zkd%hFu>+h3Is-2 zj5?rNM-0*01LY59iJ5!FFrz@Tq%OeE5bSdLNoRGO*MxYQd{Sel%kx|FJdvK6W=_Ys z(%%xhUB_w_Zmvuny#p5M*QrXmg^Qw3O&`s*^fu9(-<2R9G!y+mT>P3P<4&FuTz3}S zz7C!C3aOpkx^xy?+}e-3&SxueaB*M0VzE;r%u-<|N7`2TCiO!{n>U{HeODW&;|`x* zIp=Jp#PjG7`2(Lr9lMW830kndnNx8gZVNN0wq;&V-(@IwI zeqU}X6Q?@SOFO@z*B5-)E1R#jruDN3uQppAUC;rdTqu!}QM2i1ZXMvRg98$XjRsda zG=C|P)D|FEL5obXu4x8NjK*DEAjq5%wfdNBRf+jnU6-JDW$}OudFLIEpT_G%t+n&x zT5Iu#f z)Ze=YiA$OsKPy6&H&uT>V0E4G#b?KXK%XCv(w)@q9@?`bdot`b(}`!Nr(*>~TQzb$ zLylECa0@<@Zd44e_PX^bKcK6~>4ss?7c=&n@NIjSVghr+DkN&3s>hxjbl$)1eo30S zJ0CP+;JraH07nCH5V@dBV}OpYn;i;gL`HVHAl+>JF_;p-A;`qu;5$hP<5jylcjuhW z-nds^^_l61MecY`e$C}6TX$S})VFXVv#zFKEhjLTBi#D)z;eg?x{Tj@Z73V!rnUM6 z%{o0el54N5)K0^!G;3thcBG5Za;agu&o}0+ytIi?7~^TLTkhXRWPNmYT+etU@OkH< zYaT(-ZSi(_wqd``o!<0|GMEB%F64Rmb1Cln z4S(3=-apFkDD&usGrXe0p7U^eGQEEo?8k39F`TZQxvkpo$a2sV>KOl%RAZF7or2z< zN0&O%T=Zj7@7ded&n(=-uwrV%FGa)f zPN-%s+YS13uJ6YdJ`7Syb&T%^&g;OfR8w19{3=HKl_n-%wX>NQ=?L#6!?zFTbkZDL zUNHG5x)*SkGrh_NI7yZG!dl29Is=dt-eza!z_@ z9lS1}u(M3->`*q_zTs8XA9Vd!#(2Lc9#p~t3d!KGn%N#^dU9J=H6EGC?qG!|TRyr$%5 ziw!$-Y2#(2mCjK;Qs-$Sn8k8On=8`Tn?1Ij%~T~LJJ?J*Bj5>@p@4|Qm;SQx9@Fuj zJ?e)|*C$5*8mpDL3el47bu={|)Hw2^_bR<;q@8gZTS8F1jV~{c16Owgt>Qvas^RJV zdDi{G6eHKt-magguJq3#_n|2y0m{4PPdjdQmQ;P246Ko_qMV;};8|b3eamJwQJicT zu>i|ic|hd5*p`=SNZjLN8tShCu97>w%(w5WifYaiL}4Mt<_cUubK{1)6*e) ze!q~llnP0Wbxm_+y_-B~7{&h!$}9N5YUpjzw$s7AWppg%PS^%VX7u9QTH8v~#jz)W zE1zAAjfc?5OD!JKYA0D)p@JIhG_erx_M)-BR;C;9Ohmx%nAt(eZ{&fFt z?&`7;zw~3UXC);7(+;_`UdcVF?q(m2_6g<`#ix@uf0*x%d^&wr$gkt6@# zHTi#jbW{RV4}Gl|3kdlPH}^SCI!Or$XJ-?+*ke(RFnN-7U;NO}pi)7iE{cNWAZuAvJO0z(|>;m z!+ZdOPG@JA`zuiLqe6oET%Z4`rw8&xP!OO8FR1?aGcjdqW&jV#&<4K;Htzo>A}^__ ztHZ-YK`<^n{Nwju5TTEQHoKQuOieJ_?L+u|1`rG?aKx-0^SB_*zm}MZHB2p zLW+x@#(`mIB@{DpVjTM(%!OMlJOm4*WXSc{2;1v!p*V;1-{m7Q836Il-DCHkLZS}-8Ne-adE>WEH60y!m6dQygWMfBQFs&dy_>ktoZ!myJG_LeI#O{n$`uH(T{xH9c%m(Jl03j*;IUt{ukzrx~ zbmb&mjm`^FNwEWmzF~LB*4CD#?iPzO!KBhlQbs1ujw}qGm!luS%mnNLl%Ik9)AwH_ z22j{XV4@AL=>eRQOhPEnDznmBv&X0fj7xle117~J%zGGu{b`^bUH#Q|1~e-S4>dLE z*3S0!^&L5LPiEN_-AQCi8W?wNZEfBVhP=Mf5{A-OkPB&3w;TPtE@mJ}|^6~i3#NGR0Qx1a(63T=B zeh!`83|6~$y`kjY(m{g^?M!c~dWVYFGrj(~1gv>JUf%R*40Sj_`CW zCoh$X8vGKW%qmHnm!3cldzlITnWGxIy4Nf%yJ(9j1d4zOfVIQT$$9zU=FEZ}tQ}%o zzBx+ZX&`FwMm^+p*f@p#bX0ZVFQZ&AGA34!-RJz8;o(iFGN+c^-heyAL9+VNH!z?j z*Dx?pd8*}5g8Ni-GIShlJE;v{w=p@1D<8-*v>_cdAwE{(P*GJSvCNe9D5oO80>w@8 z@{&=2iV`?Zgu@bQ4WQc%X0@<%!r2}0Y$DsDJny@BTU;+$a4m>P43g90(@OGZLJFrs{nHh?0=bZEKtGo#?a6ZbTCIm zWn)2c0f7+XdmyQTwhyxw0%$V?0^1&cg`~*HE(G0TyU!LkZjj`op;9dE<$!UJg4W?C zF<9<#8$>72O%b1V`=Ut$(FCHmPKbf{PBUEiXk7cZ zV;6g`O&&zzXJF_BtaHN$#CJ^KKMxLK;uc%>Gb=kA0$rjU8w@#EaR|P4?bN^d4_5S~UlsTHEkQK&4`E@u9)VFRQ7vOT>iRWqaZX(Z0 zy>p}j_1M+d|)0=qNly#W6*H?ndo!Sx4#P&?Q^eSg~y?3PV1NGd}@N=Uwo zG`KGg4sBLKM`S&_HG0Ut#FMM1BnE%HB?Iaw$hwn_Q8Q_`u-QpumJbgrWU0Z!gcA*l zR#Shhz3Y6pCS2{Oq9Z3GXqE0q@3g4mZc{#d zYIOY6tWw1B)soSBs{6b~A@+mU;|&*xe5g;aUcoC`T~Cua?0_pYQ$!NSLA~GbvBzCz z1RGR`mtN8q3E%7V7XA-3QaKW3!$#&c;v;xJk4G;6e=jYS$d?fkifFlZlqkWn??oFZ z-8SPc*HNI{=uY3-_!wla^HF2&CNfm!a?f`qGF8T_UPupqWH?1Zr` z=%3sGj?>=PJ2*NyT~JZ!SUs^b(VvaRS1^E*2pFD_1QKH8W+f)d zRak)@4LT#<3lXavixgSPT4VqNH*`&ZAdg19)-ya@fzjlJR}sb|O-)VR-G!)XQR9Fz zo*-(=pfCmOYGMdnZ$@V3UF75dw80*6Kpdl@u*A)o7y<~uMr2`OK>+&+7!7c1(ULq# zJ&Dsn5@gnJU%&@|V<8m-T+m}_g`x$d)iuCLqkGbFu8Yei8{~OtzkD#Rzes z3WJAzaj|RTljCSYPn|jiJ1?-g5zzwV2m^%e>{Ax$ayS+YIRwntdJRKEX2P`dI@W4^ zGzz(iB7ZgRqpJA8Gzy?bahZYa+cE9Dta^mR6TfuH$rCRO)B=qC?jaIa_1Uw>2mP(s zNEcLimngFJaw#m8urr{_YKR8+Qh|W|D_Vq6i-f{?*~3JCf9M!~I@Zgj-#f07pomoH z$GZke8Lh&^)nk&;loKw( zG%%NyBZNnfTe1CKT1pbk-R+CTxejPGc3qDwac8MAq3jK?SJavFgC&=OT`=LA{`ytn zB90RRM;>T<#GC_xT~AMsNOS123a*@X7ni~@RJGV56S4wMY8{y3^Hieqf>R~3Nya1+ zLBF;-f)l34PmXKB_Sv9n%&|6<6{OvPpVg;?g`tHb;^W%d0+z{$0Ezf=*Mws)>@c?J zC|r!Z19Vo2(1L@OmP|p^1{?r-q038@05XTc3WL$kl@IqpFT*avIY_Xrydi+n7$XD( z9s8kDlDo+aGrWZb1I%Yc*AUz?oIuHhlTWc~*8Sd;5@aceuX(SL*=@(CWdqf6(?lEA5rO zp`n;gBor%r;F{aoE-OHu3d3J*-HVocGp}1(`q3bocIx~1`ff*0x2%JHFh=c~jt;#- zdVap9nc0i-;eIj?$PdOIbH#CT5JVWu~=f1$AuF^5Q<*gV|QER zxkEZU<8Q_7@40df?vc!H+BiMxUw(NxOZ7<*Ah{z!NWb3vOk+3@I`|@)808e&n1rq? ze^GRz&!nPIoSk7OcKc;{?!>$5@J4yQ$;isWlZ*uqvVR(1Lb+G_Lzi@3K}Q0Z)L+hf zuTS$3X#&h~lAb3}yo(oGVLyU2GMH=>*8{}Jth8O6u z;}NpIfBUA%1#+7GPzSVL4WrhiV-dksfy3A-Dy&5%JG<=KIx52|w0|`H01ZGAM8kxGkMKSuk!ymCjjgS!>s@%wT#33sj|iE^-E!q5BUCBqpsA^-yi_ZW z-Jjoq`12Fax2iNss8e;4)6!7uIOC?{kVRB>BsCtoCi|g10p=jiwgGF-$XJ7>K~qZ$ z*+Z?>5J#jy&xk$|1d0`Q!j!{TD32G)}ZcB@cjMA?Bl5X%v z1m7NYxjP~m=7MT!R377wRm9!+a-_q`VDW-9YGN{s+K7}P!oz)N1~7|&=NVWx0psrX z?)^jRhI}75rv{EOvhGO6zzb3|bT*(z=I0AR(M@azXBei%Z~$(?90g>Rc^n&wYLyk| zO{{OWQhaNUiG1<|)gM=M6G%Qd%}f0zCE1*dbMZd1oRkF73kD>jYhWW2*MBFJ!HaF; z!B9g(P(gP;<&1sf388=aUusCMvkLLUuhnfjBaKEgq^X@ zi%>RU>CcH=xE$Ay1_u?EW=@8kSZw0~P|nw;Qo_>fR%sN_G2x&na(T4N;a(Hr6Z7-; z<_1K4*+^Rohka(oHCPE676x7j$8iZb#OtJ}8!%#0g5uBjiS|?on3$ba)ltJ?UqJ0p z>#c1GkhS<#p8F`hr)~+7iXbIu+01y^3qnvY?xCba?;-jvVghqrUEPSV-x|Bq1;F{+ z`tosyR6JUlk<1=+Tew?KV`2=nw0^-m8PN{ni>KM7Lnk3>L@}B`qneSKi6hcPg@j1Q zSNH|1CS6`#U9zO`Ge^65}NZqYfDuztZO?}SXU9QtD2W_RVvTrLI5poshOmV<;gJi#c2nO?(9#~Lp>>BgF>@U$pmc(QuyavgdgM=09M2t;J?|KsmrxxmdNw`tWp zjqb^~`D)TaymD5N-jzC>RAG_cCpTpVBv^(z#qJIWDf7lF#6(9kM$09ZIhdJc6&Bjc zYh&w%uUa@wl7CUifv|`>E1J z>HEXK+wDoqK9G*{f9Hc0DV!3f>76oM)IK?(HfxH#c52As`#XZ_o_UI2g#w$fG(1e) zrfMzre#NE7#fED4PXcKK><7N}5TOu=zWhHT#l-PMT01FfAzAuklUF*{34G8Wcn&!* zaK>s3cVK6sjMqg|)#6GN270})s&8FJ_zy36^CbBsY>(yZ9}vtmY|wuFO0yzn?csbl zi2bC(kyf7wp2_A|5WL=zx>EGV3};IU;bi@KbS5V6=3V=imK;u7QIEPQ8Kg%$ThpFn z)xm0Dh3>su=qc|g_q?4FT*`J9NR#1jPCU(zwM<={icw)>E66zZJ6tJl3V(l>(PS*G zw~>OBG`moe9_s=^Jf(+hV2I^p53jL~B>soI8y;_&^{s@Xo!1?Y*thxWQ1sg1-$EAs>Jh^B(ETTJ4!2H zmfrGcq$<@9ahVQ*ur#!y8hiWIUSZx|yS)uOb?-$Zv2aM^%Am;KVe3D}A4~ph zFEbzd)6|;Q$+7%yUs1HcbAc=9ITI9y(>v6hWb={xh0@G*b1omI1R3Gsq76;md}vb3 z6E@?RL=eMUBC%R46H@W~cF1#W8U(`mhdKW#OVXKp7#WXgshE?1+^UUDs%~l5TMB3Dci+|V_6^R-}GjZF_%&jS6N>?P})Q*mwf)oot7cX*z!zIzs;{$C$5Ex1yR#iNZ~mfsQi zysJd@hf!W0-#->HIvK?MoIBD?J=gt8;r@jU^oG1~2Oa^VPNgEDXgM3zHnE|#Fxod7(=gqF;4wS%UN3~;p<`auwTD;Pw!Q>!G8J9UpKu_98`D*y&_MZ zHO3Vw%~e@!-hUq-C+<3d7{;O5><(-qLHJ+j%*z(b8Q{;Wzb6l7SO_5&d}4_8a0NZ7 z;2fl^>&z_pmmk@HrI@RaA-(>;^Tm*CAl>Ux5N*~V3;2%q8UG71oEa0m2l!Tla%g)- zg%De)8~l5tLhb*-+xVVa--H>Z4iODEKTc{cAPcy;-umkM<#As0vkN<-)oNLi4Bf)^ zE6vR-pZG1Ym3%>Rb9}fGsFJ0osiieOHWngv%f(niLPAR`=EaK_v9a2nROEsl{tjJCp8x!NYJZLphX{#)HXezIPK%_>K3^z~zH)rNA!rjSyaJFeu;y(J(ct3{V zCa=`N=jrK2c=VMkS2!G;5&l{$|A(K4R(#x56q^y}G^mY=i#u4Wnh~hD(C#1|L0wT< zN$R`1wPih0lvHGmiAXj3yAKSLn*RwHsFZc3?i?RYXk2jjQ*x1xrqbUTQ z-t|7%l;>(NwXqR)TVq|l3GX~y`KIC->kT?HpBMS~U3-EDVNB#Wy zGb<~rxamxYF0;HmYS9DP(U&G^V{I)bFCS!UZEdZ)w1hy^(y?DInTbDQD&Neg zq=P%r#~r2SGm=Kc8_y1jA>IcS9UWa+S;=9*XvvtLpWi2KDWtn*6hGl&YQY3k?#;iY zjKO;+V2qX}L7XO9b_ep)4Zd9@_j1KYY4@^Q45`o25nd$jBVpGto@hsS2)@b4Ui7FD z$I~az0Zmw{+3`=s%Z?b#+uY-2^uKri;c7&X(d(<|4MlfPde3~DV&GuliZe>^Sd;5R zkZl)wPq`&=Wf!O)tljh`uwY_(GbP|((cGR{@~iOdfI;0r)}~g$S4&XeJ^Scep4@uCb}@Ln8>{6U0=Wd5Etfq-h7D` z>%ynmypiVcrq+l}bF$qQB=;sEdR(7S6(6hg7Xz~to)Xq^s+oi-^SjI3HA8&bocxe2 zmNh_A{O4A%!k$IX!w{IDfF%wjD~t@S2YWv^!DN{xLZ2fHj_;BNT;PD=irciSJLdb_ z$t_lyZ5sDXh&dffX;FkV`ZN5ONc_QA4K)VRuI5{??jKmUOlh6GqwdPQ3Ki?) zhecdU@T9x3t2;_x|A>f~aOW~Std<^}m_Lx9^a(`k3Li!R9bAeXeE5KavnpY|D-%$X zW4H*>r}xzKd+kW>K5H;-YF5X+Qo+Q35-olGSh1%u$5YENuy3l*V(9PXYrb9>GhgCE zuuZG*jS4r7FBM+zQAG45tg^yJH;DX}SjHgmQZbp?0U~+dGNi0oRZZrv>lElryJ14U z)?Ty$(eCpbc$B!L2;-lsC9sCyRhqePu>Rq@#*8pVJbW-CwXWmMshH2jO%DrO;g$xJ zoDP+JZb=9}@EZSLtcCQ#ypY*X!dQvdI!^?o2FETarW$R+I5!c^x}yc^?y%FoH0bao zh=0~W&r25Gg$Yj&!&0si_%OQaU}5~-Oyu7R=qBz!R`V`-uYhl=6h?|UnudMtH7N{Y zUFAR0!zTu~(`=9Ks9w?bXEi1m6&XuOo5iI#y!We!?Ef*Yf8K#h6qE{It#;07=WNQh zItS3y?K!YtzQhD8VZ`b#r*yqf@%Kq3o@iG*I7HZ%aW^;aMFr{r!cPZT|AAj@NEkW6 zIfgBv|gnLaN&>|@8^gyewo1`BE?h$8Z6;O=yNu?|1p8I`Qj?o$>}MD2rD=O zz^>RY`)#_T2!a6G%iP>eG_Mm6kGxz&ja`O0NypfPy+&;ftG-Q7C!f?2b_Lti=kNod z;>3=<3m)K)38?VKyQrurt71P55ht+{2L_Y|@z!F{Ot5nDS=W{!Ozuj>!dLNM2h0v{ zuE9{cVWGGgLbcCSL=G;|UAdwkn_JfI=PXzIr%AN&A1|GyOPfJ_;?^kW5Uctf(m4`h z(T+hdcIe_g#qZ=ykxA>%^y~pv-*Oa7$1YC7V~GK_L9xDmA!5t+D0xbm2NxVRi<%l9 znLUq;jBK>VHClqjKhn^Z=4G{d$k?4*ef*`6eA&45VB#&|O_KFy1$?8^wU@>CUw9YX zIovm}ql@`8wCn+hpyGM?|qYNMvmnJuzj#l%`isHrlQ)PByd(%(2e<~z? zsr_hzNNj>>Nrd#K%21U}DyOB3>}(E=9p z-@eJ|=}p8G5tkEdYHA7y2pko)Uc!oZ%Zhr;QAgr6@+~i}A<<*kug~awu_tMwe7kD$ zB>klwRf?Bq0#VAjV~svLX=v-gH#=fOGj? zB{>ag;-FwRd&+oJ#EIb33%zXY;c*y$$Ku_)cTb-_eeAQV=~APXcd|EOANlTGKwzLn zZ?bL6+C8JbYa%s|xVDR;`U|q$Fh63&dLn;S#PVnz@M0Uy>Mo?XuM8L3j1=(;3Kr+( z<x}=E~_s1@O$XtB7x~KT&(__p$46d$Hl~3!CoOuU~U>b2-W>cC1(z zs9PJ*YY{U{yOuWo1r}A3s*7==`~rlk6#XtLDx^ zDgd7{i(WhH`Q>Hn?5|eZMo;q#3Rcc_vyFHQI>M}(YTnDTSlDaa`F&lgg;W|ZH?K34y~>PH$Cd}5 zg~!=T2Ui-8%gah*KFn?i!AUh_@4SVjC@s$yCSXyBJ(UFWd3^8Qy(qQ0We#CsuY=7w zMJ;)DYC${wJI7d$Vp>bd+cLk0;CwY3)R#K3jW?oDwXmp^M!=p3$b3!3#Cn5&&+w=S zYbl^{+HUHKvAcV<4!fxP<}83YHFb5a$Mtuu2bq(osLDCT#QfU1nwTsWob6V`Jbqk` zSIGXXM6RIGkxQ#B_qzgT`Kbgd-eWlgXU+6=#b4yPARzEd*Ov!j)U3;ZrJO8slY&AI zcFyC+)P#h6#fD8ZekZE8-m*U`Wnf}rx^?Ro9o^i3vY3mjt6fM&;MET^^*^V2W3Roq z|3z4o^IpPMQ|svnmm%NNj@e_8tZ@1oez){>>qqk>*O`tY-E7AQZ$1f$;YRdDlg5jx z>gnnE(Y^*~mzbEi35!1cvorS6#fuO|H_9*-QY^bmOG`cRQZ!DM>24iX#a89F?A96Q zSwvvl$3G)WJ66!xtTIEc%3^vw3=F}+C>dt9#Kq{0yOxn4YHVdSrqNSVhkjMHNJv;z zt{!CZoz&E^$Y7N?)K57~E}3CoH)zj^RcBPbxEd!e%+wdY<7}Rd@$tE9%K$mO*M(I6 z=6!Z|%SRqT?>`s>BC`hT@U=J9xKvZA=KB|(89{M@`J`kvE$H%r%&Q`P=5jV!pY$9< z6q_0$0uO3S>hUO9lGWZK`k%YQk{wMzZMqw_8NY!h*4M! zrP*fmhJoW{HW+E;!_5ahEC+NT*p9r4R)hK6hRLqf}6Pd z7V5>v1L72I$tr>tt4oNsODUFp`WB%}v1B!Op0t=(lLJBwh~Tq~;l1K#El(TQ2#Z`Vz?i0mF!chDFFB8)N#QP0Gr0=u1^ z-F>e;N?ubLQ&Wa7sfAr}2Ao+mN%P@h$PrpZ+^h3)AMnS9^Rr{vvfsadhYF7#br}=0 z%4#6HzTW$A>*wk5KEc(iXAq+yJtSM>aun!3V4|8MeGxj|x#@z5P`Dz=a@|+R=~oxz zgQTRSCr_TVN6^qvQtJB&N6Wd)H3t>yRh{h&D68e|w1klK^z?9ObNnB?W2a`Da|`v) zE~$kV?RvWnzY6m6Aw%pK;Pw&K=%XiM!lx2<8AhO%$Ii~~>^)hRqitX?^(~q$BO~Kc zvcNVCWOcz z^YakcgRfG_Vg1?NWkfCf{Q0{)pwDE1GTs@bN656i>R%**;5G=moK{v=286G^J|&&$ z_OI@WyBN9TC!$H$J=oi(?^LU0?X-oeYjzj0>MK`M2mgxVzYj>$hNT7?_wF zqb2jLWQNziMbL=Z+1jqn&u2`KBSS}HKGXE(6zJ_wdLR%~mfo&CCx_dCfq{_Wr~S@P zM@MzixVgAC!UU%Y6BItDW@-0nD?4Dm2wmGWhobNc$}>{E{cZbRZnAx^=u8hS#RHzS z=}aSJgY5Tnl&A4)1t%4`#3r6{f#RgOi)%M7DM|p_Q+I4t?XoiLFyF?=$T($Tle`Gr z%mif6*RNm46jNBvge#mq<1`3_sr&m)**?1gJWZP93#jIWTvu;UQB4(S7gyR&2;IG# ztC0^pL}x75`R<6Juggp zUvcPFYG`Wmj}A^v`Lw;h$<=s}{^EtSyf&^6N>{u6?X;KXK;W1*60c76nh%QU3j91U z^g)FY#Hg}HhlGTXaXkjFj*?-3?P_dnOeN&pH@5bh2&KEf%)(jl3ThD3rN~7hxKhGW zZO2cn`G|p7Rto(>h1*=>FtrzG5=>+YBcbrbxrnf&YP7i7w90xkiM>AZ*#y(cM;n6Y zF3;-vEq7W*<>Z%(znn?${|YDoV?3nA3Y!$RxQFS)TSq@_iSTbrvl@(B>rHE#6o>fm z=@dTKQsPf?(66w0GfD(e4WTF*8t9m&C75#YhM~M2EfE&*a!hx!+zJ4T3hqAHKXGP)-pwFqj5t1uHi>sXsMzm3q3{ z`l!ocsQ2`NQtjjS7Z9Il`H4}$hgI24dH#yGAdjl6sUbq*;NrTda`nA@bzA*x!QAcI zuEUohT7-BRA2G_@+#Fp8Pp68ov4s-$ckPNvaES`_#+NK4DJ?;cuq=8g%0op-i9{l` ziu7e!G>c0WSx$e1a@E(@U%Yh5MNiVQ=s?f;3HfPWQ(WucyYh=hY=&|*-Vkcg)lzK99X7|=60HRJ**>6BD_B%U~o}|b<gsAR3&x$L8J~TF?r;q)t!x!$RN^Vd<;&GIHO5SB86>qB zf>N)h3U>^EL;Eu5PBT|zPzShM%fP^ALsD^RU^K{rk3+}heB)yA;r#u9`H*XXbyA(8G|85`5Rm0536 zq|eb-RaLdJ`VD2pc%^MY>-fp({tRTsuN@uF939p2Gz{uo*MJf{Wt?0ttE{v;%T-0b z#C}Xf-I=j+M{}|5m6=wyOCn>iF~D{q24d`hKIigu)eOMr@G>(q8^7NV-ZMEl`TqTT zEIP<|5S`-VL(ux_iY=Sz7 zk_r4auR=o!VCu7LP7%g6e%f5 zQ7%L+u= zwebqbmABpsh}qKreN|LTk~n9-;cRMVlnQB$yH9{f#Sgn|3_GMhOCDXonSLc+!=MvM z2d?uQ8tC1kq2PN3QB9gY_By=evK+_Dtz)w<24{Xah+h&BcHtUb5CjX;bK$_y}n;fA^~5!m{MhSN{aFD2@pTyg&Wl*qt4 zME(Z706HT0Yw?Tf{2H@J>F(*F<}pFZFsKzfF@c#Yl~^&{qKDt!{!z1W{L z=3$t|3JR~k7U>O-t00r_^@MGkwV?x7>J>g}6pimeTKLKZ#YdhYy@hEL7$zYs(EEBG zF`~s$@{+T@BS``xtaxY*1uB-);0PQA?DOTz7wr!^-QU3-#QdFe z>wT&7NVzP}S^SN;?+u$|f*q0Zosr3MneL61x4qc~YKuV$>>fbzTQ z`HYsB4)l!=dvC9&N(6!vdLcU|AZ#|{a$UZBIhO`1DI9$KwaJ>YA~s9j{3ug3ktzoL zL8V~yc%}}O34{N>h>_QDTB{xA`v}MTl~P1~4n_?{lrcrlMy)y1iz&7>XaV^koN>0X zHg2=A*=%fJ+AR=?+YHDWc$t#@dt)Hhapm zv9CRHdM~)O;-oFrxFh;9%Zofv;#iNWpFSAvU@7bEYrSree0G>JyWUW&Uz1*#)O0=; zkYR2hmu<^rOd=bxv^-kz>P^~)&p}D$9svT;joNq=CNCqSQ*O1f_UI4lIuSBoIP61D z3a{t4f!}03A3u;b-hX;*7U8`Ya>V#;x_4EN@@&Zq1L6N7fcEL+w_QEEeLr=X&%%=d zok`>N!oGCXPe$YzRsUq_KDaKDr%7~AdVv!DxT2h|snHrK$2iBrqD_dD$qOJxe!Gs! zkveTj*F{R5Cq;ezNXhog<1w2AE_zh7Pxbc-z55Fk1Dkm#1Zb$fMn}7pMicVLe`tMu zQ=YxG1u50_phZrBfxSfVi4;3_3}G-~{5?8UN0&}8q41qsCNINKGCC%V=AN8{KbdVp zTDI1wJ)+~Z>8^nJ@gzp8jhY0caR*+?i=QIfOUtJsZ@aDL28oU14rH_6b%q+GSPp2kUUcIM7nU*uP3qW_TjUhnXCK} z=i{rr*Lw;cK9t4%<#&>-<$4un^Wry&;&{DFJ1&iP(Gp*9nNwM>@W$i#M!$(R0bBCQ zSkYGiE7fz9ome?<{U}LIrR7*3due=xlKs4c9hqMIn*^(wX~-e7dpuqFsS@w?pfoCB zKx*80!`|P;E~m+&Gl$O=I;CQ!Q|}}KnW%d?AUfWfP(U%vn+~h_{H~z_?oZQdK~BsIP}v1+ip3cGGQ>B2fZlD9&uQxmWR&(6HwOa$2ws2- z4*Hr_>P&s#(X1F1ZFnLO{#T9PeWH>}{vbQIPteMJ6M+~AU~3k3HQSPK&5XQMi@qst zz-G98(N?kRC&NpKPYbgdDTK%~&NRV-uS)_4HV8x1hmt#?PrYrD0@&?0BBL{(8rGHC#Bzq&b*YvL1 zH{0ql^tE+9Mo%sH#sm!*KcRP&mzMEs^o|84tI&wp=%+4FLH7VG0?djIm(D{_A8sJgo2^ar83%vjiT z$KUitleXI~Hwll_iPc1Q_~btzJFJq72ZH~@hY#%XeiriQ$vywxgABzQ&C0}NK2~a0 zQc}Xb`Siuj#*KX}jz1B+g}-%k6AIkmC@O|@CE52XswZ@20i4$ zM`h+fynu1h_%4P+Z@sJmzIP^CsbOS_bfp7}w%Uoi^!uR^V zc5ra8d^KYuo0bk`Ebn>$c)RHSQNcBk&=G&R+Pt;3rOTERoFxz*RMokoN(HVe5=?~x zKB)8r7aynF?u?3bT#0KJf=%F)QHdKwM01bd_UPYtu38$_%-K#f$rep(lDmNLr{h`W zw2U!}SRE7n{hLfJoQutQX;v+Att#xcW{kHf3Gib@$Wcn{bg*l9(Px|j1;7fmJh5nLT3YhiplOjN% zhJI4o>_3AwW{}8T;L*W=cj2%9jQcf=8~Vckj3^p%wK#eLaVh}&5_tamZT}kMKc0P! zlApvFc^`og;iJb!@`Zt6{tgK(r~GSZ5QD%wboZI*D3x2$MZ<2_#oZPhYjmJb z=`s~^$usRJ_HWJo=hL>dTh2&3jh~3~l1i{KFE8&koCnaEmX!_3$as=L0qvIa?Orig zSJ&oZh=HV}q>5{gOqJMMK7@rSh7MBb($tS+OP+A~XkT`gkokP)g2L7;fKQEbA3z_6uKjO3Z$B^<9p2{J%@c^8}0VuluyUrW1eW zg*IE9>Sa2nK`>P|yrmH(@y`aiaq;qBe?wnz<;7nof5UV6ro(dB2e|O}F-Pdn)#3B%WAJ`U^tIY(r$#>nttawt z!xM)7n9y|rqn6^{eP1@#lb?b!1s{M02?Jd6X=(B5uI`^N!QC6a4Rd=ad%)%#y{FIi za>S1C>NiJ-!1_MgkX-SC(;gG$YvdCwa@C%$D*Yf9$D;cnLqzRW)>+Qgbfz_<34SDL zk}DDS{(YC18=bkV@y1O$dg1~NRv!&axM_46hB0s{ zGFrrQQMoakhc8eNB6Zui_AoeH_H=9>`3p?wU$zX>GxSPuizWT zJhQ498%_5nn!*+_x7)Q?J&1~NR)}ktJ4A2azU?yt6%8J?B+HN+6*=ACe3GeVi- zDCywO{DWHx`GVM(1k`v01oppbE(CM(@Ic?>`^=0XI<0qfbTG59ph>bn4)_&T<&99| z-sj>{o~b}kJ*s|YY+{n$XmdfR)NZO48Y?Sxq^b1i{We}<eZ|0mMIe0 zkBW+ls;a6+wu`lIA|utIO;Tha%(3n?S?$2pu+x89WtAmSZPrO@VXg3u$$-5@L| zSal}Go3+y`b}r(&8VCKFQ>TZxxpaDUu7#nYow!`}gp`yg{qkJD>x5Ekn$C_ofu;WX z^=p5BsqY~u8-YVk_SqX-pQ?+Xy!UR=Lyo0)JL6&4@^Ik<#F&+(r2`0MSQNJz7)B>1 z#7=hx8es5Kbydi;zAtRg&F}i2R4Xu1+%Z=G%1MDquiWyj%Hv0mZjzG=S@xxZa^Tz8 z81&p9G0&8mg+@jWK;)YNGz)z^>>!*Vq$5_H%}eqN39S17-R7uhfp_BgFQWzDtg}!|H?5_Fk4onu;^<=Fpd{ z^}X3A^5DS(;6a;vUl7m;2nv#k-bB-7potHkXU?*(|MO5m0WT|R{;rr<(^Q?Cx}5z? zBS*9xmqD!*%T%SU9%z^DTcs~aC*MtxuR)RlmoPsUP|>b|G5x%saOn~`!& zwo6yhofiTE0+*b(IXR2a+uhM)>??w{;Uhi0f;zoDG*~N2O1^e? zcXxKq-|gw_Eco2k_pLig*ll?*uQ|u~)NL4+zP!8~6gTJ%ri%{vTr17iQ9u=c|u8 z^FiGW{*I8|q=*hg%CrtL}+-rxG!3C(IU-@@}A97zUL zXe&mJ1eA3d6!TQ}-O!vGfca%eNA#qK`3>f(HoL_hop&^!f7o=LIga?CmbaXO(d^ey zxorMP8@0Z^4$ir3h>L>*%e1RQ(}$iA1kHp5@;U8Yz18*g$2vO9*S`G(RNx2(oTJ-i zQB_fyotsOG(3N!ZIs6$5kw(;c>CI7k)0wcIii#2R(jiF0h5Gtk+x4|4(*r>EiZg#l zM=o4^d@51zgXSPyNhDIhep=z|_^OtM#^CU#^z>UC zXMq&Y2wm64H92DqtDixuSx?XFM@tBF0aohNQWZf10Qkwl!2v~>yhuPJ%3j}kdbBI- zeeiMftNT)4n!i5+b}W?1o12?~m;^YzVWFWbBgH9a%U20DpyBh6xt z&^15aTZ2W&QnP}fk2?31`wiwhM+DmyI~G->CQRh4&Kkj#k$(s#*?j1 zXr2S@4b?hfrD3_Hs<`;4cZ~AE1T@q)Z?as2t$uWPs0bJlgb>ig0mc9Kk8KK|MI-`v)16^i%x(OUgfCzmF^Xj#SS8;pZ-z)Pw_s!D~@;P#b z*kjOm*|HJ(wURHc4spfvS-sFbu|dQKEHkYb~)$0 zh{qQ6XPV26cDtoPK0jQj7n_)Pd@vsYKB*td&v$?>*xau{$OW{!i56!fjQam?zwz7) zRS`*JU_nTqcb}zFue^zii%UpgjqObF*?V+f>(ZwKBWk)Eaey3>?|Ts=(Jd{BK#aUj zpeK^vym|JkD?a8fn)ZPtgl?e4jCwdJB#U^!JJZwCAol~Dag>kYuL|;>8#lt|p%8%= zT%~?)qYv$TIAB7}HthH=GO`N7HY8-Anm~XGhi*R8K9G`nFXrb9UKmkLle`E}W@V;N zGf!i>(WfqN4g3@Kqq?AfVNp>}Z*NoYl<~K3-`2~VkTQQ8jaGq3x>;f8rDrL(aom%+ zn3)OJ)dm5mvF;L|q27Qeot$_>YkR4Uvig`;9or zm&i2Bd{4(35izlIQGPN3l^KwKslf%g8wwXUWy-7VSp2VI2~HlKZ1+Ns3bY&$Syj$j zNR+VzpQS~C0mdgLKn)^!`T?AEn9F19>ureI2pdt+lG z<;|N^Ur%UyF)1gX94@Bdh9-S8Ed?#yNbwNpC_#YdF!#%vH~#7RWQ|BKEDdMieoC2;L7KTsP0tzGec45_Y?S2gupWu`; zAB_V;LviU^`uiQhvnkuLBLL7%z_-vLM>iZ+F{&SUBhaS60w7=3)Eo}xsRy~dx#C!J zf<8%E{0DI9F`N*=C4e4$CDZU z*~LY-)zOlN=b7l%%H>NLSDWE-Lges=NAyu61kLZ?QzAQ*#ZEJQwV=2?eFv2Z$S$Cv zwLVt1*cE^0N!x3p5UE~~?XFzaj5GAANr$5V_hDeloYZ9Kt3SCsQ2%^HwGZq0;Vo5-hiFP z2|7-T9?D?jU)_nfMQVMHU6C(A@PJ9<{S0Ih5Q#QL!~jdl%XdO!Z|_HID3}Y%>2-LL zkZ?QL>a7f#t8MxFvTr;zhYi^*3F*4Bs_Go53QogDZvb!*O=o9k$H#Sd3l6qAxj@kk zX{yv|QE8V0-P(qT>;c9`r!fdSfW+mMm2;w4qrX9g1uF`u2Sf)Bz)yppFAIOgA}fZ5 zHY~?$NUx!xP&Fjpbx4nh=!F~#DK8n$SV1DB$C@L8?foe?@Qs=fWF>UE=j0%uK$(;r z09bA=hCX_8#RCI_B$ZN|5l&Quk z4yd<-L^$!8Z*AW$<0~49E4MR8MMZYSbHuwgS4(S683Ln5~Gl0tqoWS)Z8+%`1KVr{?BN z&VmB8`r++uZNq*gs5byQ|5_Rd37_BnS?HK??_LK>T2{+%x4SIWRSK1!%sK}4!pf&;Nx{gi>39-Z3%^zo0XHprB}HM@dD&ZM5Q9k%$86`1FHws-||2X^m6OrfFb!@ zg|?Q~@OiVUh6V$rjDgy@;~T;s&_e@s47erowsKob3p?yH5N6o_GQ>_*#X@pF1vq%e zyi2FlRMywGv9>QXGSvxzxXgn7{{lcV?dn=R+Fe7L;IC)S&UiRD<5l)4wV6dl5K`9D>}rxzQn>is^9wA@%m81GEVVT! z?{CZ$a}pgbL+Q&{^7<8SDwvl$jn58I7WzQP zGI5LG$d_iMq%$=+LEq4f=%5b!z+;IG9EdkLF;lQm9|yrA6d zNxl!PABfvQTr~}95kbF2w31CttsjoLCj$z+D{yYM|V@Z&3QL5M(~>$yH6mR9YH0y ziUm&ixNF?tPl$=pI%ffbsJR>|0ZNzUh!74AjwFEcwfsXs@!eO0B77$=Cs|E-o*o4| za}8?asD8j>xa!nb=;-KfD<@}0P1m{<00>)l1nv=bTi%O~?3|o^=mwrHLCXL-%BUA8 zjK{lpHzjm*;%ZIUJC)+N)4*V7etQ1LUvx8!%I}8P`JQ^(+S;~rBQ_~1$10- zv33fR9>IGd)g4;N%-X9J(f#l?l@2tnrb4H#NBHYXPsTc}=6 z*BiiZA&$P1x+Nwi2KgSN{aaTTJ1grJl(<`4o< Date: Fri, 9 Aug 2024 15:20:36 +1000 Subject: [PATCH 21/30] Update paper.md --- joss_paper/paper.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/joss_paper/paper.md b/joss_paper/paper.md index a9f422f6..65d4cdb1 100644 --- a/joss_paper/paper.md +++ b/joss_paper/paper.md @@ -202,6 +202,6 @@ The code used to create these results can be found [here](https://github.com/gri ## Acknowledgements -This research was partially funded by the Australian Government through the Australian Research Council (project number DP210103092). This work was also supported by computational resources provided by the Australian Government through NCI under the National Computational Merit Allocation Scheme (NCMAS). +This research was partially funded by the Australian Government through the Australian Research Council (project number DP210103092). This work was also supported by computational resources provided by the Australian Government through NCI under the National Computational Merit Allocation Scheme (NCMAS) and ANU Merit Allocation Scheme (ANUMAS). ## References From 1d106286d2739036dd0f872a1a195d9fdc8cff9a Mon Sep 17 00:00:00 2001 From: JordiManyer Date: Fri, 9 Aug 2024 17:36:38 +1000 Subject: [PATCH 22/30] Minor --- joss_paper/demo.jl | 36 ++++++++++++++++++++---------------- joss_paper/paper.md | 16 ++++++---------- 2 files changed, 26 insertions(+), 26 deletions(-) diff --git a/joss_paper/demo.jl b/joss_paper/demo.jl index 398b4f7c..be84b312 100644 --- a/joss_paper/demo.jl +++ b/joss_paper/demo.jl @@ -17,20 +17,18 @@ function add_labels!(labels) add_tag_from_tags!(labels,"walls",[1,2,3,4,5,7,8]) end -np = (2,2) -np_per_level = [np,(1,1)] -nc = (10,10) -fe_order = 2 - with_mpi() do distribute - parts = distribute(LinearIndices((prod(np),))) + np_per_level = [(2,2),(1,1)] # Number of processors per GMG level + parts = distribute(LinearIndices((prod(np_per_level[1]),))) - # Geometry - mh = CartesianModelHierarchy(parts,np_per_level,(0,1,0,1),nc;add_labels!) - model = get_model(mh,1) + # Create multi-level mesh + domain = (0,1,0,1) # Cartesian domain (xmin,xmax,ymin,ymax) + ncells = (10,10) # Number of cells + mh = CartesianModelHierarchy(parts,np_per_level,domain,ncells;add_labels!) + model = get_model(mh,1) # Finest mesh - # FE spaces - qdegree = 2*(fe_order+1) + # Create FESpaces + fe_order = 2 reffe_u = ReferenceFE(lagrangian,VectorValue{2,Float64},fe_order) reffe_p = ReferenceFE(lagrangian,Float64,fe_order-1;space=:P) @@ -39,7 +37,7 @@ with_mpi() do distribute U, V = get_fe_space(trials_u,1), get_fe_space(tests_u,1) Q = TestFESpace(model,reffe_p;conformity=:L2,constraint=:zeromean) - mfs = Gridap.MultiField.BlockMultiFieldStyle() + mfs = BlockMultiFieldStyle() X = MultiFieldFESpace([U,Q];style=mfs) Y = MultiFieldFESpace([V,Q];style=mfs) @@ -49,7 +47,8 @@ with_mpi() do distribute biform((u,p),(v,q),dΩ) = biform_u(u,v,dΩ) - ∫((∇⋅v)*p)dΩ - ∫((∇⋅u)*q)dΩ liform((v,q),dΩ) = ∫(v⋅f)dΩ - # Finest level + # Assemble linear system + qdegree = 2*(fe_order+1) # Quadrature degree Ω = Triangulation(model) dΩ = Measure(Ω,qdegree) op = AffineFEOperator((u,v)->biform(u,v,dΩ),v->liform(v,dΩ),X,Y) @@ -71,22 +70,27 @@ with_mpi() do distribute pre_smoothers=smoothers, post_smoothers=smoothers, coarsest_solver=LUSolver(), - maxiter=2,mode=:solver + maxiter=4,mode=:solver ) # PCG solver for the pressure block solver_p = CGSolver(JacobiLinearSolver();maxiter=20,atol=1e-14,rtol=1.e-6) - # Block triangular preconditioner + # 2x2 Block triangular preconditioner blocks = [LinearSystemBlock() LinearSystemBlock(); LinearSystemBlock() BiformBlock((p,q) -> ∫(p*q)dΩ,Q,Q)] P = BlockTriangularSolver(blocks,[solver_u,solver_p]) - solver = GMRESSolver(10;Pr=P,rtol=1.e-8,verbose=i_am_main(parts)) + + # Global solver + solver = FGMRESSolver(10,P;rtol=1.e-8,verbose=i_am_main(parts)) ns = numerical_setup(symbolic_setup(solver,A),A) + # Solve x = allocate_in_domain(A) fill!(x,0.0) solve!(x,ns,b) + + # Postprocess uh, ph = FEFunction(X,x) writevtk(Ω,"demo",cellfields=["uh"=>uh,"ph"=>ph]) end \ No newline at end of file diff --git a/joss_paper/paper.md b/joss_paper/paper.md index a9f422f6..5bfe4afc 100644 --- a/joss_paper/paper.md +++ b/joss_paper/paper.md @@ -114,12 +114,13 @@ with_mpi() do distribute parts = distribute(LinearIndices((prod(np_per_level[1]),))) # Create multi-level mesh - mh = CartesianModelHierarchy(parts,np_per_level,(0,1,0,1),(10,10);add_labels!) + domain = (0,1,0,1) # Cartesian domain (xmin,xmax,ymin,ymax) + ncells = (10,10) # Number of cells + mh = CartesianModelHierarchy(parts,np_per_level,domain,ncells;add_labels!) model = get_model(mh,1) # Finest mesh # Create FESpaces fe_order = 2 - qdegree = 2*(fe_order+1) reffe_u = ReferenceFE(lagrangian,VectorValue{2,Float64},fe_order) reffe_p = ReferenceFE(lagrangian,Float64,fe_order-1;space=:P) @@ -128,7 +129,7 @@ with_mpi() do distribute U, V = get_fe_space(trials_u,1), get_fe_space(tests_u,1) Q = TestFESpace(model,reffe_p;conformity=:L2,constraint=:zeromean) - mfs = Gridap.MultiField.BlockMultiFieldStyle() + mfs = BlockMultiFieldStyle() # Activates block-wise assembly X = MultiFieldFESpace([U,Q];style=mfs) Y = MultiFieldFESpace([V,Q];style=mfs) @@ -139,6 +140,7 @@ with_mpi() do distribute liform((v,q),dΩ) = ∫(v⋅f)dΩ # Assemble linear system + qdegree = 2*(fe_order+1) # Quadrature degree Ω = Triangulation(model) dΩ = Measure(Ω,qdegree) op = AffineFEOperator((u,v)->biform(u,v,dΩ),v->liform(v,dΩ),X,Y) @@ -160,7 +162,7 @@ with_mpi() do distribute pre_smoothers=smoothers, post_smoothers=smoothers, coarsest_solver=LUSolver(), - maxiter=2,mode=:solver + maxiter=4,mode=:solver ) # PCG solver for the pressure block @@ -194,12 +196,6 @@ The code used to create these results can be found [here](https://github.com/gri ![**Top**: Weak scalability for a Stokes problem in 2D. Time is given per F-GMRES iteration, as a function of the number of processors. **Middle**: Number of coarsening levels for the GMG solver, as a function of the number of processors. **Bottom**: Number of F-GMRES iterations required for convergence. \label{fig:packages}](weakScalability.png){ width=80% } -|Num Procs | 1 | 12 | 48 | 96 | 192 | 384 | 768 | 1536 | 3072 | -|:------------ |:----:|:----:|:----:|:----:|:----:|:----:|:----:|:----:|:----:| -|Time/Iter(s) | 3.2 | 5.7 | 6.3 | 6.0 | 7.1 | 7.2 | 9.8 | 10.3 | 13.3 | -|Num Levels | 2 | 2 | 2 | 2 | 3 | 3 | 4 | 4 | 5 | -|Num Iters | 20 | 12 | 10 | 11 | 9 | 10 | 8 | 9 | 7 | - ## Acknowledgements This research was partially funded by the Australian Government through the Australian Research Council (project number DP210103092). This work was also supported by computational resources provided by the Australian Government through NCI under the National Computational Merit Allocation Scheme (NCMAS). From 58a9148ad2b98ebc6bf6dc183b4a5c362633f0c6 Mon Sep 17 00:00:00 2001 From: JordiManyer Date: Fri, 16 Aug 2024 00:53:19 +1000 Subject: [PATCH 23/30] Minor --- joss_paper/paper.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/joss_paper/paper.md b/joss_paper/paper.md index 09a3f0db..c6c80fba 100644 --- a/joss_paper/paper.md +++ b/joss_paper/paper.md @@ -198,6 +198,6 @@ The code used to create these results can be found [here](https://github.com/gri ## Acknowledgements -This research was partially funded by the Australian Government through the Australian Research Council (project number DP210103092). This work was also supported by computational resources provided by the Australian Government through NCI under the National Computational Merit Allocation Scheme (NCMAS) and ANU Merit Allocation Scheme (ANUMAS). +This research was partially funded by the Australian Government through the Australian Research Council (project number DP210103092). This work was also supported by computational resources provided by the Australian Government through NCI under the National Computational Merit Allocation Scheme (NCMAS), the Monash-NCI partnership scheme and the ANU Merit Allocation Scheme (ANUMAS). ## References From bd03b57296590ddebebe42f9eecfb92afdb9eba1 Mon Sep 17 00:00:00 2001 From: JordiManyer Date: Sat, 17 Aug 2024 16:18:03 +1000 Subject: [PATCH 24/30] fixed Manifest.toml --- joss_paper/paper.bib | 24 + joss_paper/paper.md | 8 +- joss_paper/scalability/Manifest.toml | 886 +++++++++++++++++++++++++++ 3 files changed, 916 insertions(+), 2 deletions(-) create mode 100644 joss_paper/scalability/Manifest.toml diff --git a/joss_paper/paper.bib b/joss_paper/paper.bib index a0b83daf..2baa138c 100644 --- a/joss_paper/paper.bib +++ b/joss_paper/paper.bib @@ -306,4 +306,28 @@ @article{Arnold2 volume={85}, pages={197-217}, url={https://api.semanticscholar.org/CorpusID:14301688} +} + +@article{fenics-patch, + title={PCPATCH: Software for the Topological Construction of Multigrid Relaxation Methods}, + volume={47}, + ISSN={1557-7295}, + url={http://dx.doi.org/10.1145/3445791}, + DOI={10.1145/3445791}, + number={3}, + journal={ACM Transactions on Mathematical Software}, + publisher={Association for Computing Machinery (ACM)}, + author={Farrell, Patrick E. and Knepley, Matthew G. and Mitchell, Lawrence and Wechsung, Florian}, + year={2021}, + month=jun, pages={1–22} +} + +@misc{dealII-patch, + title={An implementation of tensor product patch smoothers on GPU}, + author={Cu Cui and Paul Grosse-Bley and Guido Kanschat and Robert Strzodka}, + year={2024}, + eprint={2405.19004}, + archivePrefix={arXiv}, + primaryClass={math.NA}, + url={https://arxiv.org/abs/2405.19004}, } \ No newline at end of file diff --git a/joss_paper/paper.md b/joss_paper/paper.md index c6c80fba..d3be2731 100644 --- a/joss_paper/paper.md +++ b/joss_paper/paper.md @@ -32,7 +32,7 @@ aas-journal: Journal of Open Source Software ## Summary and statement of need -The ever-increasing demand for resolution and accuracy in mathematical models of physical processes governed by systems of Partial Differential Equations (PDEs) can only be addressed using fully-parallel advanced numerical discretization methods and scalable solution methods, thus able to exploit the vast amount of computational resources in state-of-the-art supercomputers. +The ever-increasing demand for resolution and accuracy in mathematical models of physical processes governed by systems of Partial Differential Equations (PDEs) can only be addressed using fully-parallel advanced numerical discretization methods and scalable solvers, thus able to exploit the vast amount of computational resources in state-of-the-art supercomputers. One of the biggest scalability bottlenecks within Finite Element (FE) parallel codes is the solution of linear systems arising from the discretization of PDEs. The implementation of exact factorization-based solvers in parallel environments is an extremely challenging task, and even state-of-the-art libraries such as MUMPS [@MUMPS1] [@MUMPS2] or PARDISO [@PARDISO] have severe limitations in terms of scalability and memory consumption above a certain number of CPU cores. @@ -41,7 +41,9 @@ For simple problems, algebraic solvers and preconditioners (i.e based uniquelly In these cases, solvers that exploit the physics and mathematical discretization of the particular problem are required. This is the case of many multiphysics problems involving differential operators with a large kernel such as the divergence [@Arnold1] and the curl [@Arnold2]. Examples can be found amongst highly relevant problems such as Navier-Stokes, Maxwell or Darcy. Scalable solvers for this type of multiphysics problems rely on exploiting the block structure of such systems to find a spectrally equivalent block-preconditioner, and are often tied to a specific discretization of the underlying equations. -To this end, GridapSolvers is a registered Julia [@Bezanson2017] software package which provides highly scalable physics-informed solvers tailored for the FE numerical solution of PDEs on parallel computers. Emphasis is put on the modular design of the library, which easily allows new preconditioners to be designed from the user's specific problem. +As a consequence, high-quality open-source parallel finite element packages like FEniCS [@fenics-book] or deal.II [@dealII93] already provide implementations of several state-of-the-art physics-informed solvers [@fenics-patch] [@dealII-patch]. The Gridap ecosystem [@Badia2020] aims to provide a similar level of functionality within the Julia programming language [@Bezanson2017]. + +To this end, GridapSolvers is a registered Julia software package which provides highly scalable physics-informed solvers tailored for the FE numerical solution of PDEs on parallel computers within the Gridap ecosystem of packages. Emphasis is put on the modular design of the library, which easily allows new preconditioners to be designed from the user's specific problem. ## Building blocks and composability @@ -192,6 +194,8 @@ end The following section shows scalability results for the demo problem discussed above. We run our code on the Gadi supercomputer, which is part of the Australian National Computational Infrastructure (NCI). We use Intel's Cascade Lake 2x24-core Xeon Platinum 8274 nodes. Scalability is shown for up to 64 nodes, for a fixed local problem size of 48x64 quadrangle cells per processor. This amounts to a maximum size of approximately 37M cells and 415M degrees of freedom distributed amongst 3072 processors. Within the GMG solver, the number of coarsening levels is progressively increased to keep the global size of the coarsest solve (approximately) constant. The coarsest solve is then performed by a CG solver preconditioned by an Algebraic MultiGrid (AMG) solver, provided by PETSc [@petsc-user-ref] through the package GridapPETSc.jl. +The results show that the code scales relatively well up to 3072 processors, with loss in performance mostly tied to the number of GMG levels used for the velocity solver. The number of F-GMRES iterations required for convergence is also shown to be relatively constant (and even decreasing for bigger problem sizes), indicating that the preconditioner is robust with respect to the problem size. + The code used to create these results can be found [here](https://github.com/gridap/GridapSolvers.jl/tree/joss-paper/joss_paper/scalability). The exact releases for the packages used are provided by Julia's `Manifest.toml` file. ![**Top**: Weak scalability for a Stokes problem in 2D. Time is given per F-GMRES iteration, as a function of the number of processors. **Middle**: Number of coarsening levels for the GMG solver, as a function of the number of processors. **Bottom**: Number of F-GMRES iterations required for convergence. \label{fig:packages}](weakScalability.png){ width=80% } diff --git a/joss_paper/scalability/Manifest.toml b/joss_paper/scalability/Manifest.toml new file mode 100644 index 00000000..38d0db29 --- /dev/null +++ b/joss_paper/scalability/Manifest.toml @@ -0,0 +1,886 @@ +# This file is machine-generated - editing it directly is not advised + +julia_version = "1.9.4" +manifest_format = "2.0" +project_hash = "b424f735ecbb29aa1fe6e2140dce82cb9add9a26" + +[[deps.AbstractFFTs]] +deps = ["LinearAlgebra"] +git-tree-sha1 = "d92ad398961a3ed262d8bf04a1a2b8340f915fef" +uuid = "621f4979-c628-5d54-868e-fcf4e3e8185c" +version = "1.5.0" + + [deps.AbstractFFTs.extensions] + AbstractFFTsChainRulesCoreExt = "ChainRulesCore" + AbstractFFTsTestExt = "Test" + + [deps.AbstractFFTs.weakdeps] + ChainRulesCore = "d360d2e6-b24c-11e9-a2a3-2a2ae2dbcce4" + Test = "8dfed614-e22c-5e08-85e1-65c5234f0b40" + +[[deps.AbstractTrees]] +git-tree-sha1 = "2d9c9a55f9c93e8887ad391fbae72f8ef55e1177" +uuid = "1520ce14-60c1-5f80-bbc7-55ef81b5835c" +version = "0.4.5" + +[[deps.Adapt]] +deps = ["LinearAlgebra", "Requires"] +git-tree-sha1 = "6a55b747d1812e699320963ffde36f1ebdda4099" +uuid = "79e6a3ab-5dfb-504d-930d-738a2a938a0e" +version = "4.0.4" +weakdeps = ["StaticArrays"] + + [deps.Adapt.extensions] + AdaptStaticArraysExt = "StaticArrays" + +[[deps.ArgCheck]] +git-tree-sha1 = "a3a402a35a2f7e0b87828ccabbd5ebfbebe356b4" +uuid = "dce04be8-c92d-5529-be00-80e4d2c0e197" +version = "2.3.0" + +[[deps.ArgParse]] +deps = ["Logging", "TextWrap"] +git-tree-sha1 = "22cf435ac22956a7b45b0168abbc871176e7eecc" +uuid = "c7e460c6-2fb9-53a9-8c5b-16f535851c63" +version = "1.2.0" + +[[deps.ArgTools]] +uuid = "0dad84c5-d112-42e6-8d28-ef12dabb789f" +version = "1.1.1" + +[[deps.ArrayInterface]] +deps = ["Adapt", "LinearAlgebra", "Requires", "SparseArrays", "SuiteSparse"] +git-tree-sha1 = "c5aeb516a84459e0318a02507d2261edad97eb75" +uuid = "4fba245c-0d91-5ea0-9b3e-6abc04ee57a9" +version = "7.7.1" + + [deps.ArrayInterface.extensions] + ArrayInterfaceBandedMatricesExt = "BandedMatrices" + ArrayInterfaceBlockBandedMatricesExt = "BlockBandedMatrices" + ArrayInterfaceCUDAExt = "CUDA" + ArrayInterfaceGPUArraysCoreExt = "GPUArraysCore" + ArrayInterfaceStaticArraysCoreExt = "StaticArraysCore" + ArrayInterfaceTrackerExt = "Tracker" + + [deps.ArrayInterface.weakdeps] + BandedMatrices = "aae01518-5342-5314-be14-df237901396f" + BlockBandedMatrices = "ffab5731-97b5-5995-9138-79e8c1846df0" + CUDA = "052768ef-5323-5732-b1bb-66c8b64840ba" + GPUArraysCore = "46192b85-c4d5-4398-a991-12ede77f4527" + StaticArraysCore = "1e83bf80-4336-4d27-bf5d-d5a4f845583c" + Tracker = "9f7883ad-71c0-57eb-9f7f-b5c9e6d3789c" + +[[deps.ArrayLayouts]] +deps = ["FillArrays", "LinearAlgebra"] +git-tree-sha1 = "ce2ca959f932f5dad70697dd93133d1167cf1e4e" +uuid = "4c555306-a7a7-4459-81d9-ec55ddd5c99a" +version = "1.10.2" +weakdeps = ["SparseArrays"] + + [deps.ArrayLayouts.extensions] + ArrayLayoutsSparseArraysExt = "SparseArrays" + +[[deps.Artifacts]] +uuid = "56f22d72-fd6d-98f1-02f0-08ddc0907c33" + +[[deps.AutoHashEquals]] +deps = ["Pkg"] +git-tree-sha1 = "daaeb6f7f77b88c072a83a2451801818acb5c63b" +uuid = "15f4f7f2-30c1-5605-9d31-71845cf9641f" +version = "2.1.0" + +[[deps.BSON]] +git-tree-sha1 = "4c3e506685c527ac6a54ccc0c8c76fd6f91b42fb" +uuid = "fbb218c0-5317-5bc6-957e-2ee96dd4b1f0" +version = "0.3.9" + +[[deps.Base64]] +uuid = "2a0f44e3-6c83-55bd-87e4-b1978d98bd5f" + +[[deps.BlockArrays]] +deps = ["ArrayLayouts", "FillArrays", "LinearAlgebra"] +git-tree-sha1 = "9a9610fbe5779636f75229e423e367124034af41" +uuid = "8e7c35d0-a365-5155-bbbb-fb81a777f24e" +version = "0.16.43" + +[[deps.CEnum]] +git-tree-sha1 = "eb4cb44a499229b3b8426dcfb5dd85333951ff90" +uuid = "fa961155-64e5-5f13-b03f-caf6b980ea82" +version = "0.4.2" + +[[deps.CircularArrays]] +deps = ["OffsetArrays"] +git-tree-sha1 = "e24a6f390e5563583bb4315c73035b5b3f3e7ab4" +uuid = "7a955b69-7140-5f4e-a0ed-f168c5e2e749" +version = "1.4.0" + +[[deps.CodecZlib]] +deps = ["TranscodingStreams", "Zlib_jll"] +git-tree-sha1 = "bce6804e5e6044c6daab27bb533d1295e4a2e759" +uuid = "944b1d66-785c-5afd-91f1-9de20f533193" +version = "0.7.6" + +[[deps.Combinatorics]] +git-tree-sha1 = "08c8b6831dc00bfea825826be0bc8336fc369860" +uuid = "861a8166-3701-5b0c-9a16-15d98fcdc6aa" +version = "1.0.2" + +[[deps.CommonSubexpressions]] +deps = ["MacroTools", "Test"] +git-tree-sha1 = "7b8a93dba8af7e3b42fecabf646260105ac373f7" +uuid = "bbf7d656-a473-5ed7-a52c-81e309532950" +version = "0.3.0" + +[[deps.Compat]] +deps = ["TOML", "UUIDs"] +git-tree-sha1 = "8ae8d32e09f0dcf42a36b90d4e17f5dd2e4c4215" +uuid = "34da2185-b29b-5c13-b0c7-acf172513d20" +version = "4.16.0" +weakdeps = ["Dates", "LinearAlgebra"] + + [deps.Compat.extensions] + CompatLinearAlgebraExt = "LinearAlgebra" + +[[deps.CompilerSupportLibraries_jll]] +deps = ["Artifacts", "Libdl"] +uuid = "e66e0078-7015-5450-92f7-15fbd957f2ae" +version = "1.0.5+0" + +[[deps.ConstructionBase]] +deps = ["LinearAlgebra"] +git-tree-sha1 = "d8a9c0b6ac2d9081bf76324b39c78ca3ce4f0c98" +uuid = "187b0558-2788-49d3-abe0-74a17ed4e7c9" +version = "1.5.6" + + [deps.ConstructionBase.extensions] + ConstructionBaseIntervalSetsExt = "IntervalSets" + ConstructionBaseStaticArraysExt = "StaticArrays" + + [deps.ConstructionBase.weakdeps] + IntervalSets = "8197267c-284f-5f27-9208-e0e47529a953" + StaticArrays = "90137ffa-7385-5640-81b9-e52037218182" + +[[deps.DataAPI]] +git-tree-sha1 = "abe83f3a2f1b857aac70ef8b269080af17764bbe" +uuid = "9a962f9c-6df0-11e9-0e5d-c546b8b5ee8a" +version = "1.16.0" + +[[deps.DataStructures]] +deps = ["Compat", "InteractiveUtils", "OrderedCollections"] +git-tree-sha1 = "1d0a14036acb104d9e89698bd408f63ab58cdc82" +uuid = "864edb3b-99cc-5e75-8d2d-829cb0a9cfe8" +version = "0.18.20" + +[[deps.DataValueInterfaces]] +git-tree-sha1 = "bfc1187b79289637fa0ef6d4436ebdfe6905cbd6" +uuid = "e2d170a0-9d28-54be-80f0-106bbe20a464" +version = "1.0.0" + +[[deps.Dates]] +deps = ["Printf"] +uuid = "ade2ca70-3891-5945-98fb-dc099432e06a" + +[[deps.DiffResults]] +deps = ["StaticArraysCore"] +git-tree-sha1 = "782dd5f4561f5d267313f23853baaaa4c52ea621" +uuid = "163ba53b-c6d8-5494-b064-1a9d43ac40c5" +version = "1.1.0" + +[[deps.DiffRules]] +deps = ["IrrationalConstants", "LogExpFunctions", "NaNMath", "Random", "SpecialFunctions"] +git-tree-sha1 = "23163d55f885173722d1e4cf0f6110cdbaf7e272" +uuid = "b552c78f-8df3-52c6-915a-8e097449b14b" +version = "1.15.1" + +[[deps.Distances]] +deps = ["LinearAlgebra", "Statistics", "StatsAPI"] +git-tree-sha1 = "66c4c81f259586e8f002eacebc177e1fb06363b0" +uuid = "b4f34e82-e78d-54a5-968a-f98e89d6e8f7" +version = "0.10.11" + + [deps.Distances.extensions] + DistancesChainRulesCoreExt = "ChainRulesCore" + DistancesSparseArraysExt = "SparseArrays" + + [deps.Distances.weakdeps] + ChainRulesCore = "d360d2e6-b24c-11e9-a2a3-2a2ae2dbcce4" + SparseArrays = "2f01184e-e22b-5df5-ae63-d93ebab69eaf" + +[[deps.Distributed]] +deps = ["Random", "Serialization", "Sockets"] +uuid = "8ba89e20-285c-5b6f-9357-94700520ee1b" + +[[deps.DocStringExtensions]] +deps = ["LibGit2"] +git-tree-sha1 = "2fb1e02f2b635d0845df5d7c167fec4dd739b00d" +uuid = "ffbed154-4ef7-542d-bbb7-c09d3a79fcae" +version = "0.9.3" + +[[deps.Downloads]] +deps = ["ArgTools", "FileWatching", "LibCURL", "NetworkOptions"] +uuid = "f43a241f-c20a-4ad4-852c-f6b1247861c6" +version = "1.6.0" + +[[deps.DrWatson]] +deps = ["Dates", "FileIO", "JLD2", "LibGit2", "MacroTools", "Pkg", "Random", "Requires", "Scratch", "UnPack"] +git-tree-sha1 = "2d6e724fab0c57284b3d1a7473a5a62ce6aba471" +uuid = "634d3b9d-ee7a-5ddf-bec9-22491ea816e1" +version = "2.15.0" + +[[deps.FFTW]] +deps = ["AbstractFFTs", "FFTW_jll", "LinearAlgebra", "MKL_jll", "Preferences", "Reexport"] +git-tree-sha1 = "4820348781ae578893311153d69049a93d05f39d" +uuid = "7a1cc6ca-52ef-59f5-83cd-3a7055c09341" +version = "1.8.0" + +[[deps.FFTW_jll]] +deps = ["Artifacts", "JLLWrappers", "Libdl", "Pkg"] +git-tree-sha1 = "c6033cc3892d0ef5bb9cd29b7f2f0331ea5184ea" +uuid = "f5851436-0d7a-5f13-b9de-f02708fd171a" +version = "3.3.10+0" + +[[deps.FastGaussQuadrature]] +deps = ["LinearAlgebra", "SpecialFunctions", "StaticArrays"] +git-tree-sha1 = "fd923962364b645f3719855c88f7074413a6ad92" +uuid = "442a2c76-b920-505d-bb47-c5924d526838" +version = "1.0.2" + +[[deps.FileIO]] +deps = ["Pkg", "Requires", "UUIDs"] +git-tree-sha1 = "82d8afa92ecf4b52d78d869f038ebfb881267322" +uuid = "5789e2e9-d7fb-5bc7-8068-2c6fae9b9549" +version = "1.16.3" + +[[deps.FileWatching]] +uuid = "7b1f6079-737a-58dc-b8bc-7a2ca5c1b5ee" + +[[deps.FillArrays]] +deps = ["LinearAlgebra"] +git-tree-sha1 = "0653c0a2396a6da5bc4766c43041ef5fd3efbe57" +uuid = "1a297f60-69ca-5386-bcde-b61e274b549b" +version = "1.11.0" + + [deps.FillArrays.extensions] + FillArraysPDMatsExt = "PDMats" + FillArraysSparseArraysExt = "SparseArrays" + FillArraysStatisticsExt = "Statistics" + + [deps.FillArrays.weakdeps] + PDMats = "90014a1f-27ba-587c-ab20-58faa44d9150" + SparseArrays = "2f01184e-e22b-5df5-ae63-d93ebab69eaf" + Statistics = "10745b16-79ce-11e8-11f9-7d13ad32a3b2" + +[[deps.FiniteDiff]] +deps = ["ArrayInterface", "LinearAlgebra", "Requires", "Setfield", "SparseArrays"] +git-tree-sha1 = "73d1214fec245096717847c62d389a5d2ac86504" +uuid = "6a86dc24-6348-571c-b903-95158fe2bd41" +version = "2.22.0" + + [deps.FiniteDiff.extensions] + FiniteDiffBandedMatricesExt = "BandedMatrices" + FiniteDiffBlockBandedMatricesExt = "BlockBandedMatrices" + FiniteDiffStaticArraysExt = "StaticArrays" + + [deps.FiniteDiff.weakdeps] + BandedMatrices = "aae01518-5342-5314-be14-df237901396f" + BlockBandedMatrices = "ffab5731-97b5-5995-9138-79e8c1846df0" + StaticArrays = "90137ffa-7385-5640-81b9-e52037218182" + +[[deps.ForwardDiff]] +deps = ["CommonSubexpressions", "DiffResults", "DiffRules", "LinearAlgebra", "LogExpFunctions", "NaNMath", "Preferences", "Printf", "Random", "SpecialFunctions"] +git-tree-sha1 = "cf0fe81336da9fb90944683b8c41984b08793dad" +uuid = "f6369f11-7733-5829-9624-2563aa707210" +version = "0.10.36" +weakdeps = ["StaticArrays"] + + [deps.ForwardDiff.extensions] + ForwardDiffStaticArraysExt = "StaticArrays" + +[[deps.Future]] +deps = ["Random"] +uuid = "9fa8497b-333b-5362-9e8d-4d0656e87820" + +[[deps.Glob]] +git-tree-sha1 = "97285bbd5230dd766e9ef6749b80fc617126d496" +uuid = "c27321d9-0574-5035-807b-f59d2c89b15c" +version = "1.3.1" + +[[deps.Gridap]] +deps = ["AbstractTrees", "BSON", "BlockArrays", "Combinatorics", "DataStructures", "DocStringExtensions", "FastGaussQuadrature", "FileIO", "FillArrays", "ForwardDiff", "JLD2", "JSON", "LineSearches", "LinearAlgebra", "NLsolve", "NearestNeighbors", "PolynomialBases", "Preferences", "QuadGK", "Random", "SparseArrays", "SparseMatricesCSR", "StaticArrays", "Statistics", "Test", "WriteVTK"] +git-tree-sha1 = "8c735f66c5fa081c3ea32a803df434c8790e1a47" +uuid = "56d4f2e9-7ea1-5844-9cf6-b9c51ca7ce8e" +version = "0.18.4" + +[[deps.GridapDistributed]] +deps = ["BlockArrays", "FillArrays", "Gridap", "LinearAlgebra", "MPI", "PartitionedArrays", "SparseArrays", "SparseMatricesCSR", "WriteVTK"] +git-tree-sha1 = "72b7eb03a098c79860f8676bf3959449285516f5" +uuid = "f9701e48-63b3-45aa-9a63-9bc6c271f355" +version = "0.4.4" + +[[deps.GridapP4est]] +deps = ["ArgParse", "FillArrays", "Gridap", "GridapDistributed", "Libdl", "MPI", "P4est_wrapper", "PartitionedArrays", "Test"] +git-tree-sha1 = "f92d49f8b162ce9eba7543be9c075131f9ef7f5a" +uuid = "c2c8e14b-f5fd-423d-9666-1dd9ad120af9" +version = "0.3.8" + +[[deps.GridapPETSc]] +deps = ["Gridap", "GridapDistributed", "Libdl", "LinearAlgebra", "MPI", "PETSc_jll", "PartitionedArrays", "Random", "SparseArrays", "SparseMatricesCSR"] +git-tree-sha1 = "549d0dce1a8051e6cd2c6dc4edcf0422d48ceecb" +uuid = "bcdc36c2-0c3e-11ea-095a-c9dadae499f1" +version = "0.5.2" + +[[deps.GridapSolvers]] +deps = ["AbstractTrees", "BlockArrays", "FillArrays", "Gridap", "GridapDistributed", "GridapP4est", "GridapPETSc", "IterativeSolvers", "LineSearches", "LinearAlgebra", "MPI", "NLsolve", "PartitionedArrays", "Printf", "SparseArrays", "SparseMatricesCSR"] +git-tree-sha1 = "851108149119690bda0afa2dba818e80c1ca1db7" +uuid = "6d3209ee-5e3c-4db7-a716-942eb12ed534" +version = "0.4.0" + +[[deps.Hwloc_jll]] +deps = ["Artifacts", "JLLWrappers", "Libdl"] +git-tree-sha1 = "5e19e1e4fa3e71b774ce746274364aef0234634e" +uuid = "e33a78d0-f292-5ffc-b300-72abe9b543c8" +version = "2.11.1+0" + +[[deps.IntelOpenMP_jll]] +deps = ["Artifacts", "JLLWrappers", "Libdl"] +git-tree-sha1 = "14eb2b542e748570b56446f4c50fbfb2306ebc45" +uuid = "1d5cc7b8-4909-519e-a0f8-d0f5ad9712d0" +version = "2024.2.0+0" + +[[deps.InteractiveUtils]] +deps = ["Markdown"] +uuid = "b77e0a4c-d291-57a0-90e8-8db25a27a240" + +[[deps.IrrationalConstants]] +git-tree-sha1 = "630b497eafcc20001bba38a4651b327dcfc491d2" +uuid = "92d709cd-6900-40b7-9082-c6be49f344b6" +version = "0.2.2" + +[[deps.IterativeSolvers]] +deps = ["LinearAlgebra", "Printf", "Random", "RecipesBase", "SparseArrays"] +git-tree-sha1 = "59545b0a2b27208b0650df0a46b8e3019f85055b" +uuid = "42fd0dbc-a981-5370-80f2-aaf504508153" +version = "0.9.4" + +[[deps.IteratorInterfaceExtensions]] +git-tree-sha1 = "a3f24677c21f5bbe9d2a714f95dcd58337fb2856" +uuid = "82899510-4779-5014-852e-03e436cf321d" +version = "1.0.0" + +[[deps.JLD2]] +deps = ["FileIO", "MacroTools", "Mmap", "OrderedCollections", "PrecompileTools", "Reexport", "Requires", "TranscodingStreams", "UUIDs", "Unicode"] +git-tree-sha1 = "67d4690d32c22e28818a434b293a374cc78473d3" +uuid = "033835bb-8acc-5ee8-8aae-3f567f8a3819" +version = "0.4.51" + +[[deps.JLLWrappers]] +deps = ["Artifacts", "Preferences"] +git-tree-sha1 = "7e5d6779a1e09a36db2a7b6cff50942a0a7d0fca" +uuid = "692b3bcd-3c85-4b1f-b108-f13ce0eb3210" +version = "1.5.0" + +[[deps.JSON]] +deps = ["Dates", "Mmap", "Parsers", "Unicode"] +git-tree-sha1 = "31e996f0a15c7b280ba9f76636b3ff9e2ae58c9a" +uuid = "682c06a0-de6a-54ab-a142-c8b1cf79cde6" +version = "0.21.4" + +[[deps.LazyArtifacts]] +deps = ["Artifacts", "Pkg"] +uuid = "4af54fe1-eca0-43a8-85a7-787d91b784e3" + +[[deps.LibCURL]] +deps = ["LibCURL_jll", "MozillaCACerts_jll"] +uuid = "b27032c2-a3e7-50c8-80cd-2d36dbcbfd21" +version = "0.6.4" + +[[deps.LibCURL_jll]] +deps = ["Artifacts", "LibSSH2_jll", "Libdl", "MbedTLS_jll", "Zlib_jll", "nghttp2_jll"] +uuid = "deac9b47-8bc7-5906-a0fe-35ac56dc84c0" +version = "8.4.0+0" + +[[deps.LibGit2]] +deps = ["Base64", "NetworkOptions", "Printf", "SHA"] +uuid = "76f85450-5226-5b5a-8eaa-529ad045b433" + +[[deps.LibSSH2_jll]] +deps = ["Artifacts", "Libdl", "MbedTLS_jll"] +uuid = "29816b5a-b9ab-546f-933c-edad1886dfa8" +version = "1.11.0+1" + +[[deps.Libdl]] +uuid = "8f399da3-3557-5675-b5ff-fb832c97cbdb" + +[[deps.Libiconv_jll]] +deps = ["Artifacts", "JLLWrappers", "Libdl"] +git-tree-sha1 = "f9557a255370125b405568f9767d6d195822a175" +uuid = "94ce4f54-9a6c-5748-9c1c-f9c7231a4531" +version = "1.17.0+0" + +[[deps.LightXML]] +deps = ["Libdl", "XML2_jll"] +git-tree-sha1 = "3a994404d3f6709610701c7dabfc03fed87a81f8" +uuid = "9c8b4983-aa76-5018-a973-4c85ecc9e179" +version = "0.9.1" + +[[deps.LineSearches]] +deps = ["LinearAlgebra", "NLSolversBase", "NaNMath", "Parameters", "Printf"] +git-tree-sha1 = "e4c3be53733db1051cc15ecf573b1042b3a712a1" +uuid = "d3d80556-e9d4-5f37-9878-2ab0fcc64255" +version = "7.3.0" + +[[deps.LinearAlgebra]] +deps = ["Libdl", "OpenBLAS_jll", "libblastrampoline_jll"] +uuid = "37e2e46d-f89d-539d-b4ee-838fcccc9c8e" + +[[deps.LogExpFunctions]] +deps = ["DocStringExtensions", "IrrationalConstants", "LinearAlgebra"] +git-tree-sha1 = "a2d09619db4e765091ee5c6ffe8872849de0feea" +uuid = "2ab3a3ac-af41-5b50-aa03-7779005ae688" +version = "0.3.28" + + [deps.LogExpFunctions.extensions] + LogExpFunctionsChainRulesCoreExt = "ChainRulesCore" + LogExpFunctionsChangesOfVariablesExt = "ChangesOfVariables" + LogExpFunctionsInverseFunctionsExt = "InverseFunctions" + + [deps.LogExpFunctions.weakdeps] + ChainRulesCore = "d360d2e6-b24c-11e9-a2a3-2a2ae2dbcce4" + ChangesOfVariables = "9e997f8a-9a97-42d5-a9f1-ce6bfc15e2c0" + InverseFunctions = "3587e190-3f89-42d0-90ee-14403ec27112" + +[[deps.Logging]] +uuid = "56ddb016-857b-54e1-b83d-db4d58db5568" + +[[deps.MKL_jll]] +deps = ["Artifacts", "IntelOpenMP_jll", "JLLWrappers", "LazyArtifacts", "Libdl", "oneTBB_jll"] +git-tree-sha1 = "f046ccd0c6db2832a9f639e2c669c6fe867e5f4f" +uuid = "856f044c-d86e-5d09-b602-aeab76dc8ba7" +version = "2024.2.0+0" + +[[deps.MPI]] +deps = ["Distributed", "DocStringExtensions", "Libdl", "MPICH_jll", "MPIPreferences", "MPItrampoline_jll", "MicrosoftMPI_jll", "OpenMPI_jll", "PkgVersion", "PrecompileTools", "Requires", "Serialization", "Sockets"] +git-tree-sha1 = "b4d8707e42b693720b54f0b3434abee6dd4d947a" +uuid = "da04e1cc-30fd-572f-bb4f-1f8673147195" +version = "0.20.16" + + [deps.MPI.extensions] + AMDGPUExt = "AMDGPU" + CUDAExt = "CUDA" + + [deps.MPI.weakdeps] + AMDGPU = "21141c5a-9bdb-4563-92ae-f87d6854732e" + CUDA = "052768ef-5323-5732-b1bb-66c8b64840ba" + +[[deps.MPICH_jll]] +deps = ["Artifacts", "CompilerSupportLibraries_jll", "Hwloc_jll", "JLLWrappers", "LazyArtifacts", "Libdl", "MPIPreferences", "TOML"] +git-tree-sha1 = "19d4bd098928a3263693991500d05d74dbdc2004" +uuid = "7cb0a576-ebde-5e09-9194-50597f1243b4" +version = "4.2.2+0" + +[[deps.MPIPreferences]] +deps = ["Libdl", "Preferences"] +git-tree-sha1 = "c105fe467859e7f6e9a852cb15cb4301126fac07" +uuid = "3da0fdf6-3ccc-4f1b-acd9-58baa6c99267" +version = "0.1.11" + +[[deps.MPItrampoline_jll]] +deps = ["Artifacts", "CompilerSupportLibraries_jll", "JLLWrappers", "LazyArtifacts", "Libdl", "MPIPreferences", "TOML"] +git-tree-sha1 = "8c35d5420193841b2f367e658540e8d9e0601ed0" +uuid = "f1f71cc9-e9ae-5b93-9b94-4fe0e1ad3748" +version = "5.4.0+0" + +[[deps.MacroTools]] +deps = ["Markdown", "Random"] +git-tree-sha1 = "2fa9ee3e63fd3a4f7a9a4f4744a52f4856de82df" +uuid = "1914dd2f-81c6-5fcd-8719-6d5c9610ff09" +version = "0.5.13" + +[[deps.Markdown]] +deps = ["Base64"] +uuid = "d6f4376e-aef5-505a-96c1-9c027394607a" + +[[deps.MbedTLS_jll]] +deps = ["Artifacts", "Libdl"] +uuid = "c8ffd9c3-330d-5841-b78e-0817d7145fa1" +version = "2.28.2+0" + +[[deps.MicrosoftMPI_jll]] +deps = ["Artifacts", "JLLWrappers", "Libdl", "Pkg"] +git-tree-sha1 = "f12a29c4400ba812841c6ace3f4efbb6dbb3ba01" +uuid = "9237b28f-5490-5468-be7b-bb81f5f5e6cf" +version = "10.1.4+2" + +[[deps.Mmap]] +uuid = "a63ad114-7e13-5084-954f-fe012c677804" + +[[deps.MozillaCACerts_jll]] +uuid = "14a3606d-f60d-562e-9121-12d972cd8159" +version = "2022.10.11" + +[[deps.Mustache]] +deps = ["Printf", "Tables"] +git-tree-sha1 = "3b2db451a872b20519ebb0cec759d3d81a1c6bcb" +uuid = "ffc61752-8dc7-55ee-8c37-f3e9cdd09e70" +version = "1.0.20" + +[[deps.NLSolversBase]] +deps = ["DiffResults", "Distributed", "FiniteDiff", "ForwardDiff"] +git-tree-sha1 = "a0b464d183da839699f4c79e7606d9d186ec172c" +uuid = "d41bc354-129a-5804-8e4c-c37616107c6c" +version = "7.8.3" + +[[deps.NLsolve]] +deps = ["Distances", "LineSearches", "LinearAlgebra", "NLSolversBase", "Printf", "Reexport"] +git-tree-sha1 = "019f12e9a1a7880459d0173c182e6a99365d7ac1" +uuid = "2774e3e8-f4cf-5e23-947b-6d7e65073b56" +version = "4.5.1" + +[[deps.NaNMath]] +deps = ["OpenLibm_jll"] +git-tree-sha1 = "0877504529a3e5c3343c6f8b4c0381e57e4387e4" +uuid = "77ba4419-2d1f-58cd-9bb1-8ffee604a2e3" +version = "1.0.2" + +[[deps.NearestNeighbors]] +deps = ["Distances", "StaticArrays"] +git-tree-sha1 = "91a67b4d73842da90b526011fa85c5c4c9343fe0" +uuid = "b8a86587-4115-5ab1-83bc-aa920d37bbce" +version = "0.4.18" + +[[deps.NetworkOptions]] +uuid = "ca575930-c2e3-43a9-ace4-1e988b2c1908" +version = "1.2.0" + +[[deps.OffsetArrays]] +git-tree-sha1 = "1a27764e945a152f7ca7efa04de513d473e9542e" +uuid = "6fe1bfb0-de20-5000-8ca7-80f57d26f881" +version = "1.14.1" +weakdeps = ["Adapt"] + + [deps.OffsetArrays.extensions] + OffsetArraysAdaptExt = "Adapt" + +[[deps.OpenBLAS32_jll]] +deps = ["Artifacts", "CompilerSupportLibraries_jll", "JLLWrappers", "Libdl", "Pkg"] +git-tree-sha1 = "2fb9ee2dc14d555a6df2a714b86b7125178344c2" +uuid = "656ef2d0-ae68-5445-9ca0-591084a874a2" +version = "0.3.21+0" + +[[deps.OpenBLAS_jll]] +deps = ["Artifacts", "CompilerSupportLibraries_jll", "Libdl"] +uuid = "4536629a-c528-5b80-bd46-f80d51c5b363" +version = "0.3.21+4" + +[[deps.OpenLibm_jll]] +deps = ["Artifacts", "Libdl"] +uuid = "05823500-19ac-5b8b-9628-191a04bc5112" +version = "0.8.1+0" + +[[deps.OpenMPI_jll]] +deps = ["Artifacts", "CompilerSupportLibraries_jll", "Hwloc_jll", "JLLWrappers", "LazyArtifacts", "Libdl", "MPIPreferences", "TOML", "Zlib_jll"] +git-tree-sha1 = "bfce6d523861a6c562721b262c0d1aaeead2647f" +uuid = "fe0851c0-eecd-5654-98d4-656369965a5c" +version = "5.0.5+0" + +[[deps.OpenSpecFun_jll]] +deps = ["Artifacts", "CompilerSupportLibraries_jll", "JLLWrappers", "Libdl", "Pkg"] +git-tree-sha1 = "13652491f6856acfd2db29360e1bbcd4565d04f1" +uuid = "efe28fd5-8261-553b-a9e1-b2916fc3738e" +version = "0.5.5+0" + +[[deps.OrderedCollections]] +git-tree-sha1 = "dfdf5519f235516220579f949664f1bf44e741c5" +uuid = "bac558e1-5e72-5ebc-8fee-abe8a469f55d" +version = "1.6.3" + +[[deps.P4est_jll]] +deps = ["Artifacts", "JLLWrappers", "LazyArtifacts", "Libdl", "MPICH_jll", "MPIPreferences", "MPItrampoline_jll", "MicrosoftMPI_jll", "OpenMPI_jll", "Pkg", "TOML", "Zlib_jll"] +git-tree-sha1 = "70c2d9a33b8810198314a5722ee3e9520110b28d" +uuid = "6b5a15aa-cf52-5330-8376-5e5d90283449" +version = "2.8.1+2" + +[[deps.P4est_wrapper]] +deps = ["CEnum", "Libdl", "MPI", "P4est_jll"] +git-tree-sha1 = "d3730f758f8364d734dd35506ea3639db301d87a" +uuid = "3743d7c0-8adf-11ea-380b-7d33b0ecc1da" +version = "0.2.3" + +[[deps.PETSc_jll]] +deps = ["Artifacts", "CompilerSupportLibraries_jll", "JLLWrappers", "Libdl", "MPICH_jll", "MicrosoftMPI_jll", "OpenBLAS32_jll", "Pkg"] +git-tree-sha1 = "8384198eba24438cee406ec7ca8854acdcbbd2c8" +uuid = "8fa3689e-f0b9-5420-9873-adf6ccf46f2d" +version = "3.15.2+0" + +[[deps.PackageCompiler]] +deps = ["Artifacts", "Glob", "LazyArtifacts", "Libdl", "Pkg", "Printf", "RelocatableFolders", "TOML", "UUIDs", "p7zip_jll"] +git-tree-sha1 = "48d4429862157ad5500c4f61444db1b8c32e0a2b" +uuid = "9b87118b-4619-50d2-8e1e-99f35a4d4d9d" +version = "2.1.17" + +[[deps.Parameters]] +deps = ["OrderedCollections", "UnPack"] +git-tree-sha1 = "34c0e9ad262e5f7fc75b10a9952ca7692cfc5fbe" +uuid = "d96e819e-fc66-5662-9728-84c9c7592b0a" +version = "0.12.3" + +[[deps.Parsers]] +deps = ["Dates", "PrecompileTools", "UUIDs"] +git-tree-sha1 = "8489905bcdbcfac64d1daa51ca07c0d8f0283821" +uuid = "69de0a69-1ddd-5017-9359-2bf0b02dc9f0" +version = "2.8.1" + +[[deps.PartitionedArrays]] +deps = ["CircularArrays", "Distances", "FillArrays", "IterativeSolvers", "LinearAlgebra", "MPI", "Printf", "Random", "SparseArrays", "SparseMatricesCSR"] +git-tree-sha1 = "149d2287770c6a533507d74beaa73d76c0727922" +uuid = "5a9dfac6-5c52-46f7-8278-5e2210713be9" +version = "0.3.4" + +[[deps.Pkg]] +deps = ["Artifacts", "Dates", "Downloads", "FileWatching", "LibGit2", "Libdl", "Logging", "Markdown", "Printf", "REPL", "Random", "SHA", "Serialization", "TOML", "Tar", "UUIDs", "p7zip_jll"] +uuid = "44cfe95a-1eb2-52ea-b672-e2afdf69b78f" +version = "1.9.2" + +[[deps.PkgVersion]] +deps = ["Pkg"] +git-tree-sha1 = "f9501cc0430a26bc3d156ae1b5b0c1b47af4d6da" +uuid = "eebad327-c553-4316-9ea0-9fa01ccd7688" +version = "0.3.3" + +[[deps.PolynomialBases]] +deps = ["ArgCheck", "AutoHashEquals", "FFTW", "FastGaussQuadrature", "LinearAlgebra", "Requires", "SimpleUnPack", "SpecialFunctions"] +git-tree-sha1 = "b62fd0464edfffce54393cd617135af30fa47006" +uuid = "c74db56a-226d-5e98-8bb0-a6049094aeea" +version = "0.4.22" + +[[deps.PrecompileTools]] +deps = ["Preferences"] +git-tree-sha1 = "5aa36f7049a63a1528fe8f7c3f2113413ffd4e1f" +uuid = "aea7be01-6a6a-4083-8856-8a6e6704d82a" +version = "1.2.1" + +[[deps.Preferences]] +deps = ["TOML"] +git-tree-sha1 = "9306f6085165d270f7e3db02af26a400d580f5c6" +uuid = "21216c6a-2e73-6563-6e65-726566657250" +version = "1.4.3" + +[[deps.Printf]] +deps = ["Unicode"] +uuid = "de0858da-6303-5e67-8744-51eddeeeb8d7" + +[[deps.QuadGK]] +deps = ["DataStructures", "LinearAlgebra"] +git-tree-sha1 = "e237232771fdafbae3db5c31275303e056afaa9f" +uuid = "1fd47b50-473d-5c70-9696-f719f8f3bcdc" +version = "2.10.1" + +[[deps.REPL]] +deps = ["InteractiveUtils", "Markdown", "Sockets", "Unicode"] +uuid = "3fa0cd96-eef1-5676-8a61-b3b8758bbffb" + +[[deps.Random]] +deps = ["SHA", "Serialization"] +uuid = "9a3f8284-a2c9-5f02-9a11-845980a1fd5c" + +[[deps.RecipesBase]] +deps = ["PrecompileTools"] +git-tree-sha1 = "5c3d09cc4f31f5fc6af001c250bf1278733100ff" +uuid = "3cdcf5f2-1ef4-517c-9805-6587b60abb01" +version = "1.3.4" + +[[deps.Reexport]] +git-tree-sha1 = "45e428421666073eab6f2da5c9d310d99bb12f9b" +uuid = "189a3867-3050-52da-a836-e630ba90ab69" +version = "1.2.2" + +[[deps.RelocatableFolders]] +deps = ["SHA", "Scratch"] +git-tree-sha1 = "ffdaf70d81cf6ff22c2b6e733c900c3321cab864" +uuid = "05181044-ff0b-4ac5-8273-598c1e38db00" +version = "1.0.1" + +[[deps.Requires]] +deps = ["UUIDs"] +git-tree-sha1 = "838a3a4188e2ded87a4f9f184b4b0d78a1e91cb7" +uuid = "ae029012-a4dd-5104-9daa-d747884805df" +version = "1.3.0" + +[[deps.SHA]] +uuid = "ea8e919c-243c-51af-8825-aaa63cd721ce" +version = "0.7.0" + +[[deps.Scratch]] +deps = ["Dates"] +git-tree-sha1 = "3bac05bc7e74a75fd9cba4295cde4045d9fe2386" +uuid = "6c6a2e73-6563-6170-7368-637461726353" +version = "1.2.1" + +[[deps.Serialization]] +uuid = "9e88b42a-f829-5b0c-bbe9-9e923198166b" + +[[deps.Setfield]] +deps = ["ConstructionBase", "Future", "MacroTools", "StaticArraysCore"] +git-tree-sha1 = "e2cc6d8c88613c05e1defb55170bf5ff211fbeac" +uuid = "efcf1570-3423-57d1-acb7-fd33fddbac46" +version = "1.1.1" + +[[deps.SimpleUnPack]] +git-tree-sha1 = "58e6353e72cde29b90a69527e56df1b5c3d8c437" +uuid = "ce78b400-467f-4804-87d8-8f486da07d0a" +version = "1.1.0" + +[[deps.Sockets]] +uuid = "6462fe0b-24de-5631-8697-dd941f90decc" + +[[deps.SparseArrays]] +deps = ["Libdl", "LinearAlgebra", "Random", "Serialization", "SuiteSparse_jll"] +uuid = "2f01184e-e22b-5df5-ae63-d93ebab69eaf" + +[[deps.SparseMatricesCSR]] +deps = ["LinearAlgebra", "SparseArrays", "SuiteSparse"] +git-tree-sha1 = "38677ca58e80b5cad2382e5a1848f93b054ad28d" +uuid = "a0a7dd2c-ebf4-11e9-1f05-cf50bc540ca1" +version = "0.6.7" + +[[deps.SpecialFunctions]] +deps = ["IrrationalConstants", "LogExpFunctions", "OpenLibm_jll", "OpenSpecFun_jll"] +git-tree-sha1 = "2f5d4697f21388cbe1ff299430dd169ef97d7e14" +uuid = "276daf66-3868-5448-9aa4-cd146d93841b" +version = "2.4.0" + + [deps.SpecialFunctions.extensions] + SpecialFunctionsChainRulesCoreExt = "ChainRulesCore" + + [deps.SpecialFunctions.weakdeps] + ChainRulesCore = "d360d2e6-b24c-11e9-a2a3-2a2ae2dbcce4" + +[[deps.StaticArrays]] +deps = ["LinearAlgebra", "PrecompileTools", "Random", "StaticArraysCore"] +git-tree-sha1 = "eeafab08ae20c62c44c8399ccb9354a04b80db50" +uuid = "90137ffa-7385-5640-81b9-e52037218182" +version = "1.9.7" + + [deps.StaticArrays.extensions] + StaticArraysChainRulesCoreExt = "ChainRulesCore" + StaticArraysStatisticsExt = "Statistics" + + [deps.StaticArrays.weakdeps] + ChainRulesCore = "d360d2e6-b24c-11e9-a2a3-2a2ae2dbcce4" + Statistics = "10745b16-79ce-11e8-11f9-7d13ad32a3b2" + +[[deps.StaticArraysCore]] +git-tree-sha1 = "192954ef1208c7019899fbf8049e717f92959682" +uuid = "1e83bf80-4336-4d27-bf5d-d5a4f845583c" +version = "1.4.3" + +[[deps.Statistics]] +deps = ["LinearAlgebra", "SparseArrays"] +uuid = "10745b16-79ce-11e8-11f9-7d13ad32a3b2" +version = "1.9.0" + +[[deps.StatsAPI]] +deps = ["LinearAlgebra"] +git-tree-sha1 = "1ff449ad350c9c4cbc756624d6f8a8c3ef56d3ed" +uuid = "82ae8749-77ed-4fe6-ae5f-f523153014b0" +version = "1.7.0" + +[[deps.SuiteSparse]] +deps = ["Libdl", "LinearAlgebra", "Serialization", "SparseArrays"] +uuid = "4607b0f0-06f3-5cda-b6b1-a6196a1729e9" + +[[deps.SuiteSparse_jll]] +deps = ["Artifacts", "Libdl", "Pkg", "libblastrampoline_jll"] +uuid = "bea87d4a-7f5b-5778-9afe-8cc45184846c" +version = "5.10.1+6" + +[[deps.TOML]] +deps = ["Dates"] +uuid = "fa267f1f-6049-4f14-aa54-33bafae1ed76" +version = "1.0.3" + +[[deps.TableTraits]] +deps = ["IteratorInterfaceExtensions"] +git-tree-sha1 = "c06b2f539df1c6efa794486abfb6ed2022561a39" +uuid = "3783bdb8-4a98-5b6b-af9a-565f29a5fe9c" +version = "1.0.1" + +[[deps.Tables]] +deps = ["DataAPI", "DataValueInterfaces", "IteratorInterfaceExtensions", "OrderedCollections", "TableTraits"] +git-tree-sha1 = "598cd7c1f68d1e205689b1c2fe65a9f85846f297" +uuid = "bd369af6-aec1-5ad0-b16a-f7cc5008161c" +version = "1.12.0" + +[[deps.Tar]] +deps = ["ArgTools", "SHA"] +uuid = "a4e569a6-e804-4fa4-b0f3-eef7a1d5b13e" +version = "1.10.0" + +[[deps.Test]] +deps = ["InteractiveUtils", "Logging", "Random", "Serialization"] +uuid = "8dfed614-e22c-5e08-85e1-65c5234f0b40" + +[[deps.TextWrap]] +git-tree-sha1 = "43044b737fa70bc12f6105061d3da38f881a3e3c" +uuid = "b718987f-49a8-5099-9789-dcd902bef87d" +version = "1.0.2" + +[[deps.TranscodingStreams]] +git-tree-sha1 = "e84b3a11b9bece70d14cce63406bbc79ed3464d2" +uuid = "3bb67fe8-82b1-5028-8e26-92a6c54297fa" +version = "0.11.2" + +[[deps.UUIDs]] +deps = ["Random", "SHA"] +uuid = "cf7118a7-6976-5b1a-9a39-7adc72f591a4" + +[[deps.UnPack]] +git-tree-sha1 = "387c1f73762231e86e0c9c5443ce3b4a0a9a0c2b" +uuid = "3a884ed6-31ef-47d7-9d2a-63182c4928ed" +version = "1.0.2" + +[[deps.Unicode]] +uuid = "4ec0a83e-493e-50e2-b9ac-8f72acf5a8f5" + +[[deps.VTKBase]] +git-tree-sha1 = "c2d0db3ef09f1942d08ea455a9e252594be5f3b6" +uuid = "4004b06d-e244-455f-a6ce-a5f9919cc534" +version = "1.0.1" + +[[deps.WriteVTK]] +deps = ["Base64", "CodecZlib", "FillArrays", "LightXML", "TranscodingStreams", "VTKBase"] +git-tree-sha1 = "46664bb833f24e4fe561192e3753c9168c3b71b2" +uuid = "64499a7a-5c06-52f2-abe2-ccb03c286192" +version = "1.19.2" + +[[deps.XML2_jll]] +deps = ["Artifacts", "JLLWrappers", "Libdl", "Libiconv_jll", "Zlib_jll"] +git-tree-sha1 = "d9717ce3518dc68a99e6b96300813760d887a01d" +uuid = "02c8fc9c-b97f-50b9-bbe4-9be30ff0a78a" +version = "2.13.1+0" + +[[deps.Zlib_jll]] +deps = ["Libdl"] +uuid = "83775a58-1f1d-513f-b197-d71354ab007a" +version = "1.2.13+0" + +[[deps.libblastrampoline_jll]] +deps = ["Artifacts", "Libdl"] +uuid = "8e850b90-86db-534c-a0d3-1478176c7d93" +version = "5.8.0+0" + +[[deps.nghttp2_jll]] +deps = ["Artifacts", "Libdl"] +uuid = "8e850ede-7688-5339-a07c-302acd2aaf8d" +version = "1.52.0+1" + +[[deps.oneTBB_jll]] +deps = ["Artifacts", "JLLWrappers", "Libdl"] +git-tree-sha1 = "7d0ea0f4895ef2f5cb83645fa689e52cb55cf493" +uuid = "1317d2d5-d96f-522e-a858-c73665f53c3e" +version = "2021.12.0+0" + +[[deps.p7zip_jll]] +deps = ["Artifacts", "Libdl"] +uuid = "3f19e933-33d8-53b3-aaab-bd5110c3b7a0" +version = "17.4.0+0" From 1c7986373ca9e7f54fe0f38e580a592614cbe575 Mon Sep 17 00:00:00 2001 From: JordiManyer Date: Wed, 21 Aug 2024 11:51:30 +1000 Subject: [PATCH 25/30] Added github action to automatically compile paper --- .github/workflows/joss.yml | 24 ++++++++++++++++++++++++ 1 file changed, 24 insertions(+) create mode 100644 .github/workflows/joss.yml diff --git a/.github/workflows/joss.yml b/.github/workflows/joss.yml new file mode 100644 index 00000000..3938d31f --- /dev/null +++ b/.github/workflows/joss.yml @@ -0,0 +1,24 @@ +name: Joss Paper +on: [push] + +jobs: + paper: + runs-on: ubuntu-latest + name: Paper Draft Compilation + steps: + - name: Checkout + uses: actions/checkout@v4 + - name: Build draft PDF + uses: openjournals/openjournals-draft-action@master + with: + journal: joss + # This should be the path to the paper within your repo. + paper-path: joss_paper/paper.md + - name: Upload + uses: actions/upload-artifact@v4 + with: + name: paper + # This is the output path where Pandoc will write the compiled + # PDF. Note, this should be the same directory as the input + # paper.md + path: joss_paper/paper.pdf From afe9e61125fe6cf99870dcac69fb804f0d1c6fb4 Mon Sep 17 00:00:00 2001 From: JordiManyer Date: Thu, 22 Aug 2024 09:12:38 +1000 Subject: [PATCH 26/30] First round of reviews --- joss_paper/demo.pvtu | 20 -------- joss_paper/paper.bib | 65 ++++++++++++++------------ joss_paper/paper.md | 109 ++----------------------------------------- 3 files changed, 40 insertions(+), 154 deletions(-) delete mode 100755 joss_paper/demo.pvtu diff --git a/joss_paper/demo.pvtu b/joss_paper/demo.pvtu deleted file mode 100755 index 6c0ea990..00000000 --- a/joss_paper/demo.pvtu +++ /dev/null @@ -1,20 +0,0 @@ - - - - - - - - - - - - - - - - - - - - diff --git a/joss_paper/paper.bib b/joss_paper/paper.bib index 2baa138c..0255a6ba 100644 --- a/joss_paper/paper.bib +++ b/joss_paper/paper.bib @@ -13,11 +13,11 @@ @techreport{petsc-user-ref } @manual{mpi40, - author = "{Message Passing Interface Forum}", - title = "{MPI}: A Message-Passing Interface Standard Version 4.0", - url = "https://www.mpi-forum.org/docs/mpi-4.0/mpi40-report.pdf", - year = 2021, - month = jun + author = "{Message Passing Interface Forum}", + title = "{MPI}: A Message-Passing Interface Standard Version 4.0", + url = "https://www.mpi-forum.org/docs/mpi-4.0/mpi40-report.pdf", + year = 2021, + month = jun } @article{Verdugo2021, @@ -61,7 +61,7 @@ @misc{parrays } @article{Bezanson2017, -abstract = {Bridging cultures that have often been distant, Julia combines expertise from the diverse fields of computer science and computational science to create a new approach to numerical computing. Julia...}, +abstract = {Bridging cultures that have often been distant, Julia combines expertise from the diverse fields of computer science and computational science to create a new approach to numerical computing. {J}ulia...}, archivePrefix = {arXiv}, arxivId = {1411.1607}, author = {Bezanson, Jeff and Edelman, Alan and Karpinski, Stefan and Shah, Viral B.}, @@ -74,7 +74,7 @@ @article{Bezanson2017 number = {1}, pages = {65--98}, publisher = {Society for Industrial and Applied Mathematics}, -title = {{Julia: a fresh approach to numerical computing}}, +title = {{J}ulia: a fresh approach to numerical computing}, volume = {59}, year = {2017} } @@ -89,7 +89,7 @@ @article{Badia2020 number = {52}, pages = {2520}, publisher = {The Open Journal}, -title = {{Gridap: an extensible finite element toolbox in Julia}}, +title = {{G}ridap: an extensible finite element toolbox in {J}ulia}, url = {https://joss.theoj.org/papers/10.21105/joss.02520}, volume = {5}, year = {2020} @@ -108,7 +108,7 @@ @article{Badia2020a number = {6}, pages = {C436--C468}, publisher = {Society for Industrial and Applied Mathematics}, -title = {{A generic finite element framework on parallel tree-based adaptive meshes}}, +title = {A generic finite element framework on parallel tree-based adaptive meshes}, volume = {42}, year = {2020} } @@ -124,7 +124,7 @@ @article{p4est number = {3}, pages = {1103--1133}, publisher = {Society for Industrial and Applied Mathematics}, -title = {{p4est: scalable algorithms for parallel adaptive mesh refinement on forests of octrees}}, +title = {{p4est}: scalable algorithms for parallel adaptive mesh refinement on forests of octrees}, volume = {33}, year = {2011} } @@ -153,7 +153,7 @@ @article {freefem @Article{libMeshPaper, doi = {10.1007/s00366-006-0049-3}, author = {B.~S.~Kirk and J.~W.~Peterson and R.~H.~Stogner and G.~F.~Carey}, - title = {{\texttt{libMesh}: A C++ library for parallel adaptive mesh refinement/coarsening simulations}}, + title = {{\texttt{libMesh}}: A {C}++ library for parallel adaptive mesh refinement/coarsening simulations}, journal = {Engineering with Computers}, volume = 22, number = {3--4}, @@ -192,7 +192,7 @@ @article{Kirby2006 keywords = {Automation,Compiler,Finite element,Variational form}, number = {3}, pages = {417--444}, -title = {{A compiler for variational forms}}, +title = {A compiler for variational forms}, volume = {32}, year = {2006} } @@ -258,24 +258,26 @@ @Inbook{PARDISO } @article{MUMPS1, - title = {A Fully Asynchronous Multifrontal Solver Using Distributed Dynamic Scheduling}, - author = {P.R. Amestoy and I. S. Duff and J. Koster and J.-Y. L'Excellent}, - journal = {SIAM Journal on Matrix Analysis and Applications}, - volume = {23}, - number = {1}, - year = {2001}, - pages = {15-41} - } + title = {A Fully Asynchronous Multifrontal Solver Using Distributed Dynamic Scheduling}, + author = {P.R. Amestoy and I. S. Duff and J. Koster and J.-Y. L'Excellent}, + journal = {SIAM Journal on Matrix Analysis and Applications}, + volume = {23}, + number = {1}, + year = {2001}, + pages = {15-41}, + doi = {10.1137/s0895479899358194} +} @article{MUMPS2, - title = {{Performance and Scalability of the Block Low-Rank Multifrontal - Factorization on Multicore Architectures}}, + title = {Performance and Scalability of the Block Low-Rank Multifrontal + Factorization on Multicore Architectures}, author = {P.R. Amestoy and A. Buttari and J.-Y. L'Excellent and T. Mary}, journal = {ACM Transactions on Mathematical Software}, volume = 45, issue = 1, pages = {2:1--2:26}, - year={2019}, + year = {2019}, + doi = {10.1145/3242094} } @article{gridapdistributed, @@ -287,7 +289,7 @@ @article{gridapdistributed number = {74}, pages = {4157}, author = {Santiago Badia and Alberto F. Martín and Francesc Verdugo}, - title = {GridapDistributed: a massively parallel finite element toolbox in Julia}, + title = {GridapDistributed: a massively parallel finite element toolbox in {J}ulia}, journal = {Journal of Open Source Software} } @@ -295,21 +297,23 @@ @inproceedings{Arnold1 title={Preconditing in H(div) and applications}, author={D. N. Arnold and Richard S. Falk and Ragnar Winther}, year={1997}, - url={https://api.semanticscholar.org/CorpusID:12559456} + url={https://api.semanticscholar.org/CorpusID:12559456}, + doi={10.1090/S0025-5718-97-00826-0} } @article{Arnold2, - title={Multigrid in H (div) and H (curl)}, + title={Multigrid in H(div) and H(curl)}, author={D. N. Arnold and Richard S. Falk and Ragnar Winther}, journal={Numerische Mathematik}, year={2000}, volume={85}, pages={197-217}, - url={https://api.semanticscholar.org/CorpusID:14301688} + url={https://api.semanticscholar.org/CorpusID:14301688}, + doi={10.1007/PL00005386} } @article{fenics-patch, - title={PCPATCH: Software for the Topological Construction of Multigrid Relaxation Methods}, + title={{PCPATCH}: Software for the Topological Construction of Multigrid Relaxation Methods}, volume={47}, ISSN={1557-7295}, url={http://dx.doi.org/10.1145/3445791}, @@ -323,11 +327,12 @@ @article{fenics-patch } @misc{dealII-patch, - title={An implementation of tensor product patch smoothers on GPU}, + title={An implementation of tensor product patch smoothers on {GPU}}, author={Cu Cui and Paul Grosse-Bley and Guido Kanschat and Robert Strzodka}, year={2024}, eprint={2405.19004}, archivePrefix={arXiv}, primaryClass={math.NA}, - url={https://arxiv.org/abs/2405.19004}, + url={https://arxiv.org/abs/2405.19004}, + doi={10.48550/arXiv.2405.19004} } \ No newline at end of file diff --git a/joss_paper/paper.md b/joss_paper/paper.md index d3be2731..16d8ff4b 100644 --- a/joss_paper/paper.md +++ b/joss_paper/paper.md @@ -35,13 +35,13 @@ aas-journal: Journal of Open Source Software The ever-increasing demand for resolution and accuracy in mathematical models of physical processes governed by systems of Partial Differential Equations (PDEs) can only be addressed using fully-parallel advanced numerical discretization methods and scalable solvers, thus able to exploit the vast amount of computational resources in state-of-the-art supercomputers. One of the biggest scalability bottlenecks within Finite Element (FE) parallel codes is the solution of linear systems arising from the discretization of PDEs. -The implementation of exact factorization-based solvers in parallel environments is an extremely challenging task, and even state-of-the-art libraries such as MUMPS [@MUMPS1] [@MUMPS2] or PARDISO [@PARDISO] have severe limitations in terms of scalability and memory consumption above a certain number of CPU cores. +The implementation of exact factorization-based solvers in parallel environments is an extremely challenging task, and even state-of-the-art libraries such as MUMPS [@MUMPS1; @MUMPS2] or PARDISO [@PARDISO] have severe limitations in terms of scalability and memory consumption above a certain number of CPU cores. Hence the use of iterative methods is crucial to maintain scalability of FE codes. Unfortunately, the convergence of iterative methods is not guaranteed and rapidly deteriorates as the size of the linear system increases. To retain performance, the use of highly scalable preconditioners is mandatory. For simple problems, algebraic solvers and preconditioners (i.e based uniquelly on the algebraic system) are enough to obtain robust convergence. Many well-known libraries providing algebraic solvers already exist, such as PETSc [@petsc-user-ref], Trilinos [@trilinos], or Hypre [@hypre]. However, algebraic solvers are not always suited to deal with more challenging problems. In these cases, solvers that exploit the physics and mathematical discretization of the particular problem are required. This is the case of many multiphysics problems involving differential operators with a large kernel such as the divergence [@Arnold1] and the curl [@Arnold2]. Examples can be found amongst highly relevant problems such as Navier-Stokes, Maxwell or Darcy. Scalable solvers for this type of multiphysics problems rely on exploiting the block structure of such systems to find a spectrally equivalent block-preconditioner, and are often tied to a specific discretization of the underlying equations. -As a consequence, high-quality open-source parallel finite element packages like FEniCS [@fenics-book] or deal.II [@dealII93] already provide implementations of several state-of-the-art physics-informed solvers [@fenics-patch] [@dealII-patch]. The Gridap ecosystem [@Badia2020] aims to provide a similar level of functionality within the Julia programming language [@Bezanson2017]. +As a consequence, high-quality open-source parallel finite element packages like FEniCS [@fenics-book] or deal.II [@dealII93] already provide implementations of several state-of-the-art physics-informed solvers [@fenics-patch; @dealII-patch]. The Gridap ecosystem [@Badia2020] aims to provide a similar level of functionality within the Julia programming language [@Bezanson2017]. To this end, GridapSolvers is a registered Julia software package which provides highly scalable physics-informed solvers tailored for the FE numerical solution of PDEs on parallel computers within the Gridap ecosystem of packages. Emphasis is put on the modular design of the library, which easily allows new preconditioners to be designed from the user's specific problem. @@ -89,116 +89,17 @@ $$ with $A$ the velocity laplacian block, and $M$ a pressure mass matrix. The mass matrix is approximated by a Conjugate Gradient (CG) solver with Jacobi preconditioner. The eliminated velocity block is approximated by a 2-level V-cycle GMG solver, where the coarsest level is solved exactly in a single processor. -The code is setup to run in parallel with 4 MPI tasks and can be executed with the following command: `mpiexec -n 4 julia --project=. demo.jl`. - -```julia -using Gridap, Gridap.Algebra, Gridap.MultiField -using PartitionedArrays, GridapDistributed, GridapSolvers - -using GridapSolvers.LinearSolvers -using GridapSolvers.MultilevelTools -using GridapSolvers.BlockSolvers - -function get_bilinear_form(mh_lev,biform,qdegree) - model = get_model(mh_lev) - Ω = Triangulation(model) - dΩ = Measure(Ω,qdegree) - return (u,v) -> biform(u,v,dΩ) -end - -function add_labels!(labels) - add_tag_from_tags!(labels,"top",[6]) - add_tag_from_tags!(labels,"walls",[1,2,3,4,5,7,8]) -end - -with_mpi() do distribute - np_per_level = [(2,2),(1,1)] # Number of processors per GMG level - parts = distribute(LinearIndices((prod(np_per_level[1]),))) - - # Create multi-level mesh - domain = (0,1,0,1) # Cartesian domain (xmin,xmax,ymin,ymax) - ncells = (10,10) # Number of cells - mh = CartesianModelHierarchy(parts,np_per_level,domain,ncells;add_labels!) - model = get_model(mh,1) # Finest mesh - - # Create FESpaces - fe_order = 2 - reffe_u = ReferenceFE(lagrangian,VectorValue{2,Float64},fe_order) - reffe_p = ReferenceFE(lagrangian,Float64,fe_order-1;space=:P) - - tests_u = TestFESpace(mh,reffe_u,dirichlet_tags=["walls","top"]) - trials_u = TrialFESpace(tests_u,[VectorValue(0.0,0.0),VectorValue(1.0,0.0)]) - U, V = get_fe_space(trials_u,1), get_fe_space(tests_u,1) - Q = TestFESpace(model,reffe_p;conformity=:L2,constraint=:zeromean) - - mfs = BlockMultiFieldStyle() # Activates block-wise assembly - X = MultiFieldFESpace([U,Q];style=mfs) - Y = MultiFieldFESpace([V,Q];style=mfs) - - # Weak formulation - f = VectorValue(1.0,1.0) - biform_u(u,v,dΩ) = ∫(∇(v)⊙∇(u))dΩ - biform((u,p),(v,q),dΩ) = biform_u(u,v,dΩ) - ∫((∇⋅v)*p)dΩ - ∫((∇⋅u)*q)dΩ - liform((v,q),dΩ) = ∫(v⋅f)dΩ - - # Assemble linear system - qdegree = 2*(fe_order+1) # Quadrature degree - Ω = Triangulation(model) - dΩ = Measure(Ω,qdegree) - op = AffineFEOperator((u,v)->biform(u,v,dΩ),v->liform(v,dΩ),X,Y) - A, b = get_matrix(op), get_vector(op) - - # GMG preconditioner for the velocity block - biforms = map(mh) do mhl - get_bilinear_form(mhl,biform_u,qdegree) - end - smoothers = map(view(mh,1:num_levels(mh)-1)) do mhl - RichardsonSmoother(JacobiLinearSolver(),10,2.0/3.0) - end - restrictions, prolongations = setup_transfer_operators( - trials_u, qdegree; mode=:residual, solver=CGSolver(JacobiLinearSolver()) - ) - solver_u = GMGLinearSolver( - mh,trials_u,tests_u,biforms, - prolongations,restrictions, - pre_smoothers=smoothers, - post_smoothers=smoothers, - coarsest_solver=LUSolver(), - maxiter=4,mode=:solver - ) - - # PCG solver for the pressure block - solver_p = CGSolver(JacobiLinearSolver();maxiter=20,atol=1e-14,rtol=1.e-6) - - # Block triangular preconditioner - blocks = [LinearSystemBlock() LinearSystemBlock(); - LinearSystemBlock() BiformBlock((p,q) -> ∫(p*q)dΩ,Q,Q)] - P = BlockTriangularSolver(blocks,[solver_u,solver_p]) - - # Global solver - solver = FGMRESSolver(10,P;rtol=1.e-8,verbose=i_am_main(parts)) - ns = numerical_setup(symbolic_setup(solver,A),A) - - # Solve - x = allocate_in_domain(A) - fill!(x,0.0) - solve!(x,ns,b) - - # Postprocess - uh, ph = FEFunction(X,x) - writevtk(Ω,"demo",cellfields=["uh"=>uh,"ph"=>ph]) -end -``` +The code for this example can be found [here](https://github.com/gridap/GridapSolvers.jl/tree/joss-paper/joss_paper/demo.jl). It is setup to run in parallel with 4 MPI tasks and can be executed with the following command: `mpiexec -n 4 julia --project=. demo.jl`. ## Parallel scaling benchmark The following section shows scalability results for the demo problem discussed above. We run our code on the Gadi supercomputer, which is part of the Australian National Computational Infrastructure (NCI). We use Intel's Cascade Lake 2x24-core Xeon Platinum 8274 nodes. Scalability is shown for up to 64 nodes, for a fixed local problem size of 48x64 quadrangle cells per processor. This amounts to a maximum size of approximately 37M cells and 415M degrees of freedom distributed amongst 3072 processors. Within the GMG solver, the number of coarsening levels is progressively increased to keep the global size of the coarsest solve (approximately) constant. The coarsest solve is then performed by a CG solver preconditioned by an Algebraic MultiGrid (AMG) solver, provided by PETSc [@petsc-user-ref] through the package GridapPETSc.jl. -The results show that the code scales relatively well up to 3072 processors, with loss in performance mostly tied to the number of GMG levels used for the velocity solver. The number of F-GMRES iterations required for convergence is also shown to be relatively constant (and even decreasing for bigger problem sizes), indicating that the preconditioner is robust with respect to the problem size. +The results in \autoref{fig:scalability} show that the code scales relatively well up to 3072 processors, with loss in performance mostly tied to the number of GMG levels used for the velocity solver. The number of F-GMRES iterations required for convergence is also shown to be relatively constant (and even decreasing for bigger problem sizes), indicating that the preconditioner is robust with respect to the problem size. The code used to create these results can be found [here](https://github.com/gridap/GridapSolvers.jl/tree/joss-paper/joss_paper/scalability). The exact releases for the packages used are provided by Julia's `Manifest.toml` file. -![**Top**: Weak scalability for a Stokes problem in 2D. Time is given per F-GMRES iteration, as a function of the number of processors. **Middle**: Number of coarsening levels for the GMG solver, as a function of the number of processors. **Bottom**: Number of F-GMRES iterations required for convergence. \label{fig:packages}](weakScalability.png){ width=80% } +![**Top**: Weak scalability for a Stokes problem in 2D. Time is given per F-GMRES iteration, as a function of the number of processors. **Middle**: Number of coarsening levels for the GMG solver, as a function of the number of processors. **Bottom**: Number of F-GMRES iterations required for convergence. \label{fig:scalability}](weakScalability.png){ width=80% } ## Acknowledgements From af93e9c5260bbaabc84f324f58984c7fd5c9165a Mon Sep 17 00:00:00 2001 From: JordiManyer Date: Thu, 22 Aug 2024 09:33:30 +1000 Subject: [PATCH 27/30] Minor --- joss_paper/paper.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/joss_paper/paper.md b/joss_paper/paper.md index 16d8ff4b..c41fcd89 100644 --- a/joss_paper/paper.md +++ b/joss_paper/paper.md @@ -50,13 +50,13 @@ To this end, GridapSolvers is a registered Julia software package which provides \autoref{fig:packages} depicts the relation among GridapDistributed and other packages in the Julia package ecosystem. The core library Gridap [@Badia2020] provides all necessary abstraction and interfaces needed for the FE solution of PDEs [@Verdugo2021] for serial computing. GridapDistributed [@gridapdistributed] provides distributed-memory counterparts for these abstractions, while leveraging the serial implementations in Gridap to handle the local portion on each parallel task. GridapDistributed relies on PartitionedArrays [@parrays] in order to handle the parallel execution model (e.g., message-passing via the Message Passing Interface (MPI) [@mpi40]), global data distribution layout, and communication among tasks. PartitionedArrays also provides a parallel implementation of partitioned global linear systems (i.e., linear algebra vectors and sparse matrices) as needed in grid-based numerical simulations. -This parallel framework does however not include any performant solver for the resulting linear systems. This was delegated to GridapPETSc, which provides a plethora of highly-scalable and efficient algebraic solvers through a high-level interface to the Portable, Extensible Toolkit for Scientific Computation (PETSc) [@petsc-user-ref]. +This parallel framework does however not include any performant solver for the resulting linear systems. This was delegated to GridapPETSc [@gridapetsc], which provides a plethora of highly-scalable and efficient algebraic solvers through a high-level interface to the Portable, Extensible Toolkit for Scientific Computation (PETSc) [@petsc-user-ref]. GridapSolvers complements GridapPETSc with a modular and extensible interface for the design of physics-informed solvers. Some of the highlights of the library are: - A set of HPC-first implementations for popular Krylov-based iterative solvers. These solvers extend Gridap's API and are fully compatible with PartitionedArrays. - A modular, high-level interface for designing block-based preconditioners for multiphysics problems. These preconditioners can be used together with any solver compliant with Gridap's API, including those provided by GridapPETSc. -- A generic interface to handle multi-level distributed meshes, with full support for Adaptative Mesh Refinement (AMR) using p4est [@p4est] through GridapP4est. +- A generic interface to handle multi-level distributed meshes, with full support for Adaptative Mesh Refinement (AMR) using p4est [@p4est] through GridapP4est [@gridap4est]. - A modular implementation of Geometric MultiGrid (GMG) solvers [@gmg-book], allowing different types of smoothers and restriction/prolongation operators. - A generic interface for patch-based subdomain decomposition methods, and an implementation of patch-based smoothers for GMG solvers. From 7e026de74324a556fdf7053f21f583aa319b1fdb Mon Sep 17 00:00:00 2001 From: JordiManyer Date: Wed, 25 Sep 2024 12:33:35 +1000 Subject: [PATCH 28/30] Addressing first round of comments --- joss_paper/paper.bib | 8 ++++---- joss_paper/paper.md | 10 +++++----- 2 files changed, 9 insertions(+), 9 deletions(-) diff --git a/joss_paper/paper.bib b/joss_paper/paper.bib index 0255a6ba..dd3670d3 100644 --- a/joss_paper/paper.bib +++ b/joss_paper/paper.bib @@ -294,16 +294,16 @@ @article{gridapdistributed } @inproceedings{Arnold1, - title={Preconditing in H(div) and applications}, - author={D. N. Arnold and Richard S. Falk and Ragnar Winther}, + title={Preconditing in {H}(div) and applications}, + author={D. N. Arnold and R. S. Falk and R. Winther}, year={1997}, url={https://api.semanticscholar.org/CorpusID:12559456}, doi={10.1090/S0025-5718-97-00826-0} } @article{Arnold2, - title={Multigrid in H(div) and H(curl)}, - author={D. N. Arnold and Richard S. Falk and Ragnar Winther}, + title={Multigrid in {H}(div) and {H}(curl)}, + author={D. N. Arnold and R. S. Falk and R. Winther}, journal={Numerische Mathematik}, year={2000}, volume={85}, diff --git a/joss_paper/paper.md b/joss_paper/paper.md index c41fcd89..3803fa11 100644 --- a/joss_paper/paper.md +++ b/joss_paper/paper.md @@ -37,7 +37,7 @@ The ever-increasing demand for resolution and accuracy in mathematical models of One of the biggest scalability bottlenecks within Finite Element (FE) parallel codes is the solution of linear systems arising from the discretization of PDEs. The implementation of exact factorization-based solvers in parallel environments is an extremely challenging task, and even state-of-the-art libraries such as MUMPS [@MUMPS1; @MUMPS2] or PARDISO [@PARDISO] have severe limitations in terms of scalability and memory consumption above a certain number of CPU cores. Hence the use of iterative methods is crucial to maintain scalability of FE codes. Unfortunately, the convergence of iterative methods is not guaranteed and rapidly deteriorates as the size of the linear system increases. To retain performance, the use of highly scalable preconditioners is mandatory. -For simple problems, algebraic solvers and preconditioners (i.e based uniquelly on the algebraic system) are enough to obtain robust convergence. Many well-known libraries providing algebraic solvers already exist, such as PETSc [@petsc-user-ref], Trilinos [@trilinos], or Hypre [@hypre]. However, algebraic solvers are not always suited to deal with more challenging problems. +For simple problems, algebraic solvers and preconditioners (i.e based solely on the algebraic system) are enough to obtain robust convergence. Many well-known libraries providing algebraic solvers already exist, such as PETSc [@petsc-user-ref], Trilinos [@trilinos], or Hypre [@hypre]. However, algebraic solvers are not always suited to deal with more challenging problems. In these cases, solvers that exploit the physics and mathematical discretization of the particular problem are required. This is the case of many multiphysics problems involving differential operators with a large kernel such as the divergence [@Arnold1] and the curl [@Arnold2]. Examples can be found amongst highly relevant problems such as Navier-Stokes, Maxwell or Darcy. Scalable solvers for this type of multiphysics problems rely on exploiting the block structure of such systems to find a spectrally equivalent block-preconditioner, and are often tied to a specific discretization of the underlying equations. @@ -58,7 +58,7 @@ GridapSolvers complements GridapPETSc with a modular and extensible interface fo - A modular, high-level interface for designing block-based preconditioners for multiphysics problems. These preconditioners can be used together with any solver compliant with Gridap's API, including those provided by GridapPETSc. - A generic interface to handle multi-level distributed meshes, with full support for Adaptative Mesh Refinement (AMR) using p4est [@p4est] through GridapP4est [@gridap4est]. - A modular implementation of Geometric MultiGrid (GMG) solvers [@gmg-book], allowing different types of smoothers and restriction/prolongation operators. -- A generic interface for patch-based subdomain decomposition methods, and an implementation of patch-based smoothers for GMG solvers. +- A generic interface for patch-based subdomain decomposition methods, and an implementation of patch-based smoothers for GMG solvers. Here the term "patch-based" refers to the use of local overlapping subdomains (patches) built by aggregation of cells around a given vertex, face or cell. See [@fenics-patch; @dealII-patch] for more details. ![GridapSolvers and its relation to other packages in the Julia package ecosystem. In this diagram, each node represents a Julia package, while the (directed) arrows represent relations (dependencies) among packages. Dashed arrows mean the package can be used, but is not required. \label{fig:packages}](packages.png){ width=50% } @@ -72,7 +72,7 @@ $$ where $V_0$ is the space of velocity functions with homogeneous boundary conditions everywhere. -The system is block-assembled and solved using a flexible Generalised Minimum Residual (F-GMRES) solver, together with a block-triangular Shur-complement-based preconditioner. We eliminate the velocity block and approximate the resulting Shur complement by a pressure mass matrix. A more detailed overview of this preconditioner as well as it's spectral analysis can be found in [@Elman2014]. The resulting block structure for the system matrix $\mathcal{A}$ and our preconditioner $\mathcal{P}$ is +The system is block-assembled and solved using a flexible Generalised Minimum Residual (F-GMRES) solver, together with a block-triangular Schur-complement-based preconditioner. We eliminate the velocity block and approximate the resulting Schur complement by a pressure mass matrix. A more detailed overview of this preconditioner as well as its spectral analysis can be found in [@Elman2014]. The resulting block structure for the system matrix $\mathcal{A}$ and our preconditioner $\mathcal{P}$ is $$ \mathcal{A} = \begin{bmatrix} @@ -88,8 +88,8 @@ $$ with $A$ the velocity laplacian block, and $M$ a pressure mass matrix. -The mass matrix is approximated by a Conjugate Gradient (CG) solver with Jacobi preconditioner. The eliminated velocity block is approximated by a 2-level V-cycle GMG solver, where the coarsest level is solved exactly in a single processor. -The code for this example can be found [here](https://github.com/gridap/GridapSolvers.jl/tree/joss-paper/joss_paper/demo.jl). It is setup to run in parallel with 4 MPI tasks and can be executed with the following command: `mpiexec -n 4 julia --project=. demo.jl`. +Application of the above block-preconditioner requires both diagonal sub-matrices to be solved. The pressure block $M$ is solved using a Conjugate Gradient (CG) solver with Jacobi preconditioner. The velocity block $A$ is solved by a 2-level V-cycle GMG solver, where the coarsest level is solved exactly in a single processor. +The code for this example can be found [here](https://github.com/gridap/GridapSolvers.jl/tree/joss-paper/joss_paper/demo.jl). It is set up to run in parallel with 4 MPI tasks and can be executed with the following command: `mpiexec -n 4 julia --project=. demo.jl`. ## Parallel scaling benchmark From 070bd465867295ad60b8c48cb46c730d33cb6ac2 Mon Sep 17 00:00:00 2001 From: "Daniel S. Katz" Date: Fri, 4 Oct 2024 09:27:38 -0500 Subject: [PATCH 29/30] minor changes in paper.md --- joss_paper/paper.md | 20 ++++++++++---------- 1 file changed, 10 insertions(+), 10 deletions(-) diff --git a/joss_paper/paper.md b/joss_paper/paper.md index 3803fa11..eff2ddb2 100644 --- a/joss_paper/paper.md +++ b/joss_paper/paper.md @@ -32,16 +32,16 @@ aas-journal: Journal of Open Source Software ## Summary and statement of need -The ever-increasing demand for resolution and accuracy in mathematical models of physical processes governed by systems of Partial Differential Equations (PDEs) can only be addressed using fully-parallel advanced numerical discretization methods and scalable solvers, thus able to exploit the vast amount of computational resources in state-of-the-art supercomputers. +The ever-increasing demand for resolution and accuracy in mathematical models of physical processes governed by systems of Partial Differential Equations (PDEs) can only be addressed using fully-parallel advanced numerical discretization methods and scalable solvers, able to exploit the vast amount of computational resources in state-of-the-art supercomputers. One of the biggest scalability bottlenecks within Finite Element (FE) parallel codes is the solution of linear systems arising from the discretization of PDEs. -The implementation of exact factorization-based solvers in parallel environments is an extremely challenging task, and even state-of-the-art libraries such as MUMPS [@MUMPS1; @MUMPS2] or PARDISO [@PARDISO] have severe limitations in terms of scalability and memory consumption above a certain number of CPU cores. -Hence the use of iterative methods is crucial to maintain scalability of FE codes. Unfortunately, the convergence of iterative methods is not guaranteed and rapidly deteriorates as the size of the linear system increases. To retain performance, the use of highly scalable preconditioners is mandatory. -For simple problems, algebraic solvers and preconditioners (i.e based solely on the algebraic system) are enough to obtain robust convergence. Many well-known libraries providing algebraic solvers already exist, such as PETSc [@petsc-user-ref], Trilinos [@trilinos], or Hypre [@hypre]. However, algebraic solvers are not always suited to deal with more challenging problems. +The implementation of exact factorization-based solvers in parallel environments is an extremely challenging task, and even state-of-the-art libraries such as MUMPS [@MUMPS1; @MUMPS2] and PARDISO [@PARDISO] have severe limitations in terms of scalability and memory consumption above a certain number of CPU cores. +Hence, the use of iterative methods is crucial to maintain scalability of FE codes. Unfortunately, the convergence of iterative methods is not guaranteed and rapidly deteriorates as the size of the linear system increases. To retain performance, the use of highly scalable preconditioners is mandatory. +For simple problems, algebraic solvers and preconditioners (i.e., those based solely on the algebraic system) are enough to obtain robust convergence. Many well-known libraries providing algebraic solvers already exist, such as PETSc [@petsc-user-ref], Trilinos [@trilinos], and Hypre [@hypre]. However, algebraic solvers are not always suited to deal with more challenging problems. -In these cases, solvers that exploit the physics and mathematical discretization of the particular problem are required. This is the case of many multiphysics problems involving differential operators with a large kernel such as the divergence [@Arnold1] and the curl [@Arnold2]. Examples can be found amongst highly relevant problems such as Navier-Stokes, Maxwell or Darcy. Scalable solvers for this type of multiphysics problems rely on exploiting the block structure of such systems to find a spectrally equivalent block-preconditioner, and are often tied to a specific discretization of the underlying equations. +In these cases, solvers that exploit the physics and mathematical discretization of the particular problem are required. This is the case of many multiphysics problems involving differential operators with a large kernel such as the divergence [@Arnold1] and the curl [@Arnold2]. Examples can be found amongst highly relevant problems such as Navier-Stokes, Maxwell, and Darcy. Scalable solvers for this type of multiphysics problems rely on exploiting the block structure of such systems to find a spectrally equivalent block-preconditioner, and are often tied to a specific discretization of the underlying equations. -As a consequence, high-quality open-source parallel finite element packages like FEniCS [@fenics-book] or deal.II [@dealII93] already provide implementations of several state-of-the-art physics-informed solvers [@fenics-patch; @dealII-patch]. The Gridap ecosystem [@Badia2020] aims to provide a similar level of functionality within the Julia programming language [@Bezanson2017]. +As a consequence, high-quality open-source parallel finite element packages like FEniCS [@fenics-book] and deal.II [@dealII93] already provide implementations of several state-of-the-art physics-informed solvers [@fenics-patch; @dealII-patch]. The Gridap ecosystem [@Badia2020] aims to provide a similar level of functionality within the Julia programming language [@Bezanson2017]. To this end, GridapSolvers is a registered Julia software package which provides highly scalable physics-informed solvers tailored for the FE numerical solution of PDEs on parallel computers within the Gridap ecosystem of packages. Emphasis is put on the modular design of the library, which easily allows new preconditioners to be designed from the user's specific problem. @@ -58,13 +58,13 @@ GridapSolvers complements GridapPETSc with a modular and extensible interface fo - A modular, high-level interface for designing block-based preconditioners for multiphysics problems. These preconditioners can be used together with any solver compliant with Gridap's API, including those provided by GridapPETSc. - A generic interface to handle multi-level distributed meshes, with full support for Adaptative Mesh Refinement (AMR) using p4est [@p4est] through GridapP4est [@gridap4est]. - A modular implementation of Geometric MultiGrid (GMG) solvers [@gmg-book], allowing different types of smoothers and restriction/prolongation operators. -- A generic interface for patch-based subdomain decomposition methods, and an implementation of patch-based smoothers for GMG solvers. Here the term "patch-based" refers to the use of local overlapping subdomains (patches) built by aggregation of cells around a given vertex, face or cell. See [@fenics-patch; @dealII-patch] for more details. +- A generic interface for patch-based subdomain decomposition methods, and an implementation of patch-based smoothers for GMG solvers. Here the term "patch-based" refers to the use of local overlapping subdomains (patches) built by aggregation of cells around a given vertex, face or cell. See @fenics-patch and @dealII-patch for more details. ![GridapSolvers and its relation to other packages in the Julia package ecosystem. In this diagram, each node represents a Julia package, while the (directed) arrows represent relations (dependencies) among packages. Dashed arrows mean the package can be used, but is not required. \label{fig:packages}](packages.png){ width=50% } ## Demo -The following code snippet shows how to solve a 2D incompressible Stokes cavity problem in a cartesian domain $\Omega = [0,1]^2$. We discretize the velocity and pressure in $H^1(\Omega)$ and $L^2(\Omega)$ respectively, and use the well known stable element pair $Q_k \times P_{k-1}^{-}$ with $k=2$. For the cavity problem, we fix the velocity to $u_t = \hat{x}$ on the top boundary $\Gamma_t = (0,1)\times\{1\}$, and homogeneous Dirichlet boundary conditions elsewhere. We impose a zero-mean pressure constraint to have a solvable system of equations. Given discrete spaces $V \times Q_0$, we find $(u,p) \in V \times Q_0$ such that +The following code snippet shows how to solve a 2D incompressible Stokes cavity problem in a Cartesian domain $\Omega = [0,1]^2$. We discretize the velocity and pressure in $H^1(\Omega)$ and $L^2(\Omega)$ respectively, and use the well known stable element pair $Q_k \times P_{k-1}^{-}$ with $k=2$. For the cavity problem, we fix the velocity to $u_t = \hat{x}$ on the top boundary $\Gamma_t = (0,1)\times\{1\}$, and homogeneous Dirichlet boundary conditions elsewhere. We impose a zero-mean pressure constraint to have a solvable system of equations. Given discrete spaces $V \times Q_0$, we find $(u,p) \in V \times Q_0$ such that $$ \int_{\Omega} \nabla v : \nabla u - (\nabla \cdot v) p - (\nabla \cdot u) q = \int_{\Omega} v \cdot f \quad \forall v \in V_0, q \in Q_0 @@ -72,7 +72,7 @@ $$ where $V_0$ is the space of velocity functions with homogeneous boundary conditions everywhere. -The system is block-assembled and solved using a flexible Generalised Minimum Residual (F-GMRES) solver, together with a block-triangular Schur-complement-based preconditioner. We eliminate the velocity block and approximate the resulting Schur complement by a pressure mass matrix. A more detailed overview of this preconditioner as well as its spectral analysis can be found in [@Elman2014]. The resulting block structure for the system matrix $\mathcal{A}$ and our preconditioner $\mathcal{P}$ is +The system is block-assembled and solved using a flexible Generalised Minimum Residual (F-GMRES) solver, together with a block-triangular Schur-complement-based preconditioner. We eliminate the velocity block and approximate the resulting Schur complement by a pressure mass matrix. A more detailed overview of this preconditioner as well as its spectral analysis can be found in @Elman2014. The resulting block structure for the system matrix $\mathcal{A}$ and our preconditioner $\mathcal{P}$ is $$ \mathcal{A} = \begin{bmatrix} @@ -86,7 +86,7 @@ $$ \end{bmatrix} $$ -with $A$ the velocity laplacian block, and $M$ a pressure mass matrix. +with $A$ the velocity Laplacian block, and $M$ a pressure mass matrix. Application of the above block-preconditioner requires both diagonal sub-matrices to be solved. The pressure block $M$ is solved using a Conjugate Gradient (CG) solver with Jacobi preconditioner. The velocity block $A$ is solved by a 2-level V-cycle GMG solver, where the coarsest level is solved exactly in a single processor. The code for this example can be found [here](https://github.com/gridap/GridapSolvers.jl/tree/joss-paper/joss_paper/demo.jl). It is set up to run in parallel with 4 MPI tasks and can be executed with the following command: `mpiexec -n 4 julia --project=. demo.jl`. From 14a52ff6fec268825657a78b6e36878a2dccc38e Mon Sep 17 00:00:00 2001 From: "Daniel S. Katz" Date: Fri, 4 Oct 2024 09:32:01 -0500 Subject: [PATCH 30/30] minor changes in paper.bib --- joss_paper/paper.bib | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/joss_paper/paper.bib b/joss_paper/paper.bib index dd3670d3..1b18e315 100644 --- a/joss_paper/paper.bib +++ b/joss_paper/paper.bib @@ -208,7 +208,7 @@ @book{fenics-book @book{Elman2014, author = {Elman, Howard and Silvester, David and Wathen, Andy}, - title = "{Finite Elements and Fast Iterative Solvers: with Applications in Incompressible Fluid Dynamics}", + title = {Finite Elements and Fast Iterative Solvers: with Applications in Incompressible Fluid Dynamics}, publisher = {Oxford University Press}, year = {2014}, month = {06}, @@ -231,8 +231,8 @@ @book{gmg-book } @Manual{trilinos, - title = {The {T}rilinos {P}roject {W}ebsite}, - author = {, The {T}rilinos {P}roject {T}eam}, + title = {The {T}rilinos Project Website}, + author = {{Trilinos Project Team}}, year = {2020}, url = {https://trilinos.github.io} } @@ -335,4 +335,4 @@ @misc{dealII-patch primaryClass={math.NA}, url={https://arxiv.org/abs/2405.19004}, doi={10.48550/arXiv.2405.19004} -} \ No newline at end of file +}