Update algorithm for core counts for MPAS partition files #563

xylar · 2023-03-13T23:09:44Z

The old algorithm wasn't generating the breadth of partition sizes that the E3SM team needs. The updated algorithm seems likely to generate nearly all of the partition sizes that could conceivably be useful for a given mesh.

Checklist

Developer's Guide has been updated
Documentation has been built locally and changes look as expected
Document (in a comment titled Testing in this PR) any testing that was used to verify the changes

xylar · 2023-03-13T23:10:47Z

I will test this tomorrow in conjunction with #561.

xylar · 2023-03-14T17:21:43Z

@mark-petersen, would you be up for reviewing this algorithm? I can point you to the partition files it produces:

/lcrc/group/e3sm/ac.xylar/compass_1.2/chrysalis/test_20230314/files_for_e3sm_more_parts/ec30to60e2r2/ocean/global_ocean/files_for_e3sm/assembled_files/inputdata/ocn/mpas-o/EC30to60E2r2/partitions
/lcrc/group/e3sm/ac.xylar/compass_1.2/chrysalis/test_20230314/files_for_e3sm_more_parts/wc14to60e2r3/ocean/global_ocean/files_for_e3sm/assembled_files/inputdata/ocn/mpas-o/WC14to60E2r3/partitions

(The second is still running so files aren't there quite yet).

I want to put files in a subdirectory because there are ~400 of them for a given mesh.

I'll describe the algorithm in words, pointing at the code.

compass/ocean/tests/global_ocean/files_for_e3sm/graph_partition.py

xylar · 2023-03-14T17:27:02Z

compass/ocean/tests/global_ocean/files_for_e3sm/graph_partition.py

+        factors = _prime_factors(candidate)
+        twos = np.count_nonzero(factors == 2)
+        fives = np.count_nonzero(factors == 5)
+        gt_five = np.count_nonzero(factors > 5)
+        big_factor = factors.max()
+        if twos > 0 and fives <= twos and gt_five <= 1 and big_factor <= 7:
+            cores.append(candidate)


The core counts are allowed in based on their prime factors. Most require:

at least one power of 2

powers of 10 but not of 5 (the number of 5 factors has to be less than or equal to the number of 2s)

There can only be one prime factor bigger than 5

That one big prime factor can't be bigger than 7 (I had allowed that to go up to 23 in a previous test, but I think that produces a bunch of useless partitions). I realize the current verison is kind of ridiculous but we might want to back off and allow powers of 11, 13, etc. in the future.

In addition(see below) there is a small list of other allowed sizes:

small multiples of 3

divisors of the ne30 mesh size at @amametjanov's request.

xylar · 2023-03-14T17:38:14Z

compass/ocean/tests/global_ocean/files_for_e3sm/graph_partition.py

@@ -1,7 +1,7 @@
 import numpy as np


-def get_core_list(ncells, max_cells_per_core=6000, min_cells_per_core=100):
+def get_core_list(ncells, max_cells_per_core=30000, min_cells_per_core=2):


It seems like there are situations where folks really want to push the limits of how many or how few cells you can get away with. I'm including some partitions that accommodate these needs. We can back off, but it's at the risk that E3SM developers generate their own partitions (which is a little riskier with the new sea ice approach).

The old algorithm wasn't generating the breadth of parition sizes that the E3SM team needs. The updated algorithm seems likely to generate nearly all of the partition sizes that could conceivable be useful for a given mesh.

It takes too long even on the QU240 mesh to be feasible for PR testing anymore.

Generating this file is too slow for large meshes and large numbers of processor counts.

This merge points to new partition files for each of the following 4 MPAS-Ocean meshes. Each mesh has about 400 files that are expected to support nearly any conceivable core count. See MPAS-Dev/compass#563 for more details on how the core counts were determined.

This merge points to new partition files for each of the following 4 MPAS-Ocean meshes. Each mesh has about 400 files that are expected to support nearly any conceivable core count. Meshes with updated partitions: * EC30to60E2r2 * ECwISC30to60E2r1 * SOwISC12to60E2r4 * WC14to60E2r3 See MPAS-Dev/compass#563 for more details on how the core counts were determined.

This merge points to new partition files for each of the following 4 MPAS-Seaice meshes. Each mesh has about 400 files that are expected to support nearly any conceivable core count. Meshes with updated partitions: * EC30to60E2r2 * ECwISC30to60E2r1 * SOwISC12to60E2r4 * WC14to60E2r3 The partition files are better load-balanced, since each core owns cells in both polar and equatorial regions. They were created with the tools described in: http://mpas-dev.github.io/MPAS-Tools/stable/seaice/partition.html See MPAS-Dev/compass#563 for more details on how the core counts were determined.

xylar · 2023-03-21T13:16:44Z

Testing

I have used this approach to create partition files for 6 commonly used meshes:

EC30to60E2r2
EC30to60E2r3
ECwISC30to60E2r1
SOwISC12to60E2r4
WC14to60E2r3
WC14to60E2r5

See:
E3SM-Ocean-Discussion/E3SM#43
E3SM-Seaice-Discussion/E3SM#10

xylar · 2023-03-21T18:05:42Z

Can't build docs locally until I fix #567

xylar · 2023-03-21T18:05:57Z

@mark-petersen, this is ready for you to review when you can.

mark-petersen

@xylar, if you are happy with the partition files that this PR creates, I'm happy to approve it. The algorithm looks reasonable. The proof is in the list of files you created!

xylar · 2023-03-22T14:31:50Z

Thanks very much @mark-petersen!

This merge points to new partition files for each of the following 4 MPAS-Ocean meshes. Each mesh has about 400 files that are expected to support nearly any conceivable core count. Meshes with updated partitions: * EC30to60E2r2 * ECwISC30to60E2r1 * SOwISC12to60E2r4 * WC14to60E2r3 See MPAS-Dev/compass#563 for more details on how the core counts were determined.

This merge points to new partition files for each of the following 4 MPAS-Seaice meshes. Each mesh has about 400 files that are expected to support nearly any conceivable core count. Meshes with updated partitions: * EC30to60E2r2 * ECwISC30to60E2r1 * SOwISC12to60E2r4 * WC14to60E2r3 The partition files are better load-balanced, since each core owns cells in both polar and equatorial regions. They were created with the tools described in: http://mpas-dev.github.io/MPAS-Tools/stable/seaice/partition.html See MPAS-Dev/compass#563 for more details on how the core counts were determined.

xylar added enhancement New feature or request ocean labels Mar 13, 2023

xylar self-assigned this Mar 13, 2023

xylar mentioned this pull request Mar 13, 2023

Current EC30to60 ocean/ice initial condition should have bumped revision number E3SM-Project/E3SM#5516

Open

xylar force-pushed the update_algorithm_for_mesh_partitioning branch 2 times, most recently from 80e01cb to 3961f78 Compare March 14, 2023 17:17

xylar requested a review from mark-petersen March 14, 2023 17:18

xylar commented Mar 14, 2023

View reviewed changes

compass/ocean/tests/global_ocean/files_for_e3sm/graph_partition.py Outdated Show resolved Hide resolved

xylar commented Mar 14, 2023

View reviewed changes

xylar force-pushed the update_algorithm_for_mesh_partitioning branch from 3961f78 to 5b0519a Compare March 14, 2023 17:28

xylar commented Mar 14, 2023

View reviewed changes

xylar force-pushed the update_algorithm_for_mesh_partitioning branch from 5b0519a to 342c056 Compare March 14, 2023 19:47

xylar added 3 commits March 14, 2023 17:25

Update algo. for MPAS possible MPAS partition cores

b51a032

The old algorithm wasn't generating the breadth of parition sizes that the E3SM team needs. The updated algorithm seems likely to generate nearly all of the partition sizes that could conceivable be useful for a given mesh.

Put partition files in own directory

74effaf

Remove files_for_e3sm from the pr suite

ecae3d9

It takes too long even on the QU240 mesh to be feasible for PR testing anymore.

xylar force-pushed the update_algorithm_for_mesh_partitioning branch from 342c056 to ecae3d9 Compare March 14, 2023 22:26

Turn off sea-ice partition plot file by default

1931c31

Generating this file is too slow for large meshes and large numbers of processor counts.

xylar force-pushed the update_algorithm_for_mesh_partitioning branch from c6ba983 to 1931c31 Compare March 15, 2023 21:36

xylar changed the title ~~Update algo. for MPAS possible MPAS partition cores~~ Update algorithm for MPAS possible MPAS partition cores Mar 20, 2023

xylar changed the title ~~Update algorithm for MPAS possible MPAS partition cores~~ Update algorithm for core counts for MPAS partition files Mar 20, 2023

xylar mentioned this pull request Mar 20, 2023

Update partitions for 4 MPAS-Ocean meshes E3SM-Ocean-Discussion/E3SM#43

Closed

xylar mentioned this pull request Mar 20, 2023

Update partitions for 4 MPAS-Seaice meshes E3SM-Seaice-Discussion/E3SM#10

Closed

xylar mentioned this pull request Mar 21, 2023

Add files_for_e3sm steps for ocean and sea-ice meshes #570

Merged

5 tasks

Update developer's guide

dca42b8

xylar force-pushed the update_algorithm_for_mesh_partitioning branch from 593ad79 to dca42b8 Compare March 22, 2023 09:23

mark-petersen approved these changes Mar 22, 2023

View reviewed changes

xylar merged commit 5070bd3 into MPAS-Dev:main Mar 22, 2023

xylar deleted the update_algorithm_for_mesh_partitioning branch March 22, 2023 14:32

xylar mentioned this pull request Mar 29, 2023

Update partitions across cores for 4 MPAS-Ocean meshes E3SM-Project/E3SM#5568

Merged

xylar mentioned this pull request Apr 20, 2023

Update partitions for 4 MPAS-Seaice meshes E3SM-Project/E3SM#5611

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Update algorithm for core counts for MPAS partition files #563

Update algorithm for core counts for MPAS partition files #563

xylar commented Mar 13, 2023 •

edited

Loading

xylar commented Mar 13, 2023

xylar commented Mar 14, 2023

xylar Mar 14, 2023

xylar Mar 14, 2023

xylar commented Mar 21, 2023

xylar commented Mar 21, 2023

xylar commented Mar 21, 2023

mark-petersen left a comment

xylar commented Mar 22, 2023

Update algorithm for core counts for MPAS partition files #563

Update algorithm for core counts for MPAS partition files #563

Conversation

xylar commented Mar 13, 2023 • edited Loading

xylar commented Mar 13, 2023

xylar commented Mar 14, 2023

xylar Mar 14, 2023

Choose a reason for hiding this comment

xylar Mar 14, 2023

Choose a reason for hiding this comment

xylar commented Mar 21, 2023

Testing

xylar commented Mar 21, 2023

xylar commented Mar 21, 2023

mark-petersen left a comment

Choose a reason for hiding this comment

xylar commented Mar 22, 2023

xylar commented Mar 13, 2023 •

edited

Loading