Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
Reworking how errmsg is treated in device code to remove some preproc…
…essor variable substitutions through the use of new local variables. The changes in this commit affect 3 main areas of module_sf_mynn.F90: 1.) Subroutine SFCLAY_mynn 2.) Subroutine SFCLAY1D_mynn 3.) Subroutine GFS_zt_wat Each of these areas are described in more detail below. 1.) SFCLAY_mynn In the SFCLAY_mynn subroutine, it was possible to remove all #ifdef substitutions of errmsg(len=*) for errmsg(len=200) because errmsg is not used in any code regions of this subroutine that may be run on an accelerator device. Since this is the case, errmsg(len=*) is perfectly acceptable, and can be left alone. The OpenACC data statements within the subroutine were also updated to remove references to errmsg as well since, again, it was not necessary to have errmsg on the device for this subroutine. 2.) SFCLAY1D_mynn - Creation of device_errmsg and device_errflg and proper syncing with errmsg and errflg In the SFCLAY1D_mynn subroutine, it was also possible to remove all #ifdef substitutions by instead creating a new local variable called device_errmsg that is a copy of errmsg but with a fixed size of 512 such that it is acceptable for use on the device. This is necessary because at certain points in the subroutine, loops that are good to be offloaded to the device set errmsg under certain conditions. Since these areas cannot be isolated from the parent loop without a major rework of the loop, we must preserve a way for errmsg to be set on the device. Since device_errmsg is a fixed size, we can do that. However, this complicates the code a bit for error handling purposes as we now have errmsg and device_errmsg which must be synced properly to ensure error messages are returned to CCPP as expected. Therefore, we must keep track of when device_errmsg is set so we can know to sync device_errmsg with errmsg. This is done by making a new local variable called device_errflg to be device_errmsg's complement on the device as errflg is errmsg's complement on the host. When device_errflg is set to a nonzero integer, we then know that device_errmsg must be synced with errmsg. This is simple to do at the end of the subroutine after the device_errmsg on the device is copyout-ed by OpenACC, and a new IF-block has been added for this general case. - Special case of mid-loop return (line 1417), and the creation of device_special_errflg and device_special_errmsg However, there is a special case we must handle a bit differently. In the mid-loop return statement near line 1417, we also must perform this sync to ensure the proper errmsg is returned in the event this return is needed. Therefore, a similar IF-block has been created within the corresponding #ifdef near line 2027 to ensure errmsg has the proper value before the subroutine returns. However, since this block is in the middle of the entire code and only executed on the host, we must first perform an OpenACC sync operation to make sure the device_errmsg and the device_errflg on the host matches the device_errmsg and device_errflg on the host, otherwise the incorrect values could lead to the return statement not executing as expected. This special case seems simple, but an extra trap lay exposed. If device_errmsg and device_errflg is set on the device at any point now before this IF-block, then the return statement we moved out of the loop will now be executed for *ANY* error message, whether that was the intended course or not. Therefore, we need to ensure this special case is only triggered for this specific case. Unfortunately, there appears no other way than to create two additional variables (device_special_errmsg and device_special_errflg) to isolate this case from all other error cases. With these installed in place of just device_errmsg and device_errflg, this special return case is now properly handled. - Complete Ifdef/Ifndef removal not possible Overall, due to the nature of this special case, we have no choice but to leave the #ifdef and #ifndef preprocessor statements in place as they are the only things capable of moving this return statement out of the loop without additional invasive changes to how the code operates. 3.) GFS_zt_wat In the GFS_zt_wat subroutine, since this subroutine is called on the device from within the main I-loop of SFCLAY1D_mynn, we have no choice but to change all errmsg and errflg usage to device_errmsg or device_errflg, otherwise this subroutine and the entire parent loop could not be run on the device. Therefore, all errmsg and errflg lines have been commented out and new, comparable lines using device_errmsg and device_errflg added in their place. Additionally, the subroutine call variable list was updated.
- Loading branch information
1 parent
9c342a5
commit 95e9ff2