Add DEBUGGER option to env_run.xml

[ekluzek]

I'd like to add a DEBUGGER option to env_run.xml so that the mpirun command can be setup to run with a debugger easier than it is now. The abstraction of the run scripts, also makes it difficult for a user to even figure out how to go about connecting a debugger to a cime model. And the complexity of the cime models certainly makes it make sense to use a debugger as a tool to help with debugging the system.

So for example in the config_machine.xml for yellowstone you currently have...

<mpirun mpilib="mpi-serial">
  <executable></executable>
</mpirun>
<mpirun mpilib="default" threaded="false">
  <executable>TARGET_PROCESSOR_LIST=AUTO_SELECT mpirun.lsf $ENV{CESMDATAROOT}/tools/bin/launch </executable>
</mpirun>
<mpirun mpilib="default" threaded="true">
  <executable>unset MP_PE_AFFINITY; unset MP_TASK_AFFINITY; unset MP_CPU_BIND_LIST; mpirun.lsf $ENV{CESMDATAROOT}/tools/bin/hybrid_launch </executable>
</mpirun>
<mpirun compiler="gnu">
  <executable>mpirun.lsf </executable>
</mpirun>

We would add something like the following....

<mpirun mpilib="mpi-serial" debugger="ddt">
  <executable>ddt --connect</executable>
</mpirun>
<mpirun mpilib="default" debugger="ddt">
  <executable>ddt --connect mpirun.lsf </executable>
</mpirun>

It probably has to be setup individually for each machine, and for each debugger available (and wanted to be used on that machine). Having something available out of the box makes it easier to setup though, and all debuggers take some setup to work with CESM.

You could also add a DEBUGGERS list of available debuggers for a given machine. But, that's probably only important if we can show that having this option is useful to begin with.

@cacraigucar @fvitt @billsacks

[ekluzek]

OK, here's some updates on this for yellowstone...

Index: cime_config/cesm/machines/config_machines.xml
===================================================================
--- cime_config/cesm/machines/config_machines.xml   (revision 11641)
+++ cime_config/cesm/machines/config_machines.xml   (working copy)
@@ -1261,9 +1261,25 @@
     <PES_PER_NODE>15</PES_PER_NODE>
     <PROJECT_REQUIRED>TRUE</PROJECT_REQUIRED>
     <mpirun mpilib="mpi-serial" debugger="ddt">
-      <executable></executable>
+      <executable>ddt --connect --cwd=$RUNDIR --no-mpi</executable>
     </mpirun>
+    <mpirun mpilib="default" threaded="true" debugger="ddt">
+      <executable>ddt --connect --cwd=$RUNDIR</executable>
+      <arguments>
+   <arg name="num_tasks"> --np {{ num_tasks }}</arg>
+   <arg name="tasks_per_node"> --procs-per-node={{ tasks_per_node }}</arg>
+   <arg name="thread_count"> --openmp-threads={{ thread_count }}</arg>
+      </arguments>
+    </mpirun>
     <mpirun mpilib="default" threaded="false" debugger="ddt">
+      <executable>ddt --connect --cwd=$RUNDIR</executable>
+      <arguments>
+   <arg name="num_tasks"> --np {{ num_tasks }}</arg>
+   <arg name="tasks_per_node"> --procs-per-node={{ tasks_per_node }}</arg>
+      </arguments>
+    </mpirun>
     <mpirun mpilib="default" threaded="false">
       <executable>TARGET_PROCESSOR_LIST=AUTO_SELECT mpirun.lsf $ENV{CESMDATAROOT}/tools/bin/launch </executable>
     </mpirun>
     <mpirun mpilib="default" threaded="true">
@@ -1289,6 +1306,7 @@
    <command name="load">gmake/4.1</command>
    <command name="load">python</command>
    <command name="load">all-python-libs</command>
+   <command name="load">allinea/6.0.4</command>
       </modules>
       <modules compiler="intel">
    <command name="load">intel/15.0.3</command>

[ekluzek]

Today we thought a better name would be like this:

RUN_IN_DEBUGGER=ddt

[ekluzek]

The syntax has changed a bit in the latest cime versions: So now you don't use the {{ }} syntax. But, rather something like...

<mpirun mpilib="default" threaded="false">
    <executable>ddt --connect --cwd=$RUNDIR</executable>
    <arguments>
        <arg name="num_tasks"> --np $TOTALPES</arg>
        <arg name="tasks_per_node"> --procs-per-node=$MAX_TASKS_PER_NODE</arg>
    </arguments>
</mpirun>

If you use $ENV{''} it doesn't work correctly, presumably because it doesn't expand the variable correctly at that point or some such thing.

[ekluzek]

@mrnorman see the comment above on Feb 10th on how to set this up for DDT.

[mrnorman]

Thanks! So I just replace my mpirun section in env_mach_specific.xml to the block you gave, and it should work?

[ekluzek]

@mrnorman Yes, replacing the existing mpirun section and using that block should work for you. That is literally what I've been doing. Note, it is setup to assume that the mpilib is default and threaded is false. But, you could configure it for other configurations as well.

[mrnorman]

We ended up solving it through the CIME scripts in #1384 . I like that solution because we just change aprun to "ddt --connect aprun" in env_mach_specific. Tested it on Titan, and it worked for me.

[ekluzek]

Note, when running with mpi-serial, the following env variable should be set, to prevent a dialog box coming up to ask you

>   <environment_variables mpilib="mpi-serial">
>     <env name="ALLINEA_NO_MPI_AUTODETECT">TRUE</env>
>   </environment_variables>
>   <mpirun>

And you want to add "--no-mpi" to the ddt line.

[jgfouca]

preview_run and './xmlchange MPI_RUN_COMMAND=$your_cmd' should provide the functionality needed to support debuggers.

[billsacks]

Replaced with #2817