LCOV_EXCL_LINE counting

[apcraig]

We upgraded our software recently, now using
lcov: LCOV version 2.0-1

LCOV_EXCL_LINEs seems to be counted now incorrectly. An old coverage report is here,

https://apcraig.github.io/lcov_cice_231018-213830:8916b9ff2c:11:first,base,travis,decomp,reprosum,io,omp,perf,gridsys,quick,unittest/cicecore/cicedyn/general/ice_step_mod.F90.gcov.html

A new coverage report is

https://apcraig.github.io/lcov_cice_250114-025918:736c1771a8:6:first,base,travis,gridsys,quick,unittest/cicecore/cicedyn/general/ice_step_mod.F90.gcov.html

The code has changed a bit, but the coverage should be about the same. The old version reports 390 lines and 90% coverage. The new version reports 787 lines and 47% coverage. The main difference is that the new coverage reports are counting a bunch of the LCOV_EXCL_LINEs which were skipped before. The LCOV_EXCL_LINE comments are added by the coverage tool at compilation as they should be (they are continuation lines in Fortran), but then not being skipped properly. Our process is largely unchanged.

We are using gcc/12.4.0, GNU Fortran (Spack GCC) 12.4.0 and compiling with
FFLAGS += -O0 -g -fprofile-arcs -ftest-coverage
CFLAGS += -O0 -g -coverage
LDFLAGS += -g -ftest-coverage -fprofile-arcs

We generate aggregate coverage reports over multiple tests via

lcov --gcov-tool gcov -c -d /glade/derecho/scratch/tcraig/CICE_RUNS/derecho_gnu_smoke_gx3_8x2_diag1_run5day.c1c3e/compile -o derecho_gnu_smoke_gx3_8x2_diag1_run5day.c1c3e/lcov.info
if (-s derecho_gnu_smoke_gx3_8x2_diag1_run5day.c1c3e/lcov.info) then
  set lcovalist = "${lcovalist} -a derecho_gnu_smoke_gx3_8x2_diag1_run5day.c1c3e/lcov.info"
endif

lcov --gcov-tool gcov -c -d /glade/derecho/scratch/tcraig/CICE_RUNS/derecho_gnu_restart_gx3_4x2x25x29x4_dslenderX2.c1c3e/compile -o derecho_gnu_restart_gx3_4x2x25x29x4_dslenderX2.c1c3e/lcov.info
if (-s derecho_gnu_restart_gx3_4x2x25x29x4_dslenderX2.c1c3e/lcov.info) then
  set lcovalist = "${lcovalist} -a derecho_gnu_restart_gx3_4x2x25x29x4_dslenderX2.c1c3e/lcov.info"
endif

...

lcov ${lcovalist} -o total.info
genhtml -o ./${lcovhtmldir} --precision 2 -t "${report_name}" total.info

Any thoughts/suggestions?

[apcraig]

I added an .lcovrc with
omit_lines = LCOV_EXCL_LINE
without any improvements. Looking at the log file, I see

Capturing coverage data from /glade/derecho/scratch/tcraig/CICE_RUNS/derecho_gnu_smoke_gx3_8x2_diag1_run5day.clc3g/compile
geninfo cmd: '/glade/u/apps/derecho/24.12/spack/opt/spack/lcov/2.0/gcc/12.4.0/e4ri/bin/geninfo /glade/derecho/scratch/tcraig/CICE_RUNS/derecho_gnu_smoke_gx3_8x2_diag1_run5day.clc3g/compile --output-filename derecho_gnu_smoke_gx3_8x2_diag1_run5day.clc3g/lcov.info --gcov-tool gcov --omit-lines LCOV_EXCL_LINE --memory 0'
geninfo: WARNING: Duplicate specification "build-directory|b=s" for option "b"
Found gcov version: 12.4.0
Using intermediate gcov format
Writing temporary data to /tmp/geninfo_datsMwu
Scanning /glade/derecho/scratch/tcraig/CICE_RUNS/derecho_gnu_smoke_gx3_8x2_diag1_run5day.clc3g/compile for .gcda files ...
Found 88 data files in /glade/derecho/scratch/tcraig/CICE_RUNS/derecho_gnu_smoke_gx3_8x2_diag1_run5day.clc3g/compile
Processing /glade/derecho/scratch/tcraig/CICE_RUNS/derecho_gnu_smoke_gx3_8x2_diag1_run5day.clc3g/compile/ice_memusage_gptl.gcda
Processing /glade/derecho/scratch/tcraig/CICE_RUNS/derecho_gnu_smoke_gx3_8x2_diag1_run5day.clc3g/compile/icepack_therm_shared.gcda
Processing /glade/derecho/scratch/tcraig/CICE_RUNS/derecho_gnu_smoke_gx3_8x2_diag1_run5day.clc3g/compile/ice_history_write.gcda
geninfo: WARNING: /glade/work/tcraig/cice-consortium/cice_lcov_250115-032532/cicecore/cicedyn/infrastructure/io/io_netcdf/ice_history_write.F90:623: unexecuted block on non-branch line with non-zero hit count.  Use "geninfo --rc geninfo_unexecuted_blocks=1 to set count to zero.
Processing /glade/derecho/scratch/tcraig/CICE_RUNS/derecho_gnu_smoke_gx3_8x2_diag1_run5day.clc3g/compile/icepack_therm_mushy.gcda
geninfo: WARNING: /glade/work/tcraig/cice-consortium/cice_lcov_250115-032532/icepack/columnphysics/icepack_therm_mushy.F90:353: unexecuted block on non-branch line with non-zero hit count.  Use "geninfo --rc geninfo_unexecuted_blocks=1 to set count to zero.
	(use "geninfo --ignore-errors gcov,gcov ..." to suppress this warning)
...
Processing /glade/derecho/scratch/tcraig/CICE_RUNS/derecho_gnu_smoke_gx3_8x2_diag1_run5day.clc3g/compile/icepack_age.gcda
Processing /glade/derecho/scratch/tcraig/CICE_RUNS/derecho_gnu_smoke_gx3_8x2_diag1_run5day.clc3g/compile/ice_dyn_vp.gcda
Finished .info-file creation
Omitted 0 total lines matching 1 '--omit-lines' pattern
geninfo: ERROR: 'omit-lines' pattern 'LCOV_EXCL_LINE' is unused.
	(use "geninfo --ignore-errors unused ..." to bypass this error)

which suggests it's not omitting any lines. Why would this happen?

[henry2cox]

lcov/2.0 is pretty old at this point (May 2023). Current release is 2.3 (about a week ago).
Upgrading might be advised.
However: this is a pretty basic exclusion feature and hasn't changed much in quite some time - so you might not see different behavioiur.

If you happen to know perl (or want to learn), then you can execute the 'geninfo cmd' that you see in your log file via perl -d .... Set a breakpoint in ReadCurrentSource::parseLines - near line lcovutil.pm:6148 in the current TOT - and see why it isn't matching.

Or, see if you can create a small testcase which exhibits the issue.

Incidentally: you don't need the --gcov-tool gcov option. lcov looks for gcov by default; you only need the option if you want to use a different tool (see the man pages).

[apcraig]

Thanks @henry2cox, I will try to do some additional debugging.

[apcraig]

OK, I figured it out. Some tests were adding additional compiler checks as part of our test suite,

FFLAGS += fcheck=bounds -finit-real=nan -fimplicit-none -ffpe-trap=invalid,zero,overflow --std f2008

And coverage reports for tests with these flags were including 0 hit reports for lines that should have been ignored (but that were often actually hit). When we aggregated our test suite these erroneous lines were being included in the report. I modified our build so when we're doing coverage testing, these extra compiler flags will never be added. Not sure whether this is a problem in gcc or lcov. For some reason, this was not a problem in our prior version of gcc/lcov.

I will close this issue now. I think the lesson learned is that some of the compiler flags have negative interactions with the coverage flags. Thanks.

[henry2cox]

I think the lesson learned is that some of the compiler flags have negative interactions with the coverage flags.

That should not be happening, I think.
If the code is compiled to instrument for coverage: then the coverage data will be generated at compile and/or runtime.
Then lcov is invoked (completely independently) to translate the data into lcov format (with or without various types of filtering and other munging).
There should be no interactions, except that there is no coverage data for code that is not instrumented and no lcov-coverage data for gcov-generated coverage data that is not captured.

How was the coverage data for those extra tests getting extracted?
Naively - I would have expected your lcov --capture ... command on them to have filtered out the excluded lines, in the same way as your other capture calls.
So something here seems strange.

The other thing to mention is the filtering in newer lcov versions is more extensive and more flexible than in the past.
For example,: you can tell it to NOT do region filtering during capture, but to filter everything during aggregation. Or you can not filter in either capture or aggregation - but do so later, when writing a report. (This also gives you the opportunity to generate a differential report - to see what got excluded, for example.)

The other thing to note is that your code might be easier to follow if you wrap the excluded region in LCOV_EXCL_START/LCOV_EXCL_END markers (or whatever custom marker you decide you want) - rather than tagging individual lines. (Doesn't make an difference WRT numbers or reporting - purely a matter of style.)