Identify areas for additional testing

[edwardhartnett]

@jbathegit run these commands:

cmake -DCMAKE_Fortran_FLAGS="-fprofile-arcs -ftest-coverage" -DENABLE_TESTS=ON .. && make clean && make VERBOSE=1 && make test
gcovr -r ../ . --html --html-details -o example-html-details.html

Then open the file example-html-details.html to see the code coverage report.

Here's a snapshot:

The overall test coverage is 50%. This is pretty good, but we need it to be above 70% for sure. Can you identify areas for additional testing? (Alternatively, some of the code may be dead and can be removed instead of tested, which is even better.)

Some functions that leap out: src/copybf.f, src/copymg.f, src/copysb.f, src/cpbfdx.f, src/cpdxmm.f, src/cpymem.f, src/cpyupd.f are all untested...

As we identify areas for increased testing, add issues. When you have made a complete pass through the coverage report, and identified the first pass of tests that are needed, we can close this issue.

[jbathegit]

@edwardhartnett Where does the gcovr command live? I don't see it on the WCOSS - do I need to run gcovr on a different system?

Either way, I am working on some new tests as we speak ;-)

[edwardhartnett]

gcovr is a Unix utility. It can be installed with:

sudo apt-get install gcovr

It's a standard GNU tool so if you ask the sysadmins, they should be able to easily install for you. But then you have to, as you note, tar up the web pages and put them somewhere you can look at them.

I work on a stand-alone Linux box, not on the WCOSS systems.

[jbathegit]

Well, there's no way they'll install this on the WCOSS, and even though I have my own stand-alone NCWCP Linux box (emc-lw7-jator), it isn't installed there either at the moment, and as backed up as the sysadmins are right now it could be weeks/months before they could possibly get around to this.

Is your Linux box on the NCWCP network that I could maybe ssh into using my jator account? Or, failing that, would you mind just going ahead and creating a gcovr listing for me to look at (a full one, not just the snapshot shown above? ;-)

[edwardhartnett]

Sure, no problem. Some day I hope we can have these coverage reports generated on-line automatically every day. But we're not there yet. ;-)

We only need it occasionally, whenever we need to identify areas for further testing. Once we've identified a few things to test, we can get to work writing tests for them. Then we can run the coverage again to see what should come next...

[edwardhartnett]

And we did identify some additional testing needs. See #33

[jbathegit]

Yes, I know about #33. But as noted in #51 thru #55, I'm also going to be developing some additional test programs for those 5 issues, so I wanted to know if there was any "low-hanging fruit" from within your earlier coverage report that I might be able to address at the same time and within the same programs. And that's why I wanted to see a full report, rather than just a snapshot of the first 30 or so lines ;-)

[edwardhartnett]

OK testing is now up to 60%, which is good progress!

I've attached the tarball with the coverage output. Here's the first page:

cov.tar.gz

[jbathegit]

Thanks for this @edwardhartnett This will definitely help guide me in developing future test codes.

One question - is there any way to get a similar listing for all of the C subprograms (or, better yet, be able to include all of those on the same diagram)? I'd be curious to know what the numbers are for the C subprograms as well, and where we might want also want to add more testing for those.

[edwardhartnett]

Not that I know of, but if you drill down into the file, you will see the code and can look manually at each subprogram.

What I generally do, in a test campaign like this, is first get basic testing for files that are at 0%, then I start looking for big blocks of unexecuted code (10s of lines) and target them.

After taking those two passes through the tests, the test coverage should be over 70% or so. Then I would look again, this time writing or changing tests to hit small blocks of code, even individual lines, and to check all error handling. This will then get you about at maximum code coverage.

[aerorahul]

@edwardhartnett
Consider adding the utilities to the code coverage and testing. Thanks!

[edwardhartnett]

OK, the code coverage report does include the C code and the utilities as requested.

Since testing is well under way on this project, I will close this issue and more testing can be added with further issues.