is there a way to generate incremental code coverage report?

[EEDY]

i already have a full coverage tracefile, but i want to have a html report to only show the coverage of my delta code in a PR, is there a way to do this?
or can genhtml receives an input of git diff file, and generates html report only based on the diff file?

[henry2cox]

If I understand your question: yes. (If I have misunderstood, please attempt to clarify and we can try to find a way to solve)
A good place to start is to read either of the Differential Coverage papers referred to in the README. They give an overview of the concepts that are implemented here.

The approach:

1. generate 'baseline.info' - the coverage data associated with your code + then-current tests before your PR changes
2. generate 'current.info' - the coverage data associated with your current code + tests (i.e., after your PR)
3. generate 'udiff.txt' - the universal-diff format file showing what changed in your PR. The 'gitdiff' script included in the lcov release may be a place to start).
4. generate HTML: genhtml -o differential_report --baseline-file baseline.info --diff-file udiff.txt current.info
   (There are a lot of options and parameters to control the format of the report - e.g., to add headers, footers, code annotations, etc - see the man page.)
5. look at the report in your browser: mybrowser differential_report/index.html
   (Again: there are a lot of hyperlinks between pages, between tables and code, and between code; the intent is to enable you to quickly find and diagnose the coverage issues you care about. I suggest to just click around a bit to see what happens.)

An orthogonal approach is to look at the date bins in your code. The idea there is to categorize code both by coverage status and by the time it was written or last changed - on the assumption that recently written and/or recently changed code is more interesting than code from long ago. The paper discusses how differential and date categorization differ and why you might want to use both.

Hope this helps
Henry

[Vladyyy]

@henry2cox Correct me if I'm wrong, differential coverage will color code the diff data however the html report will still capture all source file (See attached screenshot from the differential coverage example)

Is there a way to limit the report to only the files / lines changed in the diff (plus some lines of context)

My real example has files that are 10000s lines of code and 100s of files. I would like to capture only the files and lines that have changed in the diff.

You can filter unwanted files via the --exclude option is there a better way to achieve this?

[henry2cox]

We haven't had too many issues with generating reports and traversing the reports to analyze results for moderately large projects.
There is a usability issue with the --show-owners table for projects with more than a handful of developers and less-than-perfect validation - as the per-user summary table can have an enormous number of rows, making pages painful to scroll through. (So we group by module and/or by team - by making suitable changes to the --annotate-script callback.)

Thus, I think I need to understand your application/what you are trying to do/what problems you are trying to solve - in order to come up with a better answer and/or to understand which features we might want to add.

If what you want is to show only files which contain source file changes - but NOT files which contain coverage changes (gains/losses), then:

• you could process the udiff file (from git diff or whatever). File names which appear, contain differences - so --include them. All other files will be excluded.

• Note that this will pick up source files with spacing or comment changes (no changed code) - so might be noisier than you want.
  To fix that, you would need to do a bit more processing to check for coverpoints in the changed region and remove files for which there are none. (I would use the lcov utilities for that - but your risk is that we don't officially export lcovutils.pm - and may change the interface at any time, for any reason. Upstreaming your utility + tests would eliminate that risk.)

• It also will show you the whole file, for every file which contains a diff - not just the location of the change + some context lines.
  So this probably does not meet your requirement.

• (war story - sorry:) you would be surprised by the number of bugs that we have found that manifested as LBC code (lost coverage) - or maybe you wouldn't be surprised. The typical root cause is an unfortunate change in some control path (somewhere) which results in <this> code no longer being executed by any testcase - but no testcase was sensitive to the difference, so all regressions except coverage were green.
  Filtering for source changes would remove those so the bugs might slip through.

Hope that helps - despite probably not answering your question.

Henry

[henry2cox]

Another observation:

• I think that what you may actually want is a filter which keeps coverpoints in some subset of categories + a few lines of context around each contiguous region of retained lines - and then generate a report which shows only that code - and possibly, only those categories.
• I suspect that the categories you want are UNC, UIC, LBC - as those are the ones which reflect un-exercised source or Makefile/configuration changes + losses.

Everything else would be the same: same command line, same input data, etc. Only request to retain certain categories.

I think, not too hard to implement (but not there now).
(There are a few nuances that make this not entirely trivial...let me know if you are planning to give it a shot, and want a brain dump.)

I'm not certain that this would make things easier or faster to analyze vs. doing it on a full report (...after a bit of experience navigating in the report structure). It would substantially reduce report size - if your validation methodology is well under control.

I will ask around, to see if any users think that this is a good idea/that it would improve efficiency.

[Vladyyy]

Thanks for you reply

Thus, I think I need to understand your application/what you are trying to do/what problems you are trying to solve - in order to come up with a better answer and/or to understand which features we might want to add.

I have a fairly typical scenario, engineers submit code reviews, we compute the differential coverage for changes captured in the code reviews. Code reviewers look at UNC and GNC metrics. However as you pointed out, LBC is also a useful metric.

you could process the udiff file (from git diff or whatever). File names which appear, contain differences - so --include them. All other files will be excluded.

Thanks, that seems to be the default behavior, however I was using the scripts/gitdiff tool which also includes unchanged files in the diff.

I think that what you may want is a filter which keeps coverpoints in some subset of categories

I wonder if this would be more helpful in the generated html where you would have a checkmark for each category and you would be able to toggle the displayed categories on / off

[henry2cox]

The default behaviour always generates a complete report, including all files found in the current.info input.
gitdiff can be asked to include an empty diff for unchanged files to help remove ambiguity in categorization - say, in the case that there are two or more source files with the same basename and we are applying a fuzzy match criteria. An empty diff tells the tool that there really were no differences in some particular source file.

Unfortunately, the generated report is static HTML. The pages are pre-computed and then displayed in your browser - so there is no opportunity to dynamically modify the view - though I agree with you that it would be extremely handy in quite a few use cases.
Total rewrite, though.

For the purposes of code review though: probably a static report of some selected subset is OK.
(Similarly: a per-user, per-module, or per-team subset report is probably OK/useful for pre-release review.
Right now: we use the full report + navigation for that - but a subset report might be easier to understand as it would contain less stuff that a particular user doesn't care about. No need to expend mental energy to ignore if the uninteresting stuff is not present.)

[henry2cox]

@Vladyyy - you may be interested :-)

Commit dd286c6 adds a genhtml --select-script .. callback feature which can be applied to the code review use case described above. See the 'Step7: generate subset report for code review' section in the lcov example directory.

The basic approach is to call a user callback - and have it tell the tool whether a particular line is interesting (and should be retained) or not interesting (and may be dropped).
The tool then retains the selected lines as well as a user-configurable number of surrounding context and generates a report holding coverpoints only in those regions.

The same feature is also useful other scenarios - for example, when I am trying to improve coverage of code 'owned' by me which I wrote within the last 6 months (say). This code may not be easily found in a full report containing thousands of files and many authors, as the navigation hyperlinks key the user, the date, or the TLA idividually; there is no direct way to navigate to unexercised lines written on a particular date, by a specific user. In contrast, my selection criteria can specify exactly that combined condition.
(Of course there is no free lunch. The fly in the ointment is that I need to generate another report for each condition I care about.)

It might be a Good Idea to move this 'use models' discussion somewhere else - perhaps, along with a discussion of other coverage use cases.

[Vladyyy]

Thanks @henry2cox I'll check it out next week. What I ended up doing for now is to add some js script in the html prolog that transforms the categories (UNC, UBC, etc) into buttons. Clicking on the button will filter out the lines from the table. If you are interested I can share my work.

The reason I went down this route is because of your earlier message of the importance of LBC.

[henry2cox]

Sure...feel free to share.
I think that just about anything you find useful, is something that others would also find useful. Nothing new under the sun.
I'm not sure whether it is a good idea or a bad idea to change the lcov report - which has been static HTML up to now.
It is pretty obvious that there is a lot that one might want to do that requires active code.

[Vladyyy]

Uploaded my prototype here:
8f6e32d

To test it you need to build the differential example:
cd lcov/example; make test_differential

The categories headers will become buttons that you can now toggle on/off. Toggling a category off will hide it in the files / folder table. If an entry (file / folder) has not data for the remaining categories it will also get hidden

[henry2cox]

I think that the update to genhtml is missing from the commit - or perhaps I am not thinking too clearly this morning.
Thanks.

[Vladyyy]

The changes are correct. All the new added code is javascript that's injected through the --html-prologThe logic would probably be simpler if some of the changes were made in genhtml code