Update summary plugin #2914

JonoYang · 2022-04-11T21:49:46Z

This PR further overhauls the summary plugin in summarycode. The previous summary plugin returned a count of all the detected license expressions, holders, programming languages, and other scan attributes. We are now evolving that, where we have scancode-toolkit automatically deduce the correct origin info for the codebase it scans using the summary plugin.

This PR also updates commoncode to version 30.1.0.

The summary codebase attribute created by the summary plugin now has these fields:

declared_license_expression

Primary license expression for a scanned codebase

license_clarity_score

The score of how well a codebase has declared its origin information

declared_holder

Primary holder of the scanned codebase

primary_language

The main programming language of the scanned codebase

other_license_expressions

Other detected license expressions from the scanned codebase, excluding the declared_license_expression

other_holders

Other detected copyright holders from the scanned codebase, excluding the declared_holder

other_languages

Other detected programming languages, excluding primary_language.

Reference: #2842 Signed-off-by: Jono Yang <jyang@nexb.com>

* Set summary2 to be new summary plugin * Update tests Signed-off-by: Jono Yang <jyang@nexb.com>

Reference: #2842 Signed-off-by: Jono Yang <jyang@nexb.com>

* Remove declared values from license_expressions, holders, programming_language summary lists and then call those lists `other_license_expressions, `other_holders`, and `other_programming_languages`, respectivly. Reference: #2842 Signed-off-by: Jono Yang <jyang@nexb.com>

* Move old summarizer code to legacy_summarizer.py Reference: #2842 Signed-off-by: Jono Yang <jyang@nexb.com>

* Remove summarizer2.py Reference: #2842 Signed-off-by: Jono Yang <jyang@nexb.com>

* Update commoncode to v30.1.0 * Update expected test results Reference: #2842 Signed-off-by: Jono Yang <jyang@nexb.com>

* We have updated commoncode, which added a new field to the headers. We need to regen any test expectations that tests scancode output Reference: #2842 Signed-off-by: Jono Yang <jyang@nexb.com>

* legacy_summarizer.py has been renamed to summarizer_legacy.py Reference: #2842 Signed-off-by: Jono Yang <jyang@nexb.com>

* Update CHANGELOG.rst Reference: #2842 Signed-off-by: Jono Yang <jyang@nexb.com>

* Determine declared license expression and declared holders from package data * Map more Microsoft common names in COMMON_NAMES Signed-off-by: Jono Yang <jyang@nexb.com>

* Update summarizer test cases using codebases of real code Reference: #2842 Signed-off-by: Jono Yang <jyang@nexb.com>

* Only consider the number of times a programming language occurs in get_primary_language Signed-off-by: Jono Yang <jyang@nexb.com>

Reference: #2842 Signed-off-by: Jono Yang <jyang@nexb.com>

* Add new summarizer tests for license ambiguity, conflicting license categories * Rename previous simple test to package * Remove full and copyright test Reference: #2842 Signed-off-by: Jono Yang <jyang@nexb.com>

Reference: #2842 Signed-off-by: Jono Yang <jyang@nexb.com>

* In the summary, the primary_programming_language field has been renamed programming_language. The other_programming_languages field has been renamed other_languages Signed-off-by: Jono Yang <jyang@nexb.com>

JonoYang · 2022-04-11T22:00:35Z

The remaining pieces left are to figure out what to do when multiple package data is detected at the root of the codebase we are scanning. Currently, we are under the assumption that the codebase we are scanning is a package itself. However, some packages have multiple package manifests at the same root directory. We have some code that picks which package data to use based on the number of code files that are of the same language for those packages, but it is not a reliable heuristic.

Signed-off-by: Jono Yang <jyang@nexb.com>

* Rename any references to summarizing in copyrights_summary.py to tallying and save as a new file Signed-off-by: Jono Yang <jyang@nexb.com>

Reference: #2842 Signed-off-by: Jono Yang <jyang@nexb.com>

* Combine all detected origin info from multiple package data and use the resulting values in the summary * Create new test for multiple package data summarization Reference: #2842 Signed-off-by: Jono Yang <jyang@nexb.com>

* Remove redundant code in summarizer.py Reference: #2842 Signed-off-by: Jono Yang <jyang@nexb.com>

* Remove all references to summarization in tallies tests Reference: #2842 Signed-off-by: Jono Yang <jyang@nexb.com>

* Remove copyright_summary.py * Update summarizer help text Reference: #2842 Signed-off-by: Jono Yang <jyang@nexb.com>

Reference: #2842 Signed-off-by: Jono Yang <jyang@nexb.com>

* Add entry for summary plugin changes Signed-off-by: Jono Yang <jyang@nexb.com>

Reference: #2842 Signed-off-by: Jono Yang <jyang@nexb.com>

* This allows us to more effectivly remove declared holders from the other holders list * Update holders summarization tests Reference: #2842 Signed-off-by: Jono Yang <jyang@nexb.com>

JonoYang · 2022-04-15T17:59:34Z

To handle the case of multiple package data present at a root of a directory, we've decided to just combine all the package data and return it as the declared origin information.

Reference: #2842 Signed-off-by: Jono Yang <jyang@nexb.com>

pombredanne

Looking great, yet I do not think this is worth updating the attribute name to tallies in the cluecode tests. Changing just the code should be enough IMHO. What do you think?

* Do not rename summaries to tallies Reference: #2842 Signed-off-by: Jono Yang <jyang@nexb.com>

JonoYang · 2022-04-20T00:22:27Z

@pombredanne In the cluecode tests, I've renamed the tallies attribute back to summary. I'm still failing a test for the CycloneDX output plugin and I'm not sure why it's doing that. I've opened a ticket for that #2917

pombredanne

Looking great and avoiding the copyright tests YAML means we have more reasonable churn.
Some expected JSON still need fixing though and IMHO we should not report thing without copyrights or holder in tallies but only things that have a value

Reference: #2842 Signed-off-by: Jono Yang <jyang@nexb.com>

Signed-off-by: Philippe Ombredanne <pombredanne@nexb.com>

JonoYang added 20 commits March 24, 2022 10:33

Add deprecation warnings to summary plugins

4a11db5

Reference: #2842 Signed-off-by: Jono Yang <jyang@nexb.com>

Add deprecation message to headers

d59d9df

Reference: #2842 Signed-off-by: Jono Yang <jyang@nexb.com>

Rename summary option to summary-legacy #2842

de4c250

* Set summary2 to be new summary plugin * Update tests Signed-off-by: Jono Yang <jyang@nexb.com>

Determine primary programming language in summary

9a77c72

Reference: #2842 Signed-off-by: Jono Yang <jyang@nexb.com>

Run license clarity scoring and summary at once

976ea12

* Move old summarizer code to legacy_summarizer.py Reference: #2842 Signed-off-by: Jono Yang <jyang@nexb.com>

Split old and new summarizer code

49f72ed

* Remove summarizer2.py Reference: #2842 Signed-off-by: Jono Yang <jyang@nexb.com>

Combine license scoring and summary plugins

3c80157

* Update commoncode to v30.1.0 * Update expected test results Reference: #2842 Signed-off-by: Jono Yang <jyang@nexb.com>

Update expected test results

1cb2c4a

* We have updated commoncode, which added a new field to the headers. We need to regen any test expectations that tests scancode output Reference: #2842 Signed-off-by: Jono Yang <jyang@nexb.com>

Rename legacy_summarizer.py

017a181

* legacy_summarizer.py has been renamed to summarizer_legacy.py Reference: #2842 Signed-off-by: Jono Yang <jyang@nexb.com>

Determine primary language from detected package

3cc279f

* Update CHANGELOG.rst Reference: #2842 Signed-off-by: Jono Yang <jyang@nexb.com>

Use package data for origin info

57710b7

* Determine declared license expression and declared holders from package data * Map more Microsoft common names in COMMON_NAMES Signed-off-by: Jono Yang <jyang@nexb.com>

Determine declared info from Package data

c9c0003

* Update summarizer test cases using codebases of real code Reference: #2842 Signed-off-by: Jono Yang <jyang@nexb.com>

Refactor get_primary_language

1f91ffb

* Only consider the number of times a programming language occurs in get_primary_language Signed-off-by: Jono Yang <jyang@nexb.com>

Remove unused functions

2cdcb76

Reference: #2842 Signed-off-by: Jono Yang <jyang@nexb.com>

Pick package in get_origin_info_from_package_data

f91cb46

Reference: #2842 Signed-off-by: Jono Yang <jyang@nexb.com>

Return empty strings if a package cannot be determined

02e4977

Reference: #2842 Signed-off-by: Jono Yang <jyang@nexb.com>

Fix bug where joined expressions were not returned

77adf3b

* Add new summarizer tests for license ambiguity, conflicting license categories * Rename previous simple test to package * Remove full and copyright test Reference: #2842 Signed-off-by: Jono Yang <jyang@nexb.com>

Simplify package scan summarization tests

f91fc24

Reference: #2842 Signed-off-by: Jono Yang <jyang@nexb.com>

Simplify programming language fields in summary

fb012aa

* In the summary, the primary_programming_language field has been renamed programming_language. The other_programming_languages field has been renamed other_languages Signed-off-by: Jono Yang <jyang@nexb.com>

JonoYang mentioned this pull request Apr 11, 2022

Add deprecation warnings to summary plugins #2899

Closed

JonoYang added 2 commits April 11, 2022 19:07

Ignore package uuid in tests

0abe6f0

Signed-off-by: Jono Yang <jyang@nexb.com>

Rename summarizer_legacy to tallies

ae0aef1

* Rename any references to summarizing in copyrights_summary.py to tallying and save as a new file Signed-off-by: Jono Yang <jyang@nexb.com>

JonoYang force-pushed the update-summary-plugin branch from 946c485 to ced0cb8 Compare April 13, 2022 01:57

JonoYang added 2 commits April 13, 2022 18:37

Remove legacy summarizer and tests

20cd6c0

Reference: #2842 Signed-off-by: Jono Yang <jyang@nexb.com>

Get origin info from multiple package data

5afa8f2

* Combine all detected origin info from multiple package data and use the resulting values in the summary * Create new test for multiple package data summarization Reference: #2842 Signed-off-by: Jono Yang <jyang@nexb.com>

JonoYang force-pushed the update-summary-plugin branch from ced0cb8 to 5afa8f2 Compare April 14, 2022 02:04

JonoYang added 2 commits April 14, 2022 12:02

Remove references to summary in tallies.py

7ce3734

* Remove redundant code in summarizer.py Reference: #2842 Signed-off-by: Jono Yang <jyang@nexb.com>

Rename tallies test files

b562e3f

* Remove all references to summarization in tallies tests Reference: #2842 Signed-off-by: Jono Yang <jyang@nexb.com>

JonoYang force-pushed the update-summary-plugin branch from 523bbe9 to b562e3f Compare April 14, 2022 19:24

JonoYang added 5 commits April 14, 2022 16:34

Update tests for copyright_tallies.py

be6b2a1

* Remove copyright_summary.py * Update summarizer help text Reference: #2842 Signed-off-by: Jono Yang <jyang@nexb.com>

Add tests for helper functions

47ec08f

Reference: #2842 Signed-off-by: Jono Yang <jyang@nexb.com>

Update CHANGELOG.rst

64db2e8

* Add entry for summary plugin changes Signed-off-by: Jono Yang <jyang@nexb.com>

Update CLI help text test expectations

222014d

Reference: #2842 Signed-off-by: Jono Yang <jyang@nexb.com>

Return holders as a list before joining

a4ea113

* This allows us to more effectivly remove declared holders from the other holders list * Update holders summarization tests Reference: #2842 Signed-off-by: Jono Yang <jyang@nexb.com>

JonoYang requested a review from pombredanne April 15, 2022 18:00

JonoYang added 2 commits April 15, 2022 11:05

Update CHANGELOG.rst

6163969

Reference: #2842 Signed-off-by: Jono Yang <jyang@nexb.com>

Merge branch 'develop' into update-summary-plugin

34720c1

pombredanne requested changes Apr 19, 2022

View reviewed changes

JonoYang force-pushed the update-summary-plugin branch from e92c8c5 to c071232 Compare April 20, 2022 00:06

Revert changes to cluecode_test_utils.CopyrightTest

72504e5

* Do not rename summaries to tallies Reference: #2842 Signed-off-by: Jono Yang <jyang@nexb.com>

JonoYang force-pushed the update-summary-plugin branch from c071232 to 72504e5 Compare April 20, 2022 00:15

pombredanne requested changes Apr 20, 2022

View reviewed changes

pombredanne mentioned this pull request Apr 20, 2022

Add system packages support in the new packages model #2909

Merged

4 tasks

JonoYang and others added 2 commits April 20, 2022 09:36

Update tallies test expectations

6a9576c

Reference: #2842 Signed-off-by: Jono Yang <jyang@nexb.com>

Regen CycloneDX expectation with metadata

e9df507

Signed-off-by: Philippe Ombredanne <pombredanne@nexb.com>

JonoYang merged commit 3caea56 into develop Apr 20, 2022

JonoYang deleted the update-summary-plugin branch April 20, 2022 18:15

This was referenced Apr 28, 2022

Design the new user experience for primary/declared data summaries in SCTK #2842

Closed

Improve/rework the consolidate/summary scan options #2258

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Update summary plugin #2914

Update summary plugin #2914

JonoYang commented Apr 11, 2022 •

edited

Loading

JonoYang commented Apr 11, 2022

JonoYang commented Apr 15, 2022

pombredanne left a comment

JonoYang commented Apr 20, 2022

pombredanne left a comment

Update summary plugin #2914

Update summary plugin #2914

Conversation

JonoYang commented Apr 11, 2022 • edited Loading

JonoYang commented Apr 11, 2022

JonoYang commented Apr 15, 2022

pombredanne left a comment

Choose a reason for hiding this comment

JonoYang commented Apr 20, 2022

pombredanne left a comment

Choose a reason for hiding this comment

JonoYang commented Apr 11, 2022 •

edited

Loading