Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Update rules with required phrases automatically #3924

Open
wants to merge 82 commits into
base: develop
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
82 commits
Select commit Hold shift + click to select a range
aa9374c
Add script for adding required phrases automatically
AyanSinhaMahapatra Feb 15, 2023
72f290f
Add required phrases automatically for mit rules
AyanSinhaMahapatra Sep 2, 2024
bd14127
Add required phrases at bsd-new rules automatically
AyanSinhaMahapatra Sep 12, 2024
20a7860
Add required phrases automatically for gpl-2.0
AyanSinhaMahapatra Sep 16, 2024
0a44594
Add required phrases for gpl-2.0-only rules automatically
AyanSinhaMahapatra Sep 16, 2024
435255b
Skip creating required phrase rules for generic licenses
AyanSinhaMahapatra Sep 16, 2024
12eaccf
Add required phrases for genreic license rules automtaically
AyanSinhaMahapatra Sep 17, 2024
ece8c33
Add required phrases to gpl-1.0-plus rules automatically
AyanSinhaMahapatra Sep 17, 2024
be94a84
Add required phrases automatically on gpl/lgpl rules
AyanSinhaMahapatra Sep 17, 2024
584b554
Add more required phrases automatically in permissive rules
AyanSinhaMahapatra Sep 18, 2024
c778ce0
Add required phrases automatically to gpl-3.0 rules
AyanSinhaMahapatra Sep 18, 2024
f5e9593
Add required phrases for lgpl rules automatically
AyanSinhaMahapatra Sep 18, 2024
63c23f0
Add required phrases for agpl rules automatically
AyanSinhaMahapatra Sep 18, 2024
40ced0c
Add required phrases to bsd rules automatically
AyanSinhaMahapatra Sep 18, 2024
518116d
Add required phrases in lgpl rules automatically
AyanSinhaMahapatra Sep 18, 2024
3583ac6
Add required phrases automatically for cc rules
AyanSinhaMahapatra Sep 18, 2024
1994095
Improve license detection rules
pombredanne Oct 7, 2024
4d1b8ac
Improve required phrases in paths
pombredanne Oct 7, 2024
ef5101e
Improve required phrases in paths
pombredanne Oct 7, 2024
8e615bb
Improve required phrases in URLs
pombredanne Oct 7, 2024
6831e90
Correct rule syntax
pombredanne Oct 7, 2024
e05ca6f
Move back required-phrase related code
pombredanne Oct 7, 2024
94cffc9
Use correct syntax for rules
pombredanne Oct 7, 2024
62643cb
Apache Software Foundation is not required phrase
pombredanne Oct 7, 2024
7150b82
Remove duplicate imports
pombredanne Oct 7, 2024
83eadec
Improve license models
pombredanne Oct 7, 2024
41be5e2
Apply minor cosmetic refactoring
pombredanne Oct 7, 2024
512de1e
Move tests to test_required_phrase.py
pombredanne Oct 7, 2024
d611f9b
Add misc comments to Token class variants
pombredanne Oct 7, 2024
7d5199a
Refine is_required_phrase license rules
pombredanne Oct 8, 2024
05f7981
Ensure license index is buildable with extras
pombredanne Oct 8, 2024
451fffb
Refactor required phrase spans collection
pombredanne Oct 8, 2024
86d69b6
Refine is_required_phrase AGPL rules
pombredanne Oct 8, 2024
4a49447
Refine is_required_phrase GPL3 rules
pombredanne Oct 8, 2024
7fed520
Refine is_required_phrase GPL rules
pombredanne Oct 8, 2024
4bf01c8
Refine is_required_phrase LGPL rules
pombredanne Oct 8, 2024
a6bbfa7
Refine is_required_phrase Apache rules
pombredanne Oct 8, 2024
9470393
Refine is_required_phrase BSD rules
pombredanne Oct 8, 2024
fd10501
Add new and improved required phrase rules
pombredanne Oct 8, 2024
9cee2cb
Refine rules skipped for required phrase
pombredanne Oct 8, 2024
8e16712
Rename Rule.skip_collecting_required_phrases
pombredanne Oct 8, 2024
1bcf3fc
Update required phrase generation
pombredanne Oct 8, 2024
c99d26a
Make rules valid
pombredanne Oct 8, 2024
28e2ef4
Improve required phrase in rules
pombredanne Oct 8, 2024
3ad7302
Update test results
pombredanne Oct 8, 2024
c275355
Correct rule ignorables
pombredanne Oct 9, 2024
1d1a99c
Set relevance for is_required_phrase rules
pombredanne Oct 9, 2024
4e598bf
Fix duplicate license rules
pombredanne Oct 9, 2024
1d503f4
Adjust rules from required phrase tests
pombredanne Oct 9, 2024
d0b33f8
Improve Apache and commons clause rules
pombredanne Oct 9, 2024
543d47c
Improve imagemagick rules with required phrases
pombredanne Oct 9, 2024
359fa84
Improve rule selectivity
pombredanne Oct 9, 2024
3c0858e
Do not add required phrase to license text
pombredanne Oct 9, 2024
b64794c
Improve license test results
pombredanne Oct 9, 2024
848dab1
Consider license clues in gibberrish filter
pombredanne Oct 11, 2024
4cfaa8a
Use correct dry_run argument name in tests
pombredanne Oct 11, 2024
2607e0d
Update license test expectation
pombredanne Oct 11, 2024
243ba18
Consider clues in false positive filter
pombredanne Oct 11, 2024
10e0181
Fix comment content
pombredanne Oct 11, 2024
f6d7fe6
Add required phrase to MIT rule
pombredanne Oct 11, 2024
2fd0dd7
Improve license test results
pombredanne Oct 11, 2024
f6b7196
Improve license rules and license tests
pombredanne Oct 12, 2024
1802857
Rename new rules to resolve merge conflicts
pombredanne Oct 12, 2024
68c92a2
Merge latest develop
pombredanne Oct 12, 2024
4b41c62
Improve rule ignorables and requried phrases
pombredanne Oct 12, 2024
34d8b4e
Adapt license test expectations
pombredanne Oct 12, 2024
5659d1c
Do not debug trace by default
pombredanne Oct 12, 2024
f4c0c6b
Improve license detection results
pombredanne Oct 12, 2024
fc46065
Refine license detection rules
pombredanne Oct 12, 2024
cb271b8
Improve license detection results
pombredanne Oct 12, 2024
4f15de1
Add new license rule for LGPL
pombredanne Oct 12, 2024
28e871f
Improve GPL detection
pombredanne Oct 13, 2024
46135a4
Improve license detection tests
pombredanne Oct 13, 2024
5635edf
Add new GPL rules
pombredanne Oct 13, 2024
28e1acb
Improve MPL tri-license detection
pombredanne Oct 13, 2024
584e3d6
Add new DCO licenses
pombredanne Oct 13, 2024
160e95b
Add new license detection rules
pombredanne Oct 13, 2024
e5bff98
Improve license detection tests
pombredanne Oct 13, 2024
6eff903
Relax required phrase filtering
pombredanne Oct 13, 2024
e3fc9c9
Correct license test expectation
pombredanne Oct 14, 2024
94a4d35
Correct license rule expression
pombredanne Oct 14, 2024
4718fed
Update expected license test result
pombredanne Oct 14, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
Improve license test results
Adjust with latest license code detection for required phrases

Signed-off-by: Philippe Ombredanne <pombredanne@nexb.com>
  • Loading branch information
pombredanne committed Oct 11, 2024
commit 2fd0dd7c37f8fa697f492dd3d748f180857e40c2
50 changes: 36 additions & 14 deletions tests/formattedcode/data/common/manifests-expected.jsonlines
Original file line number Diff line number Diff line change
Expand Up @@ -20,11 +20,11 @@
"system_environment": {
"operating_system": "linux",
"cpu_architecture": "64",
"platform": "Linux-5.15.0-116-generic-x86_64-with-glibc2.35",
"platform_version": "#126-Ubuntu SMP Mon Jul 1 10:14:24 UTC 2024",
"python_version": "3.10.12 (main, Mar 22 2024, 16:50:05) [GCC 11.4.0]"
"platform": "Linux-6.8.0-45-generic-x86_64-with-glibc2.35",
"platform_version": "#45~22.04.1-Ubuntu SMP PREEMPT_DYNAMIC Wed Sep 11 15:25:05 UTC 2",
"python_version": "3.11.8 (main, Feb 12 2024, 14:25:06) [GCC 11.4.0]"
},
"spdx_license_list_version": "3.24",
"spdx_license_list_version": "3.25",
"files_count": 4
}
}
Expand Down Expand Up @@ -819,15 +819,15 @@
"identifier": "apache_2_0-d66ab77d-a5cc-7104-e702-dc7df61fe9e8",
"license_expression": "apache-2.0",
"license_expression_spdx": "Apache-2.0",
"detection_count": 3,
"detection_count": 2,
"reference_matches": [
{
"license_expression": "apache-2.0",
"license_expression_spdx": "Apache-2.0",
"from_file": "manifests/npm-license-mapping/package.json",
"start_line": 20,
"end_line": 20,
"matcher": "2-aho",
"start_line": 1,
"end_line": 1,
"matcher": "1-hash",
"score": 100.0,
"matched_length": 3,
"match_coverage": 100.0,
Expand All @@ -837,6 +837,28 @@
}
]
},
{
"identifier": "apache_2_0-0d7a2023-aae9-2989-7f00-27b713b809bb",
"license_expression": "apache-2.0",
"license_expression_spdx": "Apache-2.0",
"detection_count": 1,
"reference_matches": [
{
"license_expression": "apache-2.0",
"license_expression_spdx": "Apache-2.0",
"from_file": "manifests/npm-license-mapping/package.json",
"start_line": 18,
"end_line": 20,
"matcher": "2-aho",
"score": 100.0,
"matched_length": 5,
"match_coverage": 100.0,
"rule_relevance": 100,
"rule_identifier": "apache-2.0_required_phrase_7.RULE",
"rule_url": "https://github.com/nexB/scancode-toolkit/tree/develop/src/licensedcode/data/rules/apache-2.0_required_phrase_7.RULE"
}
]
},
{
"identifier": "apache_2_0-ec759ae0-ea5a-f138-793e-388520e080c0",
"license_expression": "apache-2.0",
Expand Down Expand Up @@ -1614,22 +1636,22 @@
"license_expression": "apache-2.0",
"spdx_license_expression": "Apache-2.0",
"from_file": "manifests/npm-license-mapping/package.json",
"start_line": 20,
"start_line": 18,
"end_line": 20,
"matcher": "2-aho",
"score": 100.0,
"matched_length": 3,
"matched_length": 5,
"match_coverage": 100.0,
"rule_relevance": 100,
"rule_identifier": "spdx_license_id_apache-2.0_for_apache-2.0.RULE",
"rule_url": "https://github.com/nexB/scancode-toolkit/tree/develop/src/licensedcode/data/rules/spdx_license_id_apache-2.0_for_apache-2.0.RULE"
"rule_identifier": "apache-2.0_required_phrase_7.RULE",
"rule_url": "https://github.com/nexB/scancode-toolkit/tree/develop/src/licensedcode/data/rules/apache-2.0_required_phrase_7.RULE"
}
],
"identifier": "apache_2_0-d66ab77d-a5cc-7104-e702-dc7df61fe9e8"
"identifier": "apache_2_0-0d7a2023-aae9-2989-7f00-27b713b809bb"
}
],
"license_clues": [],
"percentage_of_license_text": 5.51,
"percentage_of_license_text": 7.09,
"copyrights": [],
"holders": [],
"authors": [
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -36,16 +36,16 @@
"start_line": 1,
"end_line": 1,
"matcher": "1-hash",
"score": 50.0,
"score": 100.0,
"matched_length": 5,
"match_coverage": 100.0,
"rule_relevance": 50,
"rule_relevance": 100,
"rule_identifier": "spdx_license_id_gpl-1.0-or-later_for_gpl-1.0-plus.RULE",
"rule_url": "https://github.com/nexB/scancode-toolkit/tree/develop/src/licensedcode/data/rules/spdx_license_id_gpl-1.0-or-later_for_gpl-1.0-plus.RULE",
"matched_text": "gpl-1.0-or-later"
}
],
"identifier": "gpl_1_0_plus-757c6eaa-30b2-56ce-77bb-b6bd8dc83d88"
"identifier": "gpl_1_0_plus-15a72215-873f-562d-09b8-391aba044287"
}
],
"other_license_expression": null,
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -36,16 +36,16 @@
"start_line": 1,
"end_line": 1,
"matcher": "1-hash",
"score": 50.0,
"score": 100.0,
"matched_length": 5,
"match_coverage": 100.0,
"rule_relevance": 50,
"rule_relevance": 100,
"rule_identifier": "spdx_license_id_gpl-1.0-or-later_for_gpl-1.0-plus.RULE",
"rule_url": "https://github.com/nexB/scancode-toolkit/tree/develop/src/licensedcode/data/rules/spdx_license_id_gpl-1.0-or-later_for_gpl-1.0-plus.RULE",
"matched_text": "gpl-1.0-or-later"
}
],
"identifier": "gpl_1_0_plus-757c6eaa-30b2-56ce-77bb-b6bd8dc83d88"
"identifier": "gpl_1_0_plus-15a72215-873f-562d-09b8-391aba044287"
}
],
"other_license_expression": null,
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -36,16 +36,16 @@
"start_line": 1,
"end_line": 1,
"matcher": "1-hash",
"score": 50.0,
"score": 100.0,
"matched_length": 5,
"match_coverage": 100.0,
"rule_relevance": 50,
"rule_relevance": 100,
"rule_identifier": "spdx_license_id_gpl-1.0-or-later_for_gpl-1.0-plus.RULE",
"rule_url": "https://github.com/nexB/scancode-toolkit/tree/develop/src/licensedcode/data/rules/spdx_license_id_gpl-1.0-or-later_for_gpl-1.0-plus.RULE",
"matched_text": "gpl-1.0-or-later"
}
],
"identifier": "gpl_1_0_plus-757c6eaa-30b2-56ce-77bb-b6bd8dc83d88"
"identifier": "gpl_1_0_plus-15a72215-873f-562d-09b8-391aba044287"
}
],
"other_license_expression": null,
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -406,10 +406,10 @@ other_license_detections:
start_line: 306
end_line: 309
matcher: 2-aho
score: '100.0'
score: '99.0'
matched_length: 47
match_coverage: '100.0'
rule_relevance: 100
rule_relevance: 99
rule_identifier: apache-2.0_1021.RULE
rule_url: https://github.com/nexB/scancode-toolkit/tree/develop/src/licensedcode/data/rules/apache-2.0_1021.RULE
matched_text: |
Expand Down Expand Up @@ -449,7 +449,7 @@ other_license_detections:
sources, credits must appear in the documentation.

4. This notice may not be removed or altered.
identifier: apache_2_0_and_hs_regexp-418b8fd9-1905-1ad7-6930-311a90aaf0a7
identifier: apache_2_0_and_hs_regexp-b58332f1-4f96-ae87-9f3d-691331c863c9
- license_expression: bsd-unchanged
license_expression_spdx: LicenseRef-scancode-bsd-unchanged
matches:
Expand Down