Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Ensure ORT to handle the output format 3 of the ScanCode 32 series #6617

Closed
sschuberth opened this issue Mar 6, 2023 · 4 comments
Closed
Assignees
Labels
enhancement Issues that are considered to be enhancements scanner About the scanner tool

Comments

@sschuberth
Copy link
Member

sschuberth commented Mar 6, 2023

Ok never mind, the issue is caused by scancode v32.x.x having a different output format, which ORT does not support yet.

Ping @maxhbr who I believe was experiencing something similar with the latest ScanCode pre-release. I'll create a new issue.

Originally posted by @sschuberth in #6585 (comment)

@sschuberth sschuberth added enhancement Issues that are considered to be enhancements scanner About the scanner tool labels Mar 6, 2023
@maxhbr
Copy link
Contributor

maxhbr commented Mar 6, 2023

My experienced behavior (not ran on my machine, but I did remote debugging):

using a ~2 week old ORT and https://github.com/nexB/scancode-toolkit/releases/tag/v32.0.0rc2 on windows, the scan result contained copyrights, but the licenses list was always [].
But there were no issues or problems reported and the whole pipeline just ran through silently swallowing all scanner license information.

@bennati
Copy link
Contributor

bennati commented Mar 6, 2023

The issue is caused by this line

val licenses = file["licenses"]?.asSequence().orEmpty()
which creates an empty sequence.
The reason is that the Scancode output format has changed from

{
      "path": "LICENSE",
      "type": "file",
      "licenses": [
        {
          "key": "here-proprietary",
          "score": 100.0,
          "name": "HERE Proprietary License",
          "short_name": "HERE Proprietary",
          "category": "Commercial",

To:

{
      "path": "LICENSE",
      "type": "file",
      ...
      "detected_license_expression": "here-proprietary",
      "detected_license_expression_spdx": "LicenseRef-scancode-here-proprietary",
      "license_detections": [
        {
          "license_expression": "here-proprietary",
          "matches": [
            {
              "score": 100.0,
              "start_line": 4,
              "end_line": 10,
              ...
              "license_expression": "here-proprietary",
              "rule_identifier": "here-proprietary.LICENSE",
              "rule_url": "https://github.com/nexB/scancode-toolkit/tree/develop/src/licensedcode/data/licenses/here-proprietary.LICENSE"
            }
          ]
        }
      ],

sschuberth added a commit that referenced this issue Mar 6, 2023
While parsing newer output format versions might not throw exceptions,
results may be incomplete (see #6617 for the context). Avoid confusion by
warning in that case.

Signed-off-by: Sebastian Schuberth <sschuberth@gmail.com>
sschuberth added a commit that referenced this issue Mar 6, 2023
While parsing newer output format versions might not throw exceptions,
results may be incomplete (see [1] for the context). Avoid confusion by
warning in that case.

[1]: #6617

Signed-off-by: Sebastian Schuberth <sschuberth@gmail.com>
sschuberth added a commit that referenced this issue Mar 6, 2023
While parsing newer output format versions might not throw exceptions,
results may be incomplete (see [1] for the context). Avoid confusion by
warning in that case.

[1]: #6617

Signed-off-by: Sebastian Schuberth <sschuberth@gmail.com>
sschuberth added a commit that referenced this issue Mar 6, 2023
While parsing newer output format versions might not throw exceptions,
results may be incomplete (see [1] for the context). Avoid confusion by
warning in that case.

[1]: #6617

Signed-off-by: Sebastian Schuberth <sschuberth@gmail.com>
sschuberth added a commit that referenced this issue Mar 6, 2023
While parsing newer output format versions might not throw exceptions,
results may be incomplete (see [1] for the context). Avoid confusion by
warning in that case.

[1]: #6617

Signed-off-by: Sebastian Schuberth <sschuberth@gmail.com>
@sschuberth
Copy link
Member Author

Work for this has been started in the scancode-format-3 branch.

@sschuberth sschuberth self-assigned this Jun 22, 2023
@sschuberth
Copy link
Member Author

The implementation is ready, but its merge is currently blocked by the discussion in #7300.

@sschuberth sschuberth changed the title Ensure ORT to handle the output format of the upcoming ScanCode version 32 series Ensure ORT to handle the output format 3 of the ScanCode 32 series Jul 19, 2023
sschuberth added a commit that referenced this issue Aug 21, 2023
While in previous outputs formats the primary elements for license
entries were single license keys (which needed to be grouped to
expressions), starting with output format version 3 the primary license
entries are expressions.

Resolves #6617.

Signed-off-by: Sebastian Schuberth <sschuberth@gmail.com>
sschuberth added a commit that referenced this issue Aug 31, 2023
While in previous outputs formats the primary elements for license
entries were single license keys (which needed to be grouped to
expressions), starting with output format version 3 the primary license
entries are expressions.

Resolves #6617.

Signed-off-by: Sebastian Schuberth <sschuberth@gmail.com>
sschuberth added a commit that referenced this issue Sep 1, 2023
While in previous outputs formats the primary elements for license
entries were single license keys (which needed to be grouped to
expressions), starting with output format version 3 the primary license
entries are expressions.

Resolves #6617.

Signed-off-by: Sebastian Schuberth <sschuberth@gmail.com>
sschuberth added a commit that referenced this issue Sep 1, 2023
While in previous outputs formats the primary elements for license
entries were single license keys (which needed to be grouped to
expressions), starting with output format version 3 the primary license
entries are expressions.

Resolves #6617.

Signed-off-by: Sebastian Schuberth <sschuberth@gmail.com>
sschuberth added a commit that referenced this issue Sep 1, 2023
While in previous output formats the primary elements for license
entries were single license keys (which needed to be grouped to
expressions), starting with output format version 3 the primary license
entries are expressions.

Resolves #6617.

Signed-off-by: Sebastian Schuberth <sschuberth@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement Issues that are considered to be enhancements scanner About the scanner tool
Projects
None yet
Development

No branches or pull requests

3 participants