Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

FileProvenanceFileStorage cannot find file listings #7041

Closed
MarcelBochtler opened this issue May 24, 2023 · 11 comments · Fixed by #9071
Closed

FileProvenanceFileStorage cannot find file listings #7041

MarcelBochtler opened this issue May 24, 2023 · 11 comments · Fixed by #9071
Labels
bug Issues that are considered to be bugs model About the data model scanner About the scanner tool

Comments

@MarcelBochtler
Copy link
Member

MarcelBochtler commented May 24, 2023 •
edited
Loading

Since #6970, we're seeing the following error in our scans:

ERROR org.ossreviewtoolkit.model.utils.FileProvenanceFileStorage - Could not read from 66eab51c0401c5041aa975dbf9c112cecfb9dfdd/file_list.xz: FileNotFoundException: /root/.ort/scanner/file-lists/66eab51c0401c5041aa975dbf9c112cecfb9dfdd/file_list.xz (No such file or directory)

We did not yet configure the postgres storage for the file listings, but from my understanding not configuring that, shouldn't lead to errors in the scan.

Edit: We're using ORT's legacy docker image for these scans.
@sschuberth sschuberth added bug Issues that are considered to be bugs model About the data model labels May 24, 2023
@sschuberth
Copy link
Member

@MarcelBochtler can this still be reproduced after the recent PRs #7119, #7121 and #7129 in that area?

@sschuberth sschuberth added scanner About the scanner tool needs info An issue where further information is required labels Jun 22, 2023
@MarcelBochtler
Copy link
Member Author

In the meantime we configured Postgres for the file lists, which solves this issue.

As a test I disabled it again, and yes, this can still be reproduced.

@sschuberth sschuberth removed the needs info An issue where further information is required label Jun 23, 2023
@alpianon
Copy link

I'm experiencing the same bug and using same workaround. ORT 19.1.0, built from sources with OpenJDK 11 on Ubuntu 22.04

@sschuberth
Copy link
Member

In any case, glad you're taking a look at ORT, @alpianon. Note that BitBake-related stuff is still in a fork only. Please ensure to ask question rather sooner than later, e.g. in Slack, as we're aware that ORT is not trivial to setup and configure, esp. with performance in mind.

@alpianon
Copy link

In any case, glad you're taking a look at ORT, @alpianon. Note that BitBake-related stuff is still in a fork only. Please ensure to ask question rather sooner than later, e.g. in Slack, as we're aware that ORT is not trivial to setup and configure, esp. with performance in mind.

I'm not working only on bitbake-based projects, now I have a java+npm project to scan :)

Glad that you started developing a bitbake analyzer. Maybe we can talk about it on slack

@sschuberth
Copy link
Member

Maybe we can talk about it on slack

We should absolutely talk! But I guess a dedicated meeting makes sense. I'll reach out to you privately.

@rettichschnidi
Copy link
Contributor

rettichschnidi commented Jun 7, 2024 •
edited
Loading

I am seeing this error too when doing the steps described in the tutorial:

$ ort -P ort.scanner.skipExcluded=true scan -i mime-types-ort-analyzer-results/analyzer-result.yml -o mime-types-ort-scanner-results/
 ______________________________
/        \_______   \__    ___/ The OSS Review Toolkit, version 22.0.0-049.sha.ed7fdbe,
|    |   | |       _/ |    |    built with JDK 11.0.22+7, running under Java 17.0.11.
|    |   | |    |   \ |    |    Executing 'scan' as 'username' on Linux
\________/ |____|___/ |____|    with 24 CPUs and a maximum of 24128 MiB of memory.

Environment variables:
ORT_CONFIG_DIR = /home/username/.ort/config
ORT_DATA_DIR = /home/username/.ort
HOME = /home/username
SHELL = /bin/bash
TERM = xterm-256color
GOPATH = /home/username/opt/go

Looking for ORT configuration in the following file:
        /home/username/.ort/config/config.yml (does not exist)

Scanning projects with:
        ScanCode (version 32.1.0)
Scanning packages with:
        ScanCode (version 32.1.0)
01:24:09.973 [DefaultDispatcher-worker-3] ERROR org.ossreviewtoolkit.model.utils.FileProvenanceFileStorage - Could not read from 39f76b06552989aaa626341d3c3936645bad36b3/file_list: FileNotFoundException: /home/username/.ort/scanner/file-lists/39f76b06552989aaa626341d3c3936645bad36b3/file_list (No such file or directory)
Caused by: FileNotFoundException: /home/username/.ort/scanner/file-lists/39f76b06552989aaa626341d3c3936645bad36b3/file_list.xz (No such file or directory)
01:24:31.163 [DefaultDispatcher-worker-3] ERROR org.ossreviewtoolkit.model.utils.FileProvenanceFileStorage - Could not read from c643dce0af603a22b7be53afb0844ce302fa93e6/file_list: FileNotFoundException: /home/username/.ort/scanner/file-lists/c643dce0af603a22b7be53afb0844ce302fa93e6/file_list (No such file or directory)
Caused by: FileNotFoundException: /home/username/.ort/scanner/file-lists/c643dce0af603a22b7be53afb0844ce302fa93e6/file_list.xz (No such file or directory)
Wrote scan result to '/home/username/some-path/ort-example-mime-types/mime-types-ort-scanner-results/scan-result.yml' (0.42 MiB) in 133.975115ms.
The scan took 39.348319185s.
Resolved issues: 0 errors, 0 warnings, 0 hints.
Unresolved issues: 0 errors, 0 warnings, 0 hints.

But when running the same command a 2nd time the errors are gone, as the mentioned files have been created during the first run:

$ ort -P ort.scanner.skipExcluded=true scan -i mime-types-ort-analyzer-results/analyzer-result.yml -o mime-types-ort-scanner-results/
 ______________________________
/        \_______   \__    ___/ The OSS Review Toolkit, version 22.0.0-049.sha.ed7fdbe,
|    |   | |       _/ |    |    built with JDK 11.0.22+7, running under Java 17.0.11.
|    |   | |    |   \ |    |    Executing 'scan' as 'username' on Linux
\________/ |____|___/ |____|    with 24 CPUs and a maximum of 24128 MiB of memory.

Environment variables:
ORT_CONFIG_DIR = /home/username/.ort/config
ORT_DATA_DIR = /home/username/.ort
HOME = /home/username
SHELL = /bin/bash
TERM = xterm-256color
GOPATH = /home/username/opt/go

Looking for ORT configuration in the following file:
        /home/username/.ort/config/config.yml (does not exist)

Scanning projects with:
        ScanCode (version 32.1.0)
Scanning packages with:
        ScanCode (version 32.1.0)
Wrote scan result to '/home/username/some-path/ort-example-mime-types/mime-types-ort-scanner-results/scan-result.yml' (0.42 MiB) in 209.344576ms.
The scan took 3.591540482s.
Resolved issues: 0 errors, 0 warnings, 0 hints.
Unresolved issues: 0 errors, 0 warnings, 0 hints.

@sschuberth
Copy link
Member

Are there any files present below /home/username/.ort/scanner/file-lists? I'm curious whether files or only being looked up under the wrong location maybe, or whether they're not being created in the first place.

@rettichschnidi
Copy link
Contributor

rettichschnidi commented Jun 11, 2024 •
edited
Loading

Are there any files present below /home/username/.ort/scanner/file-lists?

Yes, but not the ones ort scan was not able to find. Those files however exist after ort scan ran (and complained about them being missed).

I can reproduce the error like this:

  1. rm -rf ~/.ort/
  2. ort -P ort.scanner.skipExcluded=true scan -i mime-types-ort-analyzer-results/analyzer-result.yml -o mime-types-ort-scanner-results

Running ort -P ort.scanner.skipExcluded=true scan -i mime-types-ort-analyzer-results/analyzer-result.yml -o mime-types-ort-scanner-results a 2nd time will then be much faster and does no longer show this error.

@reuvenperetz
Copy link

Hi, is there any update about this issue? I'm experiencing the same problem...

mnonnenmacher added a commit that referenced this issue Sep 3, 2024
@mnonnenmacher
feat(model): Check if an archive exists before trying to download it
cdc6586 
Check if an archive exists in the storage before trying to download it
in `FileArchiver.unarchive` and `FileListResolver.resolve`.

This prevents an error message when calling `storage.getData()` and the
data does not exist.

Resolves #7041.

Signed-off-by: Martin Nonnenmacher <martin.nonnenmacher@bosch.com>
@mnonnenmacher mnonnenmacher mentioned this issue Sep 3, 2024
Merged
mnonnenmacher added a commit that referenced this issue Sep 3, 2024
@mnonnenmacher
feat(model): Check if an archive exists before trying to download it
3c53ae0 
Check if an archive exists in the storage before trying to read it in
`FileArchiver.unarchive` and `FileListResolver.resolve`.

This prevents an error message when calling `storage.getData()` and the
data does not exist.

Resolves #7041.

Signed-off-by: Martin Nonnenmacher <martin.nonnenmacher@bosch.com>
@mnonnenmacher
Copy link
Member

The error message was a false positive. It was not properly checked if the file list archive exists before trying to download it.

When the archive does not exist yet, it will be created, so no error message should be logged in this case.

mnonnenmacher added a commit that referenced this issue Sep 3, 2024
@mnonnenmacher
feat(model): Check if an archive exists before trying to download it
a5f784d 
Check if an archive exists in the storage before trying to read it in
`FileArchiver.unarchive` and `FileListResolver.resolve`.

This prevents an error message when calling `storage.getData()` and the
data does not exist.

Resolves #7041.

Signed-off-by: Martin Nonnenmacher <martin.nonnenmacher@bosch.com>
mnonnenmacher added a commit that referenced this issue Sep 3, 2024
@mnonnenmacher
feat(model): Check if an archive exists before trying to download it
bc6785c 
Check if an archive exists in the storage before trying to read it in
`FileArchiver.unarchive` and `FileListResolver.resolve`.

This prevents an error message when calling `storage.getData()` and the
data does not exist.

Resolves #7041.

Signed-off-by: Martin Nonnenmacher <martin.nonnenmacher@bosch.com>
sschuberth pushed a commit that referenced this issue Sep 3, 2024
@mnonnenmacher @sschuberth
feat(model): Check if an archive exists before trying to download it
29a108a 
Check if an archive exists in the storage before trying to read it in
`FileArchiver.unarchive` and `FileListResolver.resolve`.

This prevents an error message when calling `storage.getData()` and the
data does not exist.

Resolves #7041.

Signed-off-by: Martin Nonnenmacher <martin.nonnenmacher@bosch.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug Issues that are considered to be bugs model About the data model scanner About the scanner tool
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

feat(model): Check if an archive exists before trying to download it oss-review-toolkit/ort
6 participants
@sschuberth @rettichschnidi @mnonnenmacher @alpianon @MarcelBochtler @reuvenperetz