Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: support composite-based package overrides #1214

Merged
merged 12 commits into from
Sep 9, 2024

Conversation

G-Rath
Copy link
Collaborator

@G-Rath G-Rath commented Aug 29, 2024

This rewrites the package overrides logic to be composition based, granting a lot more flexibility:

# ignore everything
[[PackageOverrides]]
ignore = true

# ignore everything in this group
[[PackageOverrides]]
group = "dev"
ignore = true

# ignore everything in this ecosystem
[[PackageOverrides]]
ecosystem = "go"
ignore = true

# ignore all packages named "axios" regardless of ecosystem or group
[[PackageOverrides]]
name = "axios"
ignore = true

# ignore all packages named "axios" in the npm ecosystem that are in the dev group
[[PackageOverrides]]
name = "axios"
ecosystem = "npm"
group = "dev"
ignore = true

# ... and so on

While some of these might seem a bit extreme, ultimately I think this is probably the way to go as the logic itself is very straightforward and it gives a lot more power to the people.

Since config is a public package, I've had to deprecated the related existing public methods and there's a bit of naming & structural yuck but I figure that's not a big deal since v2 is right around the corner and again the logic itself is very straightforward.

Resolves #1211
Resolves #1155

@codecov-commenter
Copy link

codecov-commenter commented Aug 29, 2024

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 67.78%. Comparing base (1c086df) to head (b8f6209).

Additional details and impacted files
@@            Coverage Diff             @@
##             main    #1214      +/-   ##
==========================================
+ Coverage   67.64%   67.78%   +0.13%     
==========================================
  Files         174      174              
  Lines       16721    16759      +38     
==========================================
+ Hits        11311    11360      +49     
+ Misses       4778     4770       -8     
+ Partials      632      629       -3     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

@G-Rath G-Rath force-pushed the support-ignoring-groups branch 2 times, most recently from 72de007 to df9ea40 Compare August 29, 2024 20:13
@G-Rath G-Rath changed the title feat: support ignoring packages only in specific groups feat: support composite-based package overrides Aug 29, 2024
@G-Rath
Copy link
Collaborator Author

G-Rath commented Aug 29, 2024

(note that the docs should probably be updated, but I want to get confirmation that we're happy with this general change first)

@G-Rath G-Rath marked this pull request as ready for review August 29, 2024 20:26
@G-Rath
Copy link
Collaborator Author

G-Rath commented Aug 29, 2024

Note that this should also be usable to resolve #1124 - I think technically you could do that right now by e.g. "consider all packages in the test group as having a license we're happy with" but it might be better to have a special "skip" type value that does actually mean the overridden package is completely omitted from license checking code paths...

Copy link
Contributor

@cuixq cuixq left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM!

Copy link
Collaborator

@another-rex another-rex left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM! good idea making it more general. I'm not sure about using the models.PackageVulns type here, but that's probably something to consider in the v2 rework of the types, (we have too many types that roughly represent the same thing currently), so happy to merge as is.

@@ -967,7 +967,17 @@ func filterIgnoredPackages(r reporter.Reporter, packages []scannedPackage, confi
out := make([]scannedPackage, 0, len(packages))
for _, p := range packages {
configToUse := configManager.Get(r, p.Source.Path)
if ignore, ignoreLine := configToUse.ShouldIgnorePackageVersion(p.Name, p.Version, string(p.Ecosystem)); ignore {
if ignore, ignoreLine := configToUse.ShouldIgnorePackage(
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: Separate this out from the if statement as it is quite long, and just have it on it's own line.

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

done - I went with keeping the function in the if and using a variable for the struct as I think that's a bit more correct: in the long-run, we expect to be passing a whole "thing" to this function (like what we're already doing in vulnerability_result.go), so it's the models.PackageVuln part that's a temp workaround

@G-Rath G-Rath force-pushed the support-ignoring-groups branch from dc31d47 to 7979aa2 Compare September 2, 2024 22:11
docs/configuration.md Outdated Show resolved Hide resolved
```toml
# ignore everything
[[PackageOverrides]]
ignore = true
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is the use case for this just scanning for licenses? In that case can we specify that in the comment above.

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm not quite sure what you mean - I've only changed the logic around when an override should be applied to a particular package, not the supported "actions"; so the behaviour should be whatever ignore did previously...?

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

fwiw, my expectation is that this enables #1155

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I meant why someone would want to ignore everything. This was before I remembered that it only applies to the file in the same directory, so this will be pretty useful. Can you change the comment above this to specify that this is ignoring everything in the same directory?

Copy link
Collaborator

@another-rex another-rex Sep 5, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

(we should probably also change ignore to ignoreVulns, and deprecate ignore, as it is ambiguous as to whether we are still doing license scanning (we are) (turns out it does not), we can do that in a separate PR though. I'll make an issue for this.)

@G-Rath
Copy link
Collaborator Author

G-Rath commented Sep 3, 2024

@another-rex

I'm not sure about using the models.PackageVulns type here, but that's probably something to consider in the v2 rework of the types, (we have too many types that roughly represent the same thing currently), so happy to merge as is.

I think we're on exactly the same page - overall, my view here is naming wise this suffers because of already known issues which are a major epic to be addressed in v2 and that the more important thing is this change introduces a showcase of how the configuration could be improved that is very useful to have landed before v2 so it can be part of shaping v2 itself.

A good example of that imo is with files: it could be very reasonable to add the ability to ignore specific files but that's hard because models.PackageVulns doesn't have that information - so if the new interface(s) for v2 included that (which iirc I think it will?), we'd be able to very easily support that as a field here.

@G-Rath G-Rath force-pushed the support-ignoring-groups branch 2 times, most recently from 98a1141 to 0791430 Compare September 8, 2024 21:08
@G-Rath G-Rath force-pushed the support-ignoring-groups branch from 0791430 to b8f6209 Compare September 9, 2024 03:41
@another-rex another-rex merged commit 0cd2051 into google:main Sep 9, 2024
13 checks passed
@another-rex another-rex deleted the support-ignoring-groups branch September 9, 2024 04:31
another-rex added a commit that referenced this pull request Oct 22, 2024
… our osv-scanner.toml in fixtures (#1337)

Uses the new feature landed in #1214
another-rex added a commit that referenced this pull request Nov 29, 2024
… our osv-scanner.toml in fixtures (#1337)

Uses the new feature landed in #1214
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
4 participants