helper-cli: Add a command to generate a flat analyzer result

[fviernau]

The command defines a data model called SimpleDefinitionFile which is
read as input using any of the known file extensions. It outputs an
analyzer result with a single project containing the provided
dependencies as a flat list.

When a project does not use a package manager, currently the only way of
scanning it with ORT is to craft SPDX files and running the analyzer
against them. This command tries to provide a minimalistic alternative
to fullfil the requirements of the underlying need.

Note: This is a first experimental version. Furthermore, VcsInfo is deliberately not used, in order to:

      1. avoid nesting to in turn make it easier to generate the input file.
      2. make all attributes optional.

[codecov]

Codecov Report

Patch coverage: 88.54% and project coverage change: +7.07% 🎉

Comparison is base (72a5644) 60.93% compared to head (3f8030f) 68.01%.

Additional details and impacted files

@@             Coverage Diff              @@
##               main    #7305      +/-   ##
============================================
+ Coverage     60.93%   68.01%   +7.07%     
- Complexity     1968     2023      +55     
============================================
  Files           338      339       +1     
  Lines         16623    16719      +96     
  Branches       2363     2371       +8     
============================================
+ Hits          10129    11371    +1242     
+ Misses         5516     4363    -1153     
- Partials        978      985       +7     

Flag	Coverage Δ
funTest-docker	69.33% <ø> (ø)
funTest-non-docker	36.46% <88.54%> (+6.98%)	⬆️
test	36.10% <0.00%> (-0.40%)	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

Files Changed	Coverage Δ
...ands/CreateAnalyzerResultFromPackageListCommand.kt	88.54% <88.54%> (ø)

... and 52 files with indirect coverage changes

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[sschuberth]

This command tries to provide a minimalistic alternative
to fullfil the requirements of the underlying need.

Typo in "fullfil", should be "fulfill".

In general this sentence is a bit cryptic. Please be transparent about what the "underlying need" is and document it in the commit message.

[sschuberth]

currently the only way of
scanning it with ORT is to craft SPDX files and running the analyzer
against them.

I want to start by saying that I agree that writing / generating SPDX files as a substitute for a lack of supported definition files for ORT is cumbersome, mostly because SPDX is such a terrible format IMO. Maybe we should have instead used AboutCode Data or an ORT-specific "fake definition file" format that's easier to use than writing whole analyzer result files from hand.

That said, I believe this goal of this PR should be implemented differently: As a company-internal package manager plugin that uses your SimpleDefinitionFile as definition files. How do @mnonnenmacher and @MarcelBochtler think about this?

[fviernau]

That said, I believe this goal of this PR should be implemented differently: As a company-internal package manager plugin that uses your SimpleDefinitionFile as definition files

1. I thought it makes sense to try it out as helper-cli command first before turning it into a package manager plugin, as I believe it is quicker to develop and more light-weight. It is also slightly different than package managers, as it is limited to a single project and it does not require any cloning of any repo for the analysis. Anyhow, I planned to refactor this in future when it matured a bit, into a package manager (disabled by default) if it turns out to make sense.
2. Why do you think it should remain company internal? Do you think nobody would be interrested in using it in the long run?

[sschuberth]

2. Why do you think it should remain company internal?

Mostly because of the "I believe it is quicker to develop" rationale you mentioned: Doing this as a company-internal package manager plugin (which is something e.g. Bosch has done, too) allows you to fulfill your custom needs quicker without the need for justifications during a review process. And you still keep the option to open-source it later on.

Do you think nobody would be interrested in using it in the long run?

Hard to say for the long term if more features / generalizations get added, but in its current shape it indeed looks like a very specific solution that probably no one except EPAM is interested in.

[fviernau]

Mostly because of the "I believe it is quicker to develop" rationale you mentioned

I think we've added "experimental" things to ORT / helper-cli already before this PR.

[sschuberth]

Mostly because of the "I believe it is quicker to develop" rationale you mentioned

I think we've added "experimental" things to ORT / helper-cli already before this PR.

Right, but this is not about the level of "experimentality", but about where the code fits, whatever the maturity is. And to me a dedicated package manager so clearly is a better fit (if I'm getting the use-case right), that personally I'd draw a line here. Otherwise Bosch, Porsche and others could have contributed their plugins also as helper-cli commands (but we've turned them down).

[tsteenbe]

Frank forgot to mention we also want to use this new helper functionality to make it more easier to implement org-wide inbound FOSS scanning or pre-scanning.

Org-wide inbound FOSS scanning is something we started working whilst at HERE, the idea is to scan a FOSS package the moment it's introduced into the organization.

Scenario: Developer does dependencies update in his/her SW project -> packages are being fetched from internal package registry (JFrog Artifactory) -> Artifactory does not have version so retrieves package from public package registry (Maven Central, npmjs.com) and at the same time trigger inbound FOSS scan for each new version via REST API providing package url and VCS url -> inbound FOSS scan executes helper cli command to generate analyzer file defined in this PR -> Generated analyzer result is used to execute ORT scan with ScanCode -> Scan results are added to defined ORT scan storage -> when developer create pull/merge request ORT scan is done and new version of dependency is already scanned -> scan is quick resulting in better user experience and engineer productivity. If new version or package is not OK by ORT policy rules than one can configure Artifactory to block download pointing to the ORT report for details.

Similarly one can also implement pre-scanning - from Artifactory usage stats one can deduct most used FOSS package and then logic can be implement to trigger inbound FOSS ORT scan the moment package popular within the org release a new version.

[sschuberth]

inbound FOSS scan executes helper cli command to generate analyzer file defined in this PR

How's the required SimpleDefinitionFile input provided in that scenario?

[mnonnenmacher]

That said, I believe this goal of this PR should be implemented differently: As a company-internal package manager plugin that uses your SimpleDefinitionFile as definition files. How do @mnonnenmacher and @MarcelBochtler think about this?

That workflow makes sense to me, we have used it at Bosch as well, for example to implement and test snippet reporter related changes internally before contributing them upstream. This provides a lot of freedom for experiments (because no code reviews are required, for example) and makes it easier to reach a mature state that is worth contributing.

I think we've added "experimental" things to ORT / helper-cli already before this PR.

Overall I am not very happy with the state of the helper-cli, because also experimental commands still have to be maintained by the community, and because there are almost no tests. Adding more experiments doesn't improve the situation. Also, I have no idea how many of the commands are still used by anyone or if we maintain them just for the sake of it.

Why do you think it should remain company internal? Do you think nobody would be interested in using it in the long run?

At Bosch we have a very strong requirement for an alternative to the SPDX analyzer, because it makes some very unintuitive assumptions, it feels like 90% of the SPDX that has to be written is boilerplate, and it also does not support all ORT features. We have that topic on our Agenda for later this year, I would appreciate if we could discuss it in a community meeting after the summer break to collect requirements.

[tsteenbe]

inbound FOSS scan executes helper cli command to generate analyzer file defined in this PR

How's the required SimpleDefinitionFile input provided in that scenario?

SimpleDefinitionFile can be used to batch scanning multiple packages, say for instance we need to scan an update of framework or SDK consisting out of multiple packages each with their own code repository. Also from infrastructure efficiency/limits perspective batching scan jobs together where possible makes sense to A) reduce impact of overhead of starting ORT Docker container and B) stay within limit of execution environment (# of concurrrent jobs / job pool availability).

[fviernau]

I would appreciate if we could discuss it in a community meeting after the summer break to collect requirements.

I needed something quick in place to help during the next couple of weeks. I understand you guys don't want that, I'll just turn the PR into a draft and keep the branch around for while.

[fviernau]

I've added a test which tests end to end through helper main, which illustrates the input format (and the expected output).

[sschuberth]

Please note the detekt issues (and rebase to get rid of an unrelated test failure).

[sschuberth]

Very nice to have this example now as part of the tests, thank you! This makes some stuff immediately visible to me that I wouldn't have noticed otherwise.

[sschuberth]

Interesting: We usually do not support the exclusion of a dependency. I need to look further down to see how this is implemented then.

[fviernau]

There is one scope for excluded dependencies and another for non-excluded ones.

[sschuberth]

Is there a chance that we might want to support sub-projects as dependencies? If so, I'd prefer this to be a regular PackageLinkage enum value.

[fviernau]

This command targets the use case of doing a quick (as possible) scan and clearance for an arbitrary set of packages. In that use case it is unneeded. I also do not foresee to have subprojects.

From our discussion in the developer meeting, what I understood was, (from @mnonnenmacher) there is need for an analyzer which is a bit simpler than SPDX.
And that he preferred to have my simple use case be a helper-cli command, so that this simple use case would not conflict with @mnonnenmacher analyzer use case. So, given that I prefer we keep this helper-command to a minimum and add the more complex cases (like sub projects) to @mnonnenmacher 's analyzer use case if it fits there?!

Addressed.