Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add k8s.io/release/pkg/license package #1874

Merged
merged 4 commits into from
Jan 27, 2021
Merged

Conversation

puerco
Copy link
Member

@puerco puerco commented Jan 26, 2021

What type of PR is this?

/kind feature

What this PR does / why we need it:

This PR adds a new license package which, as of now, includes a new license reader
that scans a directory and identifies license files located in it.

The main source consists of three parts:
* The license reader code
* The SPDX functions and data structures
* A downloader object that gets and caches license data from spdx.org

An example test program (see bellow for sample output):

reader, err := license.NewReader(opts)
if err != nil {
        logrus.Fatal(err)
}
licenses, _, err := reader.ReadLicenses("kubernetes/vendor/")
if err != nil {
        logrus.Fatal(err)
}
logrus.Infof("License scanner returned %d licenses", len(licenses))
for _, l := range licenses {
        logrus.WithFields(logrus.Fields{"license": l.License.LicenseID}).Info(l.File)
}

After trying several solutions and even writing my own, I settled on using google's license classifier which turned out to be the most practical to use as well as the fastest and most accurate.

Which issue(s) this PR fixes:

Part of #1837

Special notes for your reviewer:

Tests and mock implementations are included. One test file contains unit tests that run in -package while the other has all unit and integration tests that run in its own package (license_test)

Does this PR introduce a user-facing change?

New `license` package adds the capability to scan source directories, locate license files and classify them to match OSS licenses in the SPDX catalog.

@k8s-ci-robot k8s-ci-robot added release-note Denotes a PR that will be considered when it comes time to generate release notes. kind/feature Categorizes issue or PR as related to a new feature. cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. needs-priority size/XXL Denotes a PR that changes 1000+ lines, ignoring generated files. labels Jan 26, 2021
@k8s-ci-robot k8s-ci-robot added area/release-eng Issues or PRs related to the Release Engineering subproject sig/release Categorizes an issue or PR as relevant to SIG Release. approved Indicates a PR has been approved by an approver from all required OWNERS files. labels Jan 26, 2021
@puerco puerco mentioned this pull request Jan 26, 2021
19 tasks
@puerco
Copy link
Member Author

puerco commented Jan 26, 2021

Example run of test script (with licenses cached):

release ]$ time go run tmp/test-license.go
INFO[0000] Loading license data from downloader         
INFO[0000] Downloading main SPDX license data           
INFO[0000] Read data for 448 licenses. Downloading.     
INFO[0000] SPDX: Got 448 licenses from downloader       
INFO[0000] Writing SPDX licenses to /tmp/license-reader-021240259/licenses 
INFO[0002] Scanning kubernetes/vendor/ for license files 
INFO[0002] 234 license files found in directory         
WARN[0002] File does not match a known license: kubernetes/vendor/github.com/JeffAshton/win_pdh/LICENSE 
WARN[0002] File does not match a known license: kubernetes/vendor/github.com/NYTimes/gziphandler/LICENSE.md 
WARN[0002] File does not match a known license: kubernetes/vendor/github.com/go-bindata/go-bindata/LICENSE 
WARN[0003] File does not match a known license: kubernetes/vendor/github.com/heketi/heketi/LICENSE 
WARN[0003] File does not match a known license: kubernetes/vendor/github.com/inconshreveable/mousetrap/LICENSE 
WARN[0003] File does not match a known license: kubernetes/vendor/github.com/jmespath/go-jmespath/LICENSE 
WARN[0003] File does not match a known license: kubernetes/vendor/github.com/pmezard/go-difflib/LICENSE 
WARN[0003] File does not match a known license: kubernetes/vendor/go.uber.org/zap/check_license.sh 
INFO[0004] License classifier recognized 226/234 (0%) os the files 
INFO[0004] License scanner returned 226 licenses        
INFO[0004] kubernetes/vendor/bitbucket.org/bertimus9/systemstat/LICENSE  license=MIT
INFO[0004] kubernetes/vendor/cloud.google.com/go/LICENSE  license=Apache-2.0
INFO[0004] kubernetes/vendor/github.com/Azure/azure-sdk-for-go/LICENSE  license=Apache-2.0
INFO[0004] kubernetes/vendor/github.com/Azure/go-ansiterm/LICENSE  license=MIT
INFO[0004] kubernetes/vendor/github.com/Azure/go-autorest/LICENSE  license=Apache-2.0
INFO[0004] kubernetes/vendor/github.com/Azure/go-autorest/autorest/LICENSE  license=Apache-2.0
INFO[0004] kubernetes/vendor/github.com/Azure/go-autorest/autorest/adal/LICENSE  license=Apache-2.0
INFO[0004] kubernetes/vendor/github.com/Azure/go-autorest/autorest/date/LICENSE  license=Apache-2.0
INFO[0004] kubernetes/vendor/github.com/Azure/go-autorest/autorest/mocks/LICENSE  license=Apache-2.0
INFO[0004] kubernetes/vendor/github.com/Azure/go-autorest/autorest/to/LICENSE  license=Apache-2.0
INFO[0004] kubernetes/vendor/github.com/Azure/go-autorest/autorest/validation/LICENSE  license=Apache-2.0
INFO[0004] kubernetes/vendor/github.com/Azure/go-autorest/logger/LICENSE  license=Apache-2.0
INFO[0004] kubernetes/vendor/github.com/Azure/go-autorest/tracing/LICENSE  license=Apache-2.0
INFO[0004] kubernetes/vendor/github.com/GoogleCloudPlatform/k8s-cloud-provider/LICENSE  license=Apache-2.0
INFO[0004] kubernetes/vendor/github.com/MakeNowJust/heredoc/LICENSE  license=MIT
INFO[0004] kubernetes/vendor/github.com/Microsoft/go-winio/LICENSE  license=MIT
INFO[0004] kubernetes/vendor/github.com/Microsoft/hcsshim/LICENSE  license=MIT
INFO[0004] kubernetes/vendor/github.com/PuerkitoBio/purell/LICENSE  license=BSD-3-Clause
INFO[0004] kubernetes/vendor/github.com/PuerkitoBio/urlesc/LICENSE  license=BSD-3-Clause
INFO[0004] kubernetes/vendor/github.com/armon/circbuf/LICENSE  license=MIT
INFO[0004] kubernetes/vendor/github.com/asaskevich/govalidator/LICENSE  license=MIT
INFO[0004] kubernetes/vendor/github.com/aws/aws-sdk-go/LICENSE.txt  license=Apache-2.0
INFO[0004] kubernetes/vendor/github.com/aws/aws-sdk-go/internal/sync/singleflight/LICENSE  license=BSD-3-Clause
INFO[0004] kubernetes/vendor/github.com/beorn7/perks/LICENSE  license=MIT
INFO[0004] kubernetes/vendor/github.com/blang/semver/LICENSE  license=MIT
INFO[0004] kubernetes/vendor/github.com/caddyserver/caddy/LICENSE.txt  license=Apache-2.0
INFO[0004] kubernetes/vendor/github.com/cespare/xxhash/v2/LICENSE.txt  license=MIT
INFO[0004] kubernetes/vendor/github.com/chai2010/gettext-go/LICENSE  license=BSD-3-Clause
INFO[0004] kubernetes/vendor/github.com/checkpoint-restore/go-criu/v4/LICENSE  license=Apache-2.0
INFO[0004] kubernetes/vendor/github.com/cilium/ebpf/LICENSE  license=MIT
INFO[0004] kubernetes/vendor/github.com/clusterhq/flocker-go/LICENSE  license=Apache-2.0
INFO[0004] kubernetes/vendor/github.com/container-storage-interface/spec/LICENSE  license=Apache-2.0
INFO[0004] kubernetes/vendor/github.com/containerd/cgroups/LICENSE  license=Apache-2.0
INFO[0004] kubernetes/vendor/github.com/containerd/console/LICENSE  license=Apache-2.0
INFO[0004] kubernetes/vendor/github.com/containerd/containerd/LICENSE  license=Apache-2.0
INFO[0004] kubernetes/vendor/github.com/containerd/ttrpc/LICENSE  license=Apache-2.0
INFO[0004] kubernetes/vendor/github.com/containernetworking/cni/LICENSE  license=Apache-2.0
INFO[0004] kubernetes/vendor/github.com/coredns/corefile-migration/LICENSE  license=Apache-2.0
INFO[0004] kubernetes/vendor/github.com/coreos/go-oidc/LICENSE  license=Apache-2.0
INFO[0004] kubernetes/vendor/github.com/coreos/go-semver/LICENSE  license=Apache-2.0
INFO[0004] kubernetes/vendor/github.com/coreos/go-systemd/LICENSE  license=Apache-2.0
INFO[0004] kubernetes/vendor/github.com/coreos/go-systemd/v22/LICENSE  license=Apache-2.0
INFO[0004] kubernetes/vendor/github.com/coreos/pkg/LICENSE  license=Apache-2.0
INFO[0004] kubernetes/vendor/github.com/cpuguy83/go-md2man/v2/LICENSE.md  license=MIT
INFO[0004] kubernetes/vendor/github.com/cyphar/filepath-securejoin/LICENSE  license=BSD-3-Clause
INFO[0004] kubernetes/vendor/github.com/davecgh/go-spew/LICENSE  license=ISC
INFO[0004] kubernetes/vendor/github.com/daviddengcn/go-colortext/LICENSE  license=MIT
INFO[0004] kubernetes/vendor/github.com/dgrijalva/jwt-go/LICENSE  license=MIT
INFO[0004] kubernetes/vendor/github.com/docker/distribution/LICENSE  license=Apache-2.0
INFO[0004] kubernetes/vendor/github.com/docker/docker/LICENSE  license=Apache-2.0
INFO[0004] kubernetes/vendor/github.com/docker/go-connections/LICENSE  license=Apache-2.0
INFO[0004] kubernetes/vendor/github.com/docker/go-units/LICENSE  license=Apache-2.0
INFO[0004] kubernetes/vendor/github.com/docker/spdystream/LICENSE  license=Apache-2.0
INFO[0004] kubernetes/vendor/github.com/docker/spdystream/LICENSE.docs  license=CC-BY-SA-4.0
INFO[0004] kubernetes/vendor/github.com/dustin/go-humanize/LICENSE  license=MIT
INFO[0004] kubernetes/vendor/github.com/elazarl/goproxy/LICENSE  license=BSD-3-Clause
INFO[0004] kubernetes/vendor/github.com/emicklei/go-restful/LICENSE  license=MIT
INFO[0004] kubernetes/vendor/github.com/euank/go-kmsg-parser/LICENSE  license=Apache-2.0
INFO[0004] kubernetes/vendor/github.com/evanphx/json-patch/LICENSE  license=BSD-3-Clause
INFO[0004] kubernetes/vendor/github.com/exponent-io/jsonpath/LICENSE  license=MIT
INFO[0004] kubernetes/vendor/github.com/fatih/camelcase/LICENSE.md  license=MIT
INFO[0004] kubernetes/vendor/github.com/form3tech-oss/jwt-go/LICENSE  license=MIT
INFO[0004] kubernetes/vendor/github.com/fsnotify/fsnotify/LICENSE  license=BSD-3-Clause
INFO[0004] kubernetes/vendor/github.com/fvbommel/sortorder/LICENSE  license=MIT
INFO[0004] kubernetes/vendor/github.com/ghodss/yaml/LICENSE  license=MIT
INFO[0004] kubernetes/vendor/github.com/go-logr/logr/LICENSE  license=Apache-2.0
INFO[0004] kubernetes/vendor/github.com/go-openapi/analysis/LICENSE  license=Apache-2.0
INFO[0004] kubernetes/vendor/github.com/go-openapi/errors/LICENSE  license=Apache-2.0
INFO[0004] kubernetes/vendor/github.com/go-openapi/jsonpointer/LICENSE  license=Apache-2.0
INFO[0004] kubernetes/vendor/github.com/go-openapi/jsonreference/LICENSE  license=Apache-2.0
INFO[0004] kubernetes/vendor/github.com/go-openapi/loads/LICENSE  license=Apache-2.0
INFO[0004] kubernetes/vendor/github.com/go-openapi/runtime/LICENSE  license=Apache-2.0
INFO[0004] kubernetes/vendor/github.com/go-openapi/spec/LICENSE  license=Apache-2.0
INFO[0004] kubernetes/vendor/github.com/go-openapi/strfmt/LICENSE  license=Apache-2.0
INFO[0004] kubernetes/vendor/github.com/go-openapi/swag/LICENSE  license=Apache-2.0
INFO[0004] kubernetes/vendor/github.com/go-openapi/validate/LICENSE  license=Apache-2.0
INFO[0004] kubernetes/vendor/github.com/go-ozzo/ozzo-validation/LICENSE  license=MIT
INFO[0004] kubernetes/vendor/github.com/go-stack/stack/LICENSE.md  license=MIT
INFO[0004] kubernetes/vendor/github.com/godbus/dbus/v5/LICENSE  license=BSD-2-Clause
INFO[0004] kubernetes/vendor/github.com/gogo/protobuf/LICENSE  license=BSD-3-Clause
INFO[0004] kubernetes/vendor/github.com/golang/groupcache/LICENSE  license=Apache-2.0
INFO[0004] kubernetes/vendor/github.com/golang/mock/LICENSE  license=Apache-2.0
INFO[0004] kubernetes/vendor/github.com/golang/protobuf/LICENSE  license=BSD-3-Clause
INFO[0004] kubernetes/vendor/github.com/google/btree/LICENSE  license=Apache-2.0
INFO[0004] kubernetes/vendor/github.com/google/cadvisor/LICENSE  license=Apache-2.0
INFO[0004] kubernetes/vendor/github.com/google/go-cmp/LICENSE  license=BSD-3-Clause
INFO[0004] kubernetes/vendor/github.com/google/gofuzz/LICENSE  license=Apache-2.0
INFO[0004] kubernetes/vendor/github.com/google/uuid/LICENSE  license=BSD-3-Clause
INFO[0004] kubernetes/vendor/github.com/googleapis/gax-go/v2/LICENSE  license=BSD-3-Clause
INFO[0004] kubernetes/vendor/github.com/googleapis/gnostic/LICENSE  license=Apache-2.0
INFO[0004] kubernetes/vendor/github.com/gophercloud/gophercloud/LICENSE  license=Apache-2.0
INFO[0004] kubernetes/vendor/github.com/gorilla/websocket/LICENSE  license=BSD-2-Clause
INFO[0004] kubernetes/vendor/github.com/gregjones/httpcache/LICENSE.txt  license=MIT
INFO[0004] kubernetes/vendor/github.com/grpc-ecosystem/go-grpc-middleware/LICENSE  license=Apache-2.0
INFO[0004] kubernetes/vendor/github.com/grpc-ecosystem/go-grpc-prometheus/LICENSE  license=Apache-2.0
INFO[0004] kubernetes/vendor/github.com/grpc-ecosystem/grpc-gateway/LICENSE.txt  license=BSD-3-Clause
INFO[0004] kubernetes/vendor/github.com/hashicorp/golang-lru/LICENSE  license=MPL-2.0-no-copyleft-exception
INFO[0004] kubernetes/vendor/github.com/hashicorp/hcl/LICENSE  license=MPL-2.0
INFO[0004] kubernetes/vendor/github.com/heketi/heketi/LICENSE-APACHE2  license=Apache-2.0
INFO[0004] kubernetes/vendor/github.com/hpcloud/tail/LICENSE.txt  license=MIT
INFO[0004] kubernetes/vendor/github.com/imdario/mergo/LICENSE  license=BSD-3-Clause
INFO[0004] kubernetes/vendor/github.com/ishidawataru/sctp/GO_LICENSE  license=BSD-3-Clause
INFO[0004] kubernetes/vendor/github.com/ishidawataru/sctp/LICENSE  license=Apache-2.0
INFO[0004] kubernetes/vendor/github.com/jonboulle/clockwork/LICENSE  license=Apache-2.0
INFO[0004] kubernetes/vendor/github.com/json-iterator/go/LICENSE  license=MIT
INFO[0004] kubernetes/vendor/github.com/karrick/godirwalk/LICENSE  license=BSD-2-Clause
INFO[0004] kubernetes/vendor/github.com/konsorten/go-windows-terminal-sequences/LICENSE  license=MIT
INFO[0004] kubernetes/vendor/github.com/libopenstorage/openstorage/LICENSE  license=Apache-2.0
INFO[0004] kubernetes/vendor/github.com/liggitt/tabwriter/LICENSE  license=BSD-3-Clause
INFO[0004] kubernetes/vendor/github.com/lithammer/dedent/LICENSE  license=MIT
INFO[0004] kubernetes/vendor/github.com/magiconair/properties/LICENSE  license=BSD-2-Clause
INFO[0004] kubernetes/vendor/github.com/mailru/easyjson/LICENSE  license=MIT
INFO[0004] kubernetes/vendor/github.com/matttproud/golang_protobuf_extensions/LICENSE  license=Apache-2.0
INFO[0004] kubernetes/vendor/github.com/miekg/dns/LICENSE  license=BSD-3-Clause
INFO[0004] kubernetes/vendor/github.com/mindprince/gonvml/LICENSE  license=Apache-2.0
INFO[0004] kubernetes/vendor/github.com/mistifyio/go-zfs/LICENSE  license=Apache-2.0
INFO[0004] kubernetes/vendor/github.com/mitchellh/go-wordwrap/LICENSE.md  license=MIT
INFO[0004] kubernetes/vendor/github.com/mitchellh/mapstructure/LICENSE  license=MIT
INFO[0004] kubernetes/vendor/github.com/moby/ipvs/LICENSE  license=Apache-2.0
INFO[0004] kubernetes/vendor/github.com/moby/sys/mountinfo/LICENSE  license=Apache-2.0
INFO[0004] kubernetes/vendor/github.com/moby/term/LICENSE  license=Apache-2.0
INFO[0004] kubernetes/vendor/github.com/modern-go/concurrent/LICENSE  license=Apache-2.0
INFO[0004] kubernetes/vendor/github.com/modern-go/reflect2/LICENSE  license=Apache-2.0
INFO[0004] kubernetes/vendor/github.com/mohae/deepcopy/LICENSE  license=MIT
INFO[0004] kubernetes/vendor/github.com/morikuni/aec/LICENSE  license=MIT
INFO[0004] kubernetes/vendor/github.com/mrunalp/fileutils/LICENSE  license=Apache-2.0
INFO[0004] kubernetes/vendor/github.com/munnerz/goautoneg/LICENSE  license=BSD-3-Clause
INFO[0004] kubernetes/vendor/github.com/mvdan/xurls/LICENSE  license=BSD-3-Clause
INFO[0004] kubernetes/vendor/github.com/mxk/go-flowrate/LICENSE  license=BSD-3-Clause
INFO[0004] kubernetes/vendor/github.com/onsi/ginkgo/LICENSE  license=MIT
INFO[0004] kubernetes/vendor/github.com/onsi/ginkgo/reporters/stenographer/support/go-colorable/LICENSE  license=MIT
INFO[0004] kubernetes/vendor/github.com/onsi/ginkgo/reporters/stenographer/support/go-isatty/LICENSE  license=MIT
INFO[0004] kubernetes/vendor/github.com/onsi/gomega/LICENSE  license=MIT
INFO[0004] kubernetes/vendor/github.com/opencontainers/go-digest/LICENSE  license=Apache-2.0
INFO[0004] kubernetes/vendor/github.com/opencontainers/go-digest/LICENSE.docs  license=CC-BY-SA-4.0
INFO[0004] kubernetes/vendor/github.com/opencontainers/image-spec/LICENSE  license=Apache-2.0
INFO[0004] kubernetes/vendor/github.com/opencontainers/runc/LICENSE  license=Apache-2.0
INFO[0004] kubernetes/vendor/github.com/opencontainers/runtime-spec/LICENSE  license=Apache-2.0
INFO[0004] kubernetes/vendor/github.com/opencontainers/selinux/LICENSE  license=Apache-2.0
INFO[0004] kubernetes/vendor/github.com/pelletier/go-toml/LICENSE  license=MIT
INFO[0004] kubernetes/vendor/github.com/peterbourgon/diskv/LICENSE  license=MIT
INFO[0004] kubernetes/vendor/github.com/pkg/errors/LICENSE  license=BSD-2-Clause
INFO[0004] kubernetes/vendor/github.com/pquerna/cachecontrol/LICENSE  license=Apache-2.0
INFO[0004] kubernetes/vendor/github.com/prometheus/client_golang/LICENSE  license=Apache-2.0
INFO[0004] kubernetes/vendor/github.com/prometheus/client_model/LICENSE  license=Apache-2.0
INFO[0004] kubernetes/vendor/github.com/prometheus/common/LICENSE  license=Apache-2.0
INFO[0004] kubernetes/vendor/github.com/prometheus/procfs/LICENSE  license=Apache-2.0
INFO[0004] kubernetes/vendor/github.com/quobyte/api/LICENSE  license=BSD-3-Clause
INFO[0004] kubernetes/vendor/github.com/robfig/cron/LICENSE  license=MIT
INFO[0004] kubernetes/vendor/github.com/rubiojr/go-vhd/LICENSE  license=MIT
INFO[0004] kubernetes/vendor/github.com/russross/blackfriday/LICENSE.txt  license=BSD-2-Clause
INFO[0004] kubernetes/vendor/github.com/russross/blackfriday/v2/LICENSE.txt  license=BSD-2-Clause
INFO[0004] kubernetes/vendor/github.com/satori/go.uuid/LICENSE  license=MIT
INFO[0004] kubernetes/vendor/github.com/seccomp/libseccomp-golang/LICENSE  license=BSD-2-Clause
INFO[0004] kubernetes/vendor/github.com/shurcooL/sanitized_anchor_name/LICENSE  license=MIT
INFO[0004] kubernetes/vendor/github.com/sirupsen/logrus/LICENSE  license=MIT
INFO[0004] kubernetes/vendor/github.com/soheilhy/cmux/LICENSE  license=Apache-2.0
INFO[0004] kubernetes/vendor/github.com/spf13/afero/LICENSE.txt  license=Apache-2.0
INFO[0004] kubernetes/vendor/github.com/spf13/cast/LICENSE  license=MIT
INFO[0004] kubernetes/vendor/github.com/spf13/cobra/LICENSE.txt  license=Apache-2.0
INFO[0004] kubernetes/vendor/github.com/spf13/jwalterweatherman/LICENSE  license=MIT
INFO[0004] kubernetes/vendor/github.com/spf13/pflag/LICENSE  license=BSD-3-Clause
INFO[0004] kubernetes/vendor/github.com/spf13/viper/LICENSE  license=MIT
INFO[0004] kubernetes/vendor/github.com/stretchr/objx/LICENSE  license=MIT
INFO[0004] kubernetes/vendor/github.com/stretchr/testify/LICENSE  license=MIT
INFO[0004] kubernetes/vendor/github.com/subosito/gotenv/LICENSE  license=MIT
INFO[0004] kubernetes/vendor/github.com/syndtr/gocapability/LICENSE  license=BSD-2-Clause
INFO[0004] kubernetes/vendor/github.com/thecodeteam/goscaleio/LICENSE  license=Apache-2.0
INFO[0004] kubernetes/vendor/github.com/tmc/grpc-websocket-proxy/LICENSE  license=MIT
INFO[0004] kubernetes/vendor/github.com/vishvananda/netlink/LICENSE  license=Apache-2.0
INFO[0004] kubernetes/vendor/github.com/vishvananda/netns/LICENSE  license=Apache-2.0
INFO[0004] kubernetes/vendor/github.com/vmware/govmomi/LICENSE.txt  license=Apache-2.0
INFO[0004] kubernetes/vendor/github.com/vmware/govmomi/vim25/xml/LICENSE  license=BSD-3-Clause
INFO[0004] kubernetes/vendor/github.com/willf/bitset/LICENSE  license=BSD-3-Clause
INFO[0004] kubernetes/vendor/github.com/xiang90/probing/LICENSE  license=MIT
INFO[0004] kubernetes/vendor/go.etcd.io/bbolt/LICENSE  license=MIT
INFO[0004] kubernetes/vendor/go.etcd.io/etcd/LICENSE  license=Apache-2.0
INFO[0004] kubernetes/vendor/go.mongodb.org/mongo-driver/LICENSE  license=Apache-2.0
INFO[0004] kubernetes/vendor/go.opencensus.io/LICENSE  license=Apache-2.0
INFO[0004] kubernetes/vendor/go.uber.org/atomic/LICENSE.txt  license=MIT
INFO[0004] kubernetes/vendor/go.uber.org/multierr/LICENSE.txt  license=MIT
INFO[0004] kubernetes/vendor/go.uber.org/zap/LICENSE.txt  license=MIT
INFO[0004] kubernetes/vendor/golang.org/x/crypto/LICENSE  license=BSD-3-Clause
INFO[0004] kubernetes/vendor/golang.org/x/mod/LICENSE  license=BSD-3-Clause
INFO[0004] kubernetes/vendor/golang.org/x/net/LICENSE  license=BSD-3-Clause
INFO[0004] kubernetes/vendor/golang.org/x/oauth2/LICENSE  license=BSD-3-Clause
INFO[0004] kubernetes/vendor/golang.org/x/sync/LICENSE  license=BSD-3-Clause
INFO[0004] kubernetes/vendor/golang.org/x/sys/LICENSE  license=BSD-3-Clause
INFO[0004] kubernetes/vendor/golang.org/x/text/LICENSE  license=BSD-3-Clause
INFO[0004] kubernetes/vendor/golang.org/x/time/LICENSE  license=BSD-3-Clause
INFO[0004] kubernetes/vendor/golang.org/x/tools/LICENSE  license=BSD-3-Clause
INFO[0004] kubernetes/vendor/golang.org/x/xerrors/LICENSE  license=BSD-3-Clause
INFO[0004] kubernetes/vendor/gonum.org/v1/gonum/LICENSE  license=BSD-3-Clause
INFO[0004] kubernetes/vendor/google.golang.org/api/LICENSE  license=BSD-3-Clause
INFO[0004] kubernetes/vendor/google.golang.org/api/internal/third_party/uritemplates/LICENSE  license=BSD-3-Clause
INFO[0004] kubernetes/vendor/google.golang.org/appengine/LICENSE  license=Apache-2.0
INFO[0004] kubernetes/vendor/google.golang.org/genproto/LICENSE  license=Apache-2.0
INFO[0004] kubernetes/vendor/google.golang.org/grpc/LICENSE  license=Apache-2.0
INFO[0004] kubernetes/vendor/google.golang.org/protobuf/LICENSE  license=BSD-3-Clause
INFO[0004] kubernetes/vendor/gopkg.in/fsnotify.v1/LICENSE  license=BSD-3-Clause
INFO[0004] kubernetes/vendor/gopkg.in/gcfg.v1/LICENSE  license=BSD-3-Clause
INFO[0004] kubernetes/vendor/gopkg.in/inf.v0/LICENSE  license=BSD-3-Clause
INFO[0004] kubernetes/vendor/gopkg.in/ini.v1/LICENSE  license=Apache-2.0
INFO[0004] kubernetes/vendor/gopkg.in/natefinch/lumberjack.v2/LICENSE  license=MIT
INFO[0004] kubernetes/vendor/gopkg.in/square/go-jose.v2/LICENSE  license=Apache-2.0
INFO[0004] kubernetes/vendor/gopkg.in/square/go-jose.v2/json/LICENSE  license=BSD-3-Clause
INFO[0004] kubernetes/vendor/gopkg.in/tomb.v1/LICENSE  license=BSD-3-Clause
INFO[0004] kubernetes/vendor/gopkg.in/warnings.v0/LICENSE  license=BSD-2-Clause
INFO[0004] kubernetes/vendor/gopkg.in/yaml.v2/LICENSE  license=Apache-2.0
INFO[0004] kubernetes/vendor/gopkg.in/yaml.v2/LICENSE.libyaml  license=MIT
INFO[0004] kubernetes/vendor/gopkg.in/yaml.v3/LICENSE  license=MIT
INFO[0004] kubernetes/vendor/k8s.io/gengo/LICENSE  license=Apache-2.0
INFO[0004] kubernetes/vendor/k8s.io/heapster/LICENSE  license=Apache-2.0
INFO[0004] kubernetes/vendor/k8s.io/klog/v2/LICENSE  license=Apache-2.0
INFO[0004] kubernetes/vendor/k8s.io/kube-openapi/LICENSE  license=Apache-2.0
INFO[0004] kubernetes/vendor/k8s.io/kube-openapi/pkg/validation/errors/LICENSE  license=Apache-2.0
INFO[0004] kubernetes/vendor/k8s.io/kube-openapi/pkg/validation/spec/LICENSE  license=Apache-2.0
INFO[0004] kubernetes/vendor/k8s.io/kube-openapi/pkg/validation/strfmt/LICENSE  license=Apache-2.0
INFO[0004] kubernetes/vendor/k8s.io/kube-openapi/pkg/validation/validate/LICENSE  license=Apache-2.0
INFO[0004] kubernetes/vendor/k8s.io/system-validators/LICENSE  license=Apache-2.0
INFO[0004] kubernetes/vendor/k8s.io/utils/LICENSE  license=Apache-2.0
INFO[0004] kubernetes/vendor/k8s.io/utils/inotify/LICENSE  license=BSD-3-Clause
INFO[0004] kubernetes/vendor/sigs.k8s.io/apiserver-network-proxy/konnectivity-client/LICENSE  license=Apache-2.0
INFO[0004] kubernetes/vendor/sigs.k8s.io/kustomize/LICENSE  license=Apache-2.0
INFO[0004] kubernetes/vendor/sigs.k8s.io/structured-merge-diff/v4/LICENSE  license=Apache-2.0
INFO[0004] kubernetes/vendor/sigs.k8s.io/yaml/LICENSE  license=MIT

real	0m4.778s
user	0m6.080s
sys	0m0.449s

@puerco
Copy link
Member Author

puerco commented Jan 26, 2021

Running times without licenses pre cached:

real	0m35.891s
user	0m6.940s
sys	0m0.981s

pkg/license/download.go Outdated Show resolved Hide resolved
pkg/license/download.go Outdated Show resolved Hide resolved
// Get the license corresponding to the ID label
license := d.spdx.GetLicense(label)
if license == nil {
return nil, unrecognizedPaths, errors.Wrapf(nil, "ID does not correspond to a valid license: %s", label)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Wrapping a nil error will always result in nil.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ooof nice catch. Thanks!

This commit adds a new `license` package which as if now includes
a new license reader that identifies licenses in a source code directory.

The source consists of three parts:

* The license reader code
* The SPDX functions and data structures
* A downloader object that gets and caches license data from spdx.org

Signed-off-by: Adolfo García Veytia (Puerco) <adolfo.garcia@uservers.net>
This commit contains the autogenerated fakes to mock the
implementations in the license package

Signed-off-by: Adolfo García Veytia (Puerco) <adolfo.garcia@uservers.net>
Tests for the license package. They are split in a couple of unit tests
performed in package and the rest, in their own package (license_test)

Signed-off-by: Adolfo García Veytia (Puerco) <adolfo.garcia@uservers.net>
Signed-off-by: Adolfo García Veytia (Puerco) <adolfo.garcia@uservers.net>
Copy link
Member

@saschagrunert saschagrunert left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

/lgtm

@k8s-ci-robot k8s-ci-robot added the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Jan 27, 2021
@k8s-ci-robot
Copy link
Contributor

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: puerco, saschagrunert

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:
  • OWNERS [puerco,saschagrunert]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@k8s-ci-robot k8s-ci-robot merged commit 1056280 into kubernetes:master Jan 27, 2021
@k8s-ci-robot k8s-ci-robot added this to the v1.21 milestone Jan 27, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
approved Indicates a PR has been approved by an approver from all required OWNERS files. area/release-eng Issues or PRs related to the Release Engineering subproject cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. kind/feature Categorizes issue or PR as related to a new feature. lgtm "Looks good to me", indicates that a PR is ready to be merged. needs-priority release-note Denotes a PR that will be considered when it comes time to generate release notes. sig/release Categorizes an issue or PR as relevant to SIG Release. size/XXL Denotes a PR that changes 1000+ lines, ignoring generated files.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants