Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Feature: Introduce golangci-lint #77

Open
wants to merge 9 commits into
base: main
Choose a base branch
from

Conversation

Okabe-Junya
Copy link
Contributor

What

  • Introduce golangci-lint
    • Add a configuration to start small and simple
  • Change the linter executed by make lint-go from go vet to golangci-lint

Background

When hacking on this codebase, I noticed that the (Go) linter run by pre-commit was failing like this:

$ go vet ./...                   
# github.com/GoogleCloudPlatform/khi/pkg/source/gcp/query/queryutil
# [github.com/GoogleCloudPlatform/khi/pkg/source/gcp/query/queryutil]
pkg/source/gcp/query/queryutil/parallel.go:91:2: the cancel function is not used on all paths (possible context leak)
pkg/source/gcp/query/queryutil/parallel.go:140:3: this return statement may be reached without using the cancel var defined on line 91

In addition, to detect several potential bugs in the codebase, I want to introduce a more comprehensive linter (or linter runner), such as golangci-lint.

Results

$ make lint-go
golangci-lint run
execution result
pkg/common/errorreport/reporter.go:37:15: fieldalignment: struct with 48 pointer bytes could be 24 (govet)
type Reporter struct {
              ^
pkg/common/errorreport/reporter_test.go:35:17: fieldalignment: struct with 32 pointer bytes could be 24 (govet)
        testCases := []struct {
                       ^
pkg/inspection/inspectiondata/result_repository.go:36:22: fieldalignment: struct with 32 pointer bytes could be 16 (govet)
type FileSystemStore struct {
                     ^
pkg/source/gcp/task/gke/gke_audit/parser.go:173:2: ifElseChain: rewrite if-else to switch statement (gocritic)
        if isFirst && !isLast {
        ^
pkg/testutil/log/log.go:29:28: fieldalignment: struct with 64 pointer bytes could be 48 (govet)
type mockLogFieldExtractor struct {
                           ^
pkg/testutil/log/log.go:88:9: nilness: impossible condition: nil != nil (govet)
        if err != nil {
               ^
pkg/common/collection.go:32:27: unlambda: replace `func(a, b string) int { return strings.Compare(a, b) }` with `strings.Compare` (gocritic)
        slices.SortFunc(deduped, func(a, b string) int { return strings.Compare(a, b) })
                                 ^
pkg/common/time_test.go:25:17: fieldalignment: struct with 40 pointer bytes could be 32 (govet)
        testCases := []struct {
                       ^
pkg/common/collection.go:54:4: shadow: declaration of "m" shadows declaration at line 37 (govet)
                        m := min(d[i-1][j]+1, d[i][j-1]+1)
                        ^
pkg/inspection/logger/format.go:105:27: commentFormatting: put a space between `//` and comment text (gocritic)
                colorBegin = "\033[90m" //bright black
                                        ^
pkg/inspection/logger/format.go:108:27: commentFormatting: put a space between `//` and comment text (gocritic)
                colorBegin = "\033[96m" //bright cyan
                                        ^
pkg/inspection/logger/format.go:111:27: commentFormatting: put a space between `//` and comment text (gocritic)
                colorBegin = "\033[93m" //bright yellow
                                        ^
pkg/inspection/logger/format.go:42:26: fieldalignment: struct with 32 pointer bytes could be 24 (govet)
type KHILogFormatHandler struct {
                         ^
pkg/inspection/logger/logger.go:32:26: fieldalignment: struct with 64 pointer bytes could be 56 (govet)
type globalLoggerHandler struct {
                         ^
pkg/source/gcp/task/cloud-composer/paeser_test.go:26:17: fieldalignment: struct with 40 pointer bytes could be 32 (govet)
        testCases := []struct {
                       ^
pkg/source/gcp/task/cloud-composer/paeser_test.go:122:17: fieldalignment: struct with 40 pointer bytes could be 32 (govet)
        testCases := []struct {
                       ^
pkg/testutil/util_test.go:78:29: response body must be closed (bodyclose)
                        got := ResponseFromString(tt.args.code, tt.args.response)
                                                 ^
pkg/testutil/util_test.go:56:12: fieldalignment: struct with 16 pointer bytes could be 8 (govet)
        type args struct {
                  ^
pkg/testutil/io.go:34:9: shadow: declaration of "err" shadows declaration at line 30 (govet)
                if _, err := os.Stat(filepath.Join(cwd, ROOT_INDICATOR_FILE)); err == nil {
                      ^
scripts/frontend-codegen/main.go:52:6: shadow: declaration of "templateInput" shadows declaration at line 41 (govet)
        var templateInput templateInput = templateInput{
            ^
pkg/log/cached_log_field_extractor.go:26:30: fieldalignment: struct with 112 pointer bytes could be 96 (govet)
type CachedLogFieldExtractor struct {
                             ^
pkg/log/cached_log_field_extractor_test.go:24:34: fieldalignment: struct with 80 pointer bytes could be 64 (govet)
type commonLogFieldExtractorStub struct {
                                 ^
pkg/task/definition_set.go:198:12: elseif: can replace 'else {if cond {}}' with 'else if cond {}' (gocritic)
                                } else {
                                       ^
pkg/task/definition_set.go:361:26: unlambda: replace `func(a, b string) int { return strings.Compare(a, b) }` with `strings.Compare` (gocritic)
        slices.SortFunc(result, func(a, b string) int { return strings.Compare(a, b) })
                                ^
pkg/task/definition.go:78:29: fieldalignment: struct with 72 pointer bytes could be 56 (govet)
type ConstantDefinitionImpl struct {
                            ^
pkg/task/label.go:56:17: fieldalignment: struct with 32 pointer bytes could be 24 (govet)
type labelAdder struct {
                ^
pkg/task/label.go:98:23: fieldalignment: struct with 40 pointer bytes could be 24 (govet)
type equalLabelFilter struct {
                      ^
pkg/task/runner.go:36:18: fieldalignment: struct with 88 pointer bytes could be 64 (govet)
type LocalRunner struct {
                 ^
pkg/task/runner.go:47:26: fieldalignment: struct with 80 pointer bytes could be 72 (govet)
type LocalRunnerTaskStat struct {
                         ^
pkg/task/cached_processor_test.go:42:18: fieldalignment: struct with 16 pointer bytes could be 8 (govet)
        taskSeries := []struct {
                        ^
pkg/task/definition_set_test.go:28:25: fieldalignment: struct with 80 pointer bytes could be 64 (govet)
type testTaskDefinition struct {
                        ^
pkg/task/processor_test.go:85:17: fieldalignment: struct with 48 pointer bytes could be 32 (govet)
        cmpValues := []struct {
                       ^
pkg/task/runner_test.go:27:20: fieldalignment: struct with 24 pointer bytes could be 8 (govet)
type debugRunnable struct {
                   ^
pkg/inspection/metadata/form/form.go:35:16: fieldalignment: struct with 160 pointer bytes could be 136 (govet)
type FormField struct {
               ^
pkg/inspection/metadata/form/form.go:50:19: fieldalignment: struct with 32 pointer bytes could be 8 (govet)
type FormFieldSet struct {
                  ^
pkg/inspection/metadata/header/header.go:27:13: fieldalignment: struct with 64 pointer bytes could be 40 (govet)
type Header struct {
            ^
pkg/log/structure/merger/merged.go:646:16: appendAssign: append result not assigned to the same slice (gocritic)
                                liveList = append(liveOnlyList, directiveElem)
                                           ^
pkg/log/structure/merger/merged.go:95:2: ifElseChain: rewrite if-else to switch statement (gocritic)
        if ty == structuredata.StructuredTypeMap {
        ^
pkg/log/structure/merger/merged.go:217:3: ifElseChain: rewrite if-else to switch statement (gocritic)
                if prevFound && patchFound {
                ^
pkg/log/structure/merger/merged.go:54:35: fieldalignment: struct with 80 pointer bytes could be 72 (govet)
type StrategicMergedStructureData struct {
                                  ^
pkg/log/structure/merger/keydiff_test.go:24:16: fieldalignment: struct with 72 pointer bytes could be 56 (govet)
        testCase := []struct {
                      ^
pkg/log/structure/merger/merged_test.go:112:17: fieldalignment: struct with 96 pointer bytes could be 80 (govet)
        testCases := []struct {
                       ^
pkg/log/structure/merger/merged_test.go:458:17: fieldalignment: struct with 72 pointer bytes could be 64 (govet)
        testCases := []struct {
                       ^
pkg/log/structure/merger/merged_test.go:1110:17: fieldalignment: struct with 48 pointer bytes could be 32 (govet)
        testCases := []struct {
                       ^
pkg/log/structure/merger/merged.go:335:11: shadow: declaration of "err" shadows declaration at line 309 (govet)
                                vany, err := d.prev.Value(key)
                                      ^
pkg/log/structure/merger/merged.go:349:14: shadow: declaration of "err" shadows declaration at line 309 (govet)
                        keyValue, err := d.readScalar(field, mergeKey)
                                  ^
pkg/log/structure/merger/merged.go:363:14: shadow: declaration of "err" shadows declaration at line 309 (govet)
                        keyValue, err := d.readScalar(field, mergeKey)
                                  ^
pkg/log/structure/merger/merged_test.go:436:12: shadow: declaration of "err" shadows declaration at line 422 (govet)
                                child, err := merged.Value(tc.childField)
                                       ^
pkg/source/gcp/task/gke/k8s_event/parser.go:70:9: elseif: can replace 'else {if cond {}}' with 'else if cond {}' (gocritic)
        } else {
               ^
pkg/source/gcp/task/gke/k8s_event/query_test.go:27:17: fieldalignment: struct with 104 pointer bytes could be 96 (govet)
        testCases := []struct {
                       ^
pkg/source/gcp/task/gke/k8s_event/query_test.go:60:17: fieldalignment: struct with 56 pointer bytes could be 48 (govet)
        testCases := []struct {
                       ^
pkg/source/gcp/task/gke/k8s_event/parser.go:63:19: shadow: declaration of "err" shadows declaration at line 61 (govet)
                if textPayload, err := l.GetString("textPayload"); err == nil {
                                ^
pkg/model/k8s.go:83:28: fieldalignment: struct with 64 pointer bytes could be 56 (govet)
type EndpointSliceEndpoint struct {
                           ^
pkg/model/k8s.go:91:20: fieldalignment: struct with 32 pointer bytes could be 16 (govet)
type EndpointSlice struct {
                   ^
pkg/model/enum/parent_relationship.go:36:41: fieldalignment: struct with 80 pointer bytes could be 72 (govet)
type ParentRelationshipFrontendMetadata struct {
                                        ^
pkg/source/gcp/task/gke/k8s_audit/recorder/endpointslicerecorder/recorder.go:36:32: fieldalignment: struct with 24 pointer bytes could be 8 (govet)
type singleEndpointParseResult struct {
                               ^
pkg/source/gcp/task/gke/k8s_audit/recorder/endpointslicerecorder/recorder.go:45:27: fieldalignment: struct with 24 pointer bytes could be 8 (govet)
type endpointsParseResult struct {
                          ^
cmd/kubernetes-history-inspector/main.go:104:3: exitAfterDefer: os.Exit will exit, and `defer errorreport.CheckAndReportPanic()` will not run (gocritic)
                os.Exit(1)
                ^
cmd/kubernetes-history-inspector/main.go:115:6: shadow: declaration of "err" shadows declaration at line 101 (govet)
                if err := profiler.Start(cfg); err != nil {
                   ^
cmd/kubernetes-history-inspector/main.go:154:4: shadow: declaration of "err" shadows declaration at line 101 (govet)
                        err := accesstoken.DefaultOAuthTokenResolver.SetServer(engine)
                        ^
cmd/kubernetes-history-inspector/main.go:194:23: shadow: declaration of "err" shadows declaration at line 179 (govet)
                        availableFeatures, err := t.FeatureList()
                                           ^
pkg/model/history/history.go:55:2: deprecatedComment: use `Deprecated: ` (note the casing) instead of `DEPRECATED: ` (gocritic)
        // DEPRECATED: This field is no longer used. Will be removed in near future.
        ^
pkg/model/history/builder.go:43:14: fieldalignment: struct with 72 pointer bytes could be 64 (govet)
type Builder struct {
             ^
pkg/model/history/changeset.go:29:16: fieldalignment: struct with 88 pointer bytes could be 64 (govet)
type ChangeSet struct {
               ^
pkg/model/history/resourceinfo/endpoint_slices.go:163:11: elseif: can replace 'else {if cond {}}' with 'else if cond {}' (gocritic)
                        } else {
                               ^
pkg/model/history/resourceinfo/endpoint_slices.go:104:6: S1009: should omit nil check; len() for nil slices is defined as zero (gosimple)
                if current.Endpoints == nil || len(current.Endpoints) == 0 {
                   ^
pkg/model/binarychunk/builder_test.go:85:6: ineffectual assignment to err (ineffassign)
                _, err := b.Build(context.Background(), &result, progress.NewTaskProgress("foo"))
                   ^
pkg/model/binarychunk/builder_test.go:107:18: ineffectual assignment to err (ineffassign)
                        decompressed, err := io.ReadAll(gzipReader)
                                      ^
pkg/testutil/task/util.go:23:1: deprecatedComment: the proper format is `Deprecated: <text>` (gocritic)
// Deprecated. Use testtask package instead.
^
pkg/testutil/task/util.go:30:1: deprecatedComment: the proper format is `Deprecated: <text>` (gocritic)
// Deprecated. Use testtask package instead.
^
pkg/source/gcp/api/quotaproject/header_provider_test.go:48:31: should rewrite http.NewRequestWithContext or add (*Request).WithContext (noctx)
                        req, err := http.NewRequest(http.MethodGet, "https://example.com", nil)
                                                   ^
pkg/inspection/server.go:87:30: appendAssign: append result not assigned to the same slice (gocritic)
        inspectionTypesCandidate := append(s.inspectionTypes, &newInspectionType)
                                    ^
pkg/source/gcp/task/multicloud_api/parser.go:74:2: ineffectual assignment to operationResourcePath (ineffassign)
        operationResourcePath := resourcepath.ResourcePath{}
        ^
pkg/server/server_test.go:698:29: should rewrite http.NewRequestWithContext or add (*Request).WithContext (noctx)
                        req, _ := http.NewRequest(step.RequestMethod, path, requestReader)
                                                 ^
pkg/server/server_test.go:782:29: should rewrite http.NewRequestWithContext or add (*Request).WithContext (noctx)
                        req, _ := http.NewRequest(tc.requestMethod, tc.requestPath, bytes.NewReader([]byte{}))
                                                 ^
pkg/common/httpclient/basic_http_client_test.go:51:36: response body must be closed (bodyclose)
                resp, err := client.DoWithContext(context.Background(), req)
                                                 ^
pkg/common/httpclient/basic_http_client_test.go:89:36: response body must be closed (bodyclose)
                resp, err := client.DoWithContext(context.Background(), req)
                                                 ^
pkg/source/gcp/query/queryutil/set_filter.go:130:36: unlambda: replace `func(a, b string) int { return strings.Compare(a, b) }` with `strings.Compare` (gocritic)
        slices.SortFunc(result.Additives, func(a, b string) int { return strings.Compare(a, b) })
                                          ^
make: *** [lint-go] Error 1

Next steps

  • Fix linter errors
  • Run the linter in CI

@Okabe-Junya Okabe-Junya marked this pull request as ready for review February 19, 2025 16:37
@kyasbal kyasbal requested a review from jyane February 20, 2025 05:54
@kyasbal
Copy link
Member

kyasbal commented Feb 20, 2025

@jyane I'm not super familiar with linting around Golang. Do you have any opinion or comments about this PR?

@jyane
Copy link
Collaborator

jyane commented Feb 20, 2025

I'm also not familar with the tools, and I don't have any preference.

My 2 cents is that:

  1. The errcheck will produce 150 extra lines, could have more diffs that makes "git blame" dirty.
  2. I'm not really sure how long the linter keep common, changing the linter again could produce more diffs.
  3. The lint prevents some error-prone implementations.
$ golangci-lint run | wc -l
234
// after enabling errcheck
$ golangci-lint run | wc -l
384

@Okabe-Junya
Copy link
Contributor Author

The errcheck will produce 150 extra lines, could have more diffs that makes "git blame" dirty.

I completely agree. I also had concerns about enabling errcheck.
Anyway, errcheck probably shouldn’t be enabled; I disabled it in f040de0.

I'm not really sure how long the linter keep common, changing the linter again could produce more diffs.
The lint prevents some error-prone implementations.

As you mentioned, it's a trade-off between the diffs needed to fix linter errors and improving the overall quality of the codebase.
However, I think that the sooner we introduce the linter, the more effective it will be and the easier the fixes will become.

TBH, I don't have any particular preference for golangci-lint or individual linters (and their configurations) too;
I simply want to continuously improve the quality of the codebase - As can be seen from the results of golangci-lint, only go vet feels a bit lacking.

If there are any exemplary configurations from your company or other OSS projects, we might consider adopting those settings.

@kyasbal kyasbal self-requested a review February 23, 2025 01:42
@kyasbal
Copy link
Member

kyasbal commented Feb 23, 2025

I simply want to continuously improve the quality of the codebase - As can be seen from the results of golangci-lint, only go vet feels a bit lacking.

I agree with it. I think it's would be better to have the linter.

If there are any exemplary configurations from your company or other OSS projects, we might consider adopting those settings.

We have the Go style guide but it doesn't limit us to use the other linter tool. And golangci-lint itself seems not a linter, but it contains multiple linter including the one provided from the official Golang. The direction to use golangci-lint won't be against our policies around Golang.


Overall, I agree with using golangci-lint. But I want to merge PRs into main once it can pass the CIs.

Can you fix the CI failing? If it shows a lot of errors and takes time, I'll fix these lint errors in this PR if you want.

@kyasbal kyasbal removed the request for review from jyane February 23, 2025 01:57
Okabe-Junya and others added 3 commits February 23, 2025 22:51
Signed-off-by: Junya Okabe <86868255+Okabe-Junya@users.noreply.github.com>
@Okabe-Junya
Copy link
Contributor Author

Thank you, I fixed some CI errors (excluding the linter error fixes) (200d287, 9b20056),

And I fixed some linter errors in 03c4a3e, but it seems there are still many errors – many of which are related to fieldalignment and shadow.

I don’t have time to fix these immediately, but if you can wait a little longer – about 1~2 weeks – I can address them.

Of course, if you decide to work with them sooner, it’s perfectly fine to merge this PR. In that case, there is a workaround to avoid CI failure by setting the exit code of golangci-lint to 0.

FYI: https://golangci-lint.run/usage/configuration/#run-configuration

Thanks!

Comment on lines -21 to -26
// RawToken is the actual token in string representation.
RawToken string
// ValidAtLeastUntil holds the expiration time when the information is available.
// The token refreshers won't refresh token if this value exists and later than now.
// The default value will be `January 1, year 1, 00:00:00.000000000 UTC` and it is earlier than the possible time.Now().
// The default value naturally means the token is expired.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could you keep the comments?

@@ -52,12 +52,9 @@ type InspectionRunResult struct {

// InspectionTaskServer manages tasks and provides apis to get task related information in JSON convertible type.
type InspectionTaskServer struct {
// rootTaskSet is the set of the all definitions in KHI.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please keep comments on fields

State enum.RevisionState `json:"state"`

// DEPRECATED: This field is no longer used. Will be removed in near future.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Keep comment

Comment on lines -64 to +62
// Common fields assigned by log entity
Timestamp time.Time `json:"ts"`
// ID is an actual unique ID of this log. This field must be unique. KHI uses `insertId`-`timestamp` for GCP log.
ID string `json:"id"`
// Display ID is a log ID directly visible to user. This field no need to be unique. KHI uses `insertId` for GCP log.
DisplayId string `json:"displayId"`
Body *binarychunk.BinaryReference `json:"body"`

// These fields are managed by each parsers
Type enum.LogType `json:"type"`
Timestamp time.Time `json:"ts"`
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Keep comments?

// records lease history of NEG id to ServiceNetworkEndpointGroup
NEGs *resourcelease.ResourceLeaseHistory[*resourcelease.K8sResourceLeaseHolder]

nodeNames map[string]struct{}
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Keep comments

// Type of input field. Currently, only `text` or `popup_redirect` is the supported value.
Type string
// Description of this form.
Options map[string]string `json:"options"`
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Keep commnts

Comment on lines 38 to +39
Subtractives []string
ValidationError string
// The default mode is additive mode. The empty filter means none should be selected.
// SubtractMode will match anything. The empty filter means anything should be selected, this field will be true when user specify @any in the field.
SubtractMode bool
SubtractMode bool
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Keep comments

Comment on lines -26 to -47
// Current Log
Log *log.LogEntity
// ResourceName field of the log
ResourceName string
// MethodName field of the log
MethodName string
// PrincipalEmail field of the log
PrincipalEmail string
// Kubernetes operation read from resource name and method name
Operation *model.KubernetesObjectOperation
// The request field of this log. This can be nil depending on the audit policy.
Request *structure.Reader
RequestType rtype.Type
// The response field of this log. This can be nil depending on the audit policy.
Response *structure.Reader
ResponseType rtype.Type
// Current resource body canged by this request.
ResourceBodyReader *structure.Reader
ResourceBodyYaml string
// The response code.
Code int

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Keep comment

@kyasbal
Copy link
Member

kyasbal commented Feb 23, 2025

Thank you for fixing these lint error. The CI seems to output better lint errors.

And I fixed some linter errors in 03c4a3e, but it seems there are still many errors – many of which are related to fieldalignment and shadow.

I feel fieldalignment in the unnamed structs in test codes are too much. Do we have a way to disable the rule only for testing codes?

I don’t have time to fix these immediately, but if you can wait a little longer – about 1~2 weeks – I can address them.

Please take your time. If you think the remaining part of lint errors should be fixed from my side, please feel free to assign this PR to me. Then I will work fixing the remaining part.
I'm sorry to ask these chores in addition to write these rules 🙇

@kyasbal
Copy link
Member

kyasbal commented Feb 23, 2025

Or how do you think to create another Github Action task to perform golangci-lint?
If we add golangci-lint directly, we need to address all the issue before merging them. But if you create another task for golangci-lint, I'll keep the CI task to be optional for a while. Then, fix problematic codes gracefully.
In the case, I would like to retain go vet code in backend-tests Github Action in the meantime to fix the lint errors.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants