-
Notifications
You must be signed in to change notification settings - Fork 1.8k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Scan GitHub and GitLab refs that aren't cloned by default #1918
base: main
Are you sure you want to change the base?
Conversation
f64e0a3
to
e2fb273
Compare
b2e724c
to
ec2de50
Compare
ec2de50
to
438418c
Compare
438418c
to
7cb8af2
Compare
7cb8af2
to
92c0f83
Compare
956b38d
to
c9a7acd
Compare
c9a7acd
to
9ac8dbe
Compare
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This is awesome! Just to make sure I understand: This PR has two discrete changes, right? (Pulling down all the refs and printing the source ref of found secrets.)
9ac8dbe
to
d526837
Compare
@rgmz @bplaxco alright after banging my head against this git source I think it's better if we use "remote.origin.fetch=+refs/:refs/remotes/origin/". Introducing |
983d41e
to
f057784
Compare
I've rebased this onto #2988. This now only has changes related to reporting ref provenance. |
f057784
to
87956c7
Compare
005d5e1
to
805f5dc
Compare
805f5dc
to
9d902a0
Compare
8d64793
to
139ebe2
Compare
139ebe2
to
83eee14
Compare
83eee14
to
74653a7
Compare
74653a7
to
bef8f77
Compare
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This is pretty neat - I'm only requesting changes because of the fact that the PR adds the expanded refs functionality unilaterally, no matter whether it's enabled. (I suspect this is a rebase artifact.)
Email: sanitizer.UTF8(comment.GetUser().GetEmail()), | ||
Timestamp: sanitizer.UTF8(comment.GetCreatedAt().String()), | ||
Link: sanitizer.UTF8(comment.GetHTMLURL()), |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Can I ask why you reordered these fields everywhere, or, alternatively, how the new order was picked? This isn't blocking or anything but it does add review noise so I'm curious about whether there's a reason for it I'm not picking up on.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
It's been a while; IIRC, I ran into issues caused by SourceMetadataFunc
being a long list of strings with an arbitrary order. I tried to order the fields semantically (based on git log
) so that it was easier to grok at a glance and compare uses of SourceMetadataFunc
like-for-like.
commit 8d2c7e6760b9bdc4fe36fee11cb3f28d7e469203 (HEAD -> feat/additional-refspecs)
Author: Richard Gomez <32133502+rgmz@users.noreply.github.com>
Date: Fri Apr 12 08:53:18 2024 -0400
feat(gitparse): track ref sources
diff --git a/hack/snifftest/main.go b/hack/snifftest/main.go
...
@@ -231,19 +233,23 @@ func (c *Parser) RepoPath( | |||
) (chan *Diff, error) { | |||
args := []string{ | |||
"-C", source, | |||
"--no-replace-objects", |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Why did you add this?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
It's from this hidden comment thread. GitHub's PR UI is a nightmare.
return "Merge request #" + string(mrNumber) | ||
} | ||
|
||
return fmt.Sprintf("%s (hidden ref)", string(ref)) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Does "hidden" just mean "not visible via the respective web UI?" My small concern here is that this also happens if we simply fail to parse the relevant ref because we made a mistake somewhere. What do you think of just returning the ref as-is in that case - would it be confusing?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Does "hidden" just mean "not visible via the respective web UI?"
It means "not visible with default Git behaviour"; sometimes these refs are only discoverable via the web UI or API. The term is inspired by this blog post.
The Git ref namespace (refs/*
) can store almost any path, however, Git and Git-adjacent tools only look at refs/heads/*
and refs/tags/*
. Other refs (if they exist) aren't fetched unless you explicitly request them from a remote.
Two examples of this are:
- Deleted GitHub/GitLab pull request branches being stored under
refs/pull/*
andrefs/merge-requests/*
- Deleted commits that are manually fetched
My small concern here is that this also happens if we simply fail to parse the relevant ref because we made a mistake somewhere. What do you think of just returning the ref as-is in that case - would it be confusing?
I think it's worth signaling that these refs are special and don't technically exist in the repository. Returning the ref as-is is already a source of confusion (#3493).
33d70b7
to
d7953e9
Compare
d7953e9
to
4176d48
Compare
d8a905d
to
00340e1
Compare
00340e1
to
5e5db88
Compare
'Hidden' refs, such as 'refs/pull/1004/head' may cause confusion if reported upon. GitHub, for example, will display a banner saying that the commit doesn't belong to the repository. This parse the output of 'git log --source' and converts it to a human-readable format, IF the ref is 'hidden'.
5e5db88
to
098735c
Compare
Description:
This fixes #1588.
In my experience, this find significantly more secrets with a negligible performance impact.
The only issue is that these secrets are technically not a part of the repository, so refactoring may be necessary to indicate that a result comes from a historical PR/MR branch.It now outputs the source pull/merge request (based ongit log --source
), in case the commit only exists in the PR history and not the actual repo history, which can happen when PRs are squashed.Checklist:
make test-community
)?make lint
this requires golangci-lint)?