Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[refactor] - Create separate handler for non-archive data #2825

Merged
merged 246 commits into from
May 15, 2024

Conversation

ahrav
Copy link
Collaborator

@ahrav ahrav commented May 10, 2024

Description:

This PR splits the original default handler into two separate handlers: default and archive. The default handler now processes non-archive files, and the archive handler manages all archive file types supported by the archiver library.

Additionally, this PR restores the git cat file functionality for binary diffs to a streaming method. This change is due to HandleFile now properly consuming the entire reader before beginning file processing.

Note: Most of the important changes are in handlers.go.

Checklist:

  • Tests passing (make test-community)?
  • Lint passing (make lint this requires golangci-lint)?

ahrav and others added 30 commits April 13, 2024 11:58
@@ -0,0 +1,192 @@
package handlers
Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Most of this code was moved from the old default handler.

@ahrav ahrav marked this pull request as ready for review May 10, 2024 19:46
@ahrav ahrav requested review from a team as code owners May 10, 2024 19:46
Copy link
Collaborator

@rosecodym rosecodym left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I have a non-blocking thought about a possible different way to structure the code (that you may have already considered) and a question about a suspicious comment.

pkg/handlers/handlers.go Outdated Show resolved Hide resolved
Comment on lines 28 to 31
format archiver.Format
mimeType mimeType
*readers.BufferedFileReader
isArchive bool
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It looks like three of these fields are only used to later choose a handler type. Did you consider just creating a field for a handler here and setting it in newFileReader, thus eliminating the need for selectHandler and therefore the fields needed for it to work? (It looks like you'd need to add some additional functionality to individual archive implementations, e.g. to support archive skipping, so I don't think this is a slam dunk, but I thought I'd ask.)

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It should also be used in archive.go L88-91 here, sorry I should've explicitly called it out since the rest of the code was a lift and shift.

pkg/handlers/rpm.go Outdated Show resolved Hide resolved
@ahrav ahrav requested review from rosecodym and a team May 13, 2024 21:30
@ahrav
Copy link
Collaborator Author

ahrav commented May 15, 2024

I have a non-blocking thought about a possible different way to structure the code (that you may have already considered) and a question about a suspicious comment.

Replied inline.

Copy link
Collaborator

@rosecodym rosecodym left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🎉

@ahrav ahrav merged commit ead9dd5 into main May 15, 2024
12 checks passed
@ahrav ahrav deleted the refactor-add-nonarchive-handler branch May 15, 2024 20:40
haraldh referenced this pull request in matter-labs/vault-auth-tee May 21, 2024
[![Mend
Renovate](https://app.renovatebot.com/images/banner.svg)](https://renovatebot.com)

This PR contains the following updates:

| Package | Type | Update | Change |
|---|---|---|---|
|
[trufflesecurity/trufflehog](https://togithub.com/trufflesecurity/trufflehog)
| action | minor | `v3.75.1` -> `v3.76.3` |

---

### Release Notes

<details>
<summary>trufflesecurity/trufflehog
(trufflesecurity/trufflehog)</summary>

###
[`v3.76.3`](https://togithub.com/trufflesecurity/trufflehog/releases/tag/v3.76.3)

[Compare
Source](https://togithub.com/trufflesecurity/trufflehog/compare/v3.76.2...v3.76.3)

#### What's Changed

- fix(deps): update module github.com/aws/aws-sdk-go to v1.53.3 by
[@&#8203;renovate](https://togithub.com/renovate) in
[https://github.com/trufflesecurity/trufflehog/pull/2849](https://togithub.com/trufflesecurity/trufflehog/pull/2849)
- Integromat detector - Deprecated by
[@&#8203;abmussani](https://togithub.com/abmussani) in
[https://github.com/trufflesecurity/trufflehog/pull/2856](https://togithub.com/trufflesecurity/trufflehog/pull/2856)
- add tolower to all keywords, and remove return on error for global
vars by [@&#8203;zricethezav](https://togithub.com/zricethezav) in
[https://github.com/trufflesecurity/trufflehog/pull/2852](https://togithub.com/trufflesecurity/trufflehog/pull/2852)
- Adding postman to sub-commands list by
[@&#8203;CarlesLlobet](https://togithub.com/CarlesLlobet) in
[https://github.com/trufflesecurity/trufflehog/pull/2813](https://togithub.com/trufflesecurity/trufflehog/pull/2813)
- \[chore] - upgrade Github dep by
[@&#8203;ahrav](https://togithub.com/ahrav) in
[https://github.com/trufflesecurity/trufflehog/pull/2858](https://togithub.com/trufflesecurity/trufflehog/pull/2858)
- \[chore] - move buffers pkg out of writers pkg by
[@&#8203;ahrav](https://togithub.com/ahrav) in
[https://github.com/trufflesecurity/trufflehog/pull/2826](https://togithub.com/trufflesecurity/trufflehog/pull/2826)
- \[refactor] - remove redundant chunking by
[@&#8203;ahrav](https://togithub.com/ahrav) in
[https://github.com/trufflesecurity/trufflehog/pull/2855](https://togithub.com/trufflesecurity/trufflehog/pull/2855)

#### New Contributors

- [@&#8203;CarlesLlobet](https://togithub.com/CarlesLlobet) made their
first contribution in
[https://github.com/trufflesecurity/trufflehog/pull/2813](https://togithub.com/trufflesecurity/trufflehog/pull/2813)

**Full Changelog**:
trufflesecurity/trufflehog@v3.76.2...v3.76.3

###
[`v3.76.2`](https://togithub.com/trufflesecurity/trufflehog/releases/tag/v3.76.2)

[Compare
Source](https://togithub.com/trufflesecurity/trufflehog/compare/v3.76.1...v3.76.2)

#### What's Changed

- \[bug] - Handle empty reader case in newFileReader by
[@&#8203;ahrav](https://togithub.com/ahrav) in
[https://github.com/trufflesecurity/trufflehog/pull/2854](https://togithub.com/trufflesecurity/trufflehog/pull/2854)

**Full Changelog**:
trufflesecurity/trufflehog@v3.76.1...v3.76.2

###
[`v3.76.1`](https://togithub.com/trufflesecurity/trufflehog/releases/tag/v3.76.1)

[Compare
Source](https://togithub.com/trufflesecurity/trufflehog/compare/v3.76.0...v3.76.1)

#### What's Changed

- chore: fix some typos in comments by
[@&#8203;cuiyourong](https://togithub.com/cuiyourong) in
[https://github.com/trufflesecurity/trufflehog/pull/2851](https://togithub.com/trufflesecurity/trufflehog/pull/2851)
- Added user's email and location in metadata for Github detector. by
[@&#8203;abmussani](https://togithub.com/abmussani) in
[https://github.com/trufflesecurity/trufflehog/pull/2850](https://togithub.com/trufflesecurity/trufflehog/pull/2850)
- \[refactor] - Create separate handler for non-archive data by
[@&#8203;ahrav](https://togithub.com/ahrav) in
[https://github.com/trufflesecurity/trufflehog/pull/2825](https://togithub.com/trufflesecurity/trufflehog/pull/2825)

#### New Contributors

- [@&#8203;cuiyourong](https://togithub.com/cuiyourong) made their first
contribution in
[https://github.com/trufflesecurity/trufflehog/pull/2851](https://togithub.com/trufflesecurity/trufflehog/pull/2851)

**Full Changelog**:
trufflesecurity/trufflehog@v3.76.0...v3.76.1

###
[`v3.76.0`](https://togithub.com/trufflesecurity/trufflehog/releases/tag/v3.76.0)

[Compare
Source](https://togithub.com/trufflesecurity/trufflehog/compare/v3.75.1...v3.76.0)

#### What's Changed

- fix(deps): update module golang.org/x/net to v0.25.0 by
[@&#8203;renovate](https://togithub.com/renovate) in
[https://github.com/trufflesecurity/trufflehog/pull/2792](https://togithub.com/trufflesecurity/trufflehog/pull/2792)
- Add webhook source protos by
[@&#8203;dustin-decker](https://togithub.com/dustin-decker) in
[https://github.com/trufflesecurity/trufflehog/pull/2789](https://togithub.com/trufflesecurity/trufflehog/pull/2789)
- Increase test chan size by
[@&#8203;rosecodym](https://togithub.com/rosecodym) in
[https://github.com/trufflesecurity/trufflehog/pull/2797](https://togithub.com/trufflesecurity/trufflehog/pull/2797)
- fix(deps): update module golang.org/x/exp to
v0.0.0-20240506185415-9bf2ced13842 by
[@&#8203;renovate](https://togithub.com/renovate) in
[https://github.com/trufflesecurity/trufflehog/pull/2795](https://togithub.com/trufflesecurity/trufflehog/pull/2795)
- fix(deps): update module github.com/brianvoe/gofakeit/v7 to v7.0.3 by
[@&#8203;renovate](https://togithub.com/renovate) in
[https://github.com/trufflesecurity/trufflehog/pull/2798](https://togithub.com/trufflesecurity/trufflehog/pull/2798)
- fix(deps): update module github.com/aws/aws-sdk-go to v1.52.4 by
[@&#8203;renovate](https://togithub.com/renovate) in
[https://github.com/trufflesecurity/trufflehog/pull/2794](https://togithub.com/trufflesecurity/trufflehog/pull/2794)
- fix(deps): update module google.golang.org/api to v0.178.0 by
[@&#8203;renovate](https://togithub.com/renovate) in
[https://github.com/trufflesecurity/trufflehog/pull/2800](https://togithub.com/trufflesecurity/trufflehog/pull/2800)
- Enterprise ReadMe Updates by
[@&#8203;theproductone](https://togithub.com/theproductone) in
[https://github.com/trufflesecurity/trufflehog/pull/2803](https://togithub.com/trufflesecurity/trufflehog/pull/2803)
- Updating Enterprise Readme Link from Contact to Product Info Page by
[@&#8203;theproductone](https://togithub.com/theproductone) in
[https://github.com/trufflesecurity/trufflehog/pull/2804](https://togithub.com/trufflesecurity/trufflehog/pull/2804)
- chore(deps): update golangci/golangci-lint-action action to v6 by
[@&#8203;renovate](https://togithub.com/renovate) in
[https://github.com/trufflesecurity/trufflehog/pull/2801](https://togithub.com/trufflesecurity/trufflehog/pull/2801)
- \[chore] - address linter by
[@&#8203;ahrav](https://togithub.com/ahrav) in
[https://github.com/trufflesecurity/trufflehog/pull/2783](https://togithub.com/trufflesecurity/trufflehog/pull/2783)
- fix(deps): update module github.com/xanzy/go-gitlab to v0.104.1 by
[@&#8203;renovate](https://togithub.com/renovate) in
[https://github.com/trufflesecurity/trufflehog/pull/2784](https://togithub.com/trufflesecurity/trufflehog/pull/2784)
- fix(deps): update module github.com/rabbitmq/amqp091-go to v1.10.0 by
[@&#8203;renovate](https://togithub.com/renovate) in
[https://github.com/trufflesecurity/trufflehog/pull/2809](https://togithub.com/trufflesecurity/trufflehog/pull/2809)
- fix(deps): update module github.com/snowflakedb/gosnowflake to v1.10.0
by [@&#8203;renovate](https://togithub.com/renovate) in
[https://github.com/trufflesecurity/trufflehog/pull/2810](https://togithub.com/trufflesecurity/trufflehog/pull/2810)
- Prevent panic from flaky test by
[@&#8203;rgmz](https://togithub.com/rgmz) in
[https://github.com/trufflesecurity/trufflehog/pull/2817](https://togithub.com/trufflesecurity/trufflehog/pull/2817)
- fix(deps): update module github.com/aws/aws-sdk-go to v1.52.6 by
[@&#8203;renovate](https://togithub.com/renovate) in
[https://github.com/trufflesecurity/trufflehog/pull/2816](https://togithub.com/trufflesecurity/trufflehog/pull/2816)
- fix(deps): update module github.com/hashicorp/go-retryablehttp to
v0.7.6 by [@&#8203;renovate](https://togithub.com/renovate) in
[https://github.com/trufflesecurity/trufflehog/pull/2819](https://togithub.com/trufflesecurity/trufflehog/pull/2819)
- fix(deps): update module github.com/charmbracelet/bubbletea to v0.26.2
by [@&#8203;renovate](https://togithub.com/renovate) in
[https://github.com/trufflesecurity/trufflehog/pull/2818](https://togithub.com/trufflesecurity/trufflehog/pull/2818)
- fix(deps): update module github.com/prometheus/client_golang to
v1.19.1 by [@&#8203;renovate](https://togithub.com/renovate) in
[https://github.com/trufflesecurity/trufflehog/pull/2821](https://togithub.com/trufflesecurity/trufflehog/pull/2821)
- Update postman flags to be less confusing by
[@&#8203;dustin-decker](https://togithub.com/dustin-decker) in
[https://github.com/trufflesecurity/trufflehog/pull/2755](https://togithub.com/trufflesecurity/trufflehog/pull/2755)
- \[refactor] - Refactor Archive Handling Logic by
[@&#8203;ahrav](https://togithub.com/ahrav) in
[https://github.com/trufflesecurity/trufflehog/pull/2703](https://togithub.com/trufflesecurity/trufflehog/pull/2703)
- fix(deps): update testcontainers-go monorepo to v0.31.0 by
[@&#8203;renovate](https://togithub.com/renovate) in
[https://github.com/trufflesecurity/trufflehog/pull/2823](https://togithub.com/trufflesecurity/trufflehog/pull/2823)
- fix(deps): update module github.com/xanzy/go-gitlab to v0.105.0 by
[@&#8203;renovate](https://togithub.com/renovate) in
[https://github.com/trufflesecurity/trufflehog/pull/2824](https://togithub.com/trufflesecurity/trufflehog/pull/2824)
- fix(deps): update module google.golang.org/api to v0.180.0 by
[@&#8203;renovate](https://togithub.com/renovate) in
[https://github.com/trufflesecurity/trufflehog/pull/2822](https://togithub.com/trufflesecurity/trufflehog/pull/2822)
- fix(deps): update module github.com/sassoftware/go-rpmutils to v0.4.0
by [@&#8203;renovate](https://togithub.com/renovate) in
[https://github.com/trufflesecurity/trufflehog/pull/2831](https://togithub.com/trufflesecurity/trufflehog/pull/2831)
- fix(deps): update module github.com/google/go-github/v61 to v62 by
[@&#8203;renovate](https://togithub.com/renovate) in
[https://github.com/trufflesecurity/trufflehog/pull/2832](https://togithub.com/trufflesecurity/trufflehog/pull/2832)
- Update results's extra data for Twilio by
[@&#8203;abmussani](https://togithub.com/abmussani) in
[https://github.com/trufflesecurity/trufflehog/pull/2807](https://togithub.com/trufflesecurity/trufflehog/pull/2807)
- fix(deps): update module github.com/aws/aws-sdk-go to v1.53.0 by
[@&#8203;renovate](https://togithub.com/renovate) in
[https://github.com/trufflesecurity/trufflehog/pull/2830](https://togithub.com/trufflesecurity/trufflehog/pull/2830)
- \[chore] - Update GitlabV2 detector by
[@&#8203;ahrav](https://togithub.com/ahrav) in
[https://github.com/trufflesecurity/trufflehog/pull/2840](https://togithub.com/trufflesecurity/trufflehog/pull/2840)
- Fixed the Now Scanning Emoji on the ReadMe by
[@&#8203;theproductone](https://togithub.com/theproductone) in
[https://github.com/trufflesecurity/trufflehog/pull/2842](https://togithub.com/trufflesecurity/trufflehog/pull/2842)
- fix(deps): update module github.com/fatih/color to v1.17.0 by
[@&#8203;renovate](https://togithub.com/renovate) in
[https://github.com/trufflesecurity/trufflehog/pull/2837](https://togithub.com/trufflesecurity/trufflehog/pull/2837)
- fix(deps): update module github.com/aws/aws-sdk-go to v1.53.1 by
[@&#8203;renovate](https://togithub.com/renovate) in
[https://github.com/trufflesecurity/trufflehog/pull/2841](https://togithub.com/trufflesecurity/trufflehog/pull/2841)
- Bump up performance test threshold to 50% by
[@&#8203;zricethezav](https://togithub.com/zricethezav) in
[https://github.com/trufflesecurity/trufflehog/pull/2839](https://togithub.com/trufflesecurity/trufflehog/pull/2839)
- Update SendGrid detector by [@&#8203;rgmz](https://togithub.com/rgmz)
in
[https://github.com/trufflesecurity/trufflehog/pull/2833](https://togithub.com/trufflesecurity/trufflehog/pull/2833)
- fix(deps): update module cloud.google.com/go/storage to v1.41.0 by
[@&#8203;renovate](https://togithub.com/renovate) in
[https://github.com/trufflesecurity/trufflehog/pull/2843](https://togithub.com/trufflesecurity/trufflehog/pull/2843)
- \[bug] - Fix case-sensitivity issue in PrefixRegex function by
[@&#8203;ahrav](https://togithub.com/ahrav) in
[https://github.com/trufflesecurity/trufflehog/pull/2811](https://togithub.com/trufflesecurity/trufflehog/pull/2811)
- switch to filesystem and specific tag when performance testing by
[@&#8203;zricethezav](https://togithub.com/zricethezav) in
[https://github.com/trufflesecurity/trufflehog/pull/2846](https://togithub.com/trufflesecurity/trufflehog/pull/2846)
- Use fake detectors in versioned detectors test by
[@&#8203;rosecodym](https://togithub.com/rosecodym) in
[https://github.com/trufflesecurity/trufflehog/pull/2847](https://togithub.com/trufflesecurity/trufflehog/pull/2847)
- \[feat] - Support bearer auth for docker scans by
[@&#8203;ahrav](https://togithub.com/ahrav) in
[https://github.com/trufflesecurity/trufflehog/pull/2848](https://togithub.com/trufflesecurity/trufflehog/pull/2848)
- Add "Intra42" detector by
[@&#8203;alexandregv](https://togithub.com/alexandregv) in
[https://github.com/trufflesecurity/trufflehog/pull/2835](https://togithub.com/trufflesecurity/trufflehog/pull/2835)

#### New Contributors

- [@&#8203;theproductone](https://togithub.com/theproductone) made their
first contribution in
[https://github.com/trufflesecurity/trufflehog/pull/2803](https://togithub.com/trufflesecurity/trufflehog/pull/2803)

**Full Changelog**:
trufflesecurity/trufflehog@v3.75.1...v3.76.0

</details>

---

### Configuration

📅 **Schedule**: Branch creation - At any time (no schedule defined),
Automerge - At any time (no schedule defined).

🚦 **Automerge**: Disabled by config. Please merge this manually once you
are satisfied.

♻ **Rebasing**: Whenever PR becomes conflicted, or you tick the
rebase/retry checkbox.

🔕 **Ignore**: Close this PR and you won't be reminded about this update
again.

---

- [ ] <!-- rebase-check -->If you want to rebase/retry this PR, check
this box

---

This PR has been generated by [Mend
Renovate](https://www.mend.io/free-developer-tools/renovate/). View
repository job log
[here](https://developer.mend.io/github/matter-labs/vault-auth-tee).

<!--renovate-debug:eyJjcmVhdGVkSW5WZXIiOiIzNy4zNTEuMiIsInVwZGF0ZWRJblZlciI6IjM3LjM2My41IiwidGFyZ2V0QnJhbmNoIjoibWFpbiIsImxhYmVscyI6W119-->
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Development

Successfully merging this pull request may close these issues.

2 participants