Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

feat(chronicleexporter): Support dynamic namespace and ingestion labels #1939

Merged
merged 11 commits into from
Nov 12, 2024
Merged

feat(chronicleexporter): Support dynamic namespace and ingestion labels #1939

merged 11 commits into from
Nov 12, 2024

Conversation

justinianvoss22
Copy link
Contributor

Proposed Change

  • Update chronicle exporter to be able to take namespace and ingestion labels from attributes if present.
  • Make logType not required

Testing

  • Unit testing
  • Testing agent on local BP
Checklist
  • [ x ] Changes are tested
  • [ x ] CI has passed

@justinianvoss22 justinianvoss22 requested a review from a team as a code owner October 30, 2024 18:27
@justinianvoss22 justinianvoss22 force-pushed the justin/bpop-847-support-namespace-and-ingestion-labels branch from 5323ccd to 1fd52e2 Compare October 30, 2024 18:28
@jsirianni jsirianni changed the title Justin/bpop 847 support namespace and ingestion labels feat(chronicleexporter): Support dynamic namespace and ingestion labels Nov 1, 2024
@jsirianni
Copy link
Member

I updated the PR name, it will help w/ release notes. Feel free to tweak it if its incorrect.

@kuiperda
Copy link
Contributor

kuiperda commented Nov 4, 2024

Took a quick high level pass and it makes sense to me, will let Dakota review though.

dpaasman00
dpaasman00 requested changes Nov 4, 2024
View reviewed changes
Copy link
Contributor

@dpaasman00 dpaasman00 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

First pass and generally looks good. Somethings to address however.

exporter/chronicleexporter/marshal.go Show resolved Hide resolved
exporter/chronicleexporter/marshal.go Outdated Show resolved Hide resolved
exporter/chronicleexporter/marshal_test.go Outdated Show resolved Hide resolved
exporter/chronicleexporter/marshal.go Show resolved Hide resolved
exporter/chronicleexporter/marshal.go Outdated Show resolved Hide resolved
@justinianvoss22 justinianvoss22 force-pushed the justin/bpop-847-support-namespace-and-ingestion-labels branch 2 times, most recently from ae94902 to 619a4a6 Compare November 7, 2024 15:14
dpaasman00
dpaasman00 requested changes Nov 8, 2024
View reviewed changes
Copy link
Contributor

@dpaasman00 dpaasman00 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Some new thoughts on the code

exporter/chronicleexporter/marshal.go Outdated Show resolved Hide resolved
exporter/chronicleexporter/marshal.go Outdated Show resolved Hide resolved
exporter/chronicleexporter/marshal.go Outdated Show resolved Hide resolved
exporter/chronicleexporter/marshal.go Outdated Show resolved Hide resolved
exporter/chronicleexporter/marshal.go Outdated Show resolved Hide resolved
exporter/chronicleexporter/marshal.go Show resolved Hide resolved
@justinianvoss22 justinianvoss22 force-pushed the justin/bpop-847-support-namespace-and-ingestion-labels branch from 626df5b to 36ec600 Compare November 8, 2024 23:19
dpaasman00
dpaasman00 requested changes Nov 11, 2024
View reviewed changes
Copy link
Contributor

@dpaasman00 dpaasman00 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Couple more thoughts, otherwise this is looking good. Going to test next.

exporter/chronicleexporter/README.md Outdated Show resolved Hide resolved
exporter/chronicleexporter/marshal.go Outdated Show resolved Hide resolved
dpaasman00
dpaasman00 approved these changes Nov 12, 2024
View reviewed changes
Copy link
Contributor

@dpaasman00 dpaasman00 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This has been working well for me.

@justinianvoss22 justinianvoss22 merged commit f12084d into release/v1.64.0 Nov 12, 2024
15 checks passed
@justinianvoss22 justinianvoss22 deleted the justin/bpop-847-support-namespace-and-ingestion-labels branch November 12, 2024 15:07
dpaasman00 added a commit that referenced this pull request Nov 12, 2024
@dpaasman00 @shazlehu
@kuiperda @mrsillydog @justinianvoss22
chore: Merge release/v1.64.0 for release prep (#1964)
1f4435e 
* chore: Update modules to v1.64.0

* chore: Add OTEL_AES_CREDENTIAL_PROVIDER key to agent description (#1938)

* Add OTEL_AES_CREDENTIAL_PROVIDER key hash to agent description

* Switch to sending key instead of hash, send in NonIdentifyingAttributes

* chore: Deprecate Marshal processor (#1945)

* delete processor and references

* remove from go.mod

* deps: Update OTel v0.113.0 (#1948)

Update OTel v0.113.0

* feat: Support manager config environment variable expansion (#1946)

* feat: Support manager config environment variable expansion

* Minor pointer cleanup

* Uncomment out unused test code

* fix: TestToTLS to use unreferenced config attributes (#1952)

* Fix test to use unreferenced config attributes

* Adjust caCertPool solution

* feat(chronicleexporter): Support dynamic namespace and ingestion labels  (#1939)

* add namespace and ingenstion logs initial commit

* working except ingestion labels

* ingestion labels from attributes

* use proper log entry batch

* namespace and ingestion logs no config overwrite

* delete OverrideNamespace and OverrideIngestionLabeles

* PR changes

* fix unit tests

* modify tests again

* marshal changes

* readme and namespace check

* deps: Bump min Go version to 1.22.7 (#1963)

bump min go version to 1.22.7, remove toolchains

* deps: Dependabot 11-12-2024 (#1962)

* bump snowflake v1.12.0

* bump jwt

* bump google.golang.org/api

* bump google.golang.org/grpc

* bump golang.org/x/oauth2

* bump github.com/mgechev/revive

* bump golang.org/x/tools

* bump golang.org/x/sys

* skip grpc update

* bump grpc

---------

Co-authored-by: Sam Hazlehurst <sam.hazlehurst@observiq.com>
Co-authored-by: Daniel Kuiper <44123852+kuiperda@users.noreply.github.com>
Co-authored-by: Ian Adams <ian.sam.adams@gmail.com>
Co-authored-by: Justin Voss <90650155+justinianvoss22@users.noreply.github.com>
Caleb-Hurshman pushed a commit that referenced this pull request Dec 10, 2024
@justinianvoss22 @Caleb-Hurshman
feat(chronicleexporter): Support dynamic namespace and ingestion labe…
323d1af 
…ls (#1939)

* add namespace and ingenstion logs initial commit

* working except ingestion labels

* ingestion labels from attributes

* use proper log entry batch

* namespace and ingestion logs no config overwrite

* delete OverrideNamespace and OverrideIngestionLabeles

* PR changes

* fix unit tests

* modify tests again

* marshal changes

* readme and namespace check
Caleb-Hurshman added a commit that referenced this pull request Dec 18, 2024
@Caleb-Hurshman @dpaasman00
@shazlehu @mrsillydog @justinianvoss22 @djaglowski
feat: SSAPI Receiver (BPS-289) (#1951)
be7ff1b 
* chore: Update modules to v1.67.0

* fix: QRadar README typo (#2028)

Fix README typo

* fix: Shut down zombie goroutine in chronicleexporter (#2029)

* Properly shut down chronicleexporter zombie goroutine

* Fix lint

* Fix the same problem for the GRPC workflow

* ssapi mvp

* lint

* tls

* WIP

* ticker, other pr feedback

* pagination functionality

* break if results earlier than earliest_time

* fix lint

* check for earliest/latest in query

* config unit tests

* package comment

* feat(chronicleexporter): Support dynamic namespace and ingestion labels  (#1939)

* add namespace and ingenstion logs initial commit

* working except ingestion labels

* ingestion labels from attributes

* use proper log entry batch

* namespace and ingestion logs no config overwrite

* delete OverrideNamespace and OverrideIngestionLabeles

* PR changes

* fix unit tests

* modify tests again

* marshal changes

* readme and namespace check

* debug logs

* rm unnecessary clauses

* fix error wording

* rm space

* wip

* client tests

* checkpoint methods

* WIP

* functional checkpoint

* debug logs, rm print

* loadCheckpoint return error

* splunk failure test

* WIP

* encode req body

* stricter query validation

* storage config test

* lint, tidy

* return error on export fail

* tidy

* receiver tests

* receiver tests

* lint

* fix TestCheckpoint

* rename structs

* exporter fail test

* fix search checkpointing

* auth token

* lint

* fix struct name

* rm prints, fix error messages

* fix tests

* default batch size

* log end of export

* readme

* how-to

* how-to example config

* change how-to conf values

* change test batch size

* fix test case

* fix client test

* fix rebase errors

* tidy

* feat: Enforce request maximum size and number of logs (#2033)

* feat: Enforce request maximum size and number of logs

* Fix lint

* Refactor to be more go-idiomatic

* Update Chronicle exporter readme with new flags

* fix: Delete empty values iterates through nested arrays (#2034)

* delete empty values processor iterates through slices

* log body implementation

* pr review

* initial feedback

* chore: Minor cleanup of chronicle exporter names (#2046)

* chore: Save component.TelemetrySettings on chronicle exporter (#2047)

* chore: Minor cleanup of chronicle exporter names

* chore: Chronicle exporter - save component.TelemetrySettings

* safe shutdown()

* chore: Localize chronicle exporter's metrics concerns (#2048)

chore: Pull metrics-specific concerns into hostMetricsReporter

* rm err checkk from time parsing

* chore: Add debug logging (#2042)

Add debug logging

* chore: Add new tests for chronicle exporter with http and grpc servers (#2049)

* ctx check, doc notes

* chore: Rename to `bindplane-otel-collector` (#2043)

* rename to bindplane-otel-collector

* fix website links

* update report card link

* fix: Shut down zombie goroutine in chronicleexporter (#2029)

* Properly shut down chronicleexporter zombie goroutine

* Fix lint

* Fix the same problem for the GRPC workflow

* chore: Save component.TelemetrySettings on chronicle exporter (#2047)

* chore: Minor cleanup of chronicle exporter names

* chore: Chronicle exporter - save component.TelemetrySettings

* chore: Localize chronicle exporter's metrics concerns (#2048)

chore: Pull metrics-specific concerns into hostMetricsReporter

* chore: Add new tests for chronicle exporter with http and grpc servers (#2049)

* fix: Rebase cleanup (#2063)

rebase cleanup

* chore: separate http and grpc exporters (#2050)

* fix: Shut down zombie goroutine in chronicleexporter (#2029)

* Properly shut down chronicleexporter zombie goroutine

* Fix lint

* Fix the same problem for the GRPC workflow

* ssapi mvp

* initial feedback

* chore: Save component.TelemetrySettings on chronicle exporter (#2047)

* chore: Minor cleanup of chronicle exporter names

* chore: Chronicle exporter - save component.TelemetrySettings

* chore: Localize chronicle exporter's metrics concerns (#2048)

chore: Pull metrics-specific concerns into hostMetricsReporter

* chore: Add new tests for chronicle exporter with http and grpc servers (#2049)

* chore: Save component.TelemetrySettings on chronicle exporter (#2047)

* chore: Minor cleanup of chronicle exporter names

* chore: Chronicle exporter - save component.TelemetrySettings

* chore: Localize chronicle exporter's metrics concerns (#2048)

chore: Pull metrics-specific concerns into hostMetricsReporter

* chore: Add new tests for chronicle exporter with http and grpc servers (#2049)

* fix: Shut down zombie goroutine in chronicleexporter (#2029)

* Properly shut down chronicleexporter zombie goroutine

* Fix lint

* Fix the same problem for the GRPC workflow

* fix rebase stuff

---------

Co-authored-by: Dakota Paasman <122491662+dpaasman00@users.noreply.github.com>
Co-authored-by: Sam Hazlehurst <sam.hazlehurst@observiq.com>
Co-authored-by: Ian Adams <ian.sam.adams@gmail.com>
Co-authored-by: Justin Voss <90650155+justinianvoss22@users.noreply.github.com>
Co-authored-by: Daniel Jaglowski <jaglows3@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@dpaasman00 dpaasman00 dpaasman00 approved these changes

@tbm48813 tbm48813 Awaiting requested review from tbm48813

@kuiperda kuiperda Awaiting requested review from kuiperda

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
4 participants
@justinianvoss22 @jsirianni @kuiperda @dpaasman00