Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Investigate usage of Confluent's parallel consumer #346

Closed
nscuro opened this issue Feb 27, 2023 · 2 comments
Closed

Investigate usage of Confluent's parallel consumer #346

nscuro opened this issue Feb 27, 2023 · 2 comments
Labels
domain/repo-meta-analysis domain/vuln-analysis enhancement New feature or request p3 Nice-to-have features performance size/M Medium effort

Comments

@nscuro
Copy link
Member

nscuro commented Feb 27, 2023

Confluent has published a wrapper around Kafka's Consumer/Producer API that enables parallel processing of messages, and supports batching and retries: https://github.com/confluentinc/parallel-consumer

parallel-consumer can be used alongside Kafka Streams in the same application. It can be hooked into vert.x (which ships with Quarkus) to perform I/O operations.

Based on what I've seen so far, parallel-consumer may be a good fit for vulnerability analysis and repo meta analysis.

When it comes to the supported ordering guarantees, I think key-based ordering should suffice. These ordering guarantees are something that Quarkus' reactive messaging implementation is currently missing, see #215 (comment)
@nscuro nscuro added enhancement New feature or request p3 Nice-to-have features domain/repo-meta-analysis domain/vuln-analysis size/M Medium effort labels Feb 27, 2023
@nscuro
Copy link
Member Author

nscuro commented Feb 27, 2023

Relates to #215

@nscuro nscuro mentioned this issue May 28, 2023
Merged
@nscuro nscuro mentioned this issue Aug 4, 2023
Closed
@nscuro nscuro mentioned this issue Nov 14, 2023
Closed
@nscuro nscuro mentioned this issue Dec 28, 2023
Closed
3 tasks
@nscuro nscuro mentioned this issue Feb 2, 2024
Open
34 tasks
nscuro added a commit to DependencyTrack/hyades-apiserver that referenced this issue Feb 2, 2024
@nscuro
Implement foundational API for parallel-consumer based Kafka processors
eb11e8b 
Decoupled from #509

This merely adds an API on top of which processors can be implemented. We can migrate processors one-by-one from Kafka Streams to this API. Majority of this work was already done in #509, but got out of date due to changed priorities. At the very least said PR is good to take inspiration from.

Relates to DependencyTrack/hyades#346
Relates to DependencyTrack/hyades#901
Relates to DependencyTrack/hyades#907

Signed-off-by: nscuro <nscuro@protonmail.com>
@nscuro nscuro mentioned this issue Feb 2, 2024
Merged
2 tasks
nscuro added a commit to DependencyTrack/hyades-apiserver that referenced this issue Feb 3, 2024
@nscuro
Implement foundational API for parallel-consumer based Kafka processors
bd2af3e 
Decoupled from #509

This merely adds an API on top of which processors can be implemented. We can migrate processors one-by-one from Kafka Streams to this API. Majority of this work was already done in #509, but got out of date due to changed priorities. At the very least said PR is good to take inspiration from.

Relates to DependencyTrack/hyades#346
Relates to DependencyTrack/hyades#901
Relates to DependencyTrack/hyades#907

Signed-off-by: nscuro <nscuro@protonmail.com>
nscuro added a commit to DependencyTrack/hyades-apiserver that referenced this issue Feb 5, 2024
@nscuro
Implement foundational API for parallel-consumer based Kafka processors
30cdcaa 
Decoupled from #509

This merely adds an API on top of which processors can be implemented. We can migrate processors one-by-one from Kafka Streams to this API. Majority of this work was already done in #509, but got out of date due to changed priorities. At the very least said PR is good to take inspiration from.

Relates to DependencyTrack/hyades#346
Relates to DependencyTrack/hyades#901
Relates to DependencyTrack/hyades#907

Signed-off-by: nscuro <nscuro@protonmail.com>
nscuro added a commit to DependencyTrack/hyades-apiserver that referenced this issue Feb 5, 2024
@nscuro
Implement foundational API for parallel-consumer based Kafka processors
45aaf80 
Decoupled from #509

This merely adds an API on top of which processors can be implemented. We can migrate processors one-by-one from Kafka Streams to this API. Majority of this work was already done in #509, but got out of date due to changed priorities. At the very least said PR is good to take inspiration from.

Relates to DependencyTrack/hyades#346
Relates to DependencyTrack/hyades#901
Relates to DependencyTrack/hyades#907

Signed-off-by: nscuro <nscuro@protonmail.com>
nscuro added a commit to DependencyTrack/hyades-apiserver that referenced this issue Feb 5, 2024
@nscuro
Migrate MirrorVulnerabilityProcessor from Kafka Streams to Parallel…
c387537 
… Consumer

Depends on #552

Relates to DependencyTrack/hyades#346
Relates to DependencyTrack/hyades#901
Relates to DependencyTrack/hyades#907

Signed-off-by: nscuro <nscuro@protonmail.com>
nscuro added a commit to DependencyTrack/hyades-apiserver that referenced this issue Feb 5, 2024
@nscuro
Migrate MirrorVulnerabilityProcessor from Kafka Streams to Parallel…
4b01f00 
… Consumer

Depends on #552

Relates to DependencyTrack/hyades#346
Relates to DependencyTrack/hyades#901
Relates to DependencyTrack/hyades#907

Signed-off-by: nscuro <nscuro@protonmail.com>
@nscuro nscuro mentioned this issue Feb 5, 2024
Merged
2 tasks
nscuro added a commit to DependencyTrack/hyades-apiserver that referenced this issue Feb 5, 2024
@nscuro
Implement foundational API for parallel-consumer based Kafka processors
f78813d 
Decoupled from #509

This merely adds an API on top of which processors can be implemented. We can migrate processors one-by-one from Kafka Streams to this API. Majority of this work was already done in #509, but got out of date due to changed priorities. At the very least said PR is good to take inspiration from.

Relates to DependencyTrack/hyades#346
Relates to DependencyTrack/hyades#901
Relates to DependencyTrack/hyades#907

Signed-off-by: nscuro <nscuro@protonmail.com>
nscuro added a commit to DependencyTrack/hyades-apiserver that referenced this issue Feb 5, 2024
@nscuro
Migrate MirrorVulnerabilityProcessor from Kafka Streams to Parallel…
6a0a5f2 
… Consumer

Depends on #552

Relates to DependencyTrack/hyades#346
Relates to DependencyTrack/hyades#901
Relates to DependencyTrack/hyades#907

Signed-off-by: nscuro <nscuro@protonmail.com>
nscuro added a commit to DependencyTrack/hyades-apiserver that referenced this issue Feb 5, 2024
@nscuro
Migrate RepositoryMetaResultProcessor from Kafka Streams to Paralle…
3a21fb6 
…l Consumer

Depends on #552

Relates to DependencyTrack/hyades#346
Relates to DependencyTrack/hyades#901
Relates to DependencyTrack/hyades#907

Signed-off-by: nscuro <nscuro@protonmail.com>
nscuro added a commit to DependencyTrack/hyades-apiserver that referenced this issue Feb 5, 2024
@nscuro
Migrate MirrorVulnerabilityProcessor from Kafka Streams to Parallel…
2d4cd8b 
… Consumer

Depends on #552

Relates to DependencyTrack/hyades#346
Relates to DependencyTrack/hyades#901
Relates to DependencyTrack/hyades#907

Signed-off-by: nscuro <nscuro@protonmail.com>
@nscuro nscuro mentioned this issue Feb 5, 2024
Merged
2 tasks
nscuro added a commit to DependencyTrack/hyades-apiserver that referenced this issue Feb 5, 2024
@nscuro
Implement foundational API for parallel-consumer based Kafka processors
0f157b5 
Decoupled from #509

This merely adds an API on top of which processors can be implemented. We can migrate processors one-by-one from Kafka Streams to this API. Majority of this work was already done in #509, but got out of date due to changed priorities. At the very least said PR is good to take inspiration from.

Relates to DependencyTrack/hyades#346
Relates to DependencyTrack/hyades#901
Relates to DependencyTrack/hyades#907

Signed-off-by: nscuro <nscuro@protonmail.com>
nscuro added a commit to DependencyTrack/hyades-apiserver that referenced this issue Feb 5, 2024
@nscuro
Migrate MirrorVulnerabilityProcessor from Kafka Streams to Parallel…
2215ea0 
… Consumer

Depends on #552

Relates to DependencyTrack/hyades#346
Relates to DependencyTrack/hyades#901
Relates to DependencyTrack/hyades#907

Signed-off-by: nscuro <nscuro@protonmail.com>
nscuro added a commit to DependencyTrack/hyades-apiserver that referenced this issue Feb 5, 2024
@nscuro
Migrate RepositoryMetaResultProcessor from Kafka Streams to Paralle…
5c21e51 
…l Consumer

Depends on #552

Relates to DependencyTrack/hyades#346
Relates to DependencyTrack/hyades#901
Relates to DependencyTrack/hyades#907

Signed-off-by: nscuro <nscuro@protonmail.com>
nscuro added a commit to DependencyTrack/hyades-apiserver that referenced this issue Feb 9, 2024
@nscuro
Migrate MirrorVulnerabilityProcessor from Kafka Streams to Parallel…
097f1cd 
… Consumer

Depends on #552

Relates to DependencyTrack/hyades#346
Relates to DependencyTrack/hyades#901
Relates to DependencyTrack/hyades#907

Signed-off-by: nscuro <nscuro@protonmail.com>
nscuro added a commit to DependencyTrack/hyades-apiserver that referenced this issue Feb 9, 2024
@nscuro
Migrate RepositoryMetaResultProcessor from Kafka Streams to Paralle…
65e7e07 
…l Consumer

Depends on #552

Relates to DependencyTrack/hyades#346
Relates to DependencyTrack/hyades#901
Relates to DependencyTrack/hyades#907

Signed-off-by: nscuro <nscuro@protonmail.com>
nscuro added a commit to DependencyTrack/hyades-apiserver that referenced this issue Feb 26, 2024
@nscuro
Migrate MirrorVulnerabilityProcessor from Kafka Streams to Parallel…
2dc996a 
… Consumer

Depends on #552

Relates to DependencyTrack/hyades#346
Relates to DependencyTrack/hyades#901
Relates to DependencyTrack/hyades#907

Signed-off-by: nscuro <nscuro@protonmail.com>
nscuro added a commit to DependencyTrack/hyades-apiserver that referenced this issue Feb 26, 2024
@nscuro
Migrate RepositoryMetaResultProcessor from Kafka Streams to Paralle…
d60491e 
…l Consumer

Depends on #552

Relates to DependencyTrack/hyades#346
Relates to DependencyTrack/hyades#901
Relates to DependencyTrack/hyades#907

Signed-off-by: nscuro <nscuro@protonmail.com>
nscuro added a commit to DependencyTrack/hyades-apiserver that referenced this issue Feb 26, 2024
@nscuro
Migrate MirrorVulnerabilityProcessor from Kafka Streams to Parallel…
0f48bdb 
… Consumer

Depends on #552

Relates to DependencyTrack/hyades#346
Relates to DependencyTrack/hyades#901
Relates to DependencyTrack/hyades#907

Signed-off-by: nscuro <nscuro@protonmail.com>
nscuro added a commit to DependencyTrack/hyades-apiserver that referenced this issue Feb 26, 2024
@nscuro
Migrate RepositoryMetaResultProcessor from Kafka Streams to Paralle…
3e655f7 
…l Consumer

Depends on #552

Relates to DependencyTrack/hyades#346
Relates to DependencyTrack/hyades#901
Relates to DependencyTrack/hyades#907

Signed-off-by: nscuro <nscuro@protonmail.com>
nscuro added a commit to DependencyTrack/hyades-apiserver that referenced this issue Mar 4, 2024
@nscuro
Migrate MirrorVulnerabilityProcessor from Kafka Streams to Parallel…
9e5b17f 
… Consumer

Depends on #552

Relates to DependencyTrack/hyades#346
Relates to DependencyTrack/hyades#901
Relates to DependencyTrack/hyades#907

Signed-off-by: nscuro <nscuro@protonmail.com>
nscuro added a commit to DependencyTrack/hyades-apiserver that referenced this issue Mar 4, 2024
@nscuro
Migrate RepositoryMetaResultProcessor from Kafka Streams to Paralle…
feb64d0 
…l Consumer

Depends on #552

Relates to DependencyTrack/hyades#346
Relates to DependencyTrack/hyades#901
Relates to DependencyTrack/hyades#907

Signed-off-by: nscuro <nscuro@protonmail.com>
nscuro added a commit to DependencyTrack/hyades-apiserver that referenced this issue Mar 19, 2024
@nscuro
Migrate MirrorVulnerabilityProcessor from Kafka Streams to Parallel…
5aae50b 
… Consumer

Depends on #552

Relates to DependencyTrack/hyades#346
Relates to DependencyTrack/hyades#901
Relates to DependencyTrack/hyades#907

Signed-off-by: nscuro <nscuro@protonmail.com>
nscuro added a commit to DependencyTrack/hyades-apiserver that referenced this issue Mar 19, 2024
@nscuro
Migrate MirrorVulnerabilityProcessor from Kafka Streams to Parallel…
4aa68ef 
… Consumer

Depends on #552

Relates to DependencyTrack/hyades#346
Relates to DependencyTrack/hyades#901
Relates to DependencyTrack/hyades#907

Signed-off-by: nscuro <nscuro@protonmail.com>
nscuro added a commit to DependencyTrack/hyades-apiserver that referenced this issue Mar 19, 2024
@nscuro
Migrate RepositoryMetaResultProcessor from Kafka Streams to Paralle…
3da1b2b 
…l Consumer

Depends on #552

Relates to DependencyTrack/hyades#346
Relates to DependencyTrack/hyades#901
Relates to DependencyTrack/hyades#907

Signed-off-by: nscuro <nscuro@protonmail.com>
nscuro added a commit to DependencyTrack/hyades-apiserver that referenced this issue Mar 22, 2024
@nscuro
Migrate RepositoryMetaResultProcessor from Kafka Streams to Paralle…
e6ad31f 
…l Consumer

Depends on #552

Relates to DependencyTrack/hyades#346
Relates to DependencyTrack/hyades#901
Relates to DependencyTrack/hyades#907

Signed-off-by: nscuro <nscuro@protonmail.com>
nscuro added a commit to DependencyTrack/hyades-apiserver that referenced this issue Mar 25, 2024
@nscuro
Migrate RepositoryMetaResultProcessor from Kafka Streams to Paralle…
f2221c9 
…l Consumer

Depends on #552

Relates to DependencyTrack/hyades#346
Relates to DependencyTrack/hyades#901
Relates to DependencyTrack/hyades#907

Signed-off-by: nscuro <nscuro@protonmail.com>
@nscuro nscuro mentioned this issue Mar 26, 2024
Merged
2 tasks
@nscuro
Copy link
Member Author

nscuro commented Jun 5, 2024

parallel-consumer was adopted in API server and notification-publisher.

@nscuro nscuro closed this as completed Jun 5, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
domain/repo-meta-analysis domain/vuln-analysis enhancement New feature or request p3 Nice-to-have features performance size/M Medium effort
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@nscuro