Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Pull-based Ingestion] Introduce the new pull-based ingestion engine, APIs, and Kafka plugin #16958

Merged
merged 40 commits into from
Jan 29, 2025
Merged
Changes from 1 commit
Commits
Show all changes
40 commits
Select commit Hold shift + click to select a range
998f530
add ingestion modules
yupeng9 Nov 10, 2024
0b012ef
stream poller wip
yupeng9 Nov 12, 2024
d849664
update ingestion engine
yupeng9 Nov 18, 2024
a76f087
kafka container
yupeng9 Nov 19, 2024
09c8501
more updates
yupeng9 Dec 8, 2024
1445566
local update
yupeng9 Dec 9, 2024
e920115
add batch_start/end to stream poller
yupeng9 Dec 12, 2024
7f21843
add index settings
yupeng9 Dec 16, 2024
f8585a4
local change
yupeng9 Dec 16, 2024
630fa6b
pass docmapper
yupeng9 Dec 19, 2024
5ffb22c
basic recovery
yupeng9 Dec 25, 2024
4231567
add kafka ingestion as plugin
yupeng9 Dec 26, 2024
a9e40ec
add integration test for kafka plugin
yupeng9 Dec 27, 2024
1fff1c8
cleanup
yupeng9 Dec 27, 2024
e8af8e4
use byte[] for message payload type
yupeng9 Dec 27, 2024
41a0280
javadocs
yupeng9 Dec 27, 2024
c0921eb
add ingestionEngineTest
yupeng9 Dec 27, 2024
be5e627
test recovery test in ingestionEngineTest
yupeng9 Dec 27, 2024
25676b8
unit tests for kafka plugin
yupeng9 Dec 28, 2024
5e938ae
style fix
yupeng9 Dec 28, 2024
7e396a5
add license
yupeng9 Dec 28, 2024
2c95b01
more unit tests
yupeng9 Dec 29, 2024
376d028
cleanup
yupeng9 Dec 30, 2024
6243ea5
use a blocking queue to pass polled messages to the processor for pro…
yupeng9 Dec 31, 2024
7cddad5
address comments also remove security policy from bootstrap files
yupeng9 Jan 9, 2025
440e014
support _op_type in message processing
yupeng9 Jan 12, 2025
07fe41b
simplify ingestion source class
yupeng9 Jan 12, 2025
1ee5afb
address more comments
yupeng9 Jan 12, 2025
eb59639
kafka client sha
yupeng9 Jan 12, 2025
bfddcf1
fix style
yupeng9 Jan 12, 2025
76d9602
more style fix
yupeng9 Jan 12, 2025
49f7cbc
add changelog
yupeng9 Jan 18, 2025
e5baca8
use latest gradle feature
yupeng9 Jan 18, 2025
4604d30
try fix thread leak
yupeng9 Jan 18, 2025
6f1f88c
comments
yupeng9 Jan 21, 2025
279b8f6
address comments
yupeng9 Jan 24, 2025
5460a01
ingestionEngine creation failure tes
yupeng9 Jan 26, 2025
8e1b8c3
suppress removal warning
yupeng9 Jan 27, 2025
b10ac20
support lucene10
yupeng9 Jan 27, 2025
acb627e
address comments
yupeng9 Jan 28, 2025
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
local change
Signed-off-by: Yupeng Fu <yupeng@uber.com>
yupeng9 authored and andrross committed Jan 29, 2025

Unverified

This user has not yet uploaded their public signing key.
commit f8585a422a780e1a47a0998c2c7cf7dd1ea4aac7
Original file line number Diff line number Diff line change
@@ -52,6 +52,8 @@
import java.util.function.Function;
import java.util.stream.Collectors;

import static org.gradle.internal.classpath.Instrumented.systemProperty;

/**
* Implementation of the "run" Gradle task used in run.gradle
*/
@@ -179,16 +181,19 @@ public void beforeStart() {
if (dataDir != null) {
node.setDataPath(getDataPath.apply(node));
}

if (debug) {
logger.lifecycle(
"Running opensearch in debug mode (client), {} expecting running debug server on port {}",
node,
debugPort
);
// node.systemProperty("java.security.manager", "");
node.jvmArgs("-agentlib:jdwp=transport=dt_socket,server=n,suspend=y,address=" + debugPort);
debugPort += 1;
} else if (debugServer) {
logger.lifecycle("Running opensearch in debug mode (server), {} running server with debug port {}", node, debugPort);
// node.systemProperty("java.security.manager", "");
node.jvmArgs("-agentlib:jdwp=transport=dt_socket,server=y,suspend=n,address=" + debugPort);
debugPort += 1;
}
2 changes: 1 addition & 1 deletion server/build.gradle
Original file line number Diff line number Diff line change
@@ -114,7 +114,7 @@ dependencies {
// kafka
api "org.slf4j:slf4j-api:${versions.slf4j}"
implementation "org.apache.logging.log4j:log4j-slf4j-impl:${versions.log4j}"
api "org.apache.kafka:kafka-clients:3.9.0"
api "org.apache.kafka:kafka-clients:2.8.2"

// test
// api "org.testcontainers:testcontainers:1.20.3"