Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

integrate and implement open telemetry tracing #18534

Merged
merged 1 commit into from
Dec 1, 2022
Merged

integrate and implement open telemetry tracing #18534

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
61 changes: 61 additions & 0 deletions pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -189,6 +189,7 @@
<module>presto-hudi</module>
<module>presto-native-execution</module>
<module>presto-router</module>
<module>presto-open-telemetry</module>
</modules>

<dependencyManagement>
Expand Down Expand Up @@ -850,6 +851,12 @@
<version>${project.version}</version>
</dependency>

<dependency>
<groupId>com.facebook.presto</groupId>
<artifactId>presto-open-telemetry</artifactId>
<version>${project.version}</version>
</dependency>

<dependency>
<groupId>com.facebook.hive</groupId>
<artifactId>hive-dwrf</artifactId>
Expand Down Expand Up @@ -2094,6 +2101,60 @@
<artifactId>stream</artifactId>
<version>2.9.5</version>
</dependency>

<dependency>
<groupId>io.opentelemetry</groupId>
<artifactId>opentelemetry-api</artifactId>
<version>1.19.0</version>
</dependency>

<dependency>
<groupId>io.opentelemetry</groupId>
<artifactId>opentelemetry-context</artifactId>
<version>1.19.0</version>
</dependency>

<dependency>
<groupId>io.opentelemetry</groupId>
<artifactId>opentelemetry-exporter-otlp</artifactId>
<version>1.19.0</version>
<exclusions>
<exclusion>
<groupId>com.squareup.okhttp3</groupId>
<artifactId>okhttp</artifactId>
</exclusion>
</exclusions>
</dependency>

<dependency>
<groupId>io.opentelemetry</groupId>
<artifactId>opentelemetry-extension-trace-propagators</artifactId>
<version>1.19.0</version>
</dependency>

<dependency>
<groupId>io.opentelemetry</groupId>
<artifactId>opentelemetry-sdk</artifactId>
<version>1.19.0</version>
</dependency>

<dependency>
<groupId>io.opentelemetry</groupId>
<artifactId>opentelemetry-sdk-common</artifactId>
<version>1.19.0</version>
</dependency>

<dependency>
<groupId>io.opentelemetry</groupId>
<artifactId>opentelemetry-sdk-trace</artifactId>
<version>1.19.0</version>
</dependency>

<dependency>
<groupId>io.opentelemetry</groupId>
<artifactId>opentelemetry-semconv</artifactId>
<version>1.19.0-alpha</version>
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is this intended? Or we have a stable release?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, no stable release exists. Versions of each package to use are stated here https://github.com/open-telemetry/opentelemetry-java

</dependency>
</dependencies>
</dependencyManagement>

Expand Down
45 changes: 33 additions & 12 deletions presto-main/src/main/java/com/facebook/presto/server/HttpRequestSessionContext.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -22,6 +22,7 @@
import com.facebook.presto.spi.security.SelectedRole;
import com.facebook.presto.spi.session.ResourceEstimates;
import com.facebook.presto.spi.tracing.Tracer;
import com.facebook.presto.spi.tracing.TracerHandle;
import com.facebook.presto.spi.tracing.TracerProvider;
import com.facebook.presto.sql.parser.ParsingException;
import com.facebook.presto.sql.parser.ParsingOptions;
Expand Down Expand Up @@ -211,19 +212,39 @@ else if (nameParts.size() == 2) {

this.sessionFunctions = parseSessionFunctionHeader(servletRequest);
this.sessionPropertyManager = requireNonNull(sessionPropertyManager, "sessionPropertyManager is null");
String tunnelTraceId = trimEmptyToNull(servletRequest.getHeader(PRESTO_TRACE_TOKEN));
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I know it's a bit anti pattern; but in order to keep the logic so we don't pollute the traceToken below. Check my comment below.


Map<String, String> requestHeaders = getRequestHeaders(servletRequest);
TracerHandle tracerHandle = tracerProvider.getHandleGenerator().apply(requestHeaders);

if (isTracingEnabled()) {
this.tracer = Optional.of(requireNonNull(tracerProvider.getNewTracer(), "tracer is null"));
this.tracer = Optional.of(requireNonNull(tracerProvider.getNewTracer(tracerHandle), "tracer is null"));
traceToken = Optional.ofNullable(this.tracer.get().getTracerId());
}
else {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Add a new branch here: else if (trimEmptyToNull(servletRequest.getHeader(PRESTO_TRACE_TOKEN)) != null) or something like that. And inside the body, we can either create a SimpleTracerProvider and get the token or directly assign traceToken from PRESTO_TRACE_TOKEN

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

updated trace token to replicate similar logic as before

this.tracer = Optional.of(NoopTracerProvider.NOOP_TRACER);

// If tunnel trace token is null, we expose the Presto tracing id.
// Otherwise we preserve the ability of trace token tunneling but
// still trace Presto internally for aggregation purposes.
traceToken = Optional.ofNullable(tunnelTraceId == null ? this.tracer.get().getTracerId() : tunnelTraceId);
String tunnelTraceId = trimEmptyToNull(servletRequest.getHeader(PRESTO_TRACE_TOKEN));
if (tunnelTraceId != null) {
traceToken = Optional.of(tunnelTraceId);
}
else {
traceToken = Optional.ofNullable(tracerHandle.getTraceToken());
}
}
else {
this.tracer = Optional.of(NoopTracerProvider.NOOP_TRACER);
traceToken = Optional.ofNullable(tunnelTraceId);
}

private static Map<String, String> getRequestHeaders(HttpServletRequest servletRequest)
{
ImmutableMap.Builder<String, String> headers = ImmutableMap.builder();
Enumeration<String> headerNames = servletRequest.getHeaderNames();
while (headerNames.hasMoreElements()) {
String header = headerNames.nextElement();
headers.put(header, servletRequest.getHeader(header));
}
return headers.build();
}

public static List<String> splitSessionHeader(Enumeration<String> headers)
Expand Down Expand Up @@ -502,7 +523,7 @@ public Optional<Tracer> getTracer()
*/
private boolean isTracingEnabled()
{
String clientValue = systemProperties.getOrDefault(DISTRIBUTED_TRACING_MODE, TracingConfig.DistributedTracingMode.NO_TRACE.name());
String clientValue = systemProperties.getOrDefault(DISTRIBUTED_TRACING_MODE, "");
Comment on lines -505 to +526
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why is this change?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

When the property isn't set by the user, it will allow the system default to be used instead of defaulting to no trace. It should be a nice feature to have so applications do not need to change their presto client request to enable tracing, but only need to change their presto application properties.


// Client session setting overrides everything.
if (clientValue.equalsIgnoreCase(TracingConfig.DistributedTracingMode.ALWAYS_TRACE.name())) {
Expand All @@ -511,13 +532,13 @@ private boolean isTracingEnabled()
if (clientValue.equalsIgnoreCase(TracingConfig.DistributedTracingMode.NO_TRACE.name())) {
return false;
}
if (clientValue.equalsIgnoreCase(TracingConfig.DistributedTracingMode.SAMPLE_BASED.name())) {
return true;
}

// Client not set, we then take system default value, and only init
// tracing if it's SAMPLE_BASED (TracingConfig prohibits you to
// configure system default to be ALWAYS_TRACE). If property manager
// not provided then false.
// Client not set, we then take system default value if ALWAYS_TRACE (SAMPLE_BASED disabled). If property manager not provided then false.
return sessionPropertyManager
.map(manager -> manager.decodeSystemPropertyValue(DISTRIBUTED_TRACING_MODE, null, TracingConfig.DistributedTracingMode.class) == TracingConfig.DistributedTracingMode.SAMPLE_BASED)
.map(manager -> manager.decodeSystemPropertyValue(DISTRIBUTED_TRACING_MODE, null, String.class).equalsIgnoreCase(TracingConfig.DistributedTracingMode.ALWAYS_TRACE.name()))
.orElse(false);
}

Expand Down
12 changes: 11 additions & 1 deletion presto-main/src/main/java/com/facebook/presto/server/PluginManager.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -39,9 +39,11 @@
import com.facebook.presto.spi.session.SessionPropertyConfigurationManagerFactory;
import com.facebook.presto.spi.statistics.HistoryBasedPlanStatisticsProvider;
import com.facebook.presto.spi.storage.TempStorageFactory;
import com.facebook.presto.spi.tracing.TracerProvider;
import com.facebook.presto.spi.ttl.ClusterTtlProviderFactory;
import com.facebook.presto.spi.ttl.NodeTtlFetcherFactory;
import com.facebook.presto.storage.TempStorageManager;
import com.facebook.presto.tracing.TracerProviderManager;
import com.facebook.presto.ttl.clusterttlprovidermanagers.ClusterTtlProviderManager;
import com.facebook.presto.ttl.nodettlfetchermanagers.NodeTtlFetcherManager;
import com.google.common.collect.ImmutableList;
Expand Down Expand Up @@ -117,6 +119,7 @@ public class PluginManager
private final AtomicBoolean pluginsLoaded = new AtomicBoolean();
private final ImmutableSet<String> disabledConnectors;
private final HistoryBasedPlanStatisticsManager historyBasedPlanStatisticsManager;
private final TracerProviderManager tracerProviderManager;

@Inject
public PluginManager(
Expand All @@ -134,7 +137,8 @@ public PluginManager(
SessionPropertyDefaults sessionPropertyDefaults,
NodeTtlFetcherManager nodeTtlFetcherManager,
ClusterTtlProviderManager clusterTtlProviderManager,
HistoryBasedPlanStatisticsManager historyBasedPlanStatisticsManager)
HistoryBasedPlanStatisticsManager historyBasedPlanStatisticsManager,
TracerProviderManager tracerProviderManager)
{
requireNonNull(nodeInfo, "nodeInfo is null");
requireNonNull(config, "config is null");
Expand Down Expand Up @@ -162,6 +166,7 @@ public PluginManager(
this.clusterTtlProviderManager = requireNonNull(clusterTtlProviderManager, "clusterTtlProviderManager is null");
this.disabledConnectors = requireNonNull(config.getDisabledConnectors(), "disabledConnectors is null");
this.historyBasedPlanStatisticsManager = requireNonNull(historyBasedPlanStatisticsManager, "historyBasedPlanStatisticsManager is null");
this.tracerProviderManager = requireNonNull(tracerProviderManager, "tracerProviderManager is null");
}

public void loadPlugins()
Expand Down Expand Up @@ -297,6 +302,11 @@ public void installPlugin(Plugin plugin)
log.info("Registering plan statistics provider %s", historyBasedPlanStatisticsProvider.getName());
historyBasedPlanStatisticsManager.addHistoryBasedPlanStatisticsProviderFactory(historyBasedPlanStatisticsProvider);
}

for (TracerProvider tracerProvider : plugin.getTracerProviders()) {
log.info("Registering tracer provider %s", tracerProvider.getName());
tracerProviderManager.addTracerProviderFactory(tracerProvider);
}
}

private URLClassLoader buildClassLoader(String plugin)
Expand Down
2 changes: 2 additions & 0 deletions presto-main/src/main/java/com/facebook/presto/server/PrestoServer.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -50,6 +50,7 @@
import com.facebook.presto.sql.parser.SqlParserOptions;
import com.facebook.presto.storage.TempStorageManager;
import com.facebook.presto.storage.TempStorageModule;
import com.facebook.presto.tracing.TracerProviderManager;
import com.facebook.presto.ttl.clusterttlprovidermanagers.ClusterTtlProviderManager;
import com.facebook.presto.ttl.clusterttlprovidermanagers.ClusterTtlProviderManagerModule;
import com.facebook.presto.ttl.nodettlfetchermanagers.NodeTtlFetcherManager;
Expand Down Expand Up @@ -174,6 +175,7 @@ public void run()
injector.getInstance(QueryPrerequisitesManager.class).loadQueryPrerequisites();
injector.getInstance(NodeTtlFetcherManager.class).loadNodeTtlFetcher();
injector.getInstance(ClusterTtlProviderManager.class).loadClusterTtlProvider();
injector.getInstance(TracerProviderManager.class).loadTracerProvider();

startAssociatedProcesses(injector);

Expand Down
16 changes: 2 additions & 14 deletions presto-main/src/main/java/com/facebook/presto/server/ServerMainModule.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -149,7 +149,6 @@
import com.facebook.presto.spi.relation.DomainTranslator;
import com.facebook.presto.spi.relation.PredicateCompiler;
import com.facebook.presto.spi.relation.VariableReferenceExpression;
import com.facebook.presto.spi.tracing.TracerProvider;
import com.facebook.presto.spiller.FileSingleStreamSpillerFactory;
import com.facebook.presto.spiller.GenericPartitioningSpillerFactory;
import com.facebook.presto.spiller.GenericSpillerFactory;
Expand Down Expand Up @@ -195,8 +194,7 @@
import com.facebook.presto.sql.tree.Expression;
import com.facebook.presto.sql.tree.FunctionCall;
import com.facebook.presto.statusservice.NodeStatusService;
import com.facebook.presto.tracing.NoopTracerProvider;
import com.facebook.presto.tracing.SimpleTracerProvider;
import com.facebook.presto.tracing.TracerProviderManager;
import com.facebook.presto.tracing.TracingConfig;
import com.facebook.presto.transaction.TransactionManagerConfig;
import com.facebook.presto.type.TypeDeserializer;
Expand Down Expand Up @@ -245,8 +243,6 @@
import static com.facebook.drift.server.guice.DriftServerBinder.driftServerBinder;
import static com.facebook.presto.execution.scheduler.NodeSchedulerConfig.NetworkTopologyType.FLAT;
import static com.facebook.presto.execution.scheduler.NodeSchedulerConfig.NetworkTopologyType.LEGACY;
import static com.facebook.presto.tracing.TracingConfig.TracerType.NOOP;
import static com.facebook.presto.tracing.TracingConfig.TracerType.SIMPLE;
import static com.google.common.base.Preconditions.checkArgument;
import static com.google.common.base.Strings.nullToEmpty;
import static com.google.common.util.concurrent.MoreExecutors.listeningDecorator;
Expand Down Expand Up @@ -741,15 +737,7 @@ public ListeningExecutorService createResourceManagerExecutor(ResourceManagerCon

// Distributed tracing
configBinder(binder).bindConfig(TracingConfig.class);
install(installModuleIf(
TracingConfig.class,
config -> !config.getEnableDistributedTracing() || NOOP.equalsIgnoreCase(config.getTracerType()),
moduleBinder -> moduleBinder.bind(TracerProvider.class).to(NoopTracerProvider.class).in(Scopes.SINGLETON)));

install(installModuleIf(
TracingConfig.class,
config -> config.getEnableDistributedTracing() && SIMPLE.equalsIgnoreCase(config.getTracerType()),
moduleBinder -> moduleBinder.bind(TracerProvider.class).to(SimpleTracerProvider.class).in(Scopes.SINGLETON)));
binder.bind(TracerProviderManager.class).in(Scopes.SINGLETON);

//Optional Status Detector
newOptionalBinder(binder, NodeStatusService.class);
Expand Down
10 changes: 5 additions & 5 deletions presto-main/src/main/java/com/facebook/presto/server/protocol/QueuedStatementResource.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -30,8 +30,8 @@
import com.facebook.presto.server.SessionContext;
import com.facebook.presto.spi.PrestoException;
import com.facebook.presto.spi.QueryId;
import com.facebook.presto.spi.tracing.TracerProvider;
import com.facebook.presto.sql.parser.SqlParserOptions;
import com.facebook.presto.tracing.TracerProviderManager;
import com.google.common.collect.ImmutableList;
import com.google.common.collect.ImmutableSet;
import com.google.common.collect.Ordering;
Expand Down Expand Up @@ -125,7 +125,7 @@ public class QueuedStatementResource
private final boolean compressionEnabled;

private final SqlParserOptions sqlParserOptions;
private final TracerProvider tracerProvider;
private final TracerProviderManager tracerProviderManager;
private final SessionPropertyManager sessionPropertyManager; // We may need some system default session property values at early query stage even before session is created.

private final QueryBlockingRateLimiter queryRateLimiter;
Expand All @@ -138,7 +138,7 @@ public QueuedStatementResource(
LocalQueryProvider queryResultsProvider,
SqlParserOptions sqlParserOptions,
ServerConfig serverConfig,
TracerProvider tracerProvider,
TracerProviderManager tracerProviderManager,
SessionPropertyManager sessionPropertyManager,
QueryBlockingRateLimiter queryRateLimiter)
{
Expand All @@ -149,7 +149,7 @@ public QueuedStatementResource(

this.responseExecutor = requireNonNull(executor, "responseExecutor is null").getExecutor();
this.timeoutExecutor = requireNonNull(executor, "timeoutExecutor is null").getScheduledExecutor();
this.tracerProvider = requireNonNull(tracerProvider, "tracerProvider is null");
this.tracerProviderManager = requireNonNull(tracerProviderManager, "tracerProviderManager is null");
this.sessionPropertyManager = sessionPropertyManager;

this.queryRateLimiter = requireNonNull(queryRateLimiter, "queryRateLimiter is null");
Expand Down Expand Up @@ -215,7 +215,7 @@ public Response postStatement(
SessionContext sessionContext = new HttpRequestSessionContext(
servletRequest,
sqlParserOptions,
tracerProvider,
tracerProviderManager.getTracerProvider(),
Optional.of(sessionPropertyManager));
Query query = new Query(statement, sessionContext, dispatchManager, queryResultsProvider, 0);
queries.put(query.getQueryId(), query);
Expand Down
4 changes: 3 additions & 1 deletion presto-main/src/main/java/com/facebook/presto/testing/LocalQueryRunner.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -201,6 +201,7 @@
import com.facebook.presto.sql.tree.Statement;
import com.facebook.presto.sql.tree.TruncateTable;
import com.facebook.presto.testing.PageConsumerOperator.PageConsumerOutputFactory;
import com.facebook.presto.tracing.TracerProviderManager;
import com.facebook.presto.tracing.TracingConfig;
import com.facebook.presto.transaction.InMemoryTransactionManager;
import com.facebook.presto.transaction.TransactionManager;
Expand Down Expand Up @@ -480,7 +481,8 @@ private LocalQueryRunner(Session defaultSession, FeaturesConfig featuresConfig,
new SessionPropertyDefaults(nodeInfo),
new ThrowingNodeTtlFetcherManager(),
new ThrowingClusterTtlProviderManager(),
historyBasedPlanStatisticsManager);
historyBasedPlanStatisticsManager,
new TracerProviderManager(new TracingConfig()));

connectorManager.addConnectorFactory(globalSystemConnectorFactory);
connectorManager.createConnection(GlobalSystemConnector.NAME, GlobalSystemConnector.NAME, ImmutableMap.of());
Expand Down
33 changes: 33 additions & 0 deletions presto-main/src/main/java/com/facebook/presto/tracing/NoopTracerHandle.java
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,33 @@
/*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package com.facebook.presto.tracing;

import com.facebook.presto.spi.tracing.TracerHandle;

public class NoopTracerHandle
implements TracerHandle
{
private final String traceToken;

public NoopTracerHandle()
{
this.traceToken = "noop_dummy_id";
}

@Override
public String getTraceToken()
{
return traceToken;
}
}
Loading