Upgrade Storage Backends to V2 Storage API

[mahadzaryab1]

This is a project proposed as part of LFX Mentorship term #6470 ⬅ read this first.

Background

Jaeger is an open-source, distributed tracing platform designed to monitor and troubleshoot microservices-based systems. A critical component of Jaeger is its storage backends, where traces captured by Jaeger are persisted for querying.

Currently, Jaeger uses a v1 Storage API, which operates on a data model specific to Jaeger. Each storage backend implements this API, requiring transformations between Jaeger's proprietary model and the OpenTelemetry Protocol (OTLP) data model, which is now the industry standard.

As part of #5079, Jaeger has introduced the more efficient v2 Storage API, which natively supports the OpenTelemetry data model (OTLP), allows batching of writes and streaming of resultes. This effort is part of a broader alignment with the OpenTelemetry Collector framework, tracked under #4843.

Objective

Upgrade Jaeger storage backends to natively implement the v2 Storage API.

• Memory
• Elasticsearch / OpenSearch
• Badger
• Cassandra
• gRPC / Remote

The chosen storage backend should be upgraded to fully implement the v2 Storage API in place. For a rough idea of how to upgrade from the v1 model to the OTLP data model, take a look at the PRs in the following issues that do a similar upgrade for other components of Jaeger:

• Transform v1 sanitizers into processors or something else #5545
• Implement adjusters to operate on OTLP data format #6344

Desired Outcomes

Upgrade Memory and Elasticsearch backends

We prioritize these two backends as they are the mostly frequently used with Jaeger and upgrading them paves a path for upgrading other backends.

Testing

• The storage implementations should be fully covered by unit tests
• There are already integration tests for the storage backend so all of them should pass without needing to be modified

Bonus: Upgrade Other Backends

If time permits, upgrade Badger and Cassandra storage backends.

Risks / Open Questions

• The v2 storage API doesn't have a distinction between primary and archive storage but v1 does. The ultimate plan is to remove the archive storage from the v1 implementation as well. That work effort is being tracked in Phase out the distinction between primary and archive storage #6065. We may want to think about how to handle the upgrades for the storages that implement the archive storage in v1 while we work on removing it. We may want to simply ignore the archive part of the storage while we resolve the aforementioned issue if that is possible.

[Devaansh-Kumar]

Hello @mahadzaryab1. I am Devaansh Kumar and I am interested in applying for this project under LFX'25. I have some experience with open source earlier as I had participated in GSoC'24 under Kubernetes.

I have a few questions:

1. What was the reason behind removing the distinction between primary and archive storage for v2 API? It seems intuitive to me to have both in v2.
2. For contributors new to Jaeger’s codebase, are there specific areas or smaller tasks you’d recommend starting with to get familiarized?

Thank you

[yurishkuro]

@Devaansh-Kumar

1. Phase out the distinction between primary and archive storage #6065
2. https://www.jaegertracing.io/get-involved/

[Devaansh-Kumar]

@yurishkuro I had one more question. Do you expect to move the above storages completely to v2 or have both v1 and v2 present side-by-side and give the user choice on which API version to use?

[yurishkuro]

Move completely. We're about to upgrade the write pipeline to use v2 storage API anyway, so we won't need v1 API in the future.

[yp969803]

@yurishkuro love to work on this project in lfx 2025!

[zhengkezhou1]

@yurishkuro @mahadzaryab1 I have gone through the READ path, where we process two types of requests: HTTP requests and gRPC requests. We have set up two processes to handle read requests.

Let's focus on HTTP: The UI sends a request to the HTTP server, and the http_handler routes it to a method, which then calls queryservice to retrieve the required data. Currently, we are still using v1 methods and models. For example, in http_handler.search, we use model.Trace and qs.spanReader.FindTraces instead of ptrace.Traces and qs.traceReader.FindTraces. The retrieved model is then serialized into a JSON object.

For the full request flow now, trace data undergoes two conversions:
Database model → Jaeger v1 model → JSON object

Questions:
Is it necessary to refactor http_handler and grpc_handler to v2 before working on this issue?
Based on Jaeger v2 for Otel collector, the read path allows us to convert the OTEL model directly into a JSON object. Can we achieve the same for our backend storages?
I found that the following storages might support this approach:
Elasticsearch
OpenSearch
Kafka
Clickhouse
Cassandra

[Manik2708]

Is it necessary to refactor http_handler and grpc_handler to v2 before working on this issue?

It is already changed according to query service v2. See these files: https://github.com/jaegertracing/jaeger/blob/main/cmd/query/app/apiv3/http_gateway.go and https://github.com/jaegertracing/jaeger/blob/main/cmd/query/app/apiv3/grpc_handler.go

[Manik2708]

@mahadzaryab1 @yurishkuro I have a doubt regarding this issue. Let's take an example of upgrading Writer from v1 to v2. Writer of v1 writes only 1 span whereas that of v2 takes traces to write. So while implementing writer for ElasticSearch, what is expected? Do we have to use bulk apis of elasticsearch or something else? Please see this: https://www.elastic.co/guide/en/elasticsearch/reference/current/docs-bulk.html Because we are not storing traces but spans. A similar question for memory storage. In memory do we only need to iterate the traces and store the spans or something like batch writes?

[zhengkezhou1]

@mahadzaryab1 @yurishkuro I have a doubt regarding this issue. Let's take an example of upgrading Writer from v1 to v2. Writer of v1 writes only 1 span whereas that of v2 takes traces to write. So while implementing writer for ElasticSearch, what is expected? Do we have to use bulk apis of elasticsearch or something else? Please see this: https://www.elastic.co/guide/en/elasticsearch/reference/current/docs-bulk.html Because we are not storing traces but spans. A similar question for memory storage. In memory do we only need to iterate the traces and store the spans or something like batch writes?

Elasticsearch already implemented a method for writing traces into storage.

[Manik2708]

@zzzk1 Thanks for this! So are you saying to take inspiration from this and implement our own method? Because I don't think we can use this directly in jaeger. And for memory we have to discuss for the implementaion because that is jaeger specific!

[zhengkezhou1]

@zzzk1 Thanks for this! So are you saying to take inspiration from this and implement our own method? Because I don't think we can use this directly in jaeger. And for memory we have to discuss for the implementaion because that is jaeger specific!

Yes, The pushTraceData method demonstrates how to send traces to Elasticsearch (ES). In Jaeger v2, built on the OpenTelemetry (OTEL) collector, data is first collected from the pipeline and efficiently(without model convert) written to storage using the v2 Writer interface via the WriteTraces(ctx context.Context, td ptrace.Traces) method.

[yurishkuro]

Is it necessary to refactor http_handler and grpc_handler to v2 before working on this issue?

@zzzk1 no, it's not. Yes, both these handlers operate on querysvc/v1, but internally that service is already instantiated with v2 storage and it just downgrades it to v1 if necessary

jaeger/cmd/query/app/querysvc/query_service.go

Lines 57 to 63 in 1ac7d6d

	func NewQueryService(traceReader tracestore.Reader, dependencyReader depstore.Reader, options QueryServiceOptions) *QueryService {
	spanReader, ok := v1adapter.GetV1Reader(traceReader)
	if !ok {
	// if the spanstore.Reader is not available, downgrade the native tracestore.Reader to
	// a spanstore.Reader
	spanReader = v1adapter.NewSpanReader(traceReader)
	}

[Manik2708]

@yurishkuro I could understand that we want to get rid of the conversion from OTEL model but do we need to change the id which is getting stored in the database? If yes, then how we can ensure backward compatibilty? Or should we move forward by converting the OTEL span id to jaeger specific id?

[yurishkuro]

the IDs are compatible, and besides Jaeger treats them as completely opaque (same as OTEL).

[adityachopra29]

@yurishkuro Is there any specific reason why we have implemented the tenant struct's services set as a map with the value section being an empty struct instead of a simpler array? Similarly, the operations map's value part is a map whose value portion is an empty struct.

jaeger/internal/storage/v1/memory/memory.go

Line 31 in c78f5c3

// Tenant is an in-memory store of traces for a single tenant

[yurishkuro]

map[string]struct{} is a common way to model a set.

[Manik2708]

@yurishkuro The changes in memory storage are not expected to be backward compatible, right? I can only imagine restarting the storage after the update which will eventually lead to loss of old data (because it's in-memory). Am I right? Or is there any backup mechanism?

[yurishkuro]

@Manik2708 being backwards compatible has nothing to do with persistence / backup mechanism. Memory storage is transient, it's a cache. Upgrading it to v2 is just changing its API, not it's features.

[metabiswadeep]

@yurishkuro I am interested in this opportunity. I am interested in both this and the UI. Can I add an application for both?

[Manik2708]

@Manik2708 being backwards compatible has nothing to do with persistence / backup mechanism. Memory storage is transient, it's a cache. Upgrading it to v2 is just changing its API, not it's features.

Actually the Tenant structure looks like this:

jaeger/internal/storage/v1/memory/memory.go

Lines 32 to 41 in 316cbe3

	type Tenant struct {
	sync.RWMutex
	ids []*model.TraceID
	traces map[model.TraceID]*model.Trace
	services map[string]struct{}
	operations map[string]map[spanstore.Operation]struct{}
	deduper adjuster.Adjuster
	config Configuration
	index int
	}

For changing the API we need to change the jaeger specific models to OTEL specific models. Or am I understanding wrong? Do we just need to change the APIs and then convert the OTEL models to jaeger models and keeping the store intact and same?

[ary82]

but shouldn't it be:

GetTraces(ctx context.Context, traceIDs ...GetTraceParams) iter.Seq2[ptrace.Traces, error] // Returning an iterator of traces rather than iterator of slice of traces

@Manik2708 I may be wrong here, but I think it's because a trace might span over multiple otlp ptrace.Traces. A ptrace.Traces contains ptrace.Spans grouped over a particular Resource/Service. But a trace can span over multiple services. So that means for a complete trace, multiple ptrace.Traces may be required.
Let's wait for the confirmation though.

[Manik2708]

Maybe you are right! That's why I wrote processing the traces. GetTraces is taking multiple trace ids (earlier it was a single trace id). This confused me because trace ID is unique so I thought why we need multiple traces for a single trace id. But then I am now doubtful that it can be possible if distinguishing factor is not ID but only resource/service.

[yurishkuro]

The method returns a slice in each iteration because the API is optimized for storage communication, and the storage may find it more efficient to return a single payload with multiple traces, eg one iteration with 10 traces.

[Manik2708]

@yurishkuro @mahadzaryab1 There is a grouping which happen when []model.Span are converted to ptrace.Traces. The difference between dbmodel.Span and model.Span is that we have batch in model but not in dbmodel. This grouping happens on these constraints:

1. If process of batch is present and is absent for span then the presedance is given to span's process. This means that span will have different resource than other spans. (This same option will be chosen when batch doesn't have process)
2. If batch has process and span doesn't then span is added to the same resource span slice as that of other spans of same batch. (In the end the we have to group spans with same process together; called as resource)

Now the problem in dbmodel is that we don't have anything like batch, so we have two options:

1. Simply remove this grouping and add every span to different resource span: The problem in this is that the spans have same process (service name and tags) will not be grouped together. This is not aligned with the OTLP as spans with same resource should be grouped together. Apparently this is what currently taking place. In V1Adapter we are sending nil for batch.Process so at the end the first contraint is being followed.
2. Add a new grouping similar to the old one. The probable route will be:
   1. Create a hash function for dbmodel process similar to model process.
   2. Make a map between hash (integer) value and []*dbmodel.Span and group the spans with same process.

How should we proceed further?

[yurishkuro]

The batch concept is irrelevant at this phase. It is only meaningful when the spans are being received by the collector from applications where the batch is used to reduce the repetition of Process struct that is usually identical for all spans coming from the same application. The equivalent concept in OTLP for that is Resource.

By the time spans get to the storage, the batch is deconstructed and Process is populated on every span. Then in v1 storage the dbspan is structured similarly to model.Span, i.e. they both have span.Process.

So for v2, when converting from OTLP (to_dbmodel), the Resource should be denormalized into Process in each individual span. When converting from_dbmodel, there are two options:

• each span.Process can be converted into its own Resource. While there may be some repetition, it's probably not worth optimizing for (or can be done later)
• if we were to optimize that, then we'd need to group spans by identical Process contents.

[Manik2708]

Ok, let's move by removing grouping!