Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[exporterhelper] Remove redundant request interfaces #8867

Merged
merged 1 commit into from
Nov 14, 2023

Conversation

dmitryax
Copy link
Member

Pass request+context through the senders pipeline similar to what we do in the collector's pipeline. Use a helper QueueRequest struct for passing requests through queues

This changes also moves the experimental Request interface to a separate package.

@dmitryax dmitryax requested review from a team and codeboten November 13, 2023 22:21
Copy link

codecov bot commented Nov 13, 2023

Codecov Report

Attention: 6 lines in your changes are missing coverage. Please review.

Comparison is base (ffe1a29) 90.87% compared to head (19b766f) 90.90%.

Files Patch % Lines
exporter/exporterhelper/retry_sender.go 83.33% 2 Missing ⚠️
...rter/exporterhelper/internal/persistent_storage.go 93.33% 1 Missing ⚠️
exporter/exporterhelper/logs.go 94.73% 1 Missing ⚠️
exporter/exporterhelper/metrics.go 94.73% 1 Missing ⚠️
exporter/exporterhelper/traces.go 94.73% 1 Missing ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##             main    #8867      +/-   ##
==========================================
+ Coverage   90.87%   90.90%   +0.02%     
==========================================
  Files         317      317              
  Lines       17243    17229      -14     
==========================================
- Hits        15670    15662       -8     
+ Misses       1283     1277       -6     
  Partials      290      290              

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

@dmitryax dmitryax force-pushed the remove-internal-request branch 3 times, most recently from 163a21a to 8ac78a7 Compare November 13, 2023 22:51
@dmitryax dmitryax changed the title [chore] [exporterhelper] Remove redundant request interfaces [exporterhelper] Remove redundant request interfaces Nov 13, 2023
@dmitryax dmitryax force-pushed the remove-internal-request branch from 8ac78a7 to 0f52e10 Compare November 13, 2023 22:57
// Request represents a single request that can be sent to an external endpoint.
// This API is at the early stage of development and may change without backward compatibility
// until https://github.com/open-telemetry/opentelemetry-collector/issues/8122 is resolved.
type Request interface {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

For the moment can we keep this internal, since the usage is still internal anyway.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe keep it in internal for the moment.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The usage is not only internal. This is a public interface used by the new experimental exporterhelper. We are moving to another place

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Let't do that in a separate PR then?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't think we need it in a separate package, that's why I don't want to have it.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think we already had everywhere using the internal.Request, and I am confused why we had to change too many things?

Copy link
Member Author

@dmitryax dmitryax Nov 14, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What I'm suggesting in this PR is to stop using the request and internal.Request and use the original Request in the senders along with the context. I think this change significantly simplifies code. The only place where we need to wrap it is the queue/internal package.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What I am saying is that I don't believe long term we want request in a separate package, so better not to put it there now.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Keep in mind that go is duck typing, which means for the moment you can define the interface twice one in exporter and one in internal and everything works.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ok, I moved the Request back to the exporterhelper package


// RequestMarshaler defines a function which takes a request and marshals it into a byte slice
type RequestMarshaler func(Request) ([]byte, error)
func (qr *QueueRequest) Context() context.Context {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why do you need this if you can access the internal member?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is being used by the queue sender for sending the context to the next sender

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why not passing it in the callback?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I tried that. In that case, the QueueRequest is left with onProcessingFinishedFunc func only. So we can pass them as three fields, but the callback function gets pretty complicated.

Callback func(ctx context.Context, req request.Request, onProcessingFinishedFunc func())

Not sure if keeping the onProcessingFinishedFunc field only warrants having the struct like

type QueueRequest struct {
	request.Request
	onProcessingFinishedFunc func()
}

Let me know WDYT

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe keep onProcessingFinishedFunc as a prop on the Request and wrap the original request in case of the PersistenQueue?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is essentially the same as what's implemented in this PR, just with removed Context field, right?

type QueueRequest struct {
	request.Request
	onProcessingFinishedFunc func()
}

Copy link
Member

@bogdandrutu bogdandrutu Nov 14, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I realized that we need the context to be embedded for the channel interface we want to have in #8828

exporter/exporterhelper/internal/request.go Outdated Show resolved Hide resolved
// Request represents a single request that can be sent to an external endpoint.
// This API is at the early stage of development and may change without backward compatibility
// until https://github.com/open-telemetry/opentelemetry-collector/issues/8122 is resolved.
type Request interface {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe keep it in internal for the moment.

// ItemsCount returns a number of basic items in the request where item is the smallest piece of data that can be
// sent. For example, for OTLP exporter, this value represents the number of spans,
// metric data points or log records.
ItemsCount() int
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

One thing that you need to make sure, especially for batching is that it is very bad for some systems to split the metric across requests. Even if the limit may be at data points, you may want to create an issue for your batcher work.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Makes sense. We can probably add comments to the batcher config recommending users to keep the difference between MinSizeItems and MaxSizeItems high and implement batching for OTLP exporter to make the best effort not to split metrics with those restrictions. WDYT?

Pass request+context through the senders pipeline similar to what we do in the collector's pipeline. Use a helper QueueRequest struct for passing requests through queues
@dmitryax dmitryax force-pushed the remove-internal-request branch from 0f52e10 to 19b766f Compare November 14, 2023 02:08
item.OnProcessingFinished()
Callback: func(qr internal.QueueRequest) {
_ = qs.nextSender.send(qr.Context, qr.Request.(Request))
// TODO: Update OnProcessingFinished to accept error and remove the retry->queue sender callback.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice :)

@bogdandrutu bogdandrutu merged commit a6c0bd4 into open-telemetry:main Nov 14, 2023
@github-actions github-actions bot added this to the next release milestone Nov 14, 2023
bogdandrutu pushed a commit that referenced this pull request Nov 14, 2023
Follow-up to
#8867.
Forgot to update the changelog entry after addressing the PR review
comments
@dmitryax dmitryax deleted the remove-internal-request branch November 14, 2023 04:15
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants