Support a limit on the number of logs

[yurishkuro]

Cf. opentracing/basictracer-go#39

// MaxLogsPerSpan limits the number of Logs in a span if set to a non-zero
// value. If a span has more logs than this value, logs in the middle of the
// span timeline are dropped as necessary and replaced with a log describing
// how many were dropped.
//
// About half of the MaxLogPerSpan logs kept are the oldest logs, and about
// half are the newest logs.
func (tracerOptions) MaxLogsPerSpan(maxLogsPerSpan int) TracerOption {
    return func(tracer *tracer) {
        tracer.options.maxLogsPerSpan = maxLogsPerSpan
    }
}

[oibe]

Do you want this implemented exactly as opentracing does it? Second half of the logs used as a circular buffer?

[yurishkuro]

@funny-falcon do you have a specific business reason to want this feature?

[funny-falcon]

Yes. We duplicate usual logging into tracing, and we have sometime very long running spans that write informational message in a loop. Span log records eat memory then. While we found one of such places, we suppose there could be more. We want just to not worry about.

[yurishkuro]

Nice. Could you share which logging API you are using to fork logs into both stream and spans?

Also, logs are usually written with log levels, so if you need to truncate them in the span it would make sense to do in the order of severity. Which makes me think that a plain maxLogs parameter is too crude of a mechanism, we probably need to use some strategy pattern and create some LogTruncator interface.

[funny-falcon]

We use custom logging library, therefore it were easy to integrate it with tracing: library has concept of exporters that could be added to logger with specified log level, and there were access to context.Context in a log record.

Some considerations from our workflow:

• spans with a lot of log records a rare, therefore we need to look at them almost never.
• if span had some problems, usually log records about a problem will be in among last log records. And setup records goes first. Therefore keeping some of first records and some of last records looks to be a good strategy that satisfies 99% uses.
• since we duplicate log in logging (that flows to ELK), we always could find truncated records there.

Therefore suggested solutions will satisfy our needs completely. And I doubt complication will pay off.