Consider SDK 2.0 Release and Regular Release Cadence

[dyladan]

It has now been almost a year since the 1.0 SDK release. In that time, we've learned a lot of lessons and uncovered some weaknesses in the project that we'd like to address. Some of the proposed changes would be considered breaking changes for the SDK. Because of this, I think it is a good time to consider starting work on a 2.0 release of the SDK. Below are some things I would like to address in the 2.0 cycle. If there is something I missed, please comment below or create an issue and add it to the 2.0 Milestone.

Old Typescript Versions

Currently we are using Typescript 4.4. Updating our Typescript installation to a more recent 5.x version would change the outputted .d.ts files in a way that some consumers may also need to upgrade their Typescript installations, making this a breaking change.

Old Runtime Support

We currently support NodeJS 12+. We previously dropped support for 8 and 10 in a minor release which resulted in some blowback from the community.

Remove namespacing to improve minification

Much of our SDK is written using classes and namespaces which make it difficult for bundlers to efficiently minify code. We should consider ways to reduce this wherever possible.

Clean up specification deviations

There are several places where we are deviated from the specification and should move back into spec compliance which would be considered breaking changes. Some of these are simple renames such as InstrumentationLibrary => InstrumentationScope, some are behavioral changes such as disallowing new Span(), and some are deprecations of legacy features such as addNetworkSpanEvents.

Rename structures to better match OTLP

Our internal data structures do not in all cases match the data structure expected by the protobuf library we use for OTLP exporters. Because of this, we require a transformation library in order to convert the data structures to the required format. This results in an extremely bloated bundle size, which is a particular issue for web browsers.

Restructure to allow experimental features in stable packages

Currently, the only way we distinguish experimental features from stable features is by version number. This means that experimental features require their own packages, even when they are meant to be eventually included in an existing stable package.

Remove deprecated features, fields, and methods

There are several places in the code where we've deprecated some fields that should be removed when we go to 2.0

---

Release Cadence Proposal

Semi-Monthly Minor Release

I propose that we schedule a minor release of all stable and experimental SDK components on the first and third wednesday of each month. The PR creation can be automated such that maintainers can simply approve and merge the PR in order to perform the release.

Yearly Major Release

I propose that we schedule a yearly major release, targeting the Fall Kubecon+CloudNativeCon North America for the release announcement. In these releases we can perform regular breaking maintenance such as deprecating old runtimes and updating our Typescript installation, as well as removing deprecated interfaces. Since this is a larger effort, it will likely be a more manual process with development in a non-main branch for a period of multiple weeks or months to ensure we're happy with the results before publishing.

[lizthegrey]

Will this break API compatibility for instrumentation, or only SDK?

[legendecas]

What would the release cadence be like for metrics and traces that have different version history? For example, the first stable version of @opentelemetry/sdk-metrics was released 8 months ago, while @opentelemetry/sdk-trace-base was released 2 years ago. Should we release 2.0 for both of them simultaneously?

[dyladan]

Will this break API compatibility for instrumentation, or only SDK?

only sdk

What would the release cadence be like for metrics and traces that have different version history? For example, the first stable version of @opentelemetry/sdk-metrics was released 8 months ago, while @opentelemetry/sdk-trace-base was released 2 years ago. Should we release 2.0 for both of them simultaneously?

I think it would be easier to release simultaneously and we would receive the most benefit (e.g. removing the transformation library)

[dyladan]

The following represents a proposal. Everything is subject to change following feedback from project contributors, maintainers, users, and leadership. I am documenting this here so that I can present it in the Maintainers meeting Monday August 28.

Timeline

The following represents our expected timeline. Dates are subject to change as the work progresses.

• September - identify areas which can be improved and add them to the 2.0 milestone
• September 30 - all breaking changes are identified, prioritized, and accepted or rejected by maintainers
• Around Kubecon NA 2023 release the first 2.0 alpha
• Around the end of the year release the first 2.0 beta
• Around the start of spring 2024 release the first 2.0 release candidate
• Around Kubecon EU 2024 release the stable 2.0 SDK

Goals

• 100% API compatibility. Existing instrumentation should continue to work.
• Simplify installation and configuration. Reduce number of packages required to be installed by end users. Hide implementation details from users.
• Deprecate support for old runtimes preventing us from moving forward.
• Remove deprecated fields, methods, and features
• Update development tooling including typescript, our test suites, and our monorepo tooling
• Better minification and smaller bundle size
• Better packaging to support both ESM and CommonJS in a simple manner
• Allow for experimental features to be released in stable packages
• Clean separation of signals

Assumptions

• Work will take a significant period of time. To get everything done we want to accomplish, we think a 6 month or more effort is expected.
• This will be the only time we have such a drastic timeline. Future major versions should happen more regularly and contain fewer breaking changes.
• 1.0 and 2.0 lines will diverge dramatically. Several of our goals require major restructures that will make it difficult to cherry pick commits from one line into the other.
• Development of unstable signals will continue as usual.
• Alpha, and later beta, versions of the 2.x line will be released on a regular schedule. During this time breaking changes are expected to occur regularly. Eventually, an RC will be released which is expected to have a reasonable level of stability.

[dyladan]

Proposal 1 - Feature freeze SDK 1.x line

In this scenario, we will stop new feature development on the 1.x lines of our stable signals: traces, metrics, baggage, and context. This will allow us to focus our effort on the 2.0 line with the goal of completing the work faster than if we had continued feature development on the 1.x line. High priority bug fixes will continue to be implemented in the 1.x line. Any pull request author adding a new feature to the 1.x line will be expected to first add it to the 2.x line, but most feature development should target the 2.x line.

[dyladan]

Proposal 2 - Develop new features on both lines

In this scenario, we would require most new features to be developed on both lines in order to be considered. This would likely extend the project timeline, but would ensure that new features continue to be released while work is in progress. The 1.x line is considered to be the “main” line and has priority status for security issues, bug fixes, and feature releases.

[Flarna]

We had some issues with exported classes if different versions of packages were installed. Solution would be to only export interfaces. Not sure if we should aim to get this done in 2.0.

I think there were also discussions to remove the @opentelemetry/core package by moving common stuff needed by some instrumentations into API (e.g. suppressTracing). Not sure if spec allows this meanwhile.

There are a few issues regarding @opentelemetry/instrumentation which should be also cleaned before moving it into GA. But I guess this is a different topic anyway. Or maybe keep autoinstrumentation experimental forever as there is no reasonable language support in JS for this as of now.

[dyladan]

We had some issues with exported classes if different versions of packages were installed. Solution would be to only export interfaces. Not sure if we should aim to get this done in 2.0.

I think there were also discussions to remove the @opentelemetry/core package by moving common stuff needed by some instrumentations into API (e.g. suppressTracing). Not sure if spec allows this meanwhile.

Both seem like reasonable goals for 2.0

There are a few issues regarding @opentelemetry/instrumentation which should be also cleaned before moving it into GA. But I guess this is a different topic anyway. Or maybe keep autoinstrumentation experimental forever as there is no reasonable language support in JS for this as of now.

The instrumentations should only depend on the API so it should be possible to keep that as a separate concern from this.

[dyladan]

I raised this today at the maintainers meeting and the other maintainers seemed to think this was a reasonable idea. The sentiment in the room was that proposal 1 (feature freeze 1.x) would be the preferred solution, although we did not discuss in detail. It was also brought up that we should probably expect to support the 1.x line for 6 months to a year after the 2.0 release.

edit: also, no other SIG has gone to 2.0 yet. The closest is Java Autoinstrumentation, which is going to 2.0 but only to reflect breaking semconv changes introduced in http

[dyladan]

Since it seems we're agreed here I will close this in favor of #4102 where the planning is being tracked.

[cijothomas]

https://github.com/open-telemetry/opentelemetry-specification/blob/main/specification/versioning-and-stability.md#a-note-on-replacing-signals -- it'd be good to remove this wording from the versioning doc.. Not sure why spec had such a language in the first place..

[dyladan]

@cijothomas I think that section still applies. We're not making any breaking changes to the API. In any case, that's not really our decision to make.