Revamp the TC

[anuraaga]

It's finally time to throw this out there.

I generally find issues in shared repos like the spec, proto, etc, to move slowly, and generally lack leadership. The norm is that an issue may move just a little once per week during a sig meeting, or not at all.

I thought this might be because of a lack of definition of responsibility but I found a list here

https://github.com/open-telemetry/community/blob/main/tech-committee-charter.md#responsibilities-of-the-technical-committee

This in particular stands out

Mediating technical discussions which have cross project impact

I don't think this happens. Perhaps the wording is too soft and I expect something more proactive - "move discussions forward". Though more important is the facet to make sure contributors are supported.

I think the TC needs to be more meaningful - one big issue seems to be the lack of any way of making sure the TC fulfills its responsibilities.

TC memberships are not time-limited. There is no maximum size of the TC. The size is expected to vary in order to ensure adequate coverage of important areas of expertise, balanced with the ability to make decisions efficiently.

With no term limit or significant participant process (I remember a huge amount of process for the last GC vote but almost nothing for TC elections) I see no way to ensure that TC members will actually implement this charter.

I want to emphasize that no one person comes to mind while writing this - it's also why I don't want to link to any specific issues / PRs that seem to move without the direction that, at least I would expect from a TC. If you need examples ping me, but if this doesn't resonate at all I would be disappointed.

I think it could be an appropriate time to reconsider and restructure the TC from scratch, focusing on community rather than vendor aspects. If not this may at least jog minds that the TC should be moving things forward in an encouraging way. Fulfil the charter.

As always, I'll link to GitHub's guide which has never felt like it's been implemented in this community (well, some will know I generally refer to OTel as a vendorfest, not a community).

https://opensource.guide/

[anuraaga]

I forgot one important point - it could be the GC or any other XC that drives the project forward. I guess the importance is to have full time commitment. But looking at the charters currently TC seems to fit the bill most appropriately.

[carlosalberto]

it's also why I don't want to link to any specific issues / PRs that seem to move without the direction that, at least I would expect from a TC. If you need examples ping me

Definitely examples would be appreciated, in order to have a very clear and concise idea, in case something has doubts regarding this (and detailed scenarios).

Overall I think I get why you (and other people in the past) feel this way. @trask and @alolita come to my mind. I think that one of the fundamental problems is that TC members do not often have as many cycles for OTel as they wanted. If there's no time to spend on the project, probably the rest of the discussion cannot take place.

Thanks for the feedback, and independently of whether we entirely revamp the TC or adjust it, I think this is good feedback, as it shows there's a lot to be done still.

[bhs]

@anuraaga thank you for raising this and for the feedback.

I'm glad to have a discussion about the structural aspects of the TC (election process, terms, etc), though I'd suggest that we start by deciding what problem we're solving.

IMO, at a project level, the problem we've been having is roadmap slippage, and – in some places – roadmap fuzziness. As a related problem, we have scope creep, though that wouldn't bother me much if we hit our dates. @anuraaga since you focused on "moving the project forward," I'm inferring that this resonates with you?

So, is that the overarching problem, particularly as it affects OTel end-users and contributors? If not, what is it?

[anuraaga]

I think I'm looking more at moving forward at a micro level than the macro level - for OSS projects I think we hope that when an issue is filed, someone owns getting that to resolution, similar for PRs or general discussions. I'd compare to projects with a single BDFL, where it tends to be obvious that that BDFL, who's naturally highly dedicated to the project, will be very proactive in responding to all the contributions, issues, and PRs, that come in. This makes contributors happy, creating a happy ecosystem that can work itself. I think our language SIGs do exhibit this well, possibly because they have fewer maintainers usually.

I don't see similar proactive behavior or commitment to ownership by the TC in TC-owned repos. Issues can be left unanswered for days or weeks, PRs often only advance in that small time window after the spec SIG each week, and it generally doesn't feel like the TC's "got our back". I suspect that is one of the goals of the TC - if not, I think it's a missing component of the project.

Indeed perhaps this is too much focus on the TC as it does make an assumption on their role - the overarching problem is probably more generally stated as there doesn't seem to be clear ownership of the contributor experience and community, and I think it's important for making sure contributors continue to help us help them. If this is in the TC's scope, then it probably needs to be made more explicit, and there needs to be confirmation that the members are committed to it. If it's not in scope, a separate team could be important.

Tangentially, I think another reason for lack of responses on issues, etc, is that OTel is extremely skewed towards vendors - looking at the TC, I guess there is one member, Yuri, who's company is on the user side of observability, though admittedly the cloud providers can be considered both perhaps. But I do see that issues that are interesting in some way to that vendor tend to be acted on most actively, even when there are other user needs getting less attention. I think things would get more balanced if there was more representation from the big observability users too rather than just the vendors.

@arminru reached out on Slack and I've pinged him some more detailed examples. I don't think any one person is doing anything wrong or being bad, so don't want to call anyone out. I think it's more structural to the current format of the TC, or possibly the need for a separate committee to augment it with a focus on users and contributors, not just vendors.

[bhs]

the overarching problem is probably more generally stated as there doesn't seem to be clear ownership of the contributor experience and community

This is a fair critique. My opinion is that this is not because much of the day-to-day project work is funded by vendors, though... rather, and as you point out above, it's more about the lack of a BDFL (or "BDFL-like entity") in OTel. Perhaps this is what you would like the TC to be or become: an entity that can make decisions quickly, even if there's going to be a necessary bias towards that entity's particular ["benevolent"] point of view.

In any case, OTel certainly has no BDFL. This is not uncommon for projects that need to find common ground between mannnyyy constituents, as we do with OTel. But it can create delays and a lack of accountability for quick, decisive outcomes, large and small.

I could write more but I'm curious to hear what others think here... cc @open-telemetry/governance-committee @open-telemetry/technical-committee

[tigrannajaryan]

@anuraaga Thanks for raising the issue.

Examples would definitely help to understand what exactly is the problem. Absent examples I can think of a few possible potential problems:

• TCs do not make enough decisions when necessary. Personally I aim to give room to maintainers and contributors to make their decisions and avoid top-down decision making. This has been my approach and I think of the TC in general. If the community believes that this is the wrong approach and we need more BDFL-style decisions coming from the TCs we can do it, but I would like the community as a whole to confirm that this is desirable. In a sense the current approach is by design, but if we as a community believe the design is wrong we can change it.
• TCs lack initiative and do not make proposals to drive the project forward. I do not think we have this problem but I may be wrong.
• TCs do not do enough to review the proposals from other contributors, are too slow when reviewing, etc. If you believe this is the case I think it would be useful to point to some example PRs or OTEPs that show this.
• TCs do not do enough cross-SIG coordination, to discover inter-SIG inconsistencies, to connect the dots between problems in different areas or signals, etc.

If you could help identify which of these problems you see it would help us to come up with the right solutions.

[tedsuo]

On a practical level, it feels to me like TC members are in two groups:

• People who have a huge code-level responsibility to OpenTelemetry. They are heavily involved in OTel on a daily basis, but have so many cycles taken up by these tasks and responsibilities that they are spread thin and simply over-clocked, and the remaining spec backlog does not get attention.
• People who have large responsibilities outside of OTel. They mean well and give valuable feedback, but their time is limited.

I'm happy to provide advice and assistance, but it feels like a time/resource issue. Ultimately, TC members need to decide how to rebalance their responsibilities so that more cycles are available to providing guidance and feedback to other contributors.

[dyladan]

As the project grows too there are more "cross-sig" responsibilities and decisions to make. Growing the TC to increase bandwidth is an option but comes at a cost. It is much easier for a smaller group to make quick decisions which are consistent with an overall project direction. What have other large open source projects done in these situations?

[lizthegrey]

I think there may be problems of diffuse accountability, to which I know other projects have done things such as electing individuals to serve as the head of a given sub-area.

[lmolkova]

Let's pick HTTP conventions as an example: open-telemetry/opentelemetry-specification#1747. We have a scope and a lot of interest to stabilize the most(?) essential convention, the process to work on it is not clear:

• who are the people to work on this? How we can find each other and build some WG? What power does this WG have?
• what's the path to stability?
• how to reach consensus (spec approvers are not necessarily HTTP experts)?
• what principles do we put behind it (perf/friendliness/backends/backward compatibility with existing instrumentations)?

As a contributor and owner of some instrumentations, I don't understand how to move it forward and while I'm happy to navigate through it and take ownership of some parts, I would appreciate TC to help organize and connect people, define processes, and set expectations.

Another thing I miss is clear feedback on issues/PRs (at least popular ones): yes/good, but needs more thought/won't fix/etc; and again bringing other interested folks to review and give constructive feedback.

[trask]

@lmolkova's comment above resonates with me.

What feels lacking sometimes from TC (or other body?) is not the technical excellence, but the technical leadership of driving spec-like things forward.

A couple recent examples that pop to mind of what this kind of technical leadership looks like (done very well imo): @reyang's leadership on metrics and @jmacd's leadership on sampling and histogram format.

[bogdandrutu]

@trask @lmolkova excellent points.

I want to point few things:

• Reaching consensus across a large open-source community is a hard problem and it needs lots of effort. It should not be surprising for anyone that it is a much harder problem than doing the same thing inside a company.
• As @trask pointed, the lack of "champions" (area leads) is probably something that we suffer. Clear example are when @reyang (not TC member) or @jmacd put the effort to lead that, things happen, otherwise will not happen.

@lmolkova for the semantic conventions: we should have a "semantic convention" SIG + maybe sub-SIGS (specific to different areas) that lead/champion these efforts. Also don't hesitate to start a temporary SIG to solve a problem, where people can meet and discuss more clearly the issues.

[tigrannajaryan]

@lmolkova Thank you for the example.

I think it is important to call out that the TC charter does not currently make TC’s responsible for leading individual PRs or issues (unless I am misreading the charter, so someone please correct me).

Historically, making progress on individual contributions has been the responsibility of the contributors. Ultimately whoever makes a PR submission is tasked with making progress on it.

Sometimes the TC members will indeed take responsibility of leading individual PRs and issues. This typically happens when:

• The topic is something the particular TC member specializes in or is excited about. For example, if you open an PR or issue about telemetry schemas or logs you will likely see me actively commenting and reviewing it and helping to drive it.
• The contributor asks a TC member to help.

Now, I don’t know if this is the right approach or no. Should the TC members instead be directly tasked with driving forward all PRs and issues? If that is what we believe in as a community then I would like to discuss how it can be achieved in practice (it is non-trivial amount of additional work) and then make it a clearly documented part of the TC charter.

Regardless of what we decide in terms of direct responsibilities I think another (arguably more scalable) solution to this problem is to improve our contributing guidelines. Make it clear how to make progress on your ideas and your PRs. Have specific guides for classes of topics (e.g. "How to propose and add a new semantic convention"). The guidelines should ideally answer the 4 questions that you posted.

Another thing I miss is clear feedback on issues/PRs (at least popular ones): yes/good, but needs more thought/won't fix/etc;

I think this is a fair criticism and we could definitely improve in this area.

In terms of feedback things usually fall into the following broad categories:

1. I love this PR/OTEP. I am going to actively help it, push it forward to conclusion.
2. Looks OK, but doesn’t seem important enough for me to spend significant time on it. Someone else should take care of it.
3. Unclear what it is since it is outside of my area of expertise. I don't know what to comment on it.
4. Doesn’t seem right, goes in wrong direction. I am going to actively comment to prevent doing the wrong thing, ask for changes, clarifications or justification or even block it.

I think it is the number (2) and (3) that we have problems with. Things that are not exciting and are not wrong enough to warrant active involvement. This is where we are likely to see lack of feedback. There are probably a few ways we could solve this but before jumping to solutions I would like to hear if I see the problems correctly.

[reyang]

Historically, making progress on individual contributions has been the responsibility of the contributors. Ultimately whoever makes a PR submission is tasked with making progress on it.

+1 with what @tigrannajaryan said.

https://github.com/open-telemetry/opentelemetry-specification/blob/main/CONTRIBUTING.md#how-to-get-your-pr-merged

If a PR has been stuck (e.g. there are lots of debates and people couldn't agree on each other), the owner should try to get people aligned...

[lmolkova]

I agree with @bogdandrutu and @tigrannajaryan that reaching consensus is hard and the responsibility of contributors. But some problems are big and important enough to help drive them. Singe contributors can't always build community and drive big things alone.

What I'm saying is that we should have more clear and better processes and guidance defined. It seems based on this issue, it's not only my impression that we don't move fast enough.

If we get back to HTTP: some great folks already started to drive it: #826. But there is no consensus within GC that it is worth a dedicated SIG (look into the comments).

I'd like to have SIG charter and scope documented:

• what's the bar for new SIG, what authority SIG has
• what are the roles within SIG and expectations on time commitment
• do we need active TC, GC members, or spec approvers to be on the SIG to get things merged
• how do we define when SIG has reached its goal (prototypes, specs, adoptions, etc)

If we had OpenTelemetry project priorities and roadmap documented (do we? I can't find it), we could ask TC members to help on top priority issues with a lack of consensus (at least facilitate the process, not necessarily actively work on it).
It could also help me as a contributor understand if my priorities align well with the community and if I should expect traction (e.g. I drive library instrumentation, but let's say everyone is working on Metrics at the moment, I'd want to know).

Another thing that can help is a registry of contributors interested in specific areas so I can tag the right people and connect with them (like informal async SIGs). It can help people who are not full-time on OTel and new contributors find each other.

@tigrannajaryan I think you captured my concerns right. And I agree that issues without strong yes/no reactions are the most problematic.
The process, how I'd like to see it, basically adds priorities, feedback, and clearer resolutions at each stage:

• e.g. contributor creates an issue (not yet a PR) for non-trivial change. E.g. this aligns well with project priorities (otherwise gets less attention, longer responses)
• it's triaged and recognized as something that is worth addressing. Let's find a way to do it without BDFL, at least for most cases, and communicate it to the contributor
• some design discussions happen - the contributor should perhaps learn that most non-trivial decisions happen during SIG meetings (it's not obvious, let's be transparent about it).
• then it's time for the PR, we should be clear about prereqs: e.g. prototypes

I'm not asking for a formal process, maybe @reyang and @jmacd can just write how they've done it and we can use it as a starting point.

[anuraaga]

@tigrannajaryan Your summary looks very good. I probably was misinterpreting the charter in thinking that making sure issues go forward is in the TC's responsibilities. Since TC-owned repos generally get contributions from more advanced members, I guess this is OK. But often a contributor can also try but make no progress in moving an issue forward - presumably the at-mention feature of GitHub is the main mechanism for trying to get consensus from TC members, but even then it's too common to not get replies. I don't think it's fair to even advanced contributors to put it all on them in the face of unresponsiveness - a mandate for the TC to at the least be responsive reactively if not going for proactive, would make these repos more comfortable. Particularly putting focus on the points 2) and 3) - even if the member isn't interested, I think it is up to them to at least do something, maybe telling a different member to respond.

[mtwo]

Following up on this after returning from vacation. We also discussed this during today's GC call.

First off, @anuraaga, thank you for raising this. I've often felt that there's some (unintentional) opaqueness with the TC's role at times, and that there aren't enough dedicated lines of communication from contributors to the TC. I really like @tigrannajaryan's most recent reply, and I think that items (2) and (3) are the main source of challenges for contributors.

One aspect that has been mentioned on this thread and since the project's inception is how we drive progress. Several notable OSS projects, most notably Linux, have an (often highly opinionated) person in charge who can drive forward progress and make quick decisions. Others, including many CNCF projects and most standards bodies, require consensus across stakeholders before committing to decisions. This can make these projects more welcoming to contributors and open to different ideas, at the cost of velocity. There's a gradient to this obviously, but OpenTelemetry is clearly in the second camp. Still, we have an opportunity to mitigate some the downsides of our structure while continuing to take advantage of the benefits.

During the GC call, we committed to the following:

1. Further clarify the TC's role so that people's expectations are met. For example, as discussed in this thread, the TC body generally isn't responsible for driving specific issues forward. This is left to contributors (who are sometimes TC members, particularly for spec work!).
2. The GC will formalize a project roadmap, ideally by the end of this month.
3. The TC will make one of their monthly meetings open and will add these to the community calendar.
4. The TC will discuss this thread in their next internal meeting and they will follow up with further actions. I think that @bogdandrutu is on point for this.

[trask]

Thanks everyone.

From the above, is it correct to say that the specification repo does not have a concept of "maintainers" similar to other repos?

E.g. maintainers in other repos dedicate a lot of time to these tasks:

• Mentor and guide approvers, reviewers, and contributors to the subproject.
• Ensure a healthy process for discussion and decision making is in place.
• Work with other maintainers to maintain the project's overall health and success holistically.

(^ from maintainer responsibilities)

[reyang]

From the above, is it correct to say that the specification repo does not have a concept of "maintainers" similar to other repos?

I had the same question last year and this is what I learned 🧙‍♀️:

• It's defined here.
• I referenced it here.

[tigrannajaryan]

@trask @reyang thanks, apparently the combination of what you found makes the TC members responsible for more actively maintaining the spec repo than it would appear just by reading the TC charter document alone. I think this is fair, however I still would like to have these more clearly written down instead of relying on one document linking to another and resulting in somewhat indirectly implied responsibilities.

I will post back here after the TC's meeting.

[iNikem]

Sorry to be late to the party, but I have some specific questions/concerns about Specification repo that I think are relevant to this discussion.

First, it oftentimes takes a lot of time (that I can understand) and several pings of TC (that is the problem IMO) to get approved PR merged in the spec repo. That totally may be a problem if diffused responsibility, if TC members don't have strong opinions about a given PR, they may be hesitant to merge it and hope that somebody else will do it.

Second, what does it mean that some spec issue is assigned to some TC member? Does it mean anything at all? One would expect that issue assignee is responsible for driving that issue to some conclusion. Currently it may be very confusing for contributors. Should they start working on some spec issue if it is already assigned to a TC member?

Third, who is responsible for taking care that there are no issues in spec repo that we actually DON'T want to spend any time on? Like open-telemetry/opentelemetry-specification#239 - clearly we are not going to do that. But seeing that issue may give an impression to some contributor that it is worthwhile starting working on that. Essentially: who is responsible for spec repo backlog grooming?

[tigrannajaryan]

Second, what does it mean that some spec issue is assigned to some TC member? Does it mean anything at all? One would expect that issue assignee is responsible for driving that issue to some conclusion. Currently it may be very confusing for contributors. Should they start working on some spec issue if it is already assigned to a TC member?

This is a good question and it is not answered by our guides in any way currently. I believe our current auto-assignment's purpose is to ensure there is a "first response" from a TC member. It does not mean that the TC member is responsible for driving the issue (otherwise it would imply every problem is resolved by TC members which is unsustainable and undesirable). We likely need to clarify what the "first response" means. It can be typically one of these:

• Reassign to the right person who is the expert in the area (can be any member who actively contributes to the spec).
• Close if the issue is invalid or is considered "won't fix". This is pretty rare in the spec repo, most typically require some discussion.
• Ask for clarifications if the issue is not clear, invite relevant members to the discussion. The issue will likely stay assigned to the TC member if there is no clear other member who should take it. This still does not imply that the TC member is going to resolve the issue, it likely simply means there is no better person to assign it to.
• Keep assigned to themselves if they intend to work on it themselves.

There are probably other outcomes that I am failing to list. In either case I think we need to capture these possibilities in our guidelines and clearly mark each individual issue to be in one of these states. You are right that without this it is unclear who is actually working on the issue and whether someone else should we working on it. I will opened an issue to come up with a process and clarify our contributing guide: open-telemetry/opentelemetry-specification#1948.

Third, who is responsible for taking care that there are no issues in spec repo that we actually DON'T want to spend any time on?

Are we talking about invalid or "won't fix" issues or just ones that we don't want to spend now but maybe will spend time in the future? These 2 categories will likely need to be handled differently.

[tigrannajaryan]

I am going to assign this to myself for now, at least until I post back here after the TC discusses this tomorrow.

[jsuereth]

I want to further comment on this question:

Second, what does it mean that some spec issue is assigned to some TC member? Does it mean anything at all? One would expect that issue assignee is responsible for driving that issue to some conclusion. Currently it may be very confusing for contributors. Should they start working on some spec issue if it is already assigned to a TC member?

In the Scala Specification work, we would assign an "owner" from the language committee whose responsibility was to help make it clear to the issue creator or PR submission what the requirements are to make progress. I.e. It was their responsibility to make sure feedback was accounted for, experts were included and a "way forward" was clear for the contributor. I've seen my own role in the Specifcation/Proto repository, and I've tried (sometimes more successfully than others) to make sure an issue owner knows either (a) whom they should be communicating with or (b) a clear set of technical issues to address to get their PR through.

I'd suggest this is what we need from the TC in the specification repository and as @tigrannajaryan suggests, if it's unclear how to move forward (or the TC member is busy) they need to find a new owner for the issue or PR.

Ultimately it's the owner's responsibility to get their PR merged, but the TC should make sure the "path" is clear (or not available, for things we decide we're not going to do).

[iNikem]

@tigrannajaryan I meant "invalid or "won't fix" issues"

[anuraaga]

It does not mean that the TC member is responsible for driving the issue (otherwise it would imply every problem is resolved by TC members which is unsustainable and undesirable).

Ultimately it's the owner's responsibility to get their PR merged

I think this discussion is generally bringing up good points. Yet still, I feel like the TC's sentiments set a really low bar. I would phrase both of these as "TC has no responsibility in the spec repo" - is this really the best it can do? Responsibility / ownership are important for making a comfortable atmosphere for the community and contributors.

[tedsuo]

I do agree that there are three items which represent needed "technical leadership," as @trask put it, which the TC should bottom-line.

1. Maintain the spec, OTEP, and proto repos. The TC is the primary maintainers of these repos, so it is reasonable that they bottom-line all the ordinary tasks that make for a high quality repo, especially backlog management.
2. Maintain a high level roadmap containing all of the major initiatives the project is currently working on. Ensure that these initiatives stay on track.
3. Ensure that implementation-focused SIGs are aware of the spec and are implementing it correctly.

A note on item 2: right now the GC is going to tackle creating a new roadmap, but IMHO the TC should be responsible for (and feel ownership of) the technical roadmap.

A note on item 3: currently there is only a TC review when a SIG creates a v1.0 release candidate. For SIGs that have less of a direct connection to the spec working group, this has led to some surprise/churn. C++ is one recent example. Besides extending this review to cover milestones past v1.0, more frequent reviews (quarterly?) would help ensure that SIGs stay on track.

Personally, I think performing these tasks well requires a lot of community visibility and represents a lot of work; having a heavy load of code writing and other responsibilities makes it very difficult. But guiding and enabling the community to make contributions generates orders of magnitude more progress than doing the work directly, even though for any given project it always feels faster to do the work yourself.

[tigrannajaryan]

The TC has met and discussed this today. The outcome of the discussion is that right now we are putting together a list of action items that we believe help to solve the problems raised. Once we have a reasonable draft I will post it publicly so that everyone in the community can review and provide feedback or suggest other action items.

[lizthegrey]

@tigrannajaryan and @open-telemetry/technical-committee, thank you for being so open to feedback and responsive. appreciate your work here :)

[tigrannajaryan]

I created an issue that captures the initial list of action items that the TC think should help solve the problems discussed in this thread.

Please review, comment, suggest changes here #847

[tigrannajaryan]

If there are additional thoughts about this issue please comment, otherwise I think we can close this and continue the discussion in #847 which lists specific action items.

[anuraaga]

Thanks for all the help @tigrannajaryan, agree that we can focus on the actions in #847 now :)

[alolita]

Thanks @tigrannajaryan and @open-telemetry/technical-committee members for starting to think about these areas and taking on action items to work through.