Experimental Features

[jkutner]

Readable

[elbandito]

@jkutner is this suggesting we support experimental 3rd party plugins, or do you envision a separate CNB owned repo where these experimental plugins could live?

[jkutner]

Yea, I was thinking 3rd party (where the most likely 3rd parties would be Salesforce, Paketo, etc). But not a CNB owned repo.

[hone]

I think a plugin system would be great independent of this RFC.

[ekcasey]

I wonder if this particular mechanism for enabling experimental features is better suited to pack than the lifecycle. Not all lifecycle features can or should be enabled with flags (think buildpack API features).

I am wondering if experimental API versions as proposed in #92 would better support this goal at the lifecycle level

[hone]

Wouldn't it go through a RFC?

[jkutner]

i guess i don't mean the process, but the mechanics. Like, do we need to enforce that they are the same flag?

[hone]

Why would it need to be the same flag? If an experimental feature "graduates" it could have changes etc. from the feedback/things we learn.

[elbandito]

I wonder if we need an experimental flag? I don't know how golang supports external plugins, but I can envision the command just being added to the existing list of commands, and the user would use it as any other command. We could then mark experimental commands in the help menu or introduce an additional command to list all the experimental commands that have been installed (probably might want to do this anyway so users can quickly identify and remove any experimental commands).

Available Commands:
  build                     Generate app image from source code
 *grow-bones (experimental) Some custom experimental thingy
  rebase                    Rebase app image with latest run image
  inspect-image             Show information about a built image
  set-run-image-mirrors     Set mirrors to other repositories for a given run image
  set-default-builder       Set default builder used by other commands

[hone]

The big concern I have with this is creating a set of "hidden" APIs that people depend on if we do decide to take it away.

[elbandito]

I agree. This seems like an unsustainable path of where we're having to constantly consider the ramification of making breaking changes to experimental features while developing mainstream features. After all, we're not exactly a stable project per se. Maybe once we achieve 1.0.0 is won't be as much of a concern.

[hone]

What about experimental features that use RFCs but just for the subteam?

[ekcasey]

Personally I think experimental features in the lifecycle should typically (but not always) have RFCs b/c of the lifecycle's place in the overall architecture. However I don't RFCs should be necessary (most of the time) for experimental pack features.

[ekcasey]

But generally I think we should trust the judgement of maintainers when deciding whether an RFC is a good idea for an experimental feature.

[hone]

I like the idea of a lighter weight process for experimental. It shouldn't need to go through the same rigor, but I do wonder if it still needs to be documented and captured in a single place. Should experiments have end dates of make or break? This reminds me of spike docs we have internally at salesforce that it's an experiment that's documented, what you want to learn and find out, and for how long. My big fear is having a collection of experimental features that exist years from now.

[hone]

FWIW, unstable features in rust (mostly) go through RFCs

[dfreilich]

👍 to the HP reference 🧹

[nebhale]

@ekcasey Can you please weigh in on the new changes?

[hone]

I would also add that this mechanism needs to make it clear this is experimental, can change, etc.

[hone]

Circling back to this. I'm behind the motivation, but I still have some concerns.

• Experiments that are experiments forever. At Heroku, we've had features that never shipped for GA, but essentially were GA since so many people ended up using it. Experiments should have an end goal to graduate/sunset.
• The "mechanism" needs to make it clear to users that this is experimental and unsupported.

[dfreilich]

Definitely agree with @hone's points. Some suggestions to make this a bit more rigorous/help with it:

• We should have a tracking issue in the relevant repo for any given experiment. Something that says what it is, what we are trying to prove, when we hope to know it by, and which allows for feedback. (@hone thought it should be fairly short, like 3 sentences total)
• It could be useful, as such, to have an expiration/suggested expiration date for an experiment. Something like 2 releases in the future, though we could adjust it on a case-by-case basis
• Currently, we don't really have explicit logs if you are using an experimental feature. Perhaps we should have a warning log, after a command concludes, with a message that you used this experimental feature
• If we do have more explicit logging, and a tracking issue, we should print the issue as part of the logs, and invite users to give feedback on the issue, or at least tell us whether they used it, and whether they liked it.

As a side note, as part of this we should consider:

• Make it easier to flip the experimental switch in your config (i.e., have a command pack config experimental/pack config experimental true and pack config experimental false), to make this more discoverable for users
• Document experimental features (Document experimental feature docs#141), and probably have a tracking document in pack with a list of running experiments

[hone]

From conversations with @dfreilich and @jkutner, I'm going to take this over to do some edits. I'm going to move this into draft while I re-work it.

[elbandito]

I agree. This seems like an unsustainable path of where we're having to constantly consider the ramification of making breaking changes to experimental features while developing mainstream features. After all, we're not exactly a stable project per se. Maybe once we achieve 1.0.0 is won't be as much of a concern.

[elbandito]

I wonder if we need an experimental flag? I don't know how golang supports external plugins, but I can envision the command just being added to the existing list of commands, and the user would use it as any other command. We could then mark experimental commands in the help menu or introduce an additional command to list all the experimental commands that have been installed (probably might want to do this anyway so users can quickly identify and remove any experimental commands).

Available Commands:
  build                     Generate app image from source code
 *grow-bones (experimental) Some custom experimental thingy
  rebase                    Rebase app image with latest run image
  inspect-image             Show information about a built image
  set-run-image-mirrors     Set mirrors to other repositories for a given run image
  set-default-builder       Set default builder used by other commands

[elbandito]

This is definitely something to consider. Especially the work to maintain the integrity of the interface and how much effort we're willing to put into make sure we don't break the interface and any plugins.

[hone]

I would mark plugins as out of scope.

[elbandito]

Circling back to this. I'm behind the motivation, but I still have some concerns.

• Experiments that are experiments forever. At Heroku, we've had features that never shipped for GA, but essentially were GA since so many people ended up using it. Experiments should have an end goal to graduate/sunset.
• The "mechanism" needs to make it clear to users that this is experimental and unsupported.

I like the idea of having an expiration date associated with an experiment to hold us to either adopting or sunsetting it. I just wonder what criteria would we follow to determine the end result of an experimental feature? Do we somehow poll how much it's being used in the community or by how valuable the subteams find it to be?

[hone]

Circling back to this. I'm behind the motivation, but I still have some concerns.

• Experiments that are experiments forever. At Heroku, we've had features that never shipped for GA, but essentially were GA since so many people ended up using it. Experiments should have an end goal to graduate/sunset.
• The "mechanism" needs to make it clear to users that this is experimental and unsupported.

I like the idea of having an expiration date associated with an experiment to hold us to either adopting or sunsetting it. I just wonder what criteria would we follow to determine the end result of an experimental feature? Do we somehow poll how much it's being used in the community or by how valuable the subteams find it to be?

That's definitely the hard question. I don't want to codify it too much since I'm not on the subteam, and I trust the team will figure out what will work over time vs. my ivory tower.

[dfreilich]

🔥

I like @elbandito's suggestion of explicitly calling out experimental in the help test for an experimental command/flag.

[hone]

I like @elbandito's suggestion of explicitly calling out experimental in the help test for an experimental command/flag.

@dfreilich yeah that's great, would you want me to add this to the RFC?

[hone]

Would love a re-review since my changes from @ekcasey @nebhale @jromero and not sure but @jkutner should vote somehow.

[jkutner]

I vote ✅

[ekcasey]

TBH I am not convinced we need an RFC for this. pack already has experimental features. I would be happy to see improvements like:

1. label experimental features clearly in the help text
2. warning text when experimental features are used
3. easier and more discoverable UX to toggle on/off experimental features
4. a process for checking in on existing experimental features and ensure we are removing and or promoting them when appropriate

However, I think we could address the above through a series of pack issues. I am leery about introducing a new/different formal voting process for experimental features. It would add logistical hurdles that feel potentially unnecessary.

[ekcasey]

Doesn't pack already have experimental features? I think the UX around pack experimental features could maybe use some improvement, but they do exist.

[jromero]

TBH I am not convinced we need an RFC for this. pack already has experimental features. I would be happy to see improvements like:

1. label experimental features clearly in the help text
2. warning text when experimental features are used
3. easier and more discoverable UX to toggle on/off experimental features
4. a process for checking in on existing experimental features and ensure we are removing and or promoting them when appropriate

However, I think we could address the above through a series of pack issues. I am leery about introducing a new/different formal voting process for experimental features. It would add logistical hurdles that feel potentially unnecessary.

I share @ekcasey's sentiment, initially this RFC included lifecycle experimental features which in that vein would require additional discussion. Now that it's been removed I'm failing to see the purpose of this RFC given that it's essentially what we are currently doing. I also agree with the suggested improvements but as mentioned could be tackled as individual pack issues.

The only specific call out I see in this RFC is the formal acknowledgement allowing for "experimental features without requiring a specification, core team review, or RFC".

Another benefit to this RFC is recognizing and discussing alternatives. In which I'd like to emphasize @elbandito's concerns about the added complexity of maintaining an additional interface and distributing the plugins themselves.

Especially the work to maintain the integrity of the interface and how much effort we're willing to put into make sure we don't break the interface and any plugins.

We would need to be very methodical on how that would be implemented that it's probably not worth the effort for this specific use case.

[hone]

TBH I am not convinced we need an RFC for this. pack already has experimental features. I would be happy to see improvements like:

1. label experimental features clearly in the help text

2. warning text when experimental features are used

3. easier and more discoverable UX to toggle on/off experimental features

4. a process for checking in on existing experimental features and ensure we are removing and or promoting them when appropriate

However, I think we could address the above through a series of pack issues. I am leery about introducing a new/different formal voting process for experimental features. It would add logistical hurdles that feel potentially unnecessary.

@ekcasey I'm happy to cut out any "voting"/formality. The suggestions listed above make up the bulk of my proposal of providing a bit more rigor around "experiments" to ensure both clarity and they are indeed experiments and not secret features that live on forever. These are the things I care the most about.

[hone]

@jromero I'm not entirely sure what you're suggesting. I do think the callout of not requiring experimental features to go through the RFC process is a change and the point of this RFC. As far as plugins, I do like that idea, but think they're out of scope of the intent of this RFC.

[hone]

@jromero after talking with @ekcasey, let's find time to chat about this RFC. I'm happy to close this if we think it's unnecessary, but my biggest concern is making sure experimental features don't languish/have an end date.

[hone]

I met with @jromero and we decided to park/close this RFC for now. While we're on the same page of what I wanted to achieve with this proposal, but it's not something that is needed right now. Instead, we believe this need is worth revisiting after pack hits 1.0 where backwards breaking changes will become harder.