Merge pull request #52 from choldgraf/new-jep-process

updating with new JEP process

.github/pull_request_template.md

@@ -0,0 +1,12 @@
+✨Proposing a new JEP?✨
+
+Make sure that you have at least done the following:
+
+1. **Create a GitHub issue**  in the Jupyter Enhancement Proposals repo that
+ 1. Briefly outlines the proposal
+ 2. Suggests a review team (optional)
+ 3. Declares why it should be a JEP (See the “JEP / Not-a-JEP Rubric” below.)
+2. **Identify a Shepherd** to see the process through. (Shepherds are assigned
+   on a round-robin basis from a set of interested engaged volunteers).
+3. **Decide if it's a JEP** according to the criteria listed above. The Shepherd decides if the JEP criteria have been met.
+4. **Used the new JEP markdown template** to help structure your thoughts. You can find it in `jupyter-enhancement-proposal-guidelines/TEMPLATE.md` in this repository.

README.md

@@ -25,9 +25,14 @@ Below is a list of JEPs that have been submitted. To view the discussion around
 | 0025 | Completed | [Enterprise Gateway](https://github.com/jupyter/enhancement-proposals/pull/25) |
 | 0026 | **Submitted** | [Add Language Server Support to Jupyter Server and jupyterlab-monaco](https://github.com/jupyter/enhancement-proposals/pull/26) |
 | 0028 | **Submitted** | [Standalone Jupyter Server](https://github.com/jupyter/enhancement-proposals/pull/28) |
-| 0029 | **Submitted** | [Jupyter Enhancement Proposal updates](https://github.com/jupyter/enhancement-proposals/pull/29)
+| 0029 | **Accepted** | [Jupyter Enhancement Proposal updates](https://github.com/jupyter/enhancement-proposals/pull/29)
 | 0042 | **Accepted** | [Voila Incorporation](https://github.com/jupyter/enhancement-proposals/pull/42)
 
 ## How do I submit a JEP?
 
-In order to submit a JEP, please read the [Jupyter Enhancement Proposal Submission Guidelines](jupyter-enhancement-proposal-guidelines/jupyter-enhancement-proposal-guidelines.md). You can copy this file and use it as the starting point for outlining your own JEP.
+In order to submit a JEP, first read the [Jupyter Enhancement Proposal Submission Guidelines](jupyter-enhancement-proposal-guidelines/jupyter-enhancement-proposal-guidelines.md) which describes the JEP process.
+
+In addition, read
+[the JEP proposal template](jupyter-enhancement-proposal-guidelines/JEP-TEMPLATE.md)
+for guidance on the questions you should answer before officially submitting
+the JEP.

jupyter-enhancement-proposal-guidelines/JEP-TEMPLATE.md

@@ -0,0 +1,84 @@
+---
+title: <your-JEP-title>
+authors: <list-of-authors>
+issue-number: <pre-proposal-issue-number>
+pr-number: <proposal-pull-request-number>
+date-started: <yyyy-mm-dd>
+---
+
+# Summary
+
+One paragraph explanation of the proposal.
+
+# Motivation
+
+Why are we doing this? What use cases does it support? What is the expected outcome?
+
+# Guide-level explanation
+
+Explain the proposal as if it was already implemented and you were
+explaining it to another community member. That generally means:
+
+- Introducing new named concepts.
+- Adding examples for how this proposal affects people's experience.
+- Explaining how others should *think* about the feature, and how it should impact the experience using Jupyter tools. It should explain the impact as concretely as possible.
+- If applicable, provide sample error messages, deprecation warnings, or migration guidance.
+- If applicable, describe the differences between teaching this to existing Jupyter members and new Jupyter members.
+
+For implementation-oriented JEPs, this section should focus on how other Jupyter
+developers should think about the change, and give examples of its concrete impact. For policy JEPs, this section should provide an example-driven introduction to the policy, and explain its impact in concrete terms.
+
+# Reference-level explanation
+
+This is the technical portion of the JEP. Explain the design in
+sufficient detail that:
+
+- Its interaction with other features is clear.
+- It is reasonably clear how the feature would be implemented.
+- Corner cases are dissected by example.
+
+The section should return to the examples given in the previous section, and explain more fully how the detailed proposal makes those examples work.
+
+# Rationale and alternatives
+
+- Why is this choice the best in the space of possible designs?
+- What other designs have been considered and what is the rationale for not choosing them?
+- What is the impact of not doing this?
+
+# Prior art
+
+Discuss prior art, both the good and the bad, in relation to this proposal.
+A few examples of what this can include are:
+
+- Does this feature exist in other tools or ecosystems, and what experience have their community had?
+- For community proposals: Is this done by some other community and what were their experiences with it?
+- For other teams: What lessons can we learn from what other communities have done here?
+- Papers: Are there any published papers or great posts that discuss this? If you have some relevant papers to refer to, this can serve as a more detailed theoretical background.
+
+This section is intended to encourage you as an author to think about the lessons from other languages, provide readers of your JEP with a fuller picture.
+If there is no prior art, that is fine - your ideas are interesting to us whether they are brand new or if it is an adaptation from other languages.
+
+
+# Unresolved questions
+
+- What parts of the design do you expect to resolve through the JEP process before this gets merged?
+- What related issues do you consider out of scope for this JEP that could be addressed in the future independently of the solution that comes out of this JEP?
+
+# Future possibilities
+
+Think about what the natural extension and evolution of your proposal would
+be and how it would affect the Jupyter community at-large. Try to use this section as a tool to more fully consider all possible
+interactions with the project and language in your proposal.
+Also consider how the this all fits into the roadmap for the project
+and of the relevant sub-team.
+
+This is also a good place to "dump ideas", if they are out of scope for the
+JEP you are writing but otherwise related.
+
+If you have tried and cannot think of any future possibilities,
+you may simply state that you cannot think of anything.
+
+Note that having something written down in the future-possibilities section
+is not a reason to accept the current or a future JEP; such notes should be
+in the section on motivation or rationale in this or subsequent JEPs.
+The section merely provides additional information.

jupyter-enhancement-proposal-guidelines/jupyter-enhancement-proposal-guidelines.md

@@ -1,61 +1,199 @@
 # Jupyter Enhancement Proposals (JEP) Guidelines
 
-## Problem
-Enhancements to the Jupyter ecosystem are currently presented across a multitude of platforms without any centralized management or discussion.
+**Note**: This JEP repo is the **canonical "source of truth"** for individual JEPs,
+the JEP process, and activity on JEPs.
 
-## Proposed Enhancement
+## Goals of the JEP process
 
-Jupyter Enhancement Proposals will be used when presenting changes or additions that affect multiple components of the Jupyter ecosystem. These changes can include things like:
-* Additions/changes to the message spec
-* API design that is consumed and produced in several contexts (e.g. the notebook kernel API)
-* The notebook format
+Project Jupyter uses the Jupyter Enhancement Proposal process (JEP)
+to address distributed collaboration and experimentation.
+The primary guiding principle of the JEP process is to
+encourage collaboration and discussion as early as possible in the lifecycle
+of a major proposed change in Jupyter, with the goal of preventing costly rework,
+competing ideas, and unnecessary forking or fragmentation of ideas.
 
-The format of this README is itself a JEP and can be duplicated for the creation of further JEPs.
+Several sub-goals exist for this process:
 
-## Detail Explanation
+- **Maximize success of large proposals** that get stalled in the wrong venue (e.g. a single PR comment thread)
+- Provide a **better alternative to “piecemeal” development** where multiple PRs
+  to build an end-to-end set of functionality are split across multiple GitHub
+  project without broad consensus, context, or guidance.
+- Provide a **clear, time-limited, and authoritative process for work proposals**,
+  to facilitate funding conversations.  (e.g. provide a concrete artifact to reference
+  in a grant proposal, roadmap item, etc.)
+- Provide a **consolidated “stream” of all proposals across the entire Jupyter community**
+  that contributors of all levels can monitor and selectively engage in.
 
-### JEP Titles
+## Tenets of the JEP process
 
-Jupyter Enhancement Proposals will be submitted with a title that is no longer than 12-words long. A JEP is uniquely identified by its title and the pull request number associated with it.
+The JEP process operates under the following tenets:
 
-### JEP Labels
+- **The JEP process is intended for changes of non-trivial scope.**
+  “Non-trivial” is addressed below in the “JEP / Not-a-JEP Rubric” of this document.
 
-The pull-request submitted with each JEP will be labeled with the following labels for easy searching:
-* `accepted` — this JEP has been accepted and is currently being implemented
-* `implemented` — this JEP has been implemented
-* `rejected` - this JEP has been rejected and will not be implemented
-* `withdrawn` - this JEP has been withdrawn by the submitter but can be re-submitted if someone is willing to champion it
-* `active` - this JEP is currently under active discussion within the community
+- **The JEP process naturally complements the PR process, but does not replace it.**
+  A thoroughly-reviewed and approved JEP is a valuable reference during a PR to
+  reduce friction, reduce time-consuming context sharing, and encapsulate decisions
+  and other discussions.  Moving a premature PR into a JEP should be a lightweight
+  process that doesn’t cause friction for the contributor.
 
-### JEP Structure
+  For example, GitHub issue and PR templates across the entire Jupyter project should have
+  references to the JEP process as a possible outcome of a given PR.
 
-When submitting an enhancement proposal, individuals will include the following information in their submission.
+- **There is one JEP repository, all Jupyter-governed projects must use it.**
+  To faciliate the easiest possible adoption and high visibility of ideas, a
+  single JEP repository will be used.  Even if a JEP only applies to a single organization.
 
-1. The problem that this enhancement addresses. If possible include code or anecdotes to describe this problem to readers.
-2. A brief (1-2 sentences) overview of the enhancement you are proposing. If possible include hypothetical code sample to describe how the solution would work to readers.
-3. A detailed explanation covering relevant algorithms, data structures, an API spec, and any other relevant technical information
-4. A list of pros that this implementation has over other potential implementations.
-5. A list of cons that this implementation has.
-6. A list of individuals who would be interested in contributing to this enhancement should it be accepted.
+- The JEP process **has multiple valid use cases**.  Each use case might have a
+  slightly different expected workflow.  Some expected use cases include:
 
-### JEP Submission Process
-1. Create a [Markdown](https://help.github.com/articles/github-flavored-markdown/) write up of the problem, proposed enhancement, detailed technical explanation, pros and cons, and interested contributors of the enhancement you are proposing.
-2. Create a fork of this repository.
-3. Create a folder with its name set to the JEP title in lower snake-case.
-3. Place the markdown file created in step 1 and any supplemental materials in that folder.
-4. Submit a pull request to the main repository with your JEP. 
-5. Once your PR is accepted, it will be labeled `active` per the guidelines above.
-6. Your JEP will be added to the JEP Index file in this repository.
+  - Non-trivial feature proposals within a single component that would benefit from
+    process.  (e.g., a non-trivial change to JupyterLab that would benefit from
+    formal process within the JupyterLab project)
+  - Non-trivial features or improvements that span multiple projects.
+  - Any proposed changes to published APIs or core specifications (e.g., nbformat)
+  - Changes to the JEP process itself.
+  - Launching a major project in one of the Jupyter GitHub organizations.
 
-## Pros and Cons
 
-Pros associated with this implementation include:
-* A higher quality discussion around enhancement proposals
-* Individuals are encourage to put more thought into an enhancement proposal before submitting it
-* Precedence exists in the form of PEPs (Python Enhancement Proposals) and IPEPs (IPython Enhancement Proposals)
+## JEP Submission Workflow
+
+The following sections describe the major checkpoints in the JEP process.
+Note that at any time, the JEP author may withdraw their JEP (and if it has
+been added to the README, its status becomes "withdrawn").
 
-Cons associated with this implementation include:
-* Existing IPEPs (IPython Enhancement Proposals) will not be included in this migrated repository
+### Phase 1: Pre-proposal
+
+This is the least formal stage of any jupyter enhancement proposals. During this
+phase, discussions on the mailing list, community forum, in-person, on github issues
+are all fine to gauge community interest, feasibility, consequences, and to
+scope out technical impact and implementation details.
+
+In order to transition out of the pre-proposal stage, the following checklist must be complete:
+
+1. **Create a GitHub issue**  in the Jupyter Enhancement Proposals repo that
+ 1. Briefly outlines the proposal
+ 2. Suggests a review team (optional)
+ 3. Declares why it should be a JEP (See the “JEP / Not-a-JEP Rubric” below.)
+2. **Identify a Shepherd** to see the process through. (Shepherds are assigned
+   on a round-robin basis from a set of interested engaged volunteers).
+3. **Decide if it's a JEP** according to the criteria listed above. The Shepherd decides if the JEP criteria have been met.
+
+**If it's a JEP**. The JEP author creates a new PR with the contents of the JEP proposal. See [this markdown template](JEP-TEMPLATE.md) for a suggested structure of the pull-request.
+
+Subsequent discussion and decisions about the JEP will happen in that PR, and
+we now **Move to the RFC phase**.
+
+**If it’s not a JEP**. The shepherd provides a reason for why the issue
+doesn't meet JEP criteria, and closes the issue.
+
+### Phase 2: Request for Comments for the JEP
+
+Phase two begins when **the JEP author submits a draft of the JEP as a PR to the JEP repository**.
+The Shepherd assigns a number to the JEP according to the pull-request number, and updates
+the README of the JEP repository with information about the now in-progress JEP.
+The Shepherd then works with the JEP author to
+**identify and notify relevant individuals in the Jupyter community to review and comment**.
+
+Once this group has been notified, the JEP process enters the RFC phase.
+
+During the **Request For Comment (RFC) phase**, the proposal is iterated on with
+feedback from the Review Team and the community at large. The Shepherd helps engage
+the Review Team and encourage productive discussion. After the Review Team members
+have signed off on the JEP, with the criteria that there are no major objections,
+and at least some of the Review Team are in favor, the Shepherd initiates the Final Comment Period.
+
+In the **Final Comment Period (FCP)**, the community at-large has at least 10 calendar days
+to provide any final comments on the JEP. A JEP may revert back to RFC if
+objections from the community are supported by the Review Team.
+
+At the end of the FCP, the JEP is either:
+
+* **approved**, in which case the PR is merged and work may commence on implementing the JEP (see Phase 3, below)
+* **rejected**, in which case the PR is closed
+* asked to return to the RFC phase.
+
+In each case, the JEP's status should be updated in the README of the
+`enhancement-proposals` repository.
+
+### Phase 3: Work Commences
+
+Once a JEP has been merged into the `jupyter/enhancement-proposal` repository,
+implementation can commence on the JEP. The JEP author does not need to implement the JEP themselves.
+
+If the JEP requires one or more pull-requests to be implemented, the author of the PRs should
+provide a reference to the JEP so that reviewer has background context on the JEP.
+As the JEP is being implemented, the JEP text should be amended with with addendums to
+denote the progress of the implementation using the following stages.
+
+1. In progress implementation via (list of PRs).
+2. Fully implemented via (list of PRs).
+
+If in the course of implementation, it is discovered that the implementation needs to
+be radically different from what was defined in the original JEP, then a pull
+request is submitted to modify the original JEP with the new necessary implementation,
+and a note citing the need for a modification to the JEP.
+This pull request should be re-approved by the original review team.
+
+### JEP Pathways and Status
+
+Below is a rough guide that describes the JEP process and its possible pathways.
+
+```
+                      +-----------+
+                      |           |
+                      | withdrawn |
+                      |           |             +-----------+
+                      +-----------+             |           |
+                  JEP may be withdrawn          |  rejected |
+                      at any stage              |           |
+                                                +-----^-----+
+                                                      |
+                                                +------+------+
+  +--------------+   +-----------+             |             |            +-------------+   +-----------+
+  |              |   |           |             | Request for |            |             |   |           |
+  | pre-proposal +---> submitted +------------->  Comments   +------------> in progress +---> completed |
+  |              |   |           |  identify   |    (RFC)    |  approved  |             |   |           |
+  +--------------+   +-----------+   review    |             |            +-------------+   +-----------+
+                                      team     +------+------+
+```
+
+
+## What qualifies as a JEP?
+
+This section contains a set of principles to help determine when something is a JEP.
+These should be used to determine when something becomes a PR during the JEP
+pre-proposal stage, as well as to determine when a PR becomes a JEP at an individual repo level.
+
+**Principles to follow**
+
+Below are a few example guidelines to follow when deciding if an idea should include
+a JEP (If yes, it requires a JEP). Under each question is a relevant example proposal.
+
+- Does the proposal/implementation require PRs across multiple orgs?
+  - **Example:** Defining a unique cell identifier
+- Does the proposal/implementation PR impact multiple orgs, or have widespread community impact?
+  - **Example:** Updating nbformat
+- Does the proposal/implementation change an invariant in one or more orgs?
+  - **Example:** Defining a unique cell identifier
+  - **Example:** Deferred kernel startup
+- Does the proposal/implementation create a new concept that will impact multiple repositories?
+  - **Example:** Sandboxed cell outputs
+- Does the proposal involve creating a new repository or sub-project?
+
+## The JEP public archive
+
+A public website contains a readable archive of all JEP proposals.
+It contains list of all JEPs that have entered a "final" state
+(e.g., "Completed", "Withdrawn", "Rejected"). The content of each JEP will
+be displayed in a readable fashion. When a JEP enters into a final state, it
+is added to this website.
+
+Note that the JEPs themselves contain the content, while the website is just a
+quick way to display them in a reading-friendly format.
+
+## Background
+
+For a background of the JEP process, and recent efforts to improve it, see
+[the meta-JEP readme](../29-jep-process/README.md).
 
-## Interested Contributors
-@captainsafia, @rgbkrk