From 8bde84fd1a22e22dc5ec9ac04fc69e49fb81c949 Mon Sep 17 00:00:00 2001 From: Marc Garcia Date: Tue, 21 Jun 2022 15:33:08 +0700 Subject: [PATCH 01/14] PDEP-1: Purpose and guidelines for pandas enhancement proposals --- web/pandas/about/roadmap.md | 89 ++++++++------ web/pandas/config.yml | 3 + .../accepted/0001-purpose-and-guidelines.md | 109 ++++++++++++++++++ web/pandas_web.py | 48 ++++++++ 4 files changed, 213 insertions(+), 36 deletions(-) create mode 100644 web/pandas/pdeps/accepted/0001-purpose-and-guidelines.md diff --git a/web/pandas/about/roadmap.md b/web/pandas/about/roadmap.md index 35a6b3361f32e..5681ea654586d 100644 --- a/web/pandas/about/roadmap.md +++ b/web/pandas/about/roadmap.md @@ -15,10 +15,53 @@ fundamental changes to the project that are likely to take months or years of developer time. Smaller-scoped items will continue to be tracked on our [issue tracker](https://github.com/pandas-dev/pandas/issues). -See [Roadmap evolution](#roadmap-evolution) for proposing -changes to this document. +The roadmap is defined as a set of major enhancement proposals named PDEPs. +For more information about PDEPs, and how to submit one, please refer to +[PEDP-1](/pdeps/accepted/0001-puropose-and-guidelines.html). -## Extensibility +## PDEPs + +### PDEPs under discussion + +{% for pdep in pdeps.under_discussion -%} +- [{{ pdep.title }}]({{ pdep.url }}) +{% else %} +There are currently no PEPs under discussion +{% endfor %} + +### Accepted PDEPs + +{% for pdep in pdeps.accepted -%} +- [{{ pdep.title }}]({{ pdep.url }}) +{% else %} +There are currently no accepted PEPs +{% endfor %} + +### Rejected PDEPs + +{% for pdep in pdeps.rejected -%} +- [{{ pdep.title }}]({{ pdep.url }}) +{% else %} +There are currently no rejected PEPs +{% endfor %} + +### Implemented PDEPs + +{% for pdep in pdeps.implemented -%} +- [{{ pdep.title }}]({{ pdep.url }}) +{% else %} +There are currently no implemented PEPs +{% endfor %} + +## Roadmap points pending a PDEP + + + +### Extensibility Pandas `extending.extension-types` allow for extending NumPy types with custom data types and array storage. @@ -33,7 +76,7 @@ library, making their behavior more consistent with the handling of NumPy arrays. We'll do this by cleaning up pandas' internals and adding new methods to the extension array interface. -## String data type +### String data type Currently, pandas stores text data in an `object` -dtype NumPy array. The current implementation has two primary drawbacks: First, `object` @@ -54,7 +97,7 @@ work, we may need to implement certain operations expected by pandas users (for example the algorithm used in, `Series.str.upper`). That work may be done outside of pandas. -## Apache Arrow interoperability +### Apache Arrow interoperability [Apache Arrow](https://arrow.apache.org) is a cross-language development platform for in-memory data. The Arrow logical types are closely aligned @@ -65,7 +108,7 @@ data types within pandas. This will let us take advantage of its I/O capabilities and provide for better interoperability with other languages and libraries using Arrow. -## Block manager rewrite +### Block manager rewrite We'd like to replace pandas current internal data structures (a collection of 1 or 2-D arrays) with a simpler collection of 1-D arrays. @@ -92,7 +135,7 @@ See [these design documents](https://dev.pandas.io/pandas2/internal-architecture.html#removal-of-blockmanager-new-dataframe-internals) for more. -## Decoupling of indexing and internals +### Decoupling of indexing and internals The code for getting and setting values in pandas' data structures needs refactoring. In particular, we must clearly separate code that @@ -107,7 +150,7 @@ Indexing is a complicated API with many subtleties. This refactor will require care and attention. More details are discussed at -## Numba-accelerated operations +### Numba-accelerated operations [Numba](https://numba.pydata.org) is a JIT compiler for Python code. We'd like to provide ways for users to apply their own Numba-jitted @@ -119,7 +162,7 @@ window contexts). This will improve the performance of user-defined-functions in these operations by staying within compiled code. -## Documentation improvements +### Documentation improvements We'd like to improve the content, structure, and presentation of the pandas documentation. Some specific goals include @@ -134,7 +177,7 @@ pandas documentation. Some specific goals include subsections of the documentation to make navigation and finding content easier. -## Performance monitoring +### Performance monitoring Pandas uses [airspeed velocity](https://asv.readthedocs.io/en/stable/) to monitor for performance regressions. ASV itself is a fabulous tool, @@ -154,29 +197,3 @@ We'd like to fund improvements and maintenance of these tools to - Build a GitHub bot to request ASV runs *before* a PR is merged. Currently, the benchmarks are only run nightly. - -## Roadmap Evolution - -Pandas continues to evolve. The direction is primarily determined by -community interest. Everyone is welcome to review existing items on the -roadmap and to propose a new item. - -Each item on the roadmap should be a short summary of a larger design -proposal. The proposal should include - -1. Short summary of the changes, which would be appropriate for - inclusion in the roadmap if accepted. -2. Motivation for the changes. -3. An explanation of why the change is in scope for pandas. -4. Detailed design: Preferably with example-usage (even if not - implemented yet) and API documentation -5. API Change: Any API changes that may result from the proposal. - -That proposal may then be submitted as a GitHub issue, where the pandas -maintainers can review and comment on the design. The [pandas mailing -list](https://mail.python.org/mailman/listinfo/pandas-dev) should be -notified of the proposal. - -When there's agreement that an implementation would be welcome, the -roadmap should be updated to include the summary and a link to the -discussion issue. diff --git a/web/pandas/config.yml b/web/pandas/config.yml index 9165456d55897..b1fa54332b46d 100644 --- a/web/pandas/config.yml +++ b/web/pandas/config.yml @@ -11,6 +11,7 @@ main: - pandas_web.Preprocessors.blog_add_posts - pandas_web.Preprocessors.maintainers_add_info - pandas_web.Preprocessors.home_add_releases + - pandas_web.Preprocessors.roadmap_pdeps markdown_extensions: - toc - tables @@ -157,3 +158,5 @@ sponsors: logo: /static/img/partners/r_studio.svg kind: partner description: "Wes McKinney" +roadmap: + pdeps_path: pdeps diff --git a/web/pandas/pdeps/accepted/0001-purpose-and-guidelines.md b/web/pandas/pdeps/accepted/0001-purpose-and-guidelines.md new file mode 100644 index 0000000000000..116c24cd4a994 --- /dev/null +++ b/web/pandas/pdeps/accepted/0001-purpose-and-guidelines.md @@ -0,0 +1,109 @@ +# PDEP-1: Purpose and guidelines + +- Date 21 June 2022 +- Status: Accepted +- Discussion: [#XXXXX](https://github.com/pandas-dev/pandas/pull/XXXXX) +- Author: [Marc Garcia](https://github.com/datapythonista) + +## PDEP definition, purpose and scope + +A PDEP (pandas enhancement proposal) is a proposal to a **major** change in +pandas, in a similar way as a Python [PEP](https://peps.python.org/pep-0001/) +or a NumPy [NEP](https://numpy.org/neps/nep-0000.html). + +Bug fixes and conceptually minor changes (e.g. adding a parameter to a function) +are out of the scope of PDEPs. A PDEP should be used for changes that are not +immediate and not obvious, and are expected to require a significant amount of +discussion and require detailed documentation before being implemented. + +PDEP are appropriate for user facing changes, internal changes and organizational +discussions. Examples of topics worth a PDEP could include moving a module from +pandas to a separate repository, a refactoring of the pandas block manager or +a proposal of a new code of conduct. + +## PDEP guidelines + +### Target audience + +A PDEP is a public document available to anyone, but the main stakeholders to +consider when writing a PDEP are: + +- The core development team, who will have the final decision on whether a PDEP + is approved or not +- Developers of pandas and other related projects, and experienced users. Their + feedback is highly encouraged and appreciated, to make sure all points of views + are taken into consideration +- The wider pandas community, in particular users, who may or may not have feedback + on the proposal, but should know and be able to understand the future direction of + the project + +### PDEP authors + +Anyone can propose a PDEP, but in most cases developers of pandas itself and related +projects are expected to author PDEPs. If you are unsure if you should be opening +an issue or creating a PDEP, it's probably save to start by +[opening an issue](https://github.com/pandas-dev/pandas/issues/new/choose), which can +be eventually moved to a PDEP. + +### Workflow + +#### Submitting a PDEP + +Proposing a PDEP is done by creating a PR adding a new file to `web/pdeps/accepted/`. +The file is a markdown file, you can use `web/pdeps/accepted/0001.md` as a reference +for the expected format. + +By default, we expect a PDEP will be accepted, so the PR of a PDEP should be done +in the `accepted` directory and contain `Status: Accepted`. If a PDEP is finally +rejected, its status and directory will be updated by the core team before merging, +once the decision is made. Please make sure you select the option +`Allow edits and access to secrets by maintainers` when opening the PR. + +#### Accepted PDEP + +A PDEP will be accepted by the core development team, and decisions will be made +based on the [pandas governance document](https://github.com/pandas-dev/pandas-governance/blob/master/governance.md). + +Once a PDEP is accepted, it is unclear how the implementation will happen. The +pandas project development, with a mix of volunteers and developers paid from +different sources, and development priorities are difficult to understand or +forecast. For companies, institutions or individuals with interest in seeing a +PDEP being implemented, or to in general see progress to the pandas roadmap, +please check how you can help in the [contributing page](/contribute.html). + +#### Implemented PDEP + +Once a PDEP is implemented and available in the main branch of pandas, its +status will be changed to implemented, so there is visibility that the PDEP +is not part of the roadmap and future plans, but a change that it already +happened. + +#### Rejected PDEP + +A PDEP can be rejected when the final decision is that its implementation is +not the best for the interests of the project. They are as useful as accepted +PDEPs, since there are discussions that are worth having, and decisions about +changes to pandas being made. They will be merged with `Status: Rejected`, so +there is visibility on what was discussed and what was the outcome of the +discussion. A PDEP can be rejected for different reasons, for example good ideas +that aren't backward-compatible, and the breaking changes aren't considered worth +implementing. + +#### Invalid PDEP + +For submitted PDEPs that do not contain proper documentation, are out of scope, or +are not useful to the community for any other reason, the PR will be closed after +discussion with the author, instead of merging them as rejected. This is to not +add noise to the list of rejected PDEPs, which should contain documentation as +good as an accepted PDEP, but where the final decision was to not implement the changes. + +## Evolution of this PDEP + +While most PDEPs aren't expected to change after accepted, in some cases like this +PDEP, they will be updated to contain its latest version, if things evolve. A log +of the summary of the changes will be kept to make it easier to see if any change +has happened. + +### PDEP-1 History + +- 21 June 2022: Initial version diff --git a/web/pandas_web.py b/web/pandas_web.py index 7dd63175e69ac..95bbbe68c6046 100755 --- a/web/pandas_web.py +++ b/web/pandas_web.py @@ -28,6 +28,7 @@ import importlib import operator import os +import pathlib import re import shutil import sys @@ -185,6 +186,53 @@ def home_add_releases(context): ) return context + @staticmethod + def roadmap_pdeps(context): + """ + PDEP's (pandas enhancement proposals) are not part of the bar + navigation. They are included as lists in the "Roadmap" page + and linked from there. This preprocessor obtains the list of + PDEP's in different status from the directory tree and GitHub. + """ + context["pdeps"] = { + "accepted": [], + "rejected": [], + "under_discussion": [], + "implemented": [], + } + # accepted, rejected and implemented + pdeps_path = (pathlib.Path(context["source_path"]) + / context["roadmap"]["pdeps_path"]) + for status in ('accepted', 'rejected', 'implemented'): + status_dir = pdeps_path / status + if not status_dir.is_dir(): + continue + for pdep in status_dir.iterdir(): + if pdep.suffix != ".md": + continue + html_file = pdep.with_suffix(".html").name + print(pdep) + with pdep.open() as f: + title = f.readline()[2:] # removing markdown title "# " + context["pdeps"][status].append({ + "title": title, + "url": f"/pdeps/{status}/{html_file}", + }) + + # under discussion + github_repo_url = context["main"]["github_repo_url"] + resp = requests.get("https://api.github.com/search/issues?" + f"q=is:pr is:open label:PDEP repo:{github_repo_url}") + if context["ignore_io_errors"] and resp.status_code == 403: + return context + resp.raise_for_status() + + for pdep in resp.json()["items"]: + context["pdeps"]["under_discussion"].append({"title": pdep["title"], + "url": pdep["url"]}) + + return context + def get_callable(obj_as_str: str) -> object: """ From 0b43492b928e920c06ff6aa4ca55e1400548e0a3 Mon Sep 17 00:00:00 2001 From: Marc Garcia Date: Tue, 21 Jun 2022 15:33:39 +0700 Subject: [PATCH 02/14] black --- web/pandas_web.py | 28 +++++++++++++++++----------- 1 file changed, 17 insertions(+), 11 deletions(-) diff --git a/web/pandas_web.py b/web/pandas_web.py index 95bbbe68c6046..9443c7ead1a3d 100755 --- a/web/pandas_web.py +++ b/web/pandas_web.py @@ -201,9 +201,10 @@ def roadmap_pdeps(context): "implemented": [], } # accepted, rejected and implemented - pdeps_path = (pathlib.Path(context["source_path"]) - / context["roadmap"]["pdeps_path"]) - for status in ('accepted', 'rejected', 'implemented'): + pdeps_path = ( + pathlib.Path(context["source_path"]) / context["roadmap"]["pdeps_path"] + ) + for status in ("accepted", "rejected", "implemented"): status_dir = pdeps_path / status if not status_dir.is_dir(): continue @@ -214,22 +215,27 @@ def roadmap_pdeps(context): print(pdep) with pdep.open() as f: title = f.readline()[2:] # removing markdown title "# " - context["pdeps"][status].append({ - "title": title, - "url": f"/pdeps/{status}/{html_file}", - }) + context["pdeps"][status].append( + { + "title": title, + "url": f"/pdeps/{status}/{html_file}", + } + ) # under discussion github_repo_url = context["main"]["github_repo_url"] - resp = requests.get("https://api.github.com/search/issues?" - f"q=is:pr is:open label:PDEP repo:{github_repo_url}") + resp = requests.get( + "https://api.github.com/search/issues?" + f"q=is:pr is:open label:PDEP repo:{github_repo_url}" + ) if context["ignore_io_errors"] and resp.status_code == 403: return context resp.raise_for_status() for pdep in resp.json()["items"]: - context["pdeps"]["under_discussion"].append({"title": pdep["title"], - "url": pdep["url"]}) + context["pdeps"]["under_discussion"].append( + {"title": pdep["title"], "url": pdep["url"]} + ) return context From 3d9a75bc3f56e0cbc0780a4aba9912c999230db5 Mon Sep 17 00:00:00 2001 From: Marc Garcia Date: Tue, 21 Jun 2022 15:47:34 +0700 Subject: [PATCH 03/14] Update PR number --- web/pandas/pdeps/accepted/0001-purpose-and-guidelines.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/web/pandas/pdeps/accepted/0001-purpose-and-guidelines.md b/web/pandas/pdeps/accepted/0001-purpose-and-guidelines.md index 116c24cd4a994..474013d6ae40c 100644 --- a/web/pandas/pdeps/accepted/0001-purpose-and-guidelines.md +++ b/web/pandas/pdeps/accepted/0001-purpose-and-guidelines.md @@ -2,7 +2,7 @@ - Date 21 June 2022 - Status: Accepted -- Discussion: [#XXXXX](https://github.com/pandas-dev/pandas/pull/XXXXX) +- Discussion: [#47444](https://github.com/pandas-dev/pandas/pull/47444) - Author: [Marc Garcia](https://github.com/datapythonista) ## PDEP definition, purpose and scope From 6d9d34bdd4f7d32cac858375cacdbccf07dbe721 Mon Sep 17 00:00:00 2001 From: Marc Garcia Date: Tue, 21 Jun 2022 20:23:27 +0700 Subject: [PATCH 04/14] Update web/pandas/pdeps/accepted/0001-purpose-and-guidelines.md Co-authored-by: Simon Hawkins --- web/pandas/pdeps/accepted/0001-purpose-and-guidelines.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/web/pandas/pdeps/accepted/0001-purpose-and-guidelines.md b/web/pandas/pdeps/accepted/0001-purpose-and-guidelines.md index 474013d6ae40c..3156885448229 100644 --- a/web/pandas/pdeps/accepted/0001-purpose-and-guidelines.md +++ b/web/pandas/pdeps/accepted/0001-purpose-and-guidelines.md @@ -1,6 +1,6 @@ # PDEP-1: Purpose and guidelines -- Date 21 June 2022 +- Date: 21 June 2022 - Status: Accepted - Discussion: [#47444](https://github.com/pandas-dev/pandas/pull/47444) - Author: [Marc Garcia](https://github.com/datapythonista) From a0e6cda2b9734a03da7a641111c18fc70afc402a Mon Sep 17 00:00:00 2001 From: Marc Garcia Date: Tue, 21 Jun 2022 20:23:37 +0700 Subject: [PATCH 05/14] Update web/pandas/pdeps/accepted/0001-purpose-and-guidelines.md Co-authored-by: Simon Hawkins --- web/pandas/pdeps/accepted/0001-purpose-and-guidelines.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/web/pandas/pdeps/accepted/0001-purpose-and-guidelines.md b/web/pandas/pdeps/accepted/0001-purpose-and-guidelines.md index 3156885448229..fd630772fe425 100644 --- a/web/pandas/pdeps/accepted/0001-purpose-and-guidelines.md +++ b/web/pandas/pdeps/accepted/0001-purpose-and-guidelines.md @@ -41,7 +41,7 @@ consider when writing a PDEP are: Anyone can propose a PDEP, but in most cases developers of pandas itself and related projects are expected to author PDEPs. If you are unsure if you should be opening -an issue or creating a PDEP, it's probably save to start by +an issue or creating a PDEP, it's probably safe to start by [opening an issue](https://github.com/pandas-dev/pandas/issues/new/choose), which can be eventually moved to a PDEP. From a0d7276f269b4e2c40bd9da425021fdd583d2b79 Mon Sep 17 00:00:00 2001 From: Marc Garcia Date: Wed, 22 Jun 2022 00:17:26 +0700 Subject: [PATCH 06/14] Update web/pandas/pdeps/accepted/0001-purpose-and-guidelines.md Co-authored-by: Matthew Roeschke --- web/pandas/pdeps/accepted/0001-purpose-and-guidelines.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/web/pandas/pdeps/accepted/0001-purpose-and-guidelines.md b/web/pandas/pdeps/accepted/0001-purpose-and-guidelines.md index fd630772fe425..f8994a3d9520d 100644 --- a/web/pandas/pdeps/accepted/0001-purpose-and-guidelines.md +++ b/web/pandas/pdeps/accepted/0001-purpose-and-guidelines.md @@ -64,7 +64,7 @@ once the decision is made. Please make sure you select the option A PDEP will be accepted by the core development team, and decisions will be made based on the [pandas governance document](https://github.com/pandas-dev/pandas-governance/blob/master/governance.md). -Once a PDEP is accepted, it is unclear how the implementation will happen. The +Once a PDEP is accepted, any contributions can be made toward the implementing the PDEP with an open-ended completion timeline . The pandas project development, with a mix of volunteers and developers paid from different sources, and development priorities are difficult to understand or forecast. For companies, institutions or individuals with interest in seeing a From a8295b8d4152fee2d8a2682c3d5234adadfc8b19 Mon Sep 17 00:00:00 2001 From: Marc Garcia Date: Wed, 22 Jun 2022 00:19:47 +0700 Subject: [PATCH 07/14] Implemented PDEPs including pandas version --- web/pandas/pdeps/accepted/0001-purpose-and-guidelines.md | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/web/pandas/pdeps/accepted/0001-purpose-and-guidelines.md b/web/pandas/pdeps/accepted/0001-purpose-and-guidelines.md index 474013d6ae40c..fcb5a0b48e3f8 100644 --- a/web/pandas/pdeps/accepted/0001-purpose-and-guidelines.md +++ b/web/pandas/pdeps/accepted/0001-purpose-and-guidelines.md @@ -76,7 +76,8 @@ please check how you can help in the [contributing page](/contribute.html). Once a PDEP is implemented and available in the main branch of pandas, its status will be changed to implemented, so there is visibility that the PDEP is not part of the roadmap and future plans, but a change that it already -happened. +happened. The first pandas version in which the PDEP implementation is +available will also be included in the PDEP. #### Rejected PDEP From ebf16877be66c2d096ce663a3cb1e328e68bfd72 Mon Sep 17 00:00:00 2001 From: Marc Garcia Date: Mon, 27 Jun 2022 11:55:40 +0700 Subject: [PATCH 08/14] Addressed feedback from reviews, couple of visualization fixes, and styles improvements --- web/pandas/about/roadmap.md | 32 ++++------------- .../accepted/0001-purpose-and-guidelines.md | 35 ++++++++++++------- web/pandas/static/css/pandas.css | 8 +++-- web/pandas_web.py | 2 +- 4 files changed, 37 insertions(+), 40 deletions(-) diff --git a/web/pandas/about/roadmap.md b/web/pandas/about/roadmap.md index 5681ea654586d..f58c365bd555c 100644 --- a/web/pandas/about/roadmap.md +++ b/web/pandas/about/roadmap.md @@ -21,36 +21,18 @@ For more information about PDEPs, and how to submit one, please refer to ## PDEPs -### PDEPs under discussion +{% for pdep_type in ["under_discussion", "accepted", "implemented", "rejected"] %} -{% for pdep in pdeps.under_discussion -%} -- [{{ pdep.title }}]({{ pdep.url }}) -{% else %} -There are currently no PEPs under discussion -{% endfor %} - -### Accepted PDEPs +

{{ pdep_type.replace("_", " ").capitalize() }}

-{% for pdep in pdeps.accepted -%} -- [{{ pdep.title }}]({{ pdep.url }}) +
    +{% for pdep in pdeps[pdep_type] %} +
  • {{ pdep.title }}
    • {% else %} -There are currently no accepted PEPs +
  • There are currently no PDEPs with this status
    • {% endfor %} +
-### Rejected PDEPs - -{% for pdep in pdeps.rejected -%} -- [{{ pdep.title }}]({{ pdep.url }}) -{% else %} -There are currently no rejected PEPs -{% endfor %} - -### Implemented PDEPs - -{% for pdep in pdeps.implemented -%} -- [{{ pdep.title }}]({{ pdep.url }}) -{% else %} -There are currently no implemented PEPs {% endfor %} ## Roadmap points pending a PDEP diff --git a/web/pandas/pdeps/accepted/0001-purpose-and-guidelines.md b/web/pandas/pdeps/accepted/0001-purpose-and-guidelines.md index 433642ef3313d..46bb32344d036 100644 --- a/web/pandas/pdeps/accepted/0001-purpose-and-guidelines.md +++ b/web/pandas/pdeps/accepted/0001-purpose-and-guidelines.md @@ -1,9 +1,10 @@ # PDEP-1: Purpose and guidelines - Date: 21 June 2022 -- Status: Accepted +- Status: Under discussion - Discussion: [#47444](https://github.com/pandas-dev/pandas/pull/47444) - Author: [Marc Garcia](https://github.com/datapythonista) +- Revision: 1 ## PDEP definition, purpose and scope @@ -54,15 +55,18 @@ The file is a markdown file, you can use `web/pdeps/accepted/0001.md` as a refer for the expected format. By default, we expect a PDEP will be accepted, so the PR of a PDEP should be done -in the `accepted` directory and contain `Status: Accepted`. If a PDEP is finally -rejected, its status and directory will be updated by the core team before merging, -once the decision is made. Please make sure you select the option -`Allow edits and access to secrets by maintainers` when opening the PR. +in the `accepted` but we will keep `Status: Under discussion` until it is ready to +be merged. If a PDEP is finally rejected, its status and directory will be updated +by the core team before merging, once the decision is made. Please make sure you +select the option `Allow edits and access to secrets by maintainers` when opening the PR. #### Accepted PDEP -A PDEP will be accepted by the core development team, and decisions will be made -based on the [pandas governance document](https://github.com/pandas-dev/pandas-governance/blob/master/governance.md). +A PDEP can only be accepted by the core development team, if the proposal is considered +worth implementing. Decisions will be made based on the process detailed in the +[pandas governance document](https://github.com/pandas-dev/pandas-governance/blob/master/governance.md). +In general, more than one approval will be needed before the PR is merged. And +there should not be any `Request changes` review at the time of merging. Once a PDEP is accepted, any contributions can be made toward the implementing the PDEP with an open-ended completion timeline . The pandas project development, with a mix of volunteers and developers paid from @@ -98,12 +102,19 @@ discussion with the author, instead of merging them as rejected. This is to not add noise to the list of rejected PDEPs, which should contain documentation as good as an accepted PDEP, but where the final decision was to not implement the changes. -## Evolution of this PDEP +## Evolution of PDEPs -While most PDEPs aren't expected to change after accepted, in some cases like this -PDEP, they will be updated to contain its latest version, if things evolve. A log -of the summary of the changes will be kept to make it easier to see if any change -has happened. +Most PDEPs aren't expected to change after accepted. Once there is agreement in the changes, +and they are implemented, the PDEP will be only useful to understand why the development happened, +and the details of the discussion. + +But in some cases, a PDEP can be updated. For example, a PDEP defining a procedure or +a policy, like this one (PDEP-1). Or cases when after attempting the implementation, +new knowledge is obtained that makes the original PDEP obsolete, and changes are +required. When there are specific changes to be made to the original PDEP, this will +be edited, its `Revision: X` label will be increased by one, and a note will be added +to the `PDEP-N history` section. This will let readers understand that the PDEP has +changed and avoid confusion. ### PDEP-1 History diff --git a/web/pandas/static/css/pandas.css b/web/pandas/static/css/pandas.css index 67955dd35587c..5083faa25a87e 100644 --- a/web/pandas/static/css/pandas.css +++ b/web/pandas/static/css/pandas.css @@ -8,15 +8,19 @@ h1 { color: #130654; } h2 { - font-size: 1.45rem; + font-size: 1.8rem; font-weight: 700; - color: black; + color: #130654; } h3 { font-size: 1.3rem; font-weight: 600; color: black; } +h3 a { + color: black; + text-decoration: underline dotted !important; +} a { color: #130654; } diff --git a/web/pandas_web.py b/web/pandas_web.py index 9443c7ead1a3d..ab81b374b17b3 100755 --- a/web/pandas_web.py +++ b/web/pandas_web.py @@ -208,7 +208,7 @@ def roadmap_pdeps(context): status_dir = pdeps_path / status if not status_dir.is_dir(): continue - for pdep in status_dir.iterdir(): + for pdep in sorted(status_dir.iterdir()): if pdep.suffix != ".md": continue html_file = pdep.with_suffix(".html").name From d20de1e6a315dacd979d6b90b8a66c498f02714d Mon Sep 17 00:00:00 2001 From: Marc Garcia Date: Sat, 30 Jul 2022 23:24:22 +0700 Subject: [PATCH 09/14] Addressing comments from reviews --- web/pandas/about/roadmap.md | 2 +- .../0001-purpose-and-guidelines.md | 44 +++++++++-------- web/pandas_web.py | 47 ++++++++++--------- 3 files changed, 51 insertions(+), 42 deletions(-) rename web/pandas/pdeps/{accepted => }/0001-purpose-and-guidelines.md (78%) diff --git a/web/pandas/about/roadmap.md b/web/pandas/about/roadmap.md index e2e5c3d82fead..6335fac7b9619 100644 --- a/web/pandas/about/roadmap.md +++ b/web/pandas/about/roadmap.md @@ -21,7 +21,7 @@ For more information about PDEPs, and how to submit one, please refer to ## PDEPs -{% for pdep_type in ["under_discussion", "accepted", "implemented", "rejected"] %} +{% for pdep_type in ["Under discussion", "Accepted", "Implemented", "Rejected"] %}

{{ pdep_type.replace("_", " ").capitalize() }}

diff --git a/web/pandas/pdeps/accepted/0001-purpose-and-guidelines.md b/web/pandas/pdeps/0001-purpose-and-guidelines.md similarity index 78% rename from web/pandas/pdeps/accepted/0001-purpose-and-guidelines.md rename to web/pandas/pdeps/0001-purpose-and-guidelines.md index 46bb32344d036..dab3882ab9833 100644 --- a/web/pandas/pdeps/accepted/0001-purpose-and-guidelines.md +++ b/web/pandas/pdeps/0001-purpose-and-guidelines.md @@ -1,6 +1,6 @@ # PDEP-1: Purpose and guidelines -- Date: 21 June 2022 +- Created: 30 July 2022 - Status: Under discussion - Discussion: [#47444](https://github.com/pandas-dev/pandas/pull/47444) - Author: [Marc Garcia](https://github.com/datapythonista) @@ -8,7 +8,7 @@ ## PDEP definition, purpose and scope -A PDEP (pandas enhancement proposal) is a proposal to a **major** change in +A PDEP (pandas enhancement proposal) is a proposal for a **major** change in pandas, in a similar way as a Python [PEP](https://peps.python.org/pep-0001/) or a NumPy [NEP](https://numpy.org/neps/nep-0000.html). @@ -31,7 +31,7 @@ consider when writing a PDEP are: - The core development team, who will have the final decision on whether a PDEP is approved or not -- Developers of pandas and other related projects, and experienced users. Their +- Contributors to pandas and other related projects, and experienced users. Their feedback is highly encouraged and appreciated, to make sure all points of views are taken into consideration - The wider pandas community, in particular users, who may or may not have feedback @@ -48,17 +48,23 @@ be eventually moved to a PDEP. ### Workflow +The possible states of a PDEP are: + +- Under discussion +- Accepted +- Implemented +- Rejected + +Next is described the workflow that PDEPs can follow. + #### Submitting a PDEP -Proposing a PDEP is done by creating a PR adding a new file to `web/pdeps/accepted/`. -The file is a markdown file, you can use `web/pdeps/accepted/0001.md` as a reference +Proposing a PDEP is done by creating a PR adding a new file to `web/pdeps/`. +The file is a markdown file, you can use `web/pdeps/0001.md` as a reference for the expected format. -By default, we expect a PDEP will be accepted, so the PR of a PDEP should be done -in the `accepted` but we will keep `Status: Under discussion` until it is ready to -be merged. If a PDEP is finally rejected, its status and directory will be updated -by the core team before merging, once the decision is made. Please make sure you -select the option `Allow edits and access to secrets by maintainers` when opening the PR. +The initial status of a PDEP will be `Status: Under discussion`. This will be changed +to `Status: Accepted` when the PDEP is ready and have the approval of the core team. #### Accepted PDEP @@ -68,17 +74,17 @@ worth implementing. Decisions will be made based on the process detailed in the In general, more than one approval will be needed before the PR is merged. And there should not be any `Request changes` review at the time of merging. -Once a PDEP is accepted, any contributions can be made toward the implementing the PDEP with an open-ended completion timeline . The -pandas project development, with a mix of volunteers and developers paid from -different sources, and development priorities are difficult to understand or -forecast. For companies, institutions or individuals with interest in seeing a -PDEP being implemented, or to in general see progress to the pandas roadmap, -please check how you can help in the [contributing page](/contribute.html). +Once a PDEP is accepted, any contributions can be made toward the implementation of the PDEP, +with an open-ended completion timeline. Development of pandas is difficult to understand and +forecast, being the contributors a mix of volunteers and developers paid from different sources, +with different priorities. For companies, institutions or individuals with interest in seeing a +PDEP being implemented, or to in general see progress to the pandas roadmap, please check how +you can help in the [contributing page](/contribute.html). #### Implemented PDEP Once a PDEP is implemented and available in the main branch of pandas, its -status will be changed to implemented, so there is visibility that the PDEP +status will be changed to `Status: Implemented`, so there is visibility that the PDEP is not part of the roadmap and future plans, but a change that it already happened. The first pandas version in which the PDEP implementation is available will also be included in the PDEP. @@ -86,7 +92,7 @@ available will also be included in the PDEP. #### Rejected PDEP A PDEP can be rejected when the final decision is that its implementation is -not the best for the interests of the project. They are as useful as accepted +not the best for the interests of the project. Rejected PDEPs are as useful as accepted PDEPs, since there are discussions that are worth having, and decisions about changes to pandas being made. They will be merged with `Status: Rejected`, so there is visibility on what was discussed and what was the outcome of the @@ -118,4 +124,4 @@ changed and avoid confusion. ### PDEP-1 History -- 21 June 2022: Initial version +- 30 July 2022: Initial version diff --git a/web/pandas_web.py b/web/pandas_web.py index ab81b374b17b3..16e9024d8d1d8 100755 --- a/web/pandas_web.py +++ b/web/pandas_web.py @@ -24,6 +24,7 @@ The rest of the items in the file will be added directly to the context. """ import argparse +import collections import datetime import importlib import operator @@ -194,33 +195,35 @@ def roadmap_pdeps(context): and linked from there. This preprocessor obtains the list of PDEP's in different status from the directory tree and GitHub. """ - context["pdeps"] = { - "accepted": [], - "rejected": [], - "under_discussion": [], - "implemented": [], - } + KNOWN_STATUS = {"Under discussion", "Accepted", "Implemented", "Rejected"} + context["pdeps"] = collections.defaultdict(list) + # accepted, rejected and implemented pdeps_path = ( pathlib.Path(context["source_path"]) / context["roadmap"]["pdeps_path"] ) - for status in ("accepted", "rejected", "implemented"): - status_dir = pdeps_path / status - if not status_dir.is_dir(): + for pdep in sorted(pdeps_path.iterdir()): + if pdep.suffix != ".md": continue - for pdep in sorted(status_dir.iterdir()): - if pdep.suffix != ".md": - continue - html_file = pdep.with_suffix(".html").name - print(pdep) - with pdep.open() as f: - title = f.readline()[2:] # removing markdown title "# " - context["pdeps"][status].append( - { - "title": title, - "url": f"/pdeps/{status}/{html_file}", - } - ) + with pdep.open() as f: + title = f.readline()[2:] # removing markdown title "# " + status = None + for line in f: + if line.startswith("- Status: "): + status = line.strip().split(": ", 1)[1] + break + if status not in KNOWN_STATUS: + raise RuntimeError( + f'PDEP "{pdep}" status "{status}" is unknown. ' + f"Should be one of: {KNOWN_STATUS}" + ) + html_file = pdep.with_suffix(".html").name + context["pdeps"][status].append( + { + "title": title, + "url": f"/pdeps/{html_file}", + } + ) # under discussion github_repo_url = context["main"]["github_repo_url"] From 9b37d113616c0b85e7d08a19b7e3ea0f11e20aac Mon Sep 17 00:00:00 2001 From: Marc Garcia Date: Wed, 3 Aug 2022 10:13:12 +0700 Subject: [PATCH 10/14] Update web/pandas/pdeps/0001-purpose-and-guidelines.md Co-authored-by: Simon Hawkins --- web/pandas/pdeps/0001-purpose-and-guidelines.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/web/pandas/pdeps/0001-purpose-and-guidelines.md b/web/pandas/pdeps/0001-purpose-and-guidelines.md index dab3882ab9833..3e4a4487da2b4 100644 --- a/web/pandas/pdeps/0001-purpose-and-guidelines.md +++ b/web/pandas/pdeps/0001-purpose-and-guidelines.md @@ -76,7 +76,7 @@ there should not be any `Request changes` review at the time of merging. Once a PDEP is accepted, any contributions can be made toward the implementation of the PDEP, with an open-ended completion timeline. Development of pandas is difficult to understand and -forecast, being the contributors a mix of volunteers and developers paid from different sources, +forecast, being that the contributors to pandas are a mix of volunteers and developers paid from different sources, with different priorities. For companies, institutions or individuals with interest in seeing a PDEP being implemented, or to in general see progress to the pandas roadmap, please check how you can help in the [contributing page](/contribute.html). From 55b38873a7c97bf35c16e7c7f967beeb807b2c34 Mon Sep 17 00:00:00 2001 From: Marc Garcia Date: Wed, 3 Aug 2022 10:13:27 +0700 Subject: [PATCH 11/14] Update web/pandas/pdeps/0001-purpose-and-guidelines.md Co-authored-by: Simon Hawkins --- web/pandas/pdeps/0001-purpose-and-guidelines.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/web/pandas/pdeps/0001-purpose-and-guidelines.md b/web/pandas/pdeps/0001-purpose-and-guidelines.md index 3e4a4487da2b4..dd29a009967cb 100644 --- a/web/pandas/pdeps/0001-purpose-and-guidelines.md +++ b/web/pandas/pdeps/0001-purpose-and-guidelines.md @@ -85,7 +85,7 @@ you can help in the [contributing page](/contribute.html). Once a PDEP is implemented and available in the main branch of pandas, its status will be changed to `Status: Implemented`, so there is visibility that the PDEP -is not part of the roadmap and future plans, but a change that it already +is not part of the roadmap and future plans, but a change that has already happened. The first pandas version in which the PDEP implementation is available will also be included in the PDEP. From 8c34db0eb2520606460ff36bf25ce4043064cd74 Mon Sep 17 00:00:00 2001 From: Marc Garcia Date: Wed, 3 Aug 2022 10:13:58 +0700 Subject: [PATCH 12/14] Update web/pandas/pdeps/0001-purpose-and-guidelines.md Co-authored-by: Simon Hawkins --- web/pandas/pdeps/0001-purpose-and-guidelines.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/web/pandas/pdeps/0001-purpose-and-guidelines.md b/web/pandas/pdeps/0001-purpose-and-guidelines.md index dd29a009967cb..782060f77b656 100644 --- a/web/pandas/pdeps/0001-purpose-and-guidelines.md +++ b/web/pandas/pdeps/0001-purpose-and-guidelines.md @@ -92,7 +92,7 @@ available will also be included in the PDEP. #### Rejected PDEP A PDEP can be rejected when the final decision is that its implementation is -not the best for the interests of the project. Rejected PDEPs are as useful as accepted +not in the best interests of the project. Rejected PDEPs are as useful as accepted PDEPs, since there are discussions that are worth having, and decisions about changes to pandas being made. They will be merged with `Status: Rejected`, so there is visibility on what was discussed and what was the outcome of the From 4f3343b443afff2455705b6ab37f883eed4bf659 Mon Sep 17 00:00:00 2001 From: Marc Garcia Date: Wed, 3 Aug 2022 10:14:15 +0700 Subject: [PATCH 13/14] Update web/pandas/pdeps/0001-purpose-and-guidelines.md Co-authored-by: Simon Hawkins --- web/pandas/pdeps/0001-purpose-and-guidelines.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/web/pandas/pdeps/0001-purpose-and-guidelines.md b/web/pandas/pdeps/0001-purpose-and-guidelines.md index 782060f77b656..c70cbc795a0fa 100644 --- a/web/pandas/pdeps/0001-purpose-and-guidelines.md +++ b/web/pandas/pdeps/0001-purpose-and-guidelines.md @@ -104,8 +104,8 @@ implementing. For submitted PDEPs that do not contain proper documentation, are out of scope, or are not useful to the community for any other reason, the PR will be closed after -discussion with the author, instead of merging them as rejected. This is to not -add noise to the list of rejected PDEPs, which should contain documentation as +discussion with the author, instead of merging them as rejected. This is to avoid +adding noise to the list of rejected PDEPs, which should contain documentation as good as an accepted PDEP, but where the final decision was to not implement the changes. ## Evolution of PDEPs From 7c1a7256c8ce3433aedfeb4e77ad3cb766498ee3 Mon Sep 17 00:00:00 2001 From: Marc Garcia Date: Wed, 3 Aug 2022 10:20:43 +0700 Subject: [PATCH 14/14] Last review comments and dates updated --- web/pandas/about/roadmap.md | 2 +- web/pandas/pdeps/0001-purpose-and-guidelines.md | 7 ++++--- 2 files changed, 5 insertions(+), 4 deletions(-) diff --git a/web/pandas/about/roadmap.md b/web/pandas/about/roadmap.md index 6335fac7b9619..6e922d01518ba 100644 --- a/web/pandas/about/roadmap.md +++ b/web/pandas/about/roadmap.md @@ -39,7 +39,7 @@ For more information about PDEPs, and how to submit one, please refer to diff --git a/web/pandas/pdeps/0001-purpose-and-guidelines.md b/web/pandas/pdeps/0001-purpose-and-guidelines.md index c70cbc795a0fa..085f675974b2e 100644 --- a/web/pandas/pdeps/0001-purpose-and-guidelines.md +++ b/web/pandas/pdeps/0001-purpose-and-guidelines.md @@ -1,6 +1,6 @@ # PDEP-1: Purpose and guidelines -- Created: 30 July 2022 +- Created: 3 August 2022 - Status: Under discussion - Discussion: [#47444](https://github.com/pandas-dev/pandas/pull/47444) - Author: [Marc Garcia](https://github.com/datapythonista) @@ -87,7 +87,8 @@ Once a PDEP is implemented and available in the main branch of pandas, its status will be changed to `Status: Implemented`, so there is visibility that the PDEP is not part of the roadmap and future plans, but a change that has already happened. The first pandas version in which the PDEP implementation is -available will also be included in the PDEP. +available will also be included in the PDEP header with for example +`Implemented: v2.0.0`. #### Rejected PDEP @@ -124,4 +125,4 @@ changed and avoid confusion. ### PDEP-1 History -- 30 July 2022: Initial version +- 3 August 2022: Initial version