Update changelog/contributors/version bump in one script #19379

huonw · 2023-06-26T11:13:16Z

This evolves the changelog.py script into start_release.py, which does the first 4-ish steps of the release process:

Update the changelog
Update CONTRIBUTORS.md
Bump the VERSION, on main
Commit the above and open a PR

This is currently just meant for running locally, but a possible next step would be packaging it up to run in CI: a manually triggered workflow that takes the new version number as an input and then runs .../start_release.py --new $version_input --release-manager $sender_login --publish, to open the PR as @WorkerPants.

This is a step towards #19279, and in particular, does most of "1", other than having it run in CI.

This proposes a few conventions:

branches from automation are scoped under automation/, e.g. these branches are automation/release/2.34.56
labels for automation start with automation:, e.g. these release prep PRs would be created with label automation:release-prep

Thoughts? We could retrofit these conventions onto the cherry-picker, if they seem nice.

Next steps after this might be:

run this in CI
finish step 3, by automating the publishing of the docs site, PEX creation and even announcements
finish step 2, by having automation that detects when the release PR is merged and cherry-picks, tags, creates branches and bumps versions (etc.), as appropriate

src/python/pants_release/start_release_test.py

huonw · 2023-06-26T11:14:20Z

src/python/pants_release/start_release.py

+        title=title,
+        body="\n\n".join([BASE_PR_BODY, formatted.internal]),
+        assignee=release_manager,
+        labels=["automation:release-prep"],


Note branch name and this label name, per the discussion of conventions in the PR description.

huonw · 2023-06-26T11:14:51Z

src/python/pants_release/start_release.py

+    # infer the changes since the most recent previous release on this branch
+    prior_tag = git("describe", "--tags", "--abbrev=0", release_ref)


This changes this script to not need the --prior argument, and instead just look for the most recent tag before the release ref.

huonw · 2023-06-26T11:15:37Z

src/python/pants_release/start_release.py

+        "--log-level",
+        default="WARNING",


For interactive use WARNING seems fine, but for running on CI, I was planning to set --log-level=DEBUG or something, to make it easier to understand what it's doing.

src/python/pants_release/git.py

thejcannon

Some drive by comments before I review thoroughly

src/python/pants_release/start_release.py

thejcannon · 2023-06-26T11:45:04Z

src/python/pants_release/start_release.py


 logger = logging.getLogger(__name__)

+BASE_PR_BODY = softwrap(


Can this instead be a comment on PR? Making it the PR body feels weird

CONTRIBUTORS.md

kaos · 2023-06-26T17:52:56Z

Another drive by observation without having looked at the code.

Given the number of steps done by this script, I think I would appreciate it if I were able to specify which ones to run. Say it gets halfway through and then aborts due to some error/issue or what not. If I'm able to address that and re-run I'd like it to be idempotent (either by detection, or perhaps more likely by me asking it to proceed at a certain step or something along those lines).

As long as the actions are local only, there's always the option to revert any changes to start over, but in case the issue is to manually tweak and then continue in order to get over a hump that could prove difficult. This is of course only a nice to have, but if it's easy enough to add.. perhaps worth having in the back of your mind at least :)

Sorry for the long rambling..

thejcannon · 2023-06-26T18:22:12Z

Piggybacking off @kaos maybe we can expose one script per step, that can each be run independently (with assumptions, of course).
And then one script-runner (maybe just a bash script?) which fires them off.

huonw · 2023-06-26T22:46:28Z

Sounds good. I personally prefer calling Python functions in Python rather than arranging python subprocesses in bash, so I'll go with @kaos's original idea of being able to specify which action as args to the script.

thejcannon · 2023-06-27T01:08:15Z

Generally, I see exposing each script's main as main and dealer's choice on how args are passed into it.

I can speak personally that my ability to reason about code goes down as line count goes up (I get easily distracted) so fewer, smaller, files makes me 😍

benjyw · 2023-06-27T06:42:19Z

I think having it be idempotent would be best, and preferable to a bunch of one-function scripts.

But note that this is really start_dev_release.py. For RCs the process is a little different - e.g., we bump the VERSION only in the release branch.

thejcannon · 2023-06-28T00:23:45Z

I think if the cherry pick script detects a CP with the proper label, it can do the version bump for the milestone branch.

huonw · 2023-06-28T02:25:27Z

I'll switch to idempotent. 👍

I think if the cherry pick script detects a CP with the proper label, it can do the version bump for the milestone branch.

I think we'll want dedicated automation for this, since it both does than the current auto cherry picker script:

more: it should also tag and potentially create a branch, see "The proposal" -> "2. Tag release" in Proposal: Smoother releases: 3 parts to have mostly-automated releases #19279 for my understanding of what it could do
less: AIUI, the release prep is only cherry picked to the specific relevant branch, not to all more recent ones, e.g. 2.16.0 release prep goes only to 2.16.x, not 2.17.x?

Of course, this non-cherrypicker automation can and should still do the version bump, if it needs to 👍

benjyw · 2023-06-29T02:18:25Z

I think we'd always want a human to create the tag? Esp. since we sign the release tags. And we need some human-triggered action to kick off a release. But scripts can then act off the tag to automate everything else.

thejcannon · 2023-07-12T17:39:33Z

@huonw what's the state of this PR? 😄

huonw · 2023-07-12T23:01:49Z

Updates:

the local changes are now idempotent (except for the external state like the date and PR labels for categorisation)
committing and opening a PR now only happens if --publish is explicitly specified
--publish uses PyGithub to open the PR and add comments etc. this requires adding a new dependency (Regenerate lockfiles for 3.9.* ICs #19454 will reduce the spurious/distracting lock file changes that happen in this PR)
change the release procedure to not use --publish, because setting up the token is a bit fiddly; the main purpose for that is running in CI

Example: huonw#6 (particularly huonw@894071b)

thejcannon · 2023-07-13T15:57:33Z

Can you comment a smidge on switching from gh wrapper to github? Is it code simplicity?

From the CI perspective, both are sufficient so that's unlikely the razor.

From the local perspective, I imagine it's much more likely a user has gh authenticated, and therefore can run the command without much headache. Versus having to make and use a token (although devil's advocate is that gh has gh auth token. So maybe the compromise is this code tries to use that if no token is configured?)

kaos · 2023-07-13T17:36:22Z

3rdparty/python/requirements.txt

+# Only used for release management
+PyGithub==2.0.0rc1


We can enforce this, by adding ("[PyGithub]", "src/python/pants_release/**", "!*"), to

pants/3rdparty/python/BUILD

Lines 36 to 49 in 00786d3

__dependents_rules__(

( # Only the explorer server may depend on these libraries

(

"[fastapi]",

"[starlette]",

"[strawberry-graphql]",

"[uvicorn]",

),

"src/python/pants/explorer/server/**",

"!*",

),

# Free for all on the rest

("*", "*"),

)

Something like:

__dependents_rules__( ( # Only the explorer server may depend on these libraries ( "[fastapi]", "[starlette]", "[strawberry-graphql]", "[uvicorn]", ), "src/python/pants/explorer/server/**", "!*", ), ("[PyGithub]", "src/python/pants_release/**", "!*"), # Free for all on the rest ("*", "*"), )

As a slight aside, should we swap so the core pants package is includelisted, and everything else is free game?
Because it shouldn't matter if a release needs fastapi (I mean why, but you get the point) or some non-included plugin uses PyGitHub (what if its a GitHub plugin?)

I've filed #19458

As a slight aside, should we swap so the core pants package is includelisted, and everything else is free game? Because it shouldn't matter if a release needs fastapi (I mean why, but you get the point) or some non-included plugin uses PyGitHub (what if its a GitHub plugin?)

That works.

huonw · 2023-07-13T21:15:08Z

Can you comment a smidge on switching from gh wrapper to github? Is it code simplicity?

It was motivated by two things:

the explicit suggestion in Update changelog/contributors/version bump in one script #19379 (comment)
writing the instructions as a comment is easier if we let someone else do the parsing of the API response into structured objects rather than needing to extract the number to use gh pr create: Update changelog/contributors/version bump in one script #19379 (comment)

From the local perspective, I imagine it's much more likely a user has gh authenticated, and therefore can run the command without much headache. Versus having to make and use a token (although devil's advocate is that gh has gh auth token. So maybe the compromise is this code tries to use that if no token is configured?)

Good catch! Using gh auth token seems perfect, not a devil's-advocate position. I'll switch to using that unconditionally, because it'll even handle GH_TOKEN=... env var. If someone wants to avoid needing gh installed, we can implement the feature then.

thejcannon

Overall LGTM. I love me some more automation.

src/python/pants_release/git.py

src/python/pants_release/start_release.py

lilatomic · 2023-07-16T17:04:57Z

src/python/pants_release/generate_release_announcement.py

    cur_version_sha, prev_version_sha = git(
-        "log", "-2", "--pretty=format:%h", "src/python/pants/VERSION"
+        "log", "-2", "--pretty=format:%h", str(VERSION_PATH)
    ).splitlines(keepends=False)


do we want to extract this into a method in git.py to keep all the git in one place?

For one-off/specialised calls like this, I don't have a particular point of view. For me, I'd lean away from wrapping this sort of thing up into a "find_two_most_recent_commits_changing_version" function unless there were more locations that needed it! 🤷

That's fair. There's always a tension between keeping everything at the same abstraction level and having a pile of hyper-specific functions. I find that naming the function by why I would use it helps me determine if it's worth wrapping. (Put another way: the view of this interface by the consumer rather than the provider.) something like get_commit_filter_for_this_release would fit more naturally in a list of steps that this function performs, like:

get git commit filter for this release

get and sort all contributors from that range

identify new contributors

...

You're right that this would make our git.py contain a lot of one-off functions.

thejcannon

I think this has been open long enough and I trust your judgement on fixing the comments. So I'll go ahead and approve, because it's good stuff.

huonw · 2023-07-17T00:16:37Z

Example/validation of the latest state of this (f0e56b1): huonw#8

huonw · 2023-07-17T23:18:41Z

I've created the automation:release-prep label. Merging based on @thejcannon's review and @kaos's 👍, but happy to iterate post-merge if there's further feedback.

huonw added 10 commits June 23, 2023 20:42

Rename changelog.py

c0b6564

Infer prior release based on git tags

db7fd89

Factor out version path

f710e53

Also bump version

469f0e7

Rename/refactor to 'update'

bcc624c

Manage CONTRIBUTORS.md in start_release.py

1ec6054

Basic github CLI wrapper

9704893

Commit and open PR

513c079

Controllable log level

d377366

Docs

2070422

huonw added the category:internal CI, fixes for not-yet-released features, etc. label Jun 26, 2023

huonw commented Jun 26, 2023

View reviewed changes

huonw requested review from benjyw and thejcannon June 26, 2023 11:19

Update test

248e7d5

thejcannon reviewed Jun 26, 2023

View reviewed changes

huonw added 2 commits July 5, 2023 12:19

Idempotent changelog replacement

714b627

Explicit --publish flag

c18b4ed

huonw added 3 commits July 13, 2023 07:49

Merge remote-tracking branch 'upstream/main' into feature/release-start

b4d8250

Add PyGithub dependency (plus distracting other changes)

9fa9c90

rm unused import

66d4269

huonw added 2 commits July 13, 2023 08:17

remove github cli wrapper

d5a9546

use pygithub to create PR

2e863fb

huonw requested a review from thejcannon July 12, 2023 23:02

Doc updates for new --publish

00786d3

kaos reviewed Jul 13, 2023

View reviewed changes

huonw mentioned this pull request Jul 13, 2023

Use visibility to enforce expected requirements for pants itself #19458

Open

Use gh auth token instead

0c60373

thejcannon reviewed Jul 14, 2023

View reviewed changes

thejcannon mentioned this pull request Jul 14, 2023

Add PBS Python provider #19462

Closed

lilatomic reviewed Jul 16, 2023

View reviewed changes

huonw added 2 commits July 17, 2023 09:48

Merge remote-tracking branch 'upstream/main' into feature/release-start

5fd9e8c

Contributors

554a15b

thejcannon approved these changes Jul 16, 2023

View reviewed changes

huonw added 4 commits July 17, 2023 10:02

rewrite gh auth token comment

62ae8b1

style

36a60cf

internal changes as comment

29520a4

Do changes while on main

f0e56b1

huonw force-pushed the feature/release-start branch from 8af8b04 to f0e56b1 Compare July 17, 2023 00:16

huonw merged commit 38cb6de into pantsbuild:main Jul 17, 2023

huonw deleted the feature/release-start branch July 17, 2023 23:18

wisechengyi mentioned this pull request Jul 24, 2023

Prepare 2.18.0.dev6 #19519

Merged

huonw mentioned this pull request Sep 15, 2024

upper bounds on s3 urllib3 version to the s3 handler easier to use #21402

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Update changelog/contributors/version bump in one script #19379

Update changelog/contributors/version bump in one script #19379

huonw commented Jun 26, 2023 •

edited

Loading

huonw Jun 26, 2023

huonw Jun 26, 2023

huonw Jun 26, 2023

thejcannon left a comment

thejcannon Jun 26, 2023

kaos commented Jun 26, 2023

thejcannon commented Jun 26, 2023

huonw commented Jun 26, 2023

thejcannon commented Jun 27, 2023

benjyw commented Jun 27, 2023

thejcannon commented Jun 28, 2023

huonw commented Jun 28, 2023

benjyw commented Jun 29, 2023

thejcannon commented Jul 12, 2023

huonw commented Jul 12, 2023 •

edited

Loading

thejcannon commented Jul 13, 2023

kaos Jul 13, 2023 •

edited

Loading

thejcannon Jul 13, 2023

huonw Jul 13, 2023

kaos Jul 14, 2023

huonw commented Jul 13, 2023

thejcannon left a comment

lilatomic Jul 16, 2023

huonw Jul 16, 2023

lilatomic Jul 19, 2023

thejcannon left a comment

huonw commented Jul 17, 2023

huonw commented Jul 17, 2023

		# infer the changes since the most recent previous release on this branch
		prior_tag = git("describe", "--tags", "--abbrev=0", release_ref)


		logger = logging.getLogger(__name__)

		BASE_PR_BODY = softwrap(

	__dependents_rules__(
	( # Only the explorer server may depend on these libraries
	(
	"[fastapi]",
	"[starlette]",
	"[strawberry-graphql]",
	"[uvicorn]",
	),
	"src/python/pants/explorer/server/**",
	"!*",
	),
	# Free for all on the rest
	("", ""),
	)

Update changelog/contributors/version bump in one script #19379

Update changelog/contributors/version bump in one script #19379

Conversation

huonw commented Jun 26, 2023 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

thejcannon left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

kaos commented Jun 26, 2023

thejcannon commented Jun 26, 2023

huonw commented Jun 26, 2023

thejcannon commented Jun 27, 2023

benjyw commented Jun 27, 2023

thejcannon commented Jun 28, 2023

huonw commented Jun 28, 2023

benjyw commented Jun 29, 2023

thejcannon commented Jul 12, 2023

huonw commented Jul 12, 2023 • edited Loading

thejcannon commented Jul 13, 2023

kaos Jul 13, 2023 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

huonw commented Jul 13, 2023

thejcannon left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

thejcannon left a comment

Choose a reason for hiding this comment

huonw commented Jul 17, 2023

huonw commented Jul 17, 2023

huonw commented Jun 26, 2023 •

edited

Loading

huonw commented Jul 12, 2023 •

edited

Loading

kaos Jul 13, 2023 •

edited

Loading