Modernise code in Tools/clinic/

[AlexWaygood]

Feature or enhancement

Modernise some Python anachronisms in Tools/clinic/, to make the code more readable and maintainable. Some things that can be easily done:

• Use dict instead of OrderedDict (gh-104683: Argument clinic: use dict over OrderedDict #104647)

• Simplify code like this by using the str.removesuffix and str.removeprefix methods, both new in Python 3.9:
  

  cpython/Tools/clinic/clinic.py

  Lines 2572 to 2575 in 06eeee9

  	name = f.__name__
  	if not name.endswith('_converter'):
  	return f
  	name = name[:-len('_converter')]

  (gh-104683: Modernise clinic.py using str.removeprefix and str.removesuffix #104685)

• Pyupgrade (run using the --py310-plus setting) also finds various things that can be modernised (gh-104683: Modernise Tools/clinic/ using pyupgrade #104684)

• Refactor CLanguage.output_templates (see Modernise code in Tools/clinic/ #104683 (comment))

• Put templating in a separate .py file, kind of like Tools/clinic/cpp.py

• Rework Argument Clinic error handling #107468

• Restructure Argument Clinic CLI #107467

Cc. @erlend-aasland

Linked PRs

• gh-104683: Argument clinic: use dict over OrderedDict #104647
• gh-104683: Modernise Tools/clinic/ using pyupgrade #104684
• gh-104683: Modernise clinic.py using str.removeprefix and str.removesuffix #104685
• gh-104683: clinic.py: Modernise parse_converter() using pattern-matching #104696
• gh-104683: clinic.py: Improve coverage for the parse_converter method #104729
• [3.11] gh-104683: clinic.py: Improve coverage for the parse_converter method (#104729) #104730
• gh-104683: Modernise Argument Clinic parameter state machine #106362
• gh-104683: Rename clinic.test => clinic.test.c #106443
• [3.12] gh-104683: Rename Lib/test/clinic.test as Lib/test/clinic.test.c (GH-106443) #106444
• [3.11] gh-104683: Rename Lib/test/clinic.test as Lib/test/clinic.test.c (GH-106443) #106445
• gh-104683: clinic.py: refactor four simple classes as dataclasses #106476
• gh-104683: clinic.py: refactor Parameter and Function as dataclasses #106477
• gh-104683: clinic.py: Don't needlessly reimplement contextlib.redirect_stdout #106478
• gh-104683: Argument clinic: Minor readability improvements for Destination.__init__ #106652
• gh-104683: Argument clinic: modernise cpp.Monitor #106698
• gh-104683: Argument clinic: use an enum to describe the different kinds of functions #106721
• gh-104683: Argument Clinic: Modernise parse_special_symbol() #106837
• gh-104683: Argument Clinic: Make most arguments to Class and Function required #107289
• gh-104683: Argument clinic: cleanup state_modulename_name() #107340
• gh-104683: Argument clinic: pass clinic as a parameter where possible #107435
• gh-104683: Argument clinic: Make the filename parameter to Clinic required #107439
• gh-104683: Argument clinic: remove the LandMine class #107541
• gh-104683: Argument clinic: cleanup DLSParser state_foo methods #107543
• gh-104683: Argument Clinic: Refactor and simplify 'add docstring' states #107550
• gh-104683: Rework Argument Clinic error handling #107551
• gh-104683: Make Argument Clinic template strings class level members #107556
• gh-104683: Argument Clinic: refactor format_docstring() #107623
• gh-104683: Argument clinic: remove some unnecessary uses of self.next() in the DSLParser #107635
• gh-104683: Argument Clinic: Use CConverter.length_name where possible #107638
• gh-104683: Improve consistency and test coverage of argument-clinic __repr__ functions #107667
• gh-104683: Add --exclude option to Argument Clinic CLI #107770
• gh-104683: Remove unused variables from Tools/clinic and tests for Tools/clinic #107771
• gh-104683: Argument Clinic: Params now render their own docstrings #107790
• gh-104683: Argument clinic: make format_docstring() a method on Function objects #107840
• gh-104683: Argument Clinic: Extract parse_function_names() helper #107964
• gh-104683: Argument Clinic: Remove unreachable code from _module_and_class() #108092
• gh-104683: Argument Clinic: Refactor the module and class resolver #108552

[erlend-aasland]

⚠️ Refactoring is a useful tool, but it is also risky business. We must make sure we've got good test coverage for all touched code. If we're touching uncovered branches, we must add new tests in a separate PR first ⚠️

[erlend-aasland]

The parameter_state state machine could be made slightly more readable by using enums, for example:

cpython/Tools/clinic/clinic.py

Lines 4711 to 4726 in 35963da

	# These rules are enforced with a single state variable:
	# "parameter_state". (Previously the code was a miasma of ifs and
	# separate boolean state variables.) The states are:
	#
	# [ [ a, b, ] c, ] d, e, f=3, [ g, h, [ i ] ] <- line
	# 01 2 3 4 5 6 <- state transitions
	#
	# 0: ps_start. before we've seen anything. legal transitions are to 1 or 3.
	# 1: ps_left_square_before. left square brackets before required parameters.
	# 2: ps_group_before. in a group, before required parameters.
	# 3: ps_required. required parameters, positional-or-keyword or positional-only
	# (we don't know yet). (renumber left groups!)
	# 4: ps_optional. positional-or-keyword or positional-only parameters that
	# now must have default values.
	# 5: ps_group_after. in a group, after required parameters.
	# 6: ps_right_square_after. right square brackets after required parameters.

[erlend-aasland]

Argument Clinic is now slightly more readable, thanks to the applied refactors. I suggest we continue to improve and modernise Argument Clinic, but now through more targeted issues; this issue has served its purpose. Thanks to everyone involved.