Compile time textwrap.dedent() equivalent for str or bytes literals

[gpshead]

BPO	36906
Nosy	@rhettinger, @terryjreedy, @gpshead, @stevendaprano, @methane, @serhiy-storchaka, @MojoVampire, @Carreau, @pablogsal, @remilapeyre, @Marco-Sulla, @iforapsy, @jtojnar
PRs	gh-81087: Add str.dedent and cache it at compile time #13445

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = 'https://github.com/gpshead'
closed_at = None
created_at = <Date 2019-05-13.18:40:31.178>
labels = ['interpreter-core', 'type-feature', '3.10']
title = 'Compile time textwrap.dedent() equivalent for str or bytes literals'
updated_at = <Date 2021-12-20.23:57:51.965>
user = 'https://github.com/gpshead'

bugs.python.org fields:

activity = <Date 2021-12-20.23:57:51.965>
actor = 'iforapsy'
assignee = 'gregory.p.smith'
closed = False
closed_date = None
closer = None
components = ['Interpreter Core']
creation = <Date 2019-05-13.18:40:31.178>
creator = 'gregory.p.smith'
dependencies = []
files = []
hgrepos = []
issue_num = 36906
keywords = ['patch']
message_count = 46.0
messages = ['342373', '342407', '342420', '342429', '342477', '342488', '342569', '342600', '342909', '342914', '342915', '342916', '342917', '342918', '342927', '342928', '342931', '342938', '342962', '342965', '342968', '342972', '343961', '343991', '350162', '356153', '356154', '356155', '356160', '356162', '356182', '356193', '356198', '356203', '356204', '365688', '381540', '381544', '381545', '381546', '381584', '381599', '381602', '381605', '382456', '398270']
nosy_count = 14.0
nosy_names = ['rhettinger', 'terry.reedy', 'gregory.p.smith', 'steven.daprano', 'methane', 'serhiy.storchaka', 'josh.r', 'mbussonn', 'pablogsal', 'remi.lapeyre', 'Marco Sulla', 'iforapsy', 'Miguel Amaral', 'jtojnar']
pr_nums = ['13445']
priority = 'normal'
resolution = None
stage = 'patch review'
status = 'open'
superseder = None
type = 'enhancement'
url = 'https://bugs.python.org/issue36906'
versions = ['Python 3.10']

[gpshead]

A Python pattern in code is to keep everything indented to look pretty while, yet when the triple quoted multiline string in question needs to not have leading whitespace, calling textwrap.dedent("""long multiline constant""") is a common pattern.

rather than doing this computation at runtime, this is something that'd make sense to do at compilation time. A natural suggestion for this would be a new letter prefix for multiline string literals that triggers this.

Probably not worth "wasting" a letter on this, so I'll understand if we reject the idea, but it'd be nice to have rather than importing textwrap and calling it all over the place just for this purpose.

There are many workarounds but an actual syntax would enable writing code that looked like this:

class Castle:
    def __init__(self, name, lyrics=None):
        if not lyrics:
            lyrics = df"""\
            We're knights of the round table
            We dance whene'er we're able
            We do routines and scenes
            With footwork impeccable.
            We dine well here in {name}
            We eat ham and jam and spam a lot.
            """
        self._name = name
        self._lyrics = lyrics

Without generating a larger temporary always in memory string literal in the code object that gets converted at runtime to the desired dedented form via a textwrap.dedent() call. I chose "d" as the the letter to mean dedent. I don't have a strong preference if we ever do make this a feature.

[rhettinger]

I agree that this is a recurring need and would be nice to have.

[stevendaprano]

+1

There's a long thread on something similar here:

https://mail.python.org/pipermail/python-ideas/2018-March/049564.html

Carrying over into the following month:

https://mail.python.org/pipermail/python-ideas/2018-April/049582.html

Here's an even older thread:

https://mail.python.org/pipermail/python-ideas/2010-November/008589.html

In the more recent thread, I suggested that we give strings a dedent method. When called on a literal, the keyhole optimizer may do the dedent at compile time. Whether it does or not is a "quality of implementation" factor.

The idea is to avoid the combinational explosion of yet another string prefix:

urd'...'  # unicode raw string dedent

while still making string dedents easily discoverable, and with a sufficiently good interpreter, string literals will be dedented at compile time avoiding any runtime cost:

https://mail.python.org/pipermail/python-ideas/2018-March/049578.html

[gpshead]

Oh good, I thought this had come up before. Your method idea that could be optimized on literals makes a lot of sense, and is notably more readable than yet another letter prefix.

[remilapeyre]

Hi, I have been looking to get more acquainted with the peephole optimizer. Is it okay if I work on this?

[gpshead]

I'd say go for it. We can't guarantee we'll accept the feature yet, but I think the .dedent() method with an optimization pass approach is worthwhile making a proof of concept of regardless.

[stevendaprano]

For the record, I just came across this proposed feature for Java:

https://openjdk.java.net/jeps/8222530

Add text blocks to the Java language. A text block is a multi-line 
string literal that avoids the need for most escape sequences, 
automatically formats the string in predictable ways, and gives the 
developer control over format when desired.

It seems to be similar to Python triple-quoted strings except that the
compiler automatically dedents the string using a "re-indentation
algorithm". (Which sounds to me similar to, if not identical, to that
used by textwrap.)

The JEP proposal says:

A study of Google's large internal repository of Java source code 
showed that 98% of string literals, once converted to text blocks 
and formatted appropriately, would require removal of incidental 
white space. If Java introduced multi-line string solution without 
support for automatically removing incidental white space, then many 
developers would write a method to remove it themselves and/or lobby 
for the String class to include a removal method.

which matches my own experience: *most* but not all of my indented
triple-quotes strings start with incidental whitespace that I don't care
about. But not quite all, so I think backwards compatibility requires
that *by default* triple-quoted strings are not dedented.

Note that there are a couple of major difference between the JEP
proposal and this:

• The JEP proposes to automatically dedent triple-quoted strings;
  this proposal requires an explicit call to .dedent().

• The JEP proposal allows the user to control the dedent by
  indenting, or not, the trailing end-quote;

• however that means that in Java you won't be able to control
  the dedent if the string doesn't end with a final blank line;

• Should the dedent method accept an optional int argument
  specifying the number of spaces to dedent by? (Defaulting to
  None, meaning "dedent by the common indent".) If so, that won't
  affect the compile-time optimization so long as the argument is
  a literal.

• the JEP performs the dedent before backslash escapes are
  interpreted; in this proposal backslash escapes will
  occur before the dedent.

The JEP also mentions considering multi-line string literals as Swift
and Rust do them:

https://github.com/apple/swift-evolution/blob/master/proposals/0168-multi-line-string-literals.md

https://stackoverflow.com/questions/29483365/what-is-the-syntax-for-a-multiline-string-literal

I mention these for completeness, not to suggest them as alternatives.

[gpshead]

Thanks, it's actually good to see this being a feature accepted in other languages.

[remilapeyre]

Hi @steven.daprano, @gregory.p.smith. I added the first version of my PR for review.

One issue with it is that in:

def f():
    return "   foo".dedent()

f will have both " foo" and "foo" in its constants even if the first is not used anymore. Removing it requires looping over the code once more while marking the constants seen in a set and I was not sure if this was ok.

[serhiy-storchaka]

Perform the optimization at the AST level, not in the peepholer.

[stevendaprano]

One issue with it is that in:
def f():
return " foo".dedent()
f will have both " foo" and "foo" in its constants even if the first is not used anymore.

That seems to be what happens with other folded constants:

>>> def f():
...     return 99.0 + 0.9
...
>>> f.__code__.co_consts
(None, 99.0, 0.9, 99.9)

so I guess that this is okay for a first draft. One difference is that
strings tend to be much larger than floats, so this will waste more
memory. We ought to consider removing unused constants at some point.

(But not me, sorry, I don't have enough C.)

Removing it requires looping over the code once more while marking
the constants seen in a set and I was not sure if this was ok.

That should probably be a new issue.

[stevendaprano]

Serhiy's message crossed with mine -- you should probably listen to
him over me :-)

[remilapeyre]

Perform the optimization at the AST level, not in the peepholer.

Thanks, this makes more sense.

[remilapeyre]

Serhiy's message crossed with mine.

And mine crossed with yours, sorry. I will update my PR shortly.

[remilapeyre]

Thanks @serhiy.storchaka, it's far easier to do here. I pushed the patch to the attached PR. Is there a reason the other optimisations in the Peephole optimizer are not done in the AST?

[serhiy-storchaka]

The optimization that can be done in the AST is done in the AST.

[serhiy-storchaka]

While the string method works pretty well, I do not think this is the best way. If 98% of multiline string will need deindenting, it is better to do it by default. For those 2% that do not need deintentation, it can be prohibited by adding the backslash followed by a newline at first position (except the start of the string). For example:

smile = '''\


 XX
 XX      X
          X
    XXX   X
          X
 XX      X
 XX

\\
'''

Yes, this is breaking change. But we have import from __future__ and FutureWarning. The plan may be:

3.9. Implement from __future__ import deindent.
3.11. Emit a FutureWarning for multiline literals that will be changed by dedending if from __future__ import deindent is not specified.
3.13. Make it the default behavior.

[stevendaprano]

While the string method works pretty well, I do not think this is the best way.

Regardless of what we do for literals, a dedent() method will help for
non-literals, so I think that this feature should go in even if we
intend to change the default behaviour in the future:

3.9. Implement "from __future__ import deindent".
3.11. Emit a FutureWarning for multiline literals that will be changed by dedending if "from __future__ import deindent" is not specified.
3.13. Make it the default behavior.

And that gives us plenty of time to decide whether or not making it the
default, rather than an explicit choice, is the right thing to do.

[gpshead]

Agreed, I'm in favor of going forward with this .dedent() optimization approach today.

If we were to attempt a default indented multi-line str and bytes literal behavior change in the future (a much harder decision to make as it is a breaking change), that is its own issue and probably PEP worthy.

[Carreau]

I've tried a bit PR 13455, I find this way nicer than textwrap.dedent(...),
though I wonder if f-string readability (and expected behavior?) might suffer a tiny bit with the order of formatting the f-string vs dedenting.

In the following it is clear that dedent is after formatting:

>> dedent(f" {stuff}")

It might be unclear for the following especially if .dedent() get sold as zero-overhead at compile time.

>> f" {stuff}".dedent()

Could it be made clearer with the peephole optimiser (and tested, I don't believe it is now), that dedent applies after-formatting ?

Alternative modifications/suggestions/notes:

• I can also see how having dedent applied before formatting with f-string could be useful or less surprising ( a d"" prefix could do that... just wondering what your actual goal is).
• Is this a supposed to deprecating textwrap.dedent ? Duck-typing and stuff, could textwrap.dedent work on non-str things and the current implementation not ( it assumes the .dedent() method exists) and thus be backward-incompatible ?

[stevendaprano]

It might be unclear for the following especially if .dedent() get
sold as zero-overhead at compile time.

Oh, please, please, please PLEASE let's not over-sell this! There is no
promise that dedent will be zero-overhead: it is a method, like any
other method, which is called at runtime. Some implementations *might*
*sometimes* be able to optimize that at compile-time, just as some
implementations *might* *sometimes* be able to optimize away long
complex arithmetic expressions and do them at compile time.

Such constant-folding optimizations can only occur with literals, since
arbitrary expressions aren't known at compile-time. F-strings aren't
string literals, they are executable code and can run thngs like this:

f"{'abc' if random.random() \> 0.5 else 'xyz'}"

So we don't know how many spaces each line begins with until after the
f-string is evaluated:

f"""{m:5d}
{n:5d}"""

Unless we over-sell the keyhole optimization part, there shouldn't be
anything more confusing about dedent than this:

x, X = 'spam', 'eggs'
f"{x}".upper()
\# returns 'SPAM' not 'eggs'

Could it be made clearer with the peephole optimiser (and tested, I
don't believe it is now), that dedent applies after-formatting ?

We should certainly make that clear that

Personally, I think we should soft-sell on the compile-time optimization
until such time that the Steering Council decides it should be a
mandatory language feature.

Alternative modifications/suggestions/notes:

• I can also see how having dedent applied **before** formatting
  with f-string could be useful or less surprising ( a d"" prefix
  could do that... just wondering what your actual goal is).

I don't see how it will make any difference in the common case. And the
idea here is to avoid yet another string prefix.

• Is this a supposed to deprecating textwrap.dedent ?

I don't think so, but eventually it might.

[Carreau]

Oh, please, please, please PLEASE let's not over-sell this!

Sorry didn't wanted to give you a heart attack. The optimisation has been mentioned, and you never know what people get excited on.

Such constant-folding ...

Well, in here we might get that, but I kind of want to see how this is taught or explain, what I want to avoid is tutorial or examples saying that .dedent() is "as if you hadn't put spaces in front".

I don't think so, but eventually it might.

Ok, thanks.

Again just being cautious, and I see this is targeted 3.9 so plenty of time.
I believe this will be a net improvement on many codebases.

[methane]

Can we dedent docstring too?

Is there any string like inspect.cleandoc(s) != inspect.cleandoc(s.dedent())?

[gpshead]

I think dedenting docstring content by default would be a great thing to do. But that's a separate issue, it isn't quite the same as .dedent() due to the first line. I filed https://bugs.python.org/issue37102 to track that.

[rhettinger]

We should consider dedicated syntax for compile-time dedenting:

d"""\
  This would be left aligned
      
      but this would only have four spaces

  And this would be left-justified.
"""   # Am not sure what to do about this last line

[gpshead]

Another option not using a new letter: A triple-backtick token.

def foo():
    value = 

 
    long multi line string i don't want indented.
    ```

A discuss thread was started so I reconnected it with this issue.  See
 https://discuss.python.org/t/trimmed-multiline-string/2600/8

[serhiy-storchaka]

I think it would be better to use use backtick quotes for f-strings instead of the f prefix. This would stress the special nature of f-strings (they are not literals, but expressions). But there was strong opposition to using backticks anywhere in Python syntax.

[MiguelAmaral]

A related issue(which I believe has no topic in this forum yet) is substituting an expression that results in a multiline string into a multiline f-string while matching its indentation.
If a new type of string prefix is made to auto-dedent, maybe the substitutions should match the local indentation.
Some related stackoverflow posts:

https://stackoverflow.com/questions/36739667/python-templates-for-generating-python-code-with-proper-multiline-indentation

https://stackoverflow.com/a/57189263/2976410

I.e. ideally we would have:

def make_g_code():
  nl='\n'
  return d"""\
    def g():
      {nl.join(something(i) for i in range(n))}
      return something_else
    """

This still has issues. Newline needs to be put into a variable, for instance. In the case of using this template for languages, great many use braces for delimiting blocks and those need to be escaped inside f-strings.

An implementation that works with spaces only (does not suit my case where mixed indentation is possible) is here:

http://code.activestate.com/recipes/578835-string-templates-with-adaptive-indenting/

Please let me know if this is the wrong place to comment on this issue.

[gpshead]

(assigning to me as I want to help remi.lapeyre's .dedent() method PR move forward)

[terryjreedy]

I am opposed to more prefix characters.

Are any string literal optimizations done now at compile time, like for 'abc'.upper? It does not exactly make sense to me to do just .dedent. But without the optimization, adding str.dedent without removing textwrap.dedent does not add a whole lot. (But it does add something, so I am mildly positive on the idea.)

[terryjreedy]

I read to the end of the patch and found astfold_dedent.

A small point in favor of textwrap.dedent is immediately announcing that the string will be dedented.

[stevendaprano]

A string prefix would be a large language change that would need to go through Python-Ideas, a PEP and Steering Council approval. Let's not go there :-)

A new string method is a comparatively small new feature that probably won't need a PEP or Steering Council approval.

A compile-time optimization is an implementation feature, not a language feature. Whether devs want to follow up with more string optimizations like 'spam'.upper() is entirely up to them.

Backwards compatibility implies that textwrap.dedent is not going away any time soon, but it should probably become a thin wrapper around str.dedent at some point.

[gpshead]

I expect several phases here:

(1) add a .dedent() method to str (and bytes?) - behaviors to consider mirroring are textwrap.dedent() and inspect.cleandoc(). Given their utility and similarities, it makes sense to offer both behaviors; behavior could be selected by a kwarg passed to the method.

https://docs.python.org/3/library/textwrap.html#textwrap.dedent

https://docs.python.org/3/library/inspect.html#inspect.cleandoc

(2a) Ponder the d" prefix - but in general I expect sentiment to be against yet another letter prefix. They aren't pretty. This would need a PEP. Someone would need to champion it.

(2b) Ponder making cleandoc dedenting automatic for docstrings. This would be enabled on a per-file basis via a from __future__ import docstring_dedent or similar as Serhiy suggested. No prefix letter required. Several releases later we could consider making it the default. This will also need needs a PEP.

(3) Optimizations when .dedent() is called on a constant? Nice to have, But I suggest we land (1) first as its own base implementation PR. Then consider the follow-ons in parallel.

I believe the current patch contains (1)+(3) right now. If so we should simplify it to (1) and make (3) am immediate followup as saving the runtime cost and data space is worthwhile.

Ultimate end state: probably 1+2b+3, but even 1+3 or 1+2b is a nice win.

[methane]

(1) add a .dedent() method to str (and bytes?) - behaviors to consider mirroring are textwrap.dedent() and inspect.cleandoc(). Given their utility and similarities, it makes sense to offer both behaviors; behavior could be selected by a kwarg passed to the method.

I don't think we need two algorithms here. I'm +1 to add str.dedent() which mirroring only inspect.cleandoc().

[serhiy-storchaka]

Multiline string literals were added recently in Java (https://openjdk.java.net/jeps/378). The semantic is similar to Julia: autodedent and ignoring the first blank line.

Multiline string literals have similar semantic in Swift (https://docs.swift.org/swift-book/LanguageGuide/StringsAndCharacters.html).

[methane]

I don't think we need two algorithms here. I'm +1 to add str.dedent() which mirroring only inspect.cleandoc().

I withdraw this. If we add str.dedent(), it must not be optimized for triple-quote literal.

Auto dedenting is very nice to have. It can be different from inspect.cleandoc().

We may able to cleandoc() automatically, even without from __future__. This can be different from str.dedent() or auto dedenting.

We already have a separate issue for docstring. And auto dedenting will needs PEP. How about focus on str.dedent() and change the issue title?

[jtojnar]

One benefit of using a compile time feature over a runtime method is that the former allows for more predictable dedenting by first dedenting and only then interpolating variables.

For example, the following code does not dedent the test string at all:

import textwrap

foo = "\n".join(["aaa", "bbb"])

test = textwrap.dedent(
    f"""
    block xxx:
        {foo}
    """
)

It would be much nicer if we had syntactical dedents based on the leftmost non-whitespace column in the string, as supported by Nix language, for example.

test = (
    df"""
    block xxx:
    {textwrap.indent(foo, ' '*4)}
    """
)

It would be even nicer if the interpolated strings could be indented based on the column the interpolation occurs in but that can at least be done at runtime with only minor inconvenience.

[stevendaprano]

This feature (or one very like it) has been requested again, this time with an "i" prefix:

https://discuss.python.org/t/indented-multi-line-string-literals/9846/1

[julianfortune]

What do folks think about taking an approach similar to Scala's stripMargin:

>>> my_str = """
...    |Foo
...    |     Bar
...    |  Baz
...    """.stripmargin()
Foo
    Bar
  Baz

Seems it could be achieved with only the addition of a new method to str, no?

[stevendaprano]

How is stripmargin different from dedent?

[julianfortune]

@stevendaprano my apologies, I did not realize dedent took into account shared whitespace; it seems dedent completely fills the need although it is a little more clunky that it is a separate function versus being implemented as a member of str.

[stevendaprano]

The proposal is to make dedent a string method, and then allow interpreters to optimise it into a compile-time operation (perhaps in the keyhole optimiser), *not* to use a function like this: ``` def func(x): textwrap.dedent("""blah blah blah multiple lines """ ) ``` That won't work because it won't be seen as a docstring. This issue hasn't been touched for 18 months. It looks like this issue is languishing, possibly over the question whether the method should implement textwrap.dedent or inspect.cleandoc.

[gpshead]

That question is valid but seems a mere implementation detail. A PR implementing the optimization would help motivate making that decision.

I'd still love it to be automatic for docstrings as it'd save memory for everyone in the world, but that can be considered a separate follow-up feature.

[stevendaprano]

cleandoc and dedent are functionally different, so it's not just an implementation detail, its a difference of semantics. Looking at the output of the two, I think cleandoc is the more correct behaviour for docstrings: ``` s = """First Second Third Indented fourth Fifth """ inspect.cleandoc(s) # --> 'First\nSecond\nThird\n Indented fourth\nFifth.' textwrap.dedent(s) # --> 'First\n Second\n Third\n Indented fourth\n Fifth.\n' ```

I'd still love it to be automatic for docstrings as it'd save memory for everyone in the world,

What are the consequences of just automatically dedenting docstrings? Technically its a breaking change, but is there code or people who rely on the current leading whitespace? help() already reformats the docstring, as does inspect, so maybe we should seriously consider just changing the rules for docstrings to automatically dedent them at function build time. It would probably have to go through a future import first.

[methane]

Looking at the output of the two, I think cleandoc is the more correct behaviour for docstrings:

This issue is premierly about general multine string dedenting.
So please forget about docstring now and consider what is the best dedent behevior for general multiline strings like SQL.

s = """First
    Second
    Third
        Indented fourth
    Fifth
    """

inspect.cleandoc(s) # --> 'First\nSecond\nThird\n    Indented fourth\nFifth.'
textwrap.dedent(s)  # --> 'First\n    Second\n    Third\n        Indented fourth\n    Fifth.\n'

I don't think special casing first line is not needed for general dedenting.

[stevendaprano]

On Tue, Jan 10, 2023 at 10:02:47PM -0800, Inada Naoki wrote: > Looking at the output of the two, I think cleandoc is the more correct behaviour for docstrings: This issue is premierly about general multine string dedenting.

But we already have a solution for that: the textwrap module. The textwrap module is inconvenient to use with docstrings, which is also when we want to put extra indents in the text block to make it look good in the .py file, but we need to remove that leading space before displaying the text. I agree that we should *also* think about other uses of dedent, but we should not forget docstrings. They are an important use-case, and as far as I am concerned, they are the driving motivation for moving dedent into the str type as a method.

[methane]

I agree that we should also think about other uses of dedent, but we
should not forget docstrings. They are an important use-case, and as far
as I am concerned, they are the driving motivation for moving dedent
into the str type as a method.

As I wrote in this comment, new syntax can be different from inspect.cleandoc().

I prefer having the best multiline syntax by learning from Julia, Java, Swift, etc...

Please read this comment too. Compile time docstring cleandoc is tracked in #81283.

[terryjreedy]

@Carreau "Is this a supposed to deprecating textwrap.dedent?"
@stevendaprano "I don't think so, but eventually it might."
I definitely feel that this should NOT be part of the initial PR without being discussed here.

One of the removed uses in the PR is a 16 line, about 500 char, """-quoted string literal in idlelib. At the momemt I don't really like having 'dedent' moved from the starting line to the last. (I could use inspect.cleandoc instead. Or literal concatenation as elsewhere in the file. )