WIP: Parser: Add tokens for inner blocks in inner HTML

[dmsnell]

DO NOT MERGE

Motivated by #8760
Necessary for #10463, #10108

Abstract

Add tokens in innerHTML of parsed post_content indicating where each of a block's innerBlocks came from. e.g. innerHTML: "Before <!-- {0} --> Middle <!-- {1} --> After"

Status

• the code editor for posts is showing the tokens currently which means that we have to update the full parsing stack to make this PR ready to merge

Summary

Inner blocks, or nested blocks, or blocks-within-blocks, can exist in Gutenberg posts. They are serialized in post_content in-place as normal blocks which exist in between another block's comment delimiters.

<!-- wp:outerBlock -->
Check out my
<!-- wp:voidInnerBlock /-->
and my other
<!-- wp:innerBlock -->
with its own content.
<!-- /wp:innerBlock -->
<!-- /wp:outerBlock -->

The way this gets parsed leaves us in a quandary: we cannot reconstruct the original post_content after parsing because we lose the origin location information for each inner block since they are only passed as an array of inner blocks.

{
	"blockName": "core/outerBlock",
	"attrs": {},
	"innerBlocks": [
		{
			"blockName": "core/voidInnerBlock",
			"attrs": {},
			"innerBlocks": [],
			"innerHTML": ""
		},
		{
			"blockName": "core/innerBlock",
			"attrs": {},
			"innerBlocks": [],
			"innerHTML": "\nwith its own content.\n"
		}
	],
	"innerHTML": "\nCheck out my\n\nand my other\n\n"
}

At this point we have parsed the blocks and prepared them for attaching into the JavaScript block code that interprets them but we have lost our reverse transformation.

In this PR I'd like to introduce a new mechanism which should only marginally break existing functionality but which will enable us to go back and forth isomorphically between the post_content and first stage of parsing. If we can tear apart a Gutenberg post and reassemble then it will let us to structurally-informed processing of the posts without needing to be aware of all the block JavaScript.

The proposed mechanism is a form of tombstone or token inserted into innerHTML where the innerBlocks were found.

{
	"blockName": "core/outerBlock",
	"attrs": {},
	"innerBlocks": [
		{
			"blockName": "core/voidInnerBlock",
			"attrs": {},
			"innerBlocks": [],
			"blockMarkers": [],
			"innerHTML": ""
		},
		{
			"blockName": "core/innerBlock",
			"attrs": {},
			"innerBlocks": [],
			"blockMarkers": [],
			"innerHTML": "\nwith its own content.\n"
		}
	],
	"innerHTML": "\nCheck out my\n<!-- {0} -->\nand my other\n<!-- {1} -->\n"
}

The token is a normal HTML comment of trivial structure which (if accidentally not removed before rendering) should not display in the rendered page, which should be trivial to substitute with a RegEx-based replacement, and which should remain straightforward for plugin authors and translators to deal with.

These tokens won't exist in post_content and they will be stripped before rendering the page. They exist only to restore isomorphism to the parser/printer pair.

[georgestephanis]

Incidentally, if accidental collision with content is a concern, we could use a little used emoji internally to designate our own stuff -- even potentially a utf-8 code point that we can identify isn't in the content being generated, that we then swap back after it's rendered.

https://twitter.com/leastUsedEmoji/status/1057612046261514240

🔠

[dmsnell]

Closing in favor of #11334