-
Notifications
You must be signed in to change notification settings - Fork 133
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Escape Nunjucks for special tags #1049
Conversation
I changed the approach from when I first started. This partially address #762. The main value of this PR is that together with #1047 that escapes tags, we can completely escape content that are meant to be parsed by external libraries (diagrams, latex) which commonly uses Previously, the attempt was more ambitious. I attempted to tokenize and capture nunjuck variable tags, and conditionally escape those if they are not a variable. This works generally well for user-defined variables within Markbind. The issue arrives when the user defines variables that are not within Markbind, for example in Markbind user guide. These variables are very hard to detect, and are not explicitly handled within Nunjucks itself see mozilla/nunjucks#557. Currently, we allow the user to define and escape it's own For instance: const MarkdownIt = require('markdown-it');
const md = new MarkdownIt().use(require('markdown-it-attrs'));
console.log(md.render('{%raw%}content{%endraw%}')); // <p %endraw%="">{%raw%}content</p>
// For new lines
console.log(md.render(`<div>{%raw%}
content
{%endraw%}</div>`));
// <p %raw%=""><div></p>
// <p>content</p>
// <p>{%endraw%}</div></p> |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Clean solution 👍
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
It looks like there is still some work pending for this PR. Shall we label is [WIP] in the meantime?
This approach looks promising, hopefully we can iron out the remaining issues and fix the bug :)
test/unit/nunjuckUtils.test.js
Outdated
|
||
expect(escapedContent).toBe(escapedString); | ||
}); | ||
|
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Do you think this nested escapes are a case we need to account for?
test('Escaping nested nunjucks raw tags', () => {
const escapedString = 'This is a content with nested escaped data {%raw%} {%raw%} CONTENT {%endraw%} {%endraw%}';
const escapedContent = nunjuckUtils.renderEscaped(nunjucks, escapedString);
expect(escapedContent).toBe(escapedString);
});
I can imagine this might be useful for documenting how to use to use the escape tags in the first place
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Nesting with stuff in between doesn't seem to work correctly 😅
test('Escaping nested nunjucks raw tags', () => {
const escapedString = 'Nested escaped data: {% raw %}BEFORE{% raw %} CONTENT {% endraw %}AFTER{% endraw %}';
const escapedContent = nunjuckUtils.renderEscaped(nunjucks, escapedString);
expect(escapedContent).toBe(escapedString);
});
Expected: "Nested escaped data: {% raw %}BEFORE{% raw %} CONTENT {% endraw %}AFTER{% endraw %}"
Received: "Nested escaped data: {% raw %}BEFORE{% raw %} CONTENT {% endraw %}{% endraw %}AFTER"
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
It feels like as long as we go with the opentag/closetag approach, we would always run into this issue:
$OPEN$ $CLOSE$ $CLOSE$
<- how do we match the $OPEN
to the right $CLOSE
?
Do you think having a special tag that escapes the next character would help? This would "break" the syntax so the original special tag wont be treated as such.
The scope of this PR will not address rendering of displaying I would rebase this PR on top of #1047, will be adding an integration test to ensure that nunjuck syntax are properly escaped for plugins. |
#1047 deals with markbind-it-attr already, since the entire special tag is processed as a html block. For vue, perhaps after this PR we could extend #1047's api with this PR's functionality. The regex would then also comb for these additional patterns and append For general use, we could create a special component, something like The same could be done for plugin defined components sparingly. |
Yep. Thanks to good work in #1047, special tags are isolated from markdown-it and markdown-it-attr. I was considering about how to handle other cases when the nunjuck escapes are processed by
Right now we probably don't, but potentially plugins like latex will start having
Something we could certainly explore |
be7d653
to
638379e
Compare
Ready for review. Integration test added hints at how this can be used with special tags, it naively replaced all mustaches with |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
On how this can be used with special tags, would it be better to keep it to a separate PR since it hasn't been implemented?
For now, the test focuses on ensuring that a special tag plugin could retrieve an untouchedescaped unparsed mustache syntax. Before we have the special component, it would be good to ensure that there will be no regression by having the integration test.
Would be great to have something like that. When we do, we could add other tests that checks this or even replace the current one. |
638379e
to
f4ca1c2
Compare
My bad, realised I edited your comment instead of making a new one 😅
Ahh I see the point now, thanks!
Reason I was confused is that if we are going in this direction, the test would likely be replaced as you mentioned. I'm fine with including it here as well in the meanwhile though. A brief glance:
|
No worries 😄. Thank you for your review! I reviewed some of the existing I have added additional tests to ensure that the behavior of rendering nunjucks to remove brackets is retained even when raw tags is used. This helps to prevent backward compatibility issues where we actually need the mustaches to be removed so that Vue can work properly until we have the component mentioned above. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thank you for your review! I reviewed some of the existing
nunjucks.RenderString
that exists, and I agree that it will get confusing when there are 2 ways of using nunjucks. Actually, there is no difference betweennunjucks.RenderString
andnunjuckUtils.renderEscaped
afternunjucksUtils.removeNunjucksEscapes
is called.
It wasn't confusing per se, more a matter of functionality. ( eg. the renderString
in insertHeaderFile
wasn't wrapped before, so {% raw %}
would be erased before the final nunjucks call for the page's content )
On that note, and {{
still getting erased, shouldn't the final nunjucks call for the page / external dependency's output be the raw call to nunjucks.renderString
still?
Perhaps nunjucks.removeNunjucksEscapes
is not necessary in this sense, as nunjucks already removes {% raw %}... {% endraw %}
a7961c8
to
0bb9ee9
Compare
True 👍 , we should remove any additional
We would still require this. As explained above, we still need to ensure |
0bb9ee9
to
7dd6028
Compare
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
We would still require this.
I see that {% raw %}
conflicts with markdown-it-attrs
now, and the reason for the scope of this pr.
In this sense, is removeNunjucksEscape
's purpose is only to prevent conflict with the md.render
call in page.renderFile
?
If its the case, and since the scope of this PR targets special tags, should the functional test be removed, and removeNunjucksEscapes
along with it? ( since it wouldn't conflict if it is wrapped by a special tag anyway )
Once we have a patch for markdown-it-attrs
and markdown-it
, we could then reintroduce a similar test that tests the functionality of {% raw %}...{% endraw %}
when not wrapped by a special tag, without the need for removeNunjucksEscapes
.
test/functional/test_site_special_tags/_markbind/plugins/testSpecialTag.js
Show resolved
Hide resolved
test/functional/test_site_special_tags/_markbind/plugins/testSpecialTag.js
Show resolved
Hide resolved
584e907
to
96e0d05
Compare
👍 . I removed
Yea we could do that. Removing |
96e0d05
to
06070b2
Compare
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Looks good 👍, thanks for the explanations thus far!
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Nesting still seems to be an issue I think :( let me know if I'm misunderstanding this!
Also, just to clarify, does this work only for plugins? Would it work if we want to actually escape the variable syntax (#1086)?
Just tested it out, I think I finally get what you mean by the syntax clashing with vue.
Do you happen to know if we currently have any other way to escape the {{ VARIABLE }}
syntax?
test/unit/nunjuckUtils.test.js
Outdated
|
||
expect(escapedContent).toBe(escapedString); | ||
}); | ||
|
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Nesting with stuff in between doesn't seem to work correctly 😅
test('Escaping nested nunjucks raw tags', () => {
const escapedString = 'Nested escaped data: {% raw %}BEFORE{% raw %} CONTENT {% endraw %}AFTER{% endraw %}';
const escapedContent = nunjuckUtils.renderEscaped(nunjucks, escapedString);
expect(escapedContent).toBe(escapedString);
});
Expected: "Nested escaped data: {% raw %}BEFORE{% raw %} CONTENT {% endraw %}AFTER{% endraw %}"
Received: "Nested escaped data: {% raw %}BEFORE{% raw %} CONTENT {% endraw %}{% endraw %}AFTER"
test/unit/nunjuckUtils.test.js
Outdated
|
||
expect(escapedContent).toBe(escapedString); | ||
}); | ||
|
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
It feels like as long as we go with the opentag/closetag approach, we would always run into this issue:
$OPEN$ $CLOSE$ $CLOSE$
<- how do we match the $OPEN
to the right $CLOSE
?
Do you think having a special tag that escapes the next character would help? This would "break" the syntax so the original special tag wont be treated as such.
@marvinchin Thank you so much for your review. I updated it and fixed some of the styles to make it consistent. I removed the test that was supporting nested tags because there is no need for the user to input nested escapes in any case. It is much better to remove the test than to suggest to future developers that we are officially supporting and maintaining an unnecessary interaction.
There are two main approaches for it:
Hope that helps to explain! |
@marvinchin ready for review! Sorry for the double ping 🙇 |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM :)
* 'master' of https://github.com/MarkBind/markbind: Update test files (MarkBind#1138) Remove OK button from modals (MarkBind#1134) Add start from line number functionality to code blocks (MarkBind#1115) Allow special tags to be self-closing (MarkBind#1101) Simplify baseUrl resolving process (MarkBind#1087) Remove redundant file writing (MarkBind#1090) Bump acorn from 5.7.3 to 5.7.4 (MarkBind#1121) Bump acorn from 7.1.0 to 7.1.1 in /src/lib/markbind (MarkBind#1120) Unify markdown-it parser variants (MarkBind#1056) Remove dynamic include feature (MarkBind#1037) Fix flex shrink not applying in content wrapper (MarkBind#1135) Escape Nunjucks for special tags (MarkBind#1049) Update documentation on icon slot for boxes (MarkBind#1123)
…x-pageNav * 'master' of https://github.com/MarkBind/markbind: Update test files (MarkBind#1138) Remove OK button from modals (MarkBind#1134) Add start from line number functionality to code blocks (MarkBind#1115) Allow special tags to be self-closing (MarkBind#1101) Simplify baseUrl resolving process (MarkBind#1087) Remove redundant file writing (MarkBind#1090) Bump acorn from 5.7.3 to 5.7.4 (MarkBind#1121) Bump acorn from 7.1.0 to 7.1.1 in /src/lib/markbind (MarkBind#1120) Unify markdown-it parser variants (MarkBind#1056) Remove dynamic include feature (MarkBind#1037) Fix flex shrink not applying in content wrapper (MarkBind#1135) Escape Nunjucks for special tags (MarkBind#1049) Update documentation on icon slot for boxes (MarkBind#1123) Convert code in boxes to code block (MarkBind#1086)
* Allow nunjucks that is called multiple times to support escaping for special tags * Remove the need to prune raw tags, simplify logic of replacement * Fix style and remove nested nunjucks escape test
@crphang what's the outcome of this PR, from author POV?
|
@damithc This works when used in specialTags, such as puml. This allows curly braces To actually show the curly braces:
This could be a separate plugin (which can be registered as special tag) that also notifies Vue that it is raw with |
From an author POV, nothing should change right now, but it allows developers to define more components that could consist of conflicting syntax, in this case curly braces. I would raise a seperate issue to track this, for creating a new plugin that allows author to render |
Got it. 👍 |
Initial attempt:
It escapes the tokens
}
,}}
,{{
in a single pass by adding {%raw%} {%endraw%}.In the event of
{{
, it attempts to check whether the content is a variable or not, if it is, it ignores it and goes to the next token. If not, it would escape content to the next token (Always safe to escape).P.S.
This PR allows
{%raw%}
block tags that are used for escaping nunjuck syntax to be used for special tags. This allows special tag plugins that rely on these syntax to escape these syntax by providing a way to escape repeatedly called nunjucks render through markbind's repository.Suggested commit message:
Provide a way to escape mustache syntax for repeatedly called nunjucks render for special tags.