Allow using {% (end)raw %} tags

Nunjuck's {% raw %} tags don’t work as expected due to a soft syntax
conflict with markdown-it-attrs.
In addition, variable interpolation syntax cannot be output in
code elements due to vue's variable interpolation.

Let's patch markdown-it-attrs to allow the slight syntax conflicts,
and automatically assign v-pre to code elements to skip vue compilation.

docs/_markbind/variables.md

@@ -1,9 +1,3 @@
-<variable name="showBaseUrlCode">
-<code>{<span></span>{ baseUrl }}</code>
-</variable>
-
-<variable name="showBaseUrlText">{<span></span>{ baseUrl }}</variable>
-
 <variable name="markbind_blue">#00B0F0</variable>
 
 <variable name="icon_arrow_down">:fas-arrow-down:</variable>

docs/userGuide/markBindSyntaxOverview.md

@@ -99,13 +99,15 @@ As MarkBind uses [VueStrap](https://bootstrap-vue.js.org/docs/components/alert/)
 
 [Nunjucks](https://mozilla.github.io/nunjucks/) is a JavaScript based templating tool. Here is a simple example:
 
-<box><span>
-`<ul>`<br>
-<code>{<span></span>% for item in [1, 2, 3, 4] %<span></span>}</code><br>
-&nbsp;&nbsp;`<li>`<code>Item {<span></span>{ item }}</code>`</li>`<br>
-<code>{<span></span>% endfor %<span></span>}</code><br>
-`</ul>`
-</span></box>
+{% raw %}
+```html { .no-line-numbers }
+<ul>
+{% for item in [1, 2, 3, 4] %}
+  <li>Item {{ item }}</li>
+{% endfor %}
+</ul>
+```
+{% endraw %}
 
 {{ icon_arrow_down }}
 

docs/userGuide/syntax/icons.mbdf

@@ -13,8 +13,9 @@ The advantage of font icons over emojis is font icons can be _styled_ to fit you
 </include>
 
 <box type="important">
-The syntax for icons has changed, and the earlier {<span></span>{ prefix_name }} syntax has been deprecated. <br>
-Please use the new :prefix-name: syntax instead.
+
+The syntax for icons has changed, and the earlier {%raw%}`{{ prefix_name }}`{%endraw%} syntax has been deprecated. <br>
+Please use the new `:prefix-name:` syntax instead.
 </box>
 
 ###### Using Font Awesome Icons

docs/userGuide/syntax/includes.mbdf

@@ -130,10 +130,12 @@ In other words, **`<include>` interprets the reused code relative to the origina
 
 In `article.md`:
 
-<code>
-# {<span></span>{ title }}<br>
-Author: {<span></span>{ author }}
-</code>
+{% raw %}
+```html
+# {{ title }}<br>
+Author: {{ author }}
+```
+{% endraw %}
 </div>
 
 These variables work the same way as variables in `_markbind/variables.md`, except that they only apply to the included file. They allow the included file to be reused as a template, for different source files using different variable values.

docs/userGuide/syntax/links.mbdf

@@ -32,16 +32,19 @@ Links to files of the generated site (e.g., an HTML page or an image file) can b
 
 Absolute paths:
 <div class="indented">
-Links should start with {{ showBaseUrlCode }} (which represents the root directory of the project). 
 
-{{ icon_example }} Here's how to specify a link to (1) a page, and (2) an image, using the {{ showBaseUrlCode }}:
+Links should start with {% raw %}`{{ baseUrl }}`{% endraw %} (which represents the root directory of the project). 
 
-1. <code>Click [here]({{ showBaseUrlCode }}/userGuide/reusingContents.html).</code>
-2. `![](`{{ showBaseUrlCode }}`/images/preview.png)`
+{{ icon_example }} Here's how to specify a link to (1) a page, and (2) an image, using the {% raw %}`{{ baseUrl }}`:
+
+1. `Click [here]({{ baseUrl }}/userGuide/reusingContents.html).`
+2. `![]({{ baseUrl }}/images/preview.png)`
 
 <box type="important">
-To ensure that links in the <code>_markbind/</code> folder work correctly across the entire site, they should be written as absolute paths, prepended with <code><span>{</span>{ baseUrl }}</code>. 
+
+To ensure that links in the <code>_markbind/</code> folder work correctly across the entire site, they should be written as absolute paths, prepended with `{{ baseUrl }}`. 
 </box>
+{% endraw %}
 </div>
 
 Relative paths:

docs/userGuide/syntax/pageLayouts.mbdf

@@ -1,10 +1,3 @@
-<!---
-TODO: Change to a proper escape tag once PR#1049 is ready. 
--->
-<variable name="mainContentBody">
-<code>{<span></span>{ MAIN_CONTENT_BODY }}</code>
-</variable>
-
 ## Page Layouts
 
 **A _layout_ is a set of page-tweaks that can be applied to a page (or group of pages) in one go.**
@@ -110,7 +103,8 @@ afterSetup(() => {
 In the `page.md` file of your layouts, it should come with the following reserved variable:
 
 <box>
-  {{ mainContentBody }}
+
+  {%raw%}`{{ MAIN_CONTENT_BODY }}`{%endraw%}
 </box>
 
 which injects the actual page content in every page. This allows you to build layouts in different ways.
@@ -124,7 +118,7 @@ which injects the actual page content in every page. This allows you to build la
   ```
 
   ```html {heading="page.md"}
-    {{ mainContentBody }}
+    {%raw%}{{ MAIN_CONTENT_BODY }}{%endraw%}
     <panel header="Statistics Formula for the class" type="primary">
       <img src="path_to_your_formula.png" />
     </panel>

docs/userGuide/tipsAndTricks.md

@@ -9,7 +9,7 @@
 
 <span id="escapingCharacters">
 
-##### Tip: Escaping Characters
+##### :fas-lightbulb: Escaping Characters
 
 For Markdown syntax: To display a literal character that are normally used for Markdown formatting, add a backslash (`\`) in front of the character.
 
@@ -26,7 +26,26 @@ For Markdown syntax: To display a literal character that are normally used for M
 
 </span>
 
-##### Problem: Unwanted starting space in links and triggers
+{% raw %}
+
+##### :fas-lightbulb: Using {% raw %}{% endraw %} to display `{{ content }}`
+
+
+To display the raw variable interpolation syntax using `{% raw %}{% endraw %}`, you would also need to add
+the `v-pre` attribute using markdown-it-attrs or as a html attribute.
+
+<box type="info">
+
+This isn't necessary for `<code>` elements, markdown code fences and inline code though, which markbind automatically
+adds `v-pre` for.
+
+However, this does not change the need for `{% raw %}{% endraw %}`. Meaning, you can still use variables within your code!
+</box>
+
+
+{% endraw %}
+
+##### :fas-info: Unwanted starting space in links and triggers
 
 When you use links or triggers, you may encounter a situation where an unwanted space is being generated by MarkBind:
 

docs/userGuide/usingPlugins.md

@@ -200,7 +200,8 @@ removing such potential conflicts.
   - Should return an array of string tag names to be blacklisted, with each tag name being at least 2 characters long.
 
 <box type="important">
-Note however, that variable interpolation syntax <code>{<span>{</span> variable_name <span>}</span>}</code> will act as per normal.
+
+Note however, that variable interpolation syntax {% raw %}`{{ variable_name }}`{% endraw %} will act as per normal.
 Meaning, the user would still be able to use variables in your special tags!
 </box>
 

src/lib/markbind/src/lib/markdown-it/index.js

@@ -14,7 +14,7 @@ markdownIt.use(require('markdown-it-mark'))
   .use(require('markdown-it-table-of-contents'))
   .use(require('markdown-it-task-lists'), {enabled: true})
   .use(require('markdown-it-linkify-images'), {imgClass: 'img-fluid'})
-  .use(require('markdown-it-attrs'))
+  .use(require('./patches/markdown-it-attrs-nunjucks'))
   .use(require('./markdown-it-dimmed'))
   .use(require('./markdown-it-radio-button'))
   .use(require('./markdown-it-block-embed'))

src/lib/markbind/src/lib/markdown-it/patches/markdown-it-attrs-nunjucks.js

@@ -0,0 +1,78 @@
+/*
+ Patch for markdown-it-attrs and nunjuck's usage of {% ... %}
+ */
+
+const mdAttrsUtils = require('markdown-it-attrs/utils');
+
+mdAttrsUtils.hasDelimiters = function (where, options) {
+
+  if (!where) {
+    throw new Error('Parameter `where` not passed. Should be "start", "middle", "end" or "only".');
+  }
+
+  /**
+   * @param {string} str
+   * @return {boolean}
+   */
+  return function (str) {
+    // we need minimum three chars, for example {b}
+    let minCurlyLength = options.leftDelimiter.length + 1 + options.rightDelimiter.length;
+    if (!str || typeof str !== 'string' || str.length < minCurlyLength) {
+      return false;
+    }
+
+    function validCurlyLength (curly) {
+      let isClass = curly.charAt(options.leftDelimiter.length) === '.';
+      let isId = curly.charAt(options.leftDelimiter.length) === '#';
+      return (isClass || isId)
+        ? curly.length >= (minCurlyLength + 1)
+        : curly.length >= minCurlyLength;
+    }
+
+    let start, end, slice, nextChar;
+    let rightDelimiterMinimumShift = minCurlyLength - options.rightDelimiter.length;
+    switch (where) {
+    case 'start':
+      // first char should be {, } found in char 2 or more
+      slice = str.slice(0, options.leftDelimiter.length);
+      start = slice === options.leftDelimiter ? 0 : -1;
+      end = start === -1 ? -1 : str.indexOf(options.rightDelimiter, rightDelimiterMinimumShift);
+      // check if next character is not one of the delimiters
+      nextChar = str.charAt(end + options.rightDelimiter.length);
+      if (nextChar && options.rightDelimiter.indexOf(nextChar) !== -1) {
+        end = -1;
+      }
+      break;
+
+    case 'end':
+      // last char should be }
+      start = str.lastIndexOf(options.leftDelimiter);
+      end = start === -1 ? -1 : str.indexOf(options.rightDelimiter, start + rightDelimiterMinimumShift);
+      end = end === str.length - options.rightDelimiter.length ? end : -1;
+      break;
+
+    case 'only':
+      // '{.a}'
+      slice = str.slice(0, options.leftDelimiter.length);
+      start = slice === options.leftDelimiter ? 0 : -1;
+      slice = str.slice(str.length - options.rightDelimiter.length);
+      end = slice === options.rightDelimiter ? str.length - options.rightDelimiter.length : -1;
+      break;
+    }
+
+    /*
+     Simple patch here - abort if the delimiters wrap around % %
+     */
+    const isCharAfterStartPercent = str.charAt(start + 1) === '%';
+    const isCharBeforeEndPercent = str.charAt(end - 1) === '%';
+    if (isCharAfterStartPercent && isCharBeforeEndPercent) {
+      return false;
+    }
+
+    return start !== -1
+      && end !== -1
+      && validCurlyLength(str.substring(start, end + options.rightDelimiter.length));
+  };
+};
+
+module.exports = require('markdown-it-attrs');

src/lib/markbind/src/parsers/componentParser.js

@@ -432,6 +432,9 @@ function _parseThumbnailAttributes(node) {
 function parseComponents(node) {
   try {
     switch (node.name) {
+    case 'code':
+      node.attribs['v-pre'] = '';
+      break;
     case 'panel':
       _parsePanelAttributes(node);
       break;

src/lib/markbind/src/utils/nunjuckUtils.js

@@ -6,10 +6,57 @@ const unescapedEnv = nunjucks.configure({ autoescape: false }).addFilter('date',
 
 const START_ESCAPE_STR = '{% raw %}';
 const END_ESCAPE_STR = '{% endraw %}';
-const REGEX = new RegExp('{% *raw *%}(.*?){% *endraw *%}', 'gs');
+const RAW_TAG_REGEX = new RegExp('{% *(end)?raw *%}', 'g');
 
+/**
+ * Pads the outermost {% raw %} {% endraw %} pairs with {% raw %} {% endraw %} again.
+ * This allows variables and other nunjuck syntax inside {% raw %} {% endraw %} tags
+ * to be ignored by nunjucks until the final renderString call.
+ */
 function preEscapeRawTags(pageData) {
-  return pageData.replace(REGEX, `${START_ESCAPE_STR}$&${END_ESCAPE_STR}`);
+  // TODO simplify using re.matchAll once node v10 reaches 'eol'
+  // https://github.com/nodejs/Release#nodejs-release-working-group
+  const tagMatches = [];
+  let tagMatch = RAW_TAG_REGEX.exec(pageData);
+  while (tagMatch !== null) {
+    tagMatches.push(tagMatch);
+    tagMatch = RAW_TAG_REGEX.exec(pageData);
+  }
+
+  const tagInfos = Array.from(tagMatches, match => ({
+    isStartTag: !match[0].includes('endraw'),
+    index: match.index,
+    content: match[0],
+  }));
+
+  let numStartRawTags = 0; // nesting level of {% raw %}
+  let lastTokenEnd = 0;
+  const tokens = [];
+
+  for (let i = 0; i < tagInfos.length; i += 1) {
+    const { index, isStartTag, content } = tagInfos[i];
+    const currentTokenEnd = index + content.length;
+    tokens.push(pageData.slice(lastTokenEnd, currentTokenEnd));
+    lastTokenEnd = currentTokenEnd;
+
+    if (isStartTag) {
+      if (numStartRawTags === 0) {
+        // only pad outermost {% raw %} with an extra {% raw %}
+        tokens.push(START_ESCAPE_STR);
+      }
+      numStartRawTags += 1;
+    } else {
+      if (numStartRawTags === 1) {
+        // only pad outermost {% endraw %} with an extra {% endraw %}
+        tokens.push(END_ESCAPE_STR);
+      }
+      numStartRawTags -= 1;
+    }
+  }
+  // add the last token
+  tokens.push(pageData.slice(lastTokenEnd));
+
+  return tokens.join('');
 }
 
 module.exports = {

test/functional/test_site/expected/index.html

@@ -317,8 +317,8 @@ <h4 id="focus-groups">Focus groups<a class="fa fa-anchor" href="#focus-groups" o
           <div name="Referencing specified path in boilerplate">
             <h1 id="path-within-the-boilerplate-folder-is-separately-specified">Path within the boilerplate folder is separately specified<a class="fa fa-anchor" href="#path-within-the-boilerplate-folder-is-separately-specified" onclick="event.stopPropagation()"></a></h1>
             <p>Like static include, pages within the site should be able to use files located in folders within boilerplate.</p>
-            <p>Also, the boilerplate file name (e.g. <code>inside.md</code>) and the file that it is supposed to act as (<code>notInside.md</code>) can be different.</p>
-            <p>This file should behaves as if it is in the <code>requirements</code> folder:</p>
+            <p>Also, the boilerplate file name (e.g. <code v-pre="">inside.md</code>) and the file that it is supposed to act as (<code v-pre="">notInside.md</code>) can be different.</p>
+            <p>This file should behaves as if it is in the <code v-pre="">requirements</code> folder:</p>
             <panel src="/test_site/requirements/NonFunctionalRequirements._include_.html"><template slot="_header">
                 <p>Tested with the folllowing include</p>
               </template>
@@ -347,7 +347,7 @@ <h1 id="path-within-the-boilerplate-folder-is-separately-specified">Path within
             <p><em>MarkBind supports .mbd files.</em></p>
           </div>
           <div>
-            <p><code>MarkBind supports .mbdf files.</code></p>
+            <p><code v-pre="">MarkBind supports .mbdf files.</code></p>
           </div>
           <p><strong>Include from another Markbind site</strong></p>
           <div>
@@ -630,6 +630,12 @@ <h1 id="markbind-plugin-pre-render">Markbind Plugin Pre-render<a class="fa fa-an
         </div>
         <h2 class="no-index" id="level-2-header-inside-headingsearchindex-with-no-index-attribute-should-not-be-indexed">Level 2 header (inside headingSearchIndex) with no-index attribute should not be indexed<a class="fa fa-anchor" href="#level-2-header-inside-headingsearchindex-with-no-index-attribute-should-not-be-indexed" onclick="event.stopPropagation()"></a></h2>
         <h6 class="always-index" id="level-6-header-outside-headingsearchindex-with-always-index-attribute-should-be-indexed">Level 6 header (outside headingSearchIndex) with always-index attribute should be indexed<a class="fa fa-anchor" href="#level-6-header-outside-headingsearchindex-with-always-index-attribute-should-be-indexed" onclick="event.stopPropagation()"></a></h6>
+        <p><strong>Test nunjucks raw tags</strong></p>
+        <p></p>
+        <div v-pre="">{{ variable interpolation syntax can be used with v-pre }}</div>
+        <div v-pre="">{{ nonExistentVariable }}</div>
+        <code v-pre="">{{ code elements should automatically be assigned v-pre }}</code>
+        <p></p>
         <hr class="footnotes-sep">
         <section class="footnotes">
           <ol class="footnotes-list">

test/functional/test_site/expected/requirements/UserStories._include_.html

@@ -1,10 +1,10 @@
 <p>User stories are brief (typically, 1-3 sentences) descriptions of what the system can do for the users,
   written in the customers’ language. Often, user stories are written by the customers themselves.</p>
 <p>A commonly used format for writing user stories is:<br>
-  <strong><code>As a</code></strong> <code>&lt;use type/role&gt;</code> <strong><code>I can</code></strong> <code>&lt;function&gt;</code> <strong><code>so that</code></strong> <code>&lt;benefit&gt;</code></p>
+  <strong><code v-pre="">As a</code></strong> <code v-pre="">&lt;use type/role&gt;</code> <strong><code v-pre="">I can</code></strong> <code v-pre="">&lt;function&gt;</code> <strong><code v-pre="">so that</code></strong> <code v-pre="">&lt;benefit&gt;</code></p>
 <p>Here are some examples of user stories for the IVLE system:</p>
-<pre><code class="hljs bat"><span>* As a student, I can download files uploaded by lecturers, so that I can get my own <span class="hljs-built_in">copy</span> of the files.<br></span><span>* As a lecturer, I can create discussion forums, so that students can discuss things online.<br></span><span>* As a tutor, I can <span class="hljs-built_in">print</span> attendance sheets, so that I can take attendance during the class.<br></span></code></pre>
-<p>The <code>&lt;benefit&gt;</code> can be omitted if it is obvious. E.g. As a tutor, I can print attendance sheets.
+<pre><code class="hljs bat" v-pre=""><span>* As a student, I can download files uploaded by lecturers, so that I can get my own <span class="hljs-built_in">copy</span> of the files.<br></span><span>* As a lecturer, I can create discussion forums, so that students can discuss things online.<br></span><span>* As a tutor, I can <span class="hljs-built_in">print</span> attendance sheets, so that I can take attendance during the class.<br></span></code></pre>
+<p>The <code v-pre="">&lt;benefit&gt;</code> can be omitted if it is obvious. E.g. As a tutor, I can print attendance sheets.
   User stories are mainly used for early estimation and scheduling purposes.</p>
 <p>According to <ref xp-website="">this<ref></ref>, the biggest difference between user stories and traditional requirements
     specifications is in the level of detail. User stories should only provide enough detail to make a reasonably low risk