[Question] Use async function for highlighter or how to use shiki for highlighting?

[michaeloliverx]

Hi,

I would like to highlight my codeblocks using shiki. It doesn't seem to support a synchronous API. I have tried using the following:

async function highlighter(code, lang) {
  const shiki = require("shiki");
  const highlighter = await shiki.getHighlighter({
    theme: "material-theme-palenight",
  });
  return highlighter.codeToHtml(code, lang);
}

as the value for the highlight option but it doesn't work.

Thanks for this project 🚀 its great.

[michaeloliverx]

I have also tried disabling the built in highlighter and passing a custom rehype plugin:

// https://mdsvex.com/docs#options

const visit = require("unist-util-visit");
const shiki = require("shiki");

function rehypeShiki() {
  return async (tree) => {
    const highlighter = await shiki.getHighlighter({
      theme: "material-theme-palenight",
    });
    
    visit(tree, "element", (node) => {
      if (
        node?.tagName === "code" &&
        node.properties.className[0].startsWith("language-")
      ) {
        const lang = node.properties.className[0].split("-")[1];
        const code = node.children[0].value;
        const shikiOutput = highlighter.codeToHtml(code, lang);
        node.type = "text";
        node.value = shikiOutput;
      }
    });
    return tree;
  };
}

module.exports = {
  extensions: [".svx", ".md"],
  smartypants: {
    dashes: "oldschool",
  },
  highlight: false,
  remarkPlugins: [
    [
      require("remark-github"),
      {
        // Use your own repository
        repository: "https://github.com/svelte-add/mdsvex.git",
      },
    ],
    require("remark-abbr"),
  ],
  rehypePlugins: [
    require("rehype-slug"),
    [
      require("rehype-autolink-headings"),
      {
        behavior: "wrap",
      },
    ],
    rehypeShiki,
  ],
};

Unfortunately it removes spaces existing in the code:

[pngwn]

Do you have the markdown snippet that goes along with this output, so I can test?

I've been meaning to deal with the eager code block chomping, I'll take a look at this today.

[michaeloliverx]

@pngwn

---
title: Testing shiki
---

# {title}

```python
import os
key = os.environ["SECRET_KEY"]
```

I am happy to test if it helps!

[pngwn]

@michael0liver Can you try mdsvex@0.9.0? This should add support for async custom highlight functions. Your code samples have been added as tests so this should work fine now.

It also correct an issue where markdown code nodes we chomped before remark plugins could get to them. This doesn't address rehype plugins as the code is already transformed by then but at least there is a way to get at those code blocks.

[michaeloliverx]

The async function support works but there seems to be something going on still.

JavaScript code blocks fail

Code block

```javascript
async function highlighter(code, lang) {
  const highlighter = await shiki.getHighlighter({ theme: "github-dark" });
  return highlighter.codeToHtml(code, lang || "text");
}
```

Output

21:22:00 [vite] Internal server error: Unexpected token
  Plugin: vite-plugin-svelte
  File: /Users/michaeloliver/Development/Testing-Projects/mdsvex-shiki-test/src/routes/example-markdown.md
  33: </ol>
  34: <p>By the way, you can write code in delimited blocks:</p>
  35: <pre class="shiki" style="background-color: #24292e"><code><span class="line"><span style="color: #F97583">async</span><span style="color: #E1E4E8"> </span><span style="color: #F97583">function</span><span style="color: #E1E4E8"> </span><span style="color: #B392F0">highlighter</span><span style="color: #E1E4E8">(</span><span style="color: #FFAB70">code</span><span style="color: #E1E4E8">, </span><span style="color: #FFAB70">lang</span><span style="color: #E1E4E8">) {</span></span>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                             ^
  36: <span class="line"><span style="color: #E1E4E8">  </span><span style="color: #F97583">const</span><span style="color: #E1E4E8"> </span><span style="color: #79B8FF">highlighter</span><span style="color: #E1E4E8"> </span><span style="color: #F97583">=</span><span style=

Spaces getting removed from code block

Code block

```python
from typing import Iterator

def fib(n: int) -> Iterator[int]:
    a, b = 0, 1
    while a < n:
        yield a
        a, b = b, a + b
```

Output

Interestingly when I remove the language from the code block (python in this case) then the spacing is left intact.

```
from typing import Iterator

def fib(n: int) -> Iterator[int]:
    a, b = 0, 1
    while a < n:
        yield a
        a, b = b, a + b
```

---

Here is my config:

const shiki = require("shiki");

async function highlighter(code, lang) {
  const highlighter = await shiki.getHighlighter({ theme: "github-dark" });
  return highlighter.codeToHtml(code, lang || "text");
}

module.exports = {
  extensions: [".svx", ".md"],
  highlight: {
    highlighter: highlighter,
  }
};

BTW I am using Svelte Kit.

[michaeloliverx]

Here's another example with a shell script:

```shell
#!/bin/bash

# example of using arguments to a script
echo "My first name is $1"
echo "My surname is $2"
echo "Total number of arguments is $#"
```

See how the spaces between echo and " have been removed ?

Also its quite common to use ${VAR} syntax inside a bash/shell script e.g:

```bash
VAR="MY_VAR"
echo "${MY_VAR}"
```

It looks like this conflicts with MDsveX/Svelte template syntax and never loads for me.

[pngwn]

Okay, this is because the tags that svelte interprets as expressions aren't being escaped (because it is a custom highlight function). I'll try to handle this internally but in the meantime you can do something like this:

const escape_svelty = (str) =>
  str
    .replace(
      /[{}`]/g,
      (c) => ({ '{': '{', '}': '}', '`': '`' }[c])
    )
  .replace(/\\([trn])/g, '\$1');

async function highlighter(code, lang) {
  const highlighter = await shiki.getHighlighter({ theme: "github-dark" });
  return escape_svelty(highlighter.codeToHtml(code, lang || "text"));
}

I think this should work okay.

Could you open a new issue for this?

[fzn0x]

Thanks for the workaround. Looks like we need to use the @html directives from Svelte to make this work as intended.

import { codeToHtml, getSingletonHighlighter } from 'shiki';

const THEME = 'github-dark';
const WIDTH = '800px';

/**
 * @param code {string} - code to highlight
 * @param lang {string} - code language
 * @returns {Promise<string>} - highlighted html
 */
export async function highlighter(code: string, lang: string) {
	await getSingletonHighlighter({
		langs: [lang],
		themes: [THEME]
	});
	const html = await codeToHtml(code, {
		lang,
		theme: THEME
	});

	const wrappedHtml = `
		<div style="width: ${WIDTH}; overflow-x: auto;">
			${html}
		</div>
	`;

	return `{@html \`<code>${wrappedHtml}</code>\`}`;
}

I'm not sure about the escape_svelty function, that is not needed in my case.

[ericarthurc]

Okay, this is because the tags that svelte interprets as expressions aren't being escaped (because it is a custom highlight function). I'll try to handle this internally but in the meantime you can do something like this:

const escape_svelty = (str) =>
  str
    .replace(
      /[{}`]/g,
      (c) => ({ '{': '{', '}': '}', '`': '`' }[c])
    )
  .replace(/\\([trn])/g, '\$1');

async function highlighter(code, lang) {
  const highlighter = await shiki.getHighlighter({ theme: "github-dark" });
  return escape_svelty(highlighter.codeToHtml(code, lang || "text"));
}

I think this should work okay.

Could you open a new issue for this?

Just a heads up I am using the latest sveltekit, shiki and MDsveX; I had to wrap my shiki highlighter in your escape_svelty function as I was getting pre-transform errors: Pre-transform error: _posts/post_1.md:7:469 Unexpected token

[ericarthurc]

Thanks for the workaround. Looks like we need to use the @html directives from Svelte to make this work as intended.

import { codeToHtml, getSingletonHighlighter } from 'shiki';

const THEME = 'github-dark';
const WIDTH = '800px';

/**
 * @param code {string} - code to highlight
 * @param lang {string} - code language
 * @returns {Promise<string>} - highlighted html
 */
export async function highlighter(code: string, lang: string) {
	await getSingletonHighlighter({
		langs: [lang],
		themes: [THEME]
	});
	const html = await codeToHtml(code, {
		lang,
		theme: THEME
	});

	const wrappedHtml = `
		<div style="width: ${WIDTH}; overflow-x: auto;">
			${html}
		</div>
	`;

	return `{@html \`<code>${wrappedHtml}</code>\`}`;
}

I'm not sure about the escape_svelty function, that is not needed in my case.

This also worked for me as well, wrapping my highlight functions return in @html