Allow $ to literally denote quantities of USD in chat

[dlqqq]

Description

• Fixes Improve rendering of literal dollar signs in chat #1067
• Allows for $ to literally denote multiple quantities of USD per line in the chat by escaping dollar signs in the frontend.
  • This does not escape $$, so $$ can still be used to delimit display math.
  • This also does not escape $ when it is being shown literally within a code, pre, kbd, or samp element. This list of "literal elements" was derived from: https://developer.mozilla.org/en-US/docs/Web/HTML/Element/code#see_also
• The rendering behavior now only allows for \( <math> \) to denote inline math, while allowing $$ <math> $$ and \[ <math> \] to denote display math.
  • This is aligned with MathJax's default behavior, which JupyterLab overrides in @jupyterlab/mathjax-extension: https://docs.mathjax.org/en/latest/input/tex/delimiters.html
• Edits the system prompt to request that the LLM not use $ as an inline math delimiter, but instead use LaTeX math delimiters (i.e. \( <math> \) instead of $<math>$).

Demo

[srdas]

I tested the formatting for several LLMs: all Bedrock Chat Anthropic LLMs, Llama3 models, and mistral models. The formatting looks great and does not fall afoul of TeX formatting $ limiters.

Even if you prompt it to use TeX formatting it does so by displaying the formatting and not forcing math mode, as shown here:

So the modification in this PR is robuest and all LGTM.

[dlqqq]

@srdas Thanks for the review! Actually your testing pointed out a regression. While we don’t want to treat $ for inline math, we should still allow for $$ to be used for display math. This is aligned with Mathjax’s recommendation for delimiters here: https://docs.mathjax.org/en/latest/input/tex/delimiters.html

JupyterLab overrides the default delimiters set by MathJax, which is why we have to escape single dollar signs in our application despite MathJax being the default implementation of ILatexTypesetter.

I'll update this PR to allow for double dollar signs while still escaping single dollar signs.

[dlqqq]

I just noticed another regression below. We shouldn't escape dollar signs within Markdown code blocks. The LLM is trying to write \$ in its response, but this gets transformed to \\\$ when we escape the dollar sign by adding triple backticks.

Handling this will be more tricky because I don't think a regex can match this case.

[srdas]

I just noticed another regression below. We shouldn't escape dollar signs within Markdown code blocks. The LLM is trying to write \$ in its response, but this gets transformed to \\\$ when we escape the dollar sign by adding triple backticks.

Handling this will be more tricky because I don't think a regex can match this case.

I also noticed this in my testing, because at one point it returned this sort of formatting, but I was unable to reproduce it again when I tried. Good you caught it again.

[dlqqq]

It may be easier to somehow change the MathJax config to simply not respect $ as an inline delimiter, which means we don't have to implement any complex escaping logic for $ symbols. However, the documentation is unclear at best: https://docs.mathjax.org/en/latest/web/configuration.html#configuring-mathjax-after-it-is-loaded

[dlqqq]

It may be easier to somehow change the MathJax config to simply not respect $ as an inline delimiter

This strategy proved to be too complex. I tried creating a new instance of MathJaxTypesetter with custom inline delimiters, but that also affected the rendering in notebooks. It seems only one MathJax configuration can be used at a time across the entire browser.

I've updated this PR and its description to address the 2 bugs called out earlier:

• Escapes $ but not $$
• Does not escape $ when inside of a "literal element", e.g. code, pre.

[dlqqq]

Demo showing dollar symbols are not escaped in code blocks:

[srdas]

Great use of AI to correct AI! Jupyter AI is now officially self-repairing!
LGTM!

[dlqqq]

@meeseeksdev please backport to v3-dev

[dlqqq]

cc: @jtpio @brichet re: jupyter-chat

[brichet]

cc: @jtpio @brichet re: jupyter-chat

Thanks for the ping @dlqqq