MathML incorrect for LaTeX prime notation

[castedo]

REPRO STEPS:

1. Create doc.md

cat > doc.md <<'EOF'
$$
Z^p
$$

$$
Z^+
$$

$$
Z^\prime
$$

$$
Z'
$$

$$
Z\prime
$$
EOF

2. Save an optional line of MathJax script:

cat > mathjax.md <<'EOF'
<script src="https://cdn.jsdelivr.net/npm/mathjax@3/es5/mml-chtml.js" type="text/javascript"></script>
EOF

3. Generate MathML to be viewed with MathJax:

pandoc mathjax.md doc.md -s --mathml > mathjax-with-mathml.html

4. Open mathjax-with-mathml.html in a browser.

RESULT:
The MathML generated by pandoc (in mathjax-with-mathml) is (edited for clarity):

<p> ... <msup><mi>Z</mi><mi>p</mi></msup>   <annotation>Z^p</annotation> ... </p>
<p> ... <msup><mi>Z</mi><mo>+</mo></msup>   <annotation>Z^+</annotation> ... </p>
<p> ... <msup><mi>Z</mi><mi>′</mi></msup>   <annotation>Z^\prime</annotation> ... </p>
<p> ... <mrow><mi>Z</mi><mi>′</mi></mrow>   <annotation>Z&#39;</annotation> ... </p>
<p> ... <mrow><mi>Z</mi><mi>′</mi></mrow><annotation>Z\prime</annotation> ... </p>

And then visually looking at the webpage, the result of Z' looks like Z\prime and not Z^\prime.

EXPECTED:
Output like what LaTeX and MathJax generate.

Specifically the MathML for Z' should be a <msup>, like what MathJax generates, and not <mrow>.

<p> ... <msup><mi>Z</mi><mi>p</mi></msup>   <annotation>Z^p</annotation> ... </p>
<p> ... <msup><mi>Z</mi><mo>+</mo></msup>   <annotation>Z^+</annotation> ... </p>
<p> ... <msup><mi>Z</mi><mi>′</mi></msup>   <annotation>Z^\prime</annotation> ... </p>
<p> ... <mrow><mi>Z</mi><mi>′</mi></mrow>   <annotation>Z&#39;</annotation> ... </p>
<p> ... <mrow><mi>Z</mi><mi>′</mi></mrow><annotation>Z\prime</annotation> ... </p>

Similarly, the visual result is not what LaTeX produces. The visual output of LaTeX for Z' looks like Z^\prime and not Z\prime.

I'll add more information and commentary separately in this thread.

[alerque]

Entering and rendering primes does not have a straightforward right answer.

c.f. alerque/polytype#10

[castedo]

Entering and rendering primes does not have a straightforward right answer.

c.f. alerque/polytype#10

I will read this thread and comment more here.

[castedo]

Here is some notable information in TeX stackexchange.

https://tex.stackexchange.com/questions/87134/what-is-the-advantage-of-using-f-prime-instead-of-f

[castedo]

I reproduce the LaTeX output for the above my saving a main.tex as follows:

cat > main.tex <<'EOF'
\documentclass{article}
\begin{document}
\input{doc.md}
\end{document}
EOF

and then doing pdflatex main.tex using TeX Live on Fedora 41. I can give instruction on a public Docker container image to use to repro if wanted.

[castedo]

I should also mention (recalling from memory) KaTeX parses and generates output for Z' that is the same as Z^\prime. I can double check with example steps for KaTeX if anyone wants. So I believe LaTeX, MathJax, and KaTeX, all generate output that is equivalent for Z' and Z^\prime whereas pandoc does not.

[castedo]

Entering and rendering primes does not have a straightforward right answer.
c.f. alerque/polytype#10

I will read this thread and comment more here.

@alerque OK, I've read the thread you linked. I'll steer clear from commenting about what polytype or SILE should do. I'm not a user of either and not very familiar with either. I am very familiar with LaTeX however. I suspect vastly more pandoc users are familiar with LaTeX rather than polytype or SILE.

The pandoc documentation says anything between the $ characters will be treated as "TeX math". From the TeX stackexchange thread i linked, it looks like the Z' = Z^\prime equivalence might be due to a LaTeX macro.

So perhaps the "right answer" depends on whether the TeX between $ characters should try to emulate LaTeX. To this question, I was expecting a definite YES. Although I would expect very advanced LaTeX functionality to be missing. But this doesn't seem particularly advanced. Even KaTeX emulates this functionality of LaTeX (as does MathJax).

So net, net, I think the straightforward right answer for pandoc markdown to do is what LaTeX, MathJax, and KaTeX all do, which is treat Z' as short hand for Z^\prime (when generating MathML).

[jgm]

Here's what the pandoc-generated mathml looks like when viewed directly in Safari (without mathjax):

I think the bottom two look right; using a subscripted prime puts the prime too far above.

It's possible that things look different when you use mathjax. (Though, if you're using mathjax with pandoc, I'd suggest just using mathjax's tex support and skipping the tex -> mathml translation).

[castedo]

To date, I have been relying on math encoded as TeX and rendered by KaTeX. But I am migrating to a JATS XML pipeline with MathML like PubMed Central. I'm happy to go into the details, but my particular details probably aren't as relevant as what is the target functionality of pandoc.

I've only recently started doing detailed testing with MathML, but so far it seems Safari/Chrome/Firefox are all rendering MathML slightly differently. And all notably differently than MathJax. For my complex math doc test it seems clear MathJax is performing far better and more in line with native LaTeX. I was hoping that by 2025 standard MathML rendering in major browsers would be a reality, but this seems not to be the case.

So it seems we have three relevant flavors of MathML rendering:

1. native Safari,
2. native Chrome/Edge,
3. and MathJax (on any major browser).

I am migrating to MathJax flavor MathML. It is what PubMed Central's website is using. I suspect, without great confidence, that MathJax is more compliant with the standard and the browsers are currently not. I predict that websites that care will continue to rely on MathJax and few website who care will target native browser rendering for MathML. Hopefully eventually the major browsers will match MathJax and MathJax will not be needed. But I suspect it will be a long time.

In the short-term, I am planning to write a Node.js utility to parse the JATS XML that pandoc outputs, remove all the MathML produced by pandoc, then have MathJax render the LaTeX to MathXML, and insert MathJax flavor MathML back into the JATS XML file.

This begs the question whether this is the long-term situation for pandoc. For which rendering flavor of MathML does pandoc produce MathML (in JATS XML at least)?

[castedo]

For completeness, here some details on my particular situation that may or may not be that relevant to pandoc target functionality.

I am attempting to archive encoded math for long-term preservation, similar to PubMed Central. My recent analysis on ~20K recent PubMed Central JATS XML docs with math content is that about 4/5ths of these docs only contain MathML and not TeX. The other 1/5th has both MathML and TeX. One quick investigation of a paper on PubMed Central's website had MathJax rendering from the MathML and not TeX. So at the moment, I'm betting on MathML, similar to what is archived in PubMed Central, rendered by MathJax, as the best long-term storage format. But I'm very nervous about MathML. It has not been around as long at LaTeX. But for short chunks of math, I suspect MathML has less ambiguity and is more conducive to automated testing and validation than embedded human-authored-LaTeX. For example, is \mspace safe LaTeX for the long-term? LaTeX and pandoc are happy with it, but not KaTeX which throws an exception and fails if is see \mspace which is not supported. But how easily can one test and validate which subset of LaTeX is safe to use? I suspect it's going to be easier to test and validate MathML.

[jgm]

For comparison, here is how your original input renders with a variety of methods.

via latex pandoc -o test.pdf --pdf-engine=ENGINE

pdflatex	lualatex	xelatex

via pandoc -s -o test.html --mathml

Firefox native	Chrome native	Safari native	Firefox mathjax	Chrome mathjax	Safari mathjax

[jgm]

Summary: putting the prime in msup gives undesirable results in Chrome native and Safari native. Putting the prime in mrow gives undesirable results in mathjax. The two options display identically in Firefox native and in all the latex variants.

[castedo]

Nice matrix! Particularly interesting to see how $Z\prime got rendered differently with different LaTeX engines.

I'll isolate some more cases from my complex math doc where pandoc TeX -> MathML leads to undesirable results. This case of $Z'$ was my first attempt to isolate whether an undesirable result was native browser MathML rendering or pandoc MathML generation. From my recent attempt to migrate to MathML, I found native browser MathML rendering was usually resulting in undesirable results. In contrast I got desirable results like LaTeX when MathJax rendered MathML generated from MathJax (which is not surprising).

[castedo]

There is another line of information I can help gather if you find it useful, which is what LaTeX and corresponding MathML appears in PubMed central. I have ~20K JATS XML files with math and have scripts that do analysis on them. Here's one case that I easily and quickly found:

In PubMed Central article PMC11757725 (Scientific Reports, nature) after the text "we generate a fake data" there is a X' in the <tex-math> XML element and the <mm:math> next to it is

<mml:math id="M4"><mml:msup><mml:mi>X</mml:mi><mml:mo>&#x02032;</mml:mo></mml:msup></mml:math>

Note the X and prime are wrapped in a <msup> and not an <mrow>.

There are tons and tons of these examples. Although at first glance, I suspect they are all coming from the same JATS XML pipeline which is whatever Nature is using.

Something to consider for pandoc MathML is format drift. I suspect that native web browser MathML rendering is going to drift over the next decade. In contrast, I bet the MathML in PubMed Central is not going to drift. I'm not sure which MathML flavor will be dominant in say 10 years, but I bet the native browser flavor will be a moving target whereas MathJax/PubMed flavor will stay constant.