Some indexed terms are ignored when using xelatex engine with memoir class

[ygversil]

Describe the bug
Having a document with some indexed terms, when running make latexpdf with latex_engine = 'xelatex', and 'memoir' theme in latex_documents, the index is mostly empty in the output PDF file. Most of the terms are absent.

To Reproduce

• Create a new project named Foo, with author Me, and version 0.1.0.

$ sphinx-quicktart  # project: Foo; author: Me; version: 0.1.0; leave other options unchanged

• Now edit conf.py and add the following lines.

# -- Options for LaTeX output -------------------------------------------------
latex_engine = 'xelatex'
latex_documents = [('index', 'foo.tex', project, author, 'memoir')]

• Also edit index.rst and add the following line in just before Indices and tables.

This is :index:`Foo` project.

• Finally run the command to have PDF output.

$ make latexpdf

• Now open the resulting PDF file and look at the last page: there is the Index title, but it is empty. Furthermore, standard error output shows the following warning.

WARNING: unknown attribute `hyperpage'! (ignored)

Expected behavior

The index should have the word Foo with and an hyperlink to the first page like this: Foo, 1

Environment info

• OS: Archlinux with latest packages
• Python version: 3.8.6
• Sphinx version: 3.2.1
• Sphinx extensions: None
• Extra tools:
  • Texlive: 2020.54586

Additional context

The following other combinations work by producing correct index page:

• 'pdflatex' engine with 'memoir' theme,
• 'xelatex' engine with 'manual' theme.

I think the following patch can solve the issue, but I am not sure if this is the correct way to solve the problem. Maybe it is more a xelatex or a memoir issue.

--- sphinx.xdy  2020-11-18 13:56:48.131747451 +0100
+++ sphinx.xdy.new      2020-11-18 14:02:13.304152921 +0100
@@ -4,12 +4,13 @@
 ;; configuration is a workaround, which requires to pass option
 ;; hyperindex=false to hyperref.
 ;; textit and emph not currently used, spxpagem replaces former textbf
-(define-attributes (("textbf" "textit" "emph" "spxpagem" "default")))
+(define-attributes (("textbf" "textit" "emph" "spxpagem" "default" "hyperpage")))
 (markup-locref :open "\textbf{\hyperpage{" :close "}}" :attr "textbf")
 (markup-locref :open "\textit{\hyperpage{" :close "}}" :attr "textit")
 (markup-locref :open "\emph{\hyperpage{" :close "}}" :attr "emph")
 (markup-locref :open "\spxpagem{\hyperpage{" :close "}}" :attr "spxpagem")
 (markup-locref :open "\hyperpage{" :close "}" :attr "default")
+(markup-locref :open "\hyperpage{" :close "}" :attr "hyperpage")
 
 (require "numeric-sort.xdy")

[jfbu]

Hi, thanks for the report and testing file. One can also reproduce your issue with default latex engine pdflatex, from setting latex_use_xindy to True.

Generally speaking, memoir is a big aggregate of code extracted from many packages. It is destined for use by a single user and a hand-written document. Compatibility with Sphinx borders on the miraculous.

In the case at hand, this is clearly a bug in memoir class which tries to intervene into hyperref already intricate dealings, and in so doing makes the hyperindex=false option inoperant. Relevant here is the fact that memoir does not load hyperref nor provide a class option to do it, but leaves it to user to decide. But then options passed to hyperref should work, no?

It turns out that memoir documentation does mention xindy in its section 17.2.3, MakeIndex, page 312 of v3.7n, 2020/10/04:

It turns out that xindy cannot handle a memoir hyperindex (which can be obtained with the aid of the hyperref package), although MakeIndex can do so If you are going to use xindy to process the raw index data put \xindyindex in the preamble, which will prevent hyperindexing.

Problem is that doing so (i.e. adding \xindyindex in the preamble) does not work for us. It does not work whether or not I leave
\PassOptionsToPackage{hyperindex=false}{hyperref} in the preamble. Trying to pass hyperindex=false as class option does not change anything (anyway, hyperref does receive this option, but has memoir preempts it with some code not taking care of the option setting, it simply does not work).

I found a fix which is to set forcefully in the preamble some internal non documented memoir boolean:

latex_elements = {
    'preamble': r'\memhyperindexfalse',
}

This issue is borderline to wontfix, because as I said it is miraculous if memoir works with Sphinx. The former is destined to fine-tuning manually crafter document, the latter to produce latex documents in an automated way from non latex sources and requires already quite a few latex packages, and Sphinx contributed latex macros to achieve its task.

Pondering this issue took me a disproportionate amount of time on non-Sphinx code and although the above seems to fix it, I can not test it extensively nor go to the effort of trying to set up unit tests for it, and fixing it causes us to add one line to our latex template especially for memoir class support: and there are also already some added patches for memoir elsewhere in our latex code. It is impossible to consider this a permanent task to maintain compatibility with memoir.

I can not be certain the above fix (which uses an undocumented boolean from memoir code) will not fail at next memoir release. And what happens if memoir's author becomes aware of the incomplete compatibility with hyperref options? An upstream fix might create problem so I will have to do something such as \ifdefined..\fi to protect our fix.