Upgrade to Sphinx 1.5.x

[jdemeyer]

The newer version of Sphinx needs requests.

Tarballs:

• https://pypi.python.org/packages/a7/df/4487783152b14f2b7cd0b0c9afb119b262c584bf972b90ab544b61b74c62/Sphinx-1.5.3.tar.gz
• https://pypi.python.org/packages/16/09/37b69de7c924d318e51ece1c4ceb679bf93be9d05973bb30c35babd596e2/requests-2.13.0.tar.gz

We add a patch to Sphinx to allow running without SSL: sphinx-doc/sphinx#3554

Upstream: Fixed upstream, but not in a stable release.

CC: @kiwifb

Component: packages: standard

Keywords: days85

Author: Jeroen Demeyer

Branch/Commit: f20f00a

Reviewer: François Bissey, John Palmieri

Issue created by migration from https://trac.sagemath.org/ticket/22252

[kiwifb]

comment:1

Should we update docutils to 0.13 at the same time? sphinx 1.5.2 as all the fix to work with it as far as I know, can be in a separate ticket but it is better if they are done at the same time given the comments in cschwan/sage-on-gentoo#451

[jdemeyer]

comment:2

I would say: let's do whatever is easiest. So the question is: is it easier to make Sage and Sphinx-1.5.2 work with docutils-0.12 or with docutils-0.13?

[kiwifb]

comment:3

Thinking more about it, sphinx 1.5.2 should work with both, so we can do sphinx first then docutils afterwards.

[jdemeyer]

Branch: u/jdemeyer/upgrade_to_sphinx_1_5_2

[sagetrac-git]

Branch pushed to git repo; I updated commit sha1. New commits:

6a61a49

Doc fixes for Sphinx-1.5

[sagetrac-git]

Commit: 6a61a49

[jdemeyer]

Author: Jeroen Demeyer

[jhpalmieri]

comment:8

The line

    import ssl

from sphinx/util/requests.py is going to cause problems on systems without SSL (like my Mac, until I ran make openssl and sage -f python2).

[sagetrac-git]

Changed commit from 6a61a49 to 33e4007

[sagetrac-git]

Branch pushed to git repo; I updated commit sha1. This was a forced push. New commits:

04fe397	Upgrade to Sphinx 1.5.3
33e4007	Doc fixes for Sphinx 1.5.x

[sagetrac-git]

Changed commit from 33e4007 to 9333e9a

[sagetrac-git]

Branch pushed to git repo; I updated commit sha1. This was a forced push. New commits:

9333e9a

Docbuild fixes for Sphinx 1.5.x

[sagetrac-git]

Changed commit from 9333e9a to 6c37777

[sagetrac-git]

Branch pushed to git repo; I updated commit sha1. This was a forced push. New commits:

d8cdc76	Upgrade to Sphinx 1.5.3
6c37777	Docbuild fixes for Sphinx 1.5.x

[jdemeyer]

Changed keywords from none to days85

[sagetrac-git]

Changed commit from 6c37777 to 78e6718

[sagetrac-git]

Branch pushed to git repo; I updated commit sha1. This was a forced push. New commits:

879c401	Upgrade to Sphinx 1.5.3
78e6718	Docbuild fixes for Sphinx 1.5.x

[jdemeyer]

comment:14

This last version actually manages to compile the documentation.

[jhpalmieri]

comment:16

What about the issue in [comment:8]? Is ssl going to become a dependency for building the Sage docs?

[kiwifb]

comment:17

Modulo [comment:8] it works for me.

[jpflori]

comment:18

I think SSl is going to be a Sage dependency soon anyway so that we can upgrade R.

[jpflori]

comment:19

In fact, SSL is already a Sage dependency: #22189.

[jhpalmieri]

comment:39

Replying to @jdemeyer:

I think the LC_ALL issue is orthogonal to this ticket, so I didn't do anything for that.

I agree.

[jhpalmieri]

comment:40

I am happy with this, but it needs testing on platforms other than OS X.

[kiwifb]

comment:41

Replying to @jdemeyer:

On my Gentoo system, I never managed to built the Japanese PDF documentation (any ideas, fbissey?) with or without this patch:

[docpdf] /bin/sh: extractbb: command not found

So I can make changes to the \ifPDFTeX thing but I'll have to do it "in the dark" without actually being able to check it.

So I would appreciate if somebody else could finish this ticket.

I should check that I can build it now. I changed system so I am starting from a cleaner slate. I used to have problem with russian documentation because of tex fonts problems. According to my ebuild I need the following from tex

L10N="ca de en fr hu it ja pt ru tr" USE=extra emerge -1Nv texlive

I usually do

VARTEX=$somelocations make doc-pdf

Gentoo doesn't install "unfolded" russian (and japanese?) tex fonts. If you don't specify VARTEX, it may try to do it in a system location.

[kiwifb]

comment:42

Japanese pdf doc build OK here with this ticket as is (a vanilla build rather than a sage-on-gentoo one). Cannot quite read it but the "a_tour_of_sage.pdf" looks legit compared to the english one.

[strogdon]

comment:43

I have Gentoo here and all pdfs build with this ticket. I have installed

[I] dev-texlive/texlive-langcjk
[I] dev-texlive/texlive-langcyrillic
[I] dev-texlive/texlive-langenglish
[I] dev-texlive/texlive-langeuropean
[I] dev-texlive/texlive-langfrench
[I] dev-texlive/texlive-langgerman
[I] dev-texlive/texlive-langitalian
[I] dev-texlive/texlive-langjapanese
[I] dev-texlive/texlive-langportuguese
[I] dev-texlive/texlive-langspanish

/usr/bin/extractbb is pulled in by app-text/texlive-core-2015-r1. The only thing I had to do is uncomment

%f  cid-x.map

in /etc/texmf/dvipdfmx.d/dvipdfmx.cfg so certain fonts could be generated.

Replying to @jdemeyer:

On my Gentoo system, I never managed to built the Japanese PDF documentation (any ideas, fbissey?) with or without this patch:

[docpdf] /bin/sh: extractbb: command not found

So I can make changes to the \ifPDFTeX thing but I'll have to do it "in the dark" without actually being able to check it.

So I would appreciate if somebody else could finish this ticket.

[kiwifb]

comment:44

Replying to @strogdon:

/usr/bin/extractbb is pulled in by app-text/texlive-core-2015-r1. The only thing I had to do is uncomment

%f  cid-x.map

in /etc/texmf/dvipdfmx.d/dvipdfmx.cfg so certain fonts could be generated.

Can you elaborate Steve? Do you mean that after doing that change the fonts are installed and you don't need to set VARTEX, or that I should check my build for missing fonts?

[strogdon]

comment:45

Replying to @kiwifb:

Replying to @strogdon:

/usr/bin/extractbb is pulled in by app-text/texlive-core-2015-r1. The only thing I had to do is uncomment

%f  cid-x.map

in /etc/texmf/dvipdfmx.d/dvipdfmx.cfg so certain fonts could be generated.

Can you elaborate Steve? Do you mean that after doing that change the fonts are installed and you don't need to set VARTEX, or that I should check my build for missing fonts?

This was with vanilla and not s-o-g. I just did make doc-pdf. In doing that some japanese fonts could not be generated. I uncommented the f cid-x.map line although it could have been f ckx.map in order to get the build to complete.

[strogdon]

comment:46

Replying to @strogdon:

Replying to @kiwifb:

Replying to @strogdon:

/usr/bin/extractbb is pulled in by app-text/texlive-core-2015-r1. The only thing I had to do is uncomment

%f  cid-x.map

in /etc/texmf/dvipdfmx.d/dvipdfmx.cfg so certain fonts could be generated.

Can you elaborate Steve? Do you mean that after doing that change the fonts are installed and you don't need to set VARTEX, or that I should check my build for missing fonts?

This was with vanilla and not s-o-g. I just did make doc-pdf. In doing that some japanese fonts could not be generated. I uncommented the f cid-x.map line although it could have been f ckx.map in order to get the build to complete.

It was %f cid-x.map that I had to uncomment.

[kiwifb]

comment:47

Still commented here, and yet I completed the build of the pdf documentation succesfully

%% Put additional fontmap files here (usually for Type0 fonts)
%f  cid-x.map

% the following file is generated by updmap(-sys) from the 
% KanjiMap entries in the updmap.cfg file.
f kanjix.map
% minimal example for Chinese and Korean users
% improvements please to tex-live@tug.org
f ckx.map

[strogdon]

comment:48

Replying to @kiwifb:

Still commented here, and yet I completed the build of the pdf documentation succesfully

%% Put additional fontmap files here (usually for Type0 fonts)
%f  cid-x.map

% the following file is generated by updmap(-sys) from the 
% KanjiMap entries in the updmap.cfg file.
f kanjix.map
% minimal example for Chinese and Korean users
% improvements please to tex-live@tug.org
f ckx.map

I don't think there is an issue. I'm redoing things. It will take a while. Things are quite slow.

[strogdon]

comment:49

OK, with f cid-x.map commented out I get

mktexpk --mfmode / --bdpi 600 --mag 2+180/600 --dpi 1380 gbm

in ja/tutorial/missfont.log when building ja/tutorial/tutorial-jp.pdf and I see

[docpdf] > <./a2tag.png> <./1.pngtutorial-jp.dvi -> tutorial-jp.pdf
[docpdf] > <./ab2tag.png>][1 [277
[docpdf] kpathsea: Running mktexpk --mfmode / --bdpi 600 --mag 2+180/600 --dpi 1380 gbm

...

[docpdf] Underfull \hbox (badness 10000) in paragraph at lines 22578--22581
[docpdf] []\T1/ptm/m/n/10 For de-tails, see \T1/pcr/m/n/10 ModulesWithBasis.ParentMethod
[docpdf] s.module_morphism() \T1/ptm/m/n/10 (and also \T1/pcr/m/n/10 sage.
[docpdf] [327] [328] [329] [330] [331] [332] [333] [334] [335] [336] [337] [338]gsftopk: fatal: map file `psfonts.cmz' not found.
[docpdf] 
[docpdf] Underfull \hbox (badness 7888) in paragraph at lines 23459--23467
[docpdf] []\T1/ptm/m/n/10 Again I rec-om-mend this [][]$http : / / www . scipy . org / W
[docpdf] iki / Documentation ? action = AttachFile & do = get & target = scipy _
[docpdf] [339] [340] [341] [342] [343]mktexpk: don't know how to create bitmap font for gbm.
[docpdf] mktexpk: perhaps gbm is missing from the map file.
[docpdf] kpathsea: Appending font creation commands to missfont.log.
[docpdf] 
[docpdf] dvipdfmx:warning: Could not locate a virtual/physical font for TFM "gbm".
[docpdf] dvipdfmx:warning: >> There are no valid font mapping entry for this font.
[docpdf] dvipdfmx:warning: >> Font file name "gbm" was assumed but failed to locate that font.
[docpdf] dvipdfmx:fatal: Cannot proceed without .vf or "physical" font for PDF output...
[docpdf] 
[docpdf] Output file removed.
[docpdf] make[2]: *** [Makefile:35: all-pdf-ja] Error 1
[docpdf] make[2]: Leaving directory '/64bitdev/storage/sage-git_develop/sage/local/share/doc/sage/latex/ja/tutorial'

The same missfont.log file is in ja/a_tour_of_sage. If I uncomment %f cid-x.map then everything builds. I'm not sure what's up but I can make the pdfs to build. I guess it's possible that since I didn't do

L10N="ca de en fr hu it ja pt ru tr" USE=extra emerge -1Nv texlive

I manually installed the packages - that that could be the difference? The only difference on my system is that app-text/texlive-2015 would be rebuilt with L10N="ca* de* en fr hu* it* ja* pt* ru* tr*. But all the extra languages are installed.

[kiwifb]

comment:50

Since I had to build anew I also needed USE="cjk xetex", but you would have been prompted for those I am sure. On the other hand I found /usr/share/texmf-dist/fonts/tfm/ptex/dvips/gbm.tfm on my system and it belongs to

$ equery b /usr/share/texmf-dist/fonts/tfm/ptex/dvips/gbm.tfm
 * Searching for /usr/share/texmf-dist/fonts/tfm/ptex/dvips/gbm.tfm ... 
dev-texlive/texlive-langjapanese-2015 (/usr/share/texmf-dist/fonts/tfm/ptex/dvips/gbm.tfm)

Can you check that you have that?

[kiwifb]

comment:51

On another note:

  611  rm build/pkgs/sphinx/patches/latex_memory.patch 
  612  ./sage -f sphinx
  613  make doc-clean
  614  MAKE="make -j8" make doc-pdf

So I removed the latex_memory patch from sphinx, rebuild and the pdf doc build fine. So, at least on my system, it is unnecessary.

[strogdon]

comment:52

Replying to @kiwifb:

Since I had to build anew I also needed USE="cjk xetex", but you would have been prompted for those I am sure. On the other hand I found /usr/share/texmf-dist/fonts/tfm/ptex/dvips/gbm.tfm on my system and it belongs to

$ equery b /usr/share/texmf-dist/fonts/tfm/ptex/dvips/gbm.tfm
 * Searching for /usr/share/texmf-dist/fonts/tfm/ptex/dvips/gbm.tfm ... 
dev-texlive/texlive-langjapanese-2015 (/usr/share/texmf-dist/fonts/tfm/ptex/dvips/gbm.tfm)

Can you check that you have that?

Yes, this file is on my system.

[kiwifb]

comment:53

Hum... I don't any of the stuff you quote in my log. I am wondering if it is because I have fonts (truetype not tex) that have japanase character sets on my system.

[I] media-fonts/arphicfonts (0.2.20080216.1-r2@14/02/17): Chinese TrueType Arphic Fonts
[I] media-fonts/bitstream-cyberbit (2.0@14/02/17): Cyberbit Unicode (including CJK) font
[I] media-fonts/corefonts (1-r7@14/02/17): Microsoft's TrueType core fonts
[I] media-fonts/croscorefonts (1.23.0@14/02/17): Open licensed fonts metrically compatible with MS corefonts
[I] media-fonts/dejavu (2.35@30/12/16): DejaVu fonts, bitstream vera with ISO-8859-2 characters
[I] media-fonts/droid (113-r4@14/02/17): Font family from Google's Android project
[I] media-fonts/encodings (1.0.4@30/12/16): X.Org font encodings
[I] media-fonts/font-adobe-100dpi (1.0.3@14/02/17): X.Org Adobe bitmap fonts
[I] media-fonts/font-adobe-75dpi (1.0.3@14/02/17): X.Org Adobe bitmap fonts
[I] media-fonts/font-adobe-utopia-100dpi (1.0.4@14/02/17): X.Org Adobe Utopia bitmap fonts
[I] media-fonts/font-alias (1.0.3-r1@14/02/17): X.Org font aliases
[I] media-fonts/font-bh-lucidatypewriter-100dpi (1.0.3@14/02/17): X.Org Bigelow & Holmes Lucida bitmap fonts
[I] media-fonts/font-misc-misc (1.1.2@14/02/17): X.Org miscellaneous fonts
[I] media-fonts/font-util (1.3.1@30/12/16): X.Org font utilities
[I] media-fonts/hack (2.020@30/12/16): A typeface designed for source code
[I] media-fonts/ipamonafont (1.0.8@14/02/17): Hacked version of IPA fonts, which is suitable for browsing 2ch
[I] media-fonts/ja-ipafonts (003.02-r1@14/02/17): Japanese TrueType fonts developed by IPA (Information-technology Promotion Agency, Japan)
[I] media-fonts/liberation-fonts (2.00.1-r1@30/12/16): A Helvetica/Times/Courier replacement TrueType font set, courtesy of Red Hat
[I] media-fonts/libertine (5.3.0.20120702-r2@16/01/17): Fonts from the Linux Libertine Open Fonts Project
[I] media-fonts/noto (20160305-r1@30/12/16): Google's font family that aims to support all the world's languages
[I] media-fonts/stix-fonts (1.1.1@14/02/17): Comprehensive OpenType font set of mathematical symbols and alphabets
[I] media-fonts/urw-fonts (2.4.9@30/12/16): free good quality fonts gpl'd by URW++
[I] media-fonts/wqy-microhei (0.2.0_beta@14/02/17): A droid derived Sans-Serif style CJK font
[I] media-fonts/wqy-zenhei (0.9.46@14/02/17): WenQuanYi Hei-Ti Style (sans-serif) Chinese outline font

[strogdon]

comment:54

Replying to @kiwifb:

Hum... I don't any of the stuff you quote in my log. I am wondering if it is because I have fonts (truetype not tex) that have japanase character sets on my system.

It appears that, for whatever reason, I had a

~/.texlive/texmf-var/fonts/map/dvipdfmx/updmap/kanjix.map

file that was getting in the way of the build. Moving the texmf-var folder out of the way and I appear to now have a sane build of the pdf docs.

[jdemeyer]

comment:55

Let me remind everybody that this ticket is about upgrading from Sphinx 1.4 to 1.5. Discussions about building docs in general can be good to have, but they do not belong on this ticket (unless there is a regression, i.e. something worked with Sphinx 1.4 but no longer works).

[kiwifb]

comment:56

Sorry for the noise Jeroen. We wanted to be sure we had the dependencies right.

My own experience here is that:

1. doc, including pdf and japanese pdf, in particular, build fine
2. it does so with or without latex_memory.patch so I guess we can drop it - unless someone has a different experience.

I am happy for this ticket to be sent to the bots once the offending patch is removed - or someone shows it is needed in some cases.

[jdemeyer]

comment:57

Replying to @kiwifb:

it does so with or without latex_memory.patch so I guess we can drop it - unless someone has a different experience.

Can we leave this to a follow-up ticket? The patch has been added in the past for a reason, so I'm not too keen to just drop it.

[kiwifb]

comment:58

Replying to @jdemeyer:

Replying to @kiwifb:

it does so with or without latex_memory.patch so I guess we can drop it - unless someone has a different experience.

Can we leave this to a follow-up ticket? The patch has been added in the past for a reason, so I'm not too keen to just drop it.

I am happy with that, we may well discover that is needed on some platforms. Putting it in positive review.

[jdemeyer]

comment:59

Replying to @jdemeyer:

Can we leave this to a follow-up ticket? The patch has been added in the past for a reason, so I'm not too keen to just drop it.

See #22660.

[vbraun]

Changed branch from u/jdemeyer/upgrade_to_sphinx_1_5_2 to f20f00a