texlive: 2015 -> 2016

[vcunat]

I finally got from words to deeds. Please, test your use cases!

• It's today's snapshot.
• TUG mirroring works now. It will slowly start to fail, but I've got a snapshot of all files, so we can fix that up later.

Things done

• Tested using sandboxing
  (nix.useSandbox on NixOS,
  or option build-use-sandbox in nix.conf
  on non-NixOS)
• Built on platform(s)
  • NixOS
  • OS X
  • Linux
• Tested compilation of all pkgs that depend on this change using nix-shell -p nox --run "nox-review wip"
• Tested execution of some binary files (usually in ./result/bin/)

---

Note: only basic testing has been done so far; also see FIXME items.

[jokogr]

error: attribute ‘rotating’ missing, at /home/joko/playground/nixpkgs/pkgs/tools/typesetting/tex/dblatex/default.nix:4:26

CTAN reports that "The pack­age is now part of the la­tex-graph­ics bun­dle" (Source: https://www.ctan.org/pkg/rotating?lang=en), is it ok to change this dependency ("rotating") with "graphics"?

[jokogr]

I also got:

output path ‘/nix/store/wx77rngg1yx1ixbcv8wwkc5msqld7hm2-xassoccnt.tar.xz’ has sha512 hash ‘15rcb708spq8hv65lvccq1vnv4zbsd088g7ng0cydlyl8f3xmi2kk14244md50yzvq41qyfiizvkp72722592j927xn3ndv54vdqrp4’ when ‘2xbc9yxlzszcxc00n75vfq0xdbnh367s64hwcdq9p3jn3h154s6f5rwky54spzh1pxvfrs2hp2qg9mddsxh8an095ksgmp7845fg1ag’ was expected

but after running again sudo nixos-rebuild switch, it seems to move on...

[vcunat]

Thanks, dblatex fixed up. That hash error was likely due to some transient network error, if it disappeared (or some mirrors serve a newer version of that file already).

[jokogr]

ok, I'm missing xelatex (and xetex apparently). pdflatex is here, but my work is too dependent to the former.

My relevant systemPackages are the following:

    (texlive.combine {
     inherit (texlive) scheme-basic collection-bibtexextra collection-binextra
     collection-fontsrecommended collection-genericextra
     collection-genericrecommended collection-langgreek collection-latex
     collection-latexextra collection-latexrecommended collection-metapost
     collection-publishers collection-science collection-xetex xindy glossaries;
    })

[vcunat]

The upstream machinery for creating symlinks have changed apparently, and now it's overwriting valid links to extra executables by invalid ones (and dead links are auto-cleaned by our code later).

[avnik]

@vcunat I have at least one hash mismatch during build process

output path ‘/nix/store/ck5gjbj2zf85bpjrw4vwm0n9cfn456sg-koma-script.tar.xz’ has sha512 hash ‘2lbvzgifrpfkmhvv9pcqpv4mcr3sdb4iv99qmzgwb6lagli0l8c93nzxviw2lclyy0gb69i264wcbs4nl4ax7zsq3a3n4mcjimnlcm2’ when ‘2pbcb0jwlfy76z673sxapb3gaca4hd6i15q7865gh45vkgjvv01x18xps1f15micfakzsccync0wl7mj1x2a0rmb0i006lnqrscs5if’ was expected

I still curious, why hashes in CTAN changes

[jokogr]

@avnik that's because we are using a snapshot of the up-to-date TeX Live tree, where only the latest packages are available. I don't believe it's NixOS fault, perhaps we need to contact with upstream developers, so that they maintain version numbers in their packaging.

See also what @vcunat wrote above:

That hash error was likely due to some transient network error, if it disappeared (or some mirrors serve a newer version of that file already).

[vcunat]

@avnik: upstream continuously updates packages in-place. Some yearly snapshots are there but badly accessible (without mirroring). (I'm sorry, I didn't fully read the reply above that says the same.)

[vcunat]

BTW, I did ask upstream (tldistro ML), and even the yearly snapshots seem too large for them to mirror. Distributions are expected to do so themselves.

[avnik]

@vcunat: I see that they use SVN internally, can fetchsvn give us more stable result? Or may be special downloader for TL stuff, derived from tlmgr?

Another option is returning back to big tarball based snapshots, but use them as unpacked/fixed-output, and populate with symlinks.

[vcunat]

The big tarballs have the same release timing and mirroring as the small ones. We do fix output of the unpacked packages (all the thousands of them). AFAIK the svn is only for the infrastructure like the common binaries and not for the hundreds of megabytes (compressed) of packages.

[avnik]

At least tags and branches have texmf-dist http://tug.org/svn/texlive/tags/texlive-2016.0/Master/texmf-dist/ so it have all "TL packages" stuff

[avnik]

Also everything in trunk as well -- http://tug.org/svn/texlive/trunk/Master/texmf-dist/

[vcunat]

I see. We could in theory extract individual packages from that with texlive.tlpdb as that contains file lists...

But anyway, I don't see what we could get. I don't think there's a way to fetch efficiently individual files at individual revisions from svn (http to to web frontend might be the best shot) and it's likely without a mirror as well. And, from later #10026 discussion it's apparently not a problem for us to mirror snapshots of all the compressed packages (and e.g. update them yearly)... maybe not for docs as that's ~1.3 GB but the rest is 0.7 GB.

[avnik]

Current way is enough good, as long as files not disappears/changed under foots.

[vcunat]

@jokogr: xe(la)tex and other bin/ links should be fine now, as in 2015. Thanks for noticing the problem.

[vcunat]

I also uploaded my snapshot to a temporary mirror, so you should no longer get hash mismatches.

[jokogr]

@vcunat thanks, my project started compiling... and xindy is broken again!

% xindy -V
xindy release: 2.5.1
xindy script version: 1.18
xindy: Could not execute xindy kernel: No such file or directory

I think the TeX Live maintainers changed the way xindy is packaged, they don't compile it at all. So, instead of patching xindy-?.?.?/user-commands/xindy.in for system's clisp, could you please patch directly xindy.pl? I think this will solve it.

Could you also let me know how I could get into the installation environment so that I can help you more directly, with some patches? I mean a command like nix-shell, but for texlive-new packages.

Many thanks again for your efforts so far!

[vcunat]

@jokogr: xindy fixed up now. For quick building of tex-envs I use commands like:

$ nix-build -E 'with import ./. {}; texlive.combine { inherit (texlive) scheme-small xindy; }' -Q
$ ./result/bin/xindy -V

while being in the root of a nixpkgs repo checkout. You can import <nixpkgs> {} instead to use the default location (given by $NIX_PATH etc.).

[domenkozar]

Here's the diff I had to do to get the project building again:

diff --git a/src/doc/default.nix b/src/doc/default.nix
index 673d319..e3529f8 100644
--- a/src/doc/default.nix
+++ b/src/doc/default.nix
@@ -13,7 +13,7 @@ stdenv.mkDerivation rec {

   buildInputs = [ ditaa pandoc git
    (texlive.combine {
-      inherit (texlive) scheme-small luatex luatexbase sectsty titlesec cprotect bigfoot titling droid;
+      inherit (texlive) scheme-small luatex luatexbase sectsty titlesec cprotect bigfoot titling droid luatex-def ctablestack;
     })
   ];

If we document such changes will be needed, +1 to merge this.

[vcunat]

I just found it... you could use just collection-luatex. This part seems to be changing upstream and now it's different again from our 2016 version.

[expipiplus1]

This worked for all my tex stuff (a few letters, CVs and some pandoc-generated documents) without a problem.

[jokogr]

It has also worked with my PhD thesis and a couple of publications (IEEE
templates etc.)

On Tue, 6 Sep 2016 at 14:09 Joe Hermaszewski notifications@github.com
wrote:

This worked for all my tex stuff (a few letters, CVs and some
pandoc-generated documents) without a problem.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
#16391 (comment), or mute
the thread
https://github.com/notifications/unsubscribe-auth/ABMcw1x7iL43iVj005Ac2AMU1M8fNfhWks5qnUoCgaJpZM4I6lEB
.

[edolstra]

@vcunat I've run

./maintainers/scripts/copy-tarballs.pl --expr '(import ./. {}).texlive'

but that didn't find many texlive tarballs (only http://lipa.ms.mff.cuni.cz/~cunav5am/nix/texlive-2016/texlive-20160523b-source.tar.xz)...

[vcunat]

@edolstra: I'm sorry, I purged the hashes in this PR in the meantime (perhaps a mistake to push it here). It should work on the previous refs, e.g. 3be36a5:

$ env AWS_ACCESS_KEY_ID=foo AWS_SECRET_ACCESS_KEY=bar maintainers/scripts/copy-tarballs.pl --dry-run --expr '(import ./. {}).texlive'
evaluation returned 7226 tarballs
Net::Amazon::S3: Amazon responded with 403 Forbidden

[vcunat]

Oh, but that command will make them hash-addressed, won't it? I wanted to avoid those 7k additional hashes in nixpkgs, but if you want them hash-addressed, I see no better way than me adding those hashes afterwards.

[edolstra]

I'm probably missing something here. What exactly is the problem here? Note that if the texlive files are not fetched via regular fetchurl calls, them it's pointless to mirror them on tarballs.nixos.org.

[vcunat]

The problem is that 7k sha512 hashes would take quite a lot of space in nixpkgs (they were in base16 originally, adding ~1 MB). Currently the implementation uses something similar to fetchzip, because I wanted the unpacked contents to be fixed-output (as with 2015).

@edolstra: from my point of view it would work well to have http://tarballs.nixos.org/texlive-2016/${name}.tar.xz, i.e. "copy the directory as it is". I see a couple of similar references in nixpkgs, but if you only want to support content-addressing for this, I might e.g. add a set of sha1 in base32.

[vcunat]

I staged this. We can resolve mirroring a bit later, I suppose. Hashes of the outputs shouldn't change anyway.

[cmfwyp]

@vcunat

KOMA-Script is broken in the version (3.20) currently on nixos-unstable, but the issue was fixed in 3.21 so it can be fixed by upgrading koma-script to 3.21 in Nixpkgs. I updated fixedHashes.nix and pkgs.nix, and tried rebuilding only to realize the package gets downloaded from http://lipa.ms.mff.cuni.cz/~cunav5am/nix/texlive-2016/koma-script.tar.xz, which is 3.20.

[vcunat]

I just took what was in upstream TeX Live 2016; we can individually update such problematic packages, but I don't think I can get to it within a week.

[domenkozar]

@vcunat can I ask you to write a changelog entry for this PR?

[jokogr]

@vcunat could you also post how one could update manually TeXLive packages? In addition to what @cmfwyp did, I have added a url attribute in the koma-script of pkgs.nix, but still no luck.

[vcunat]

It's just a version update, but I can write a simple entry into release notes.

It isn't possible to "just update", because there is no stable location where to download from, at least no in upstream.

[jokogr]

@vcunat e.g., if I download from http://ftp.ntua.gr/mirror/ctan/systems/texlive/tlnet/archive/koma-script.tar.xz, it currently has v3.21 as seen in the extracted tlpkg/tlpobj/koma-script.tlpobj file (catalogue-version 3.21). How to make nix compile this one?

[cmfwyp]

@jokogr My understanding is that we cannot do that because that URL will become version 3.22 as soon as it is released, and then our build will be broken.

Are we able to update the archives on http://lipa.ms.mff.cuni.cz/~cunav5am/nix/texlive-2016 to a newer version? If so, there's the solution.

[jokogr]

@cmfwyp I know that's the case, upstream URLs of future versions are going to be identical to the previous ones.

Supposing that I host the updated package on my webserver, so I have a stable location for this version, my question was how to update the respective nix expressions, so that only this package gets updated.