texlive: mirroring is bad

[utdemir]

I can not build dblatex(a dependency of nix) on nixpkgs master. It can't download some required archives. Changing mirror to ftp.math.utah.edu in pkgs/tools/typesetting/tex/texlive-new/default.nix also doesn't solve the problem.

It's not a specific dependency, on every run, a different dependency fails(depends on download order).

Steps to reproduce:

• nix-shell -I ~/nixpkgs -p dblatex on nixpkgs/master

trying ftp://tug.ctan.org/pub/tex/historic/systems/texlive/2015/tlnet-final/archive/hyph-utf8.tar.xz

trying ftp://tug.ctan.org/pub/tex/historic/systems/texlive/2015/tlnet-final/archive/gsftopk.tar.xz
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0  % Total    % Received % Xferd  Average Speed   Time    Tim
e     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0  % Total    % Received % Xferd  Average Speed   Time    Tim
e     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0  % Total    % Received % Xferd  Average Speed   Time    Tim
e     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
trying ftp://tug.ctan.org/pub/tex/historic/systems/texlive/2015/tlnet-final/archive/glyphlist.tar.xz
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:--  0:00:03 --:--:--     0curl: (67) Access denied: 530
error: cannot download hyph-utf8.tar.xz from any mirror
builder for ‘/nix/store/w9rz6n1wrixby1mycs6fck0sywj8vz71-hyph-utf8.tar.xz.drv’ failed with exit code 1
building path(s) ‘/nix/store/4l2xs6chhhzsa23g796ypbm2734lv16v-hyphen-base.tar.xz’
cannot build derivation ‘/nix/store/v7hq3p4pqgy2b1g2knhy448l5hwn73sq-texlive-hyph-utf8-687.drv’: 1 dependencies couldn't be built
building path(s) ‘/nix/store/nmnggnvhi43z4mzh3v7m2jd4xa9227y1-texlive-scripts.tar.xz’
building path(s) ‘/nix/store/swaad1cc905fh3l5lb9hid74d7f1hzxc-times.tar.xz’
building path(s) ‘/nix/store/d9pvb1fvdm6l7ax8w3760calnz21f4dp-titlesec.tar.xz’
building path(s) ‘/nix/store/2fmwf45srfqi3bgg6629k7ns0062iq34-tools.tar.xz’
building path(s) ‘/nix/store/2icg1a4x23bs7w5y63rbxbcq09rl7m05-url.tar.xz’
building path(s) ‘/nix/store/wcqf6cdilc55v4bxlwf531f0mnik1r1x-wasysym.tar.xz’
building path(s) ‘/nix/store/22yfj7scwl6a9vvzk5i8k8x83252aiih-zapfding.tar.xz’
cannot build derivation ‘/nix/store/byhr95n2lq68pciq3npiqcf5xma20rfk-texlive-combined-2015.drv’: 1 dependencies couldn't be built
error: build of ‘/nix/store/byhr95n2lq68pciq3npiqcf5xma20rfk-texlive-combined-2015.drv’ failed
/run/current-system/sw/bin/nix-shell: failed to build all dependencies

[peti]

I just built dblatex on 0d0be13, and it downloaded and compiled everything just fine:

$ nix-build --no-out-link -A dblatex
/nix/store/wxmw9fmg8sxq3wvph3bmbyrik396hwzm-dblatex-0.3.7

[utdemir]

It was giving me "Access denied", a little investigation revealed that some DOS protection mechanism on the servers was getting in my way. It's also weird that the two mirrors I tried had that exact behavior. Probably some default configuration on their FTP servers.

Giving -j 1(build-max-jobs) fixed the problem.

This is rather annoying that increasing build-max-jobs breaks something irrelevant like this, probably an option like "max. number of parallel requests per host" would be useful, but given the current implementation this is pretty hard to implement(I think).

I'm closing this issue, thanks @peti .

[utdemir]

Or, I'll just rephrase the issue as "Can't compile dblatex when build-max-jobs is 24".

Can we do anything about this?

[vcunat]

As I noted in #287, we would have to mirror it ourselves. TAN only decently mirrors the latest TeX Live packages, not yearly releases.

[vcunat]

• I now tested and both upstream URLs resolve to the same IP (v4), so switching is meaningless.
• Dependencies and dblatex itself should get into binary cache soon – you reported just hour or two after I pushed it to master 9bd0bac.

[vcunat]

probably an option like "max. number of parallel requests per host" would be useful

Maybe, but not all hosts behave the same, e.g. for cache.nixos.org we do want to utilize many more parallel requests. (At least for checking whether paths are in the cache, though there is a separate setting for that already, binary-caches-parallel-connections = 150.)

[vcunat]

Now the generic URL doesn't work anymore but http://ftp.math.utah.edu still does (pushed commits to use that). I'm afraid they might be taking action as a result of nixpkgs causing lots of downloads from them, which might end up in this sole source being cut away in future.

I believe I still have the full archive on some computer if that happens (in nix store, after I was generating the hashes after unpacking).

[Mathnerd314]

We could just update the package list daily-ish and use the CTAN mirror system; TeXLive holds back most of the updates to core packages, so at least the minimal set will work on old channel versions (and probably more).
One commit per day wouldn't be any more often than Haskell: https://github.com/NixOS/nixpkgs/commits/master?author=peti

[vcunat]

I would prefer to avoid updating faster than once a year, perhaps except some important bug fixes.

Well, the core stuff should be mirrored by us already. I think the critical point was between committing and getting to channels (and hashed-mirror, perhaps), so we might be good in that already, but I haven't really checked, I think. There's also the advantage that the unpacked stuff is made fixed-output, so updates are no-op even if stdenv changes (except for the binaries, but those are small and from a single tarball).

[utdemir]

It occured to me again:

curl: (7) Failed to connect to tug.org port 21: Connection timed out
error: cannot download texlive-20150521-source.tar.xz from any mirror

[vcunat]

No working mirror anymore

Now the situation got much worse. The upstream package archive was recently overwritten by the final state of TeX Live 2015, so there is nowhere to download from, except for the packages that are in our binary cache (mainly the closure of texlive.combined.scheme-small). People will be getting broken hashes.

I suppose we would best import it all into our binary cache. Upstream just mirrors the daily-changing state and expects distros to solve anything else on their own. Even the unpacked packages are fixed-output derivations (by sha1); I found locally all packages required for running but not those with documentation or sources.

@edolstra: is it acceptable to mirror these at least? They're not too big:

$ nix-push --dest ~/texlive-cache/ (nix-build -Q -E 'with import ./. {}; lib.filter (p: p.tlType == "run") texlive.scheme-full.pkgs')
[...]
total compressed size 589.87 MiB, 34.4%

It's likely other users have the other types of packages as well. We all know the fixed hashes, so trust shouldn't be an issue in this step...

Any other ideas?

[edolstra]

Hm, I tried mirroring to the tarball cache, but copy-tarballs.pl can't find any files to mirror:

$ DRY_RUN=1 ./maintainers/scripts/copy-tarballs.pl --expr 'with import ./. {}; lib.filter (p: p.tlType == "run") texlive.scheme-full.pkgs'
evaluation returned 0 tarballs
mirrored 0 files, already have 0 files

Uploading to the binary cache won't work very well, because the tarballs will probably get garbage-collected (not being referenced by a release).

[vcunat]

The expression assumed you're in a nixpkgs tree. I assume that's the problem. Otherwise you can substitute with import <nixpkgs> {};.

[edolstra]

The actual problem appears to be that the unpackPkg derivations don't include the source tarball as a derivation attribute, so find-tarballs can't find them.

[vcunat]

Ah, well, I don't/didn't know how copy-tarballs.pl works. These would go to tarballs.nixos.org that is only checked by fetchurl and not during substitution, right? I now checked and I don't even have all these tarballs, only the unpacked and pruned /nix/store outputs.

[vcunat]

Anyway, if someone has the tarballs, I composed:

• list of tarballs for files needed during runtime, and
• list of all tarballs, i.e. including also documentation and source packages.

These are fixed-output by their md5.

[vcunat]

@edolstra: do you have those tarballs and can upload them? Or should we be searching for someone who has them?

[edolstra]

No, I don't think so. I checked the old Hydra server (lucifer), hoping it still has them, but that's apparently not the case.

[vcunat]

If noone here has a better idea, I suppose I'll post a question on the mailing-list to find if someone still has the original tarballs, as uploading them seems the easiest way to fix stuff including all old branches. (EDITed: readability)

[peti]

I have the following ~3800 tarballs available at hydra.cryp.to: http://pastebin.com/Q8SEyr5P. It's not much, but I hope it helpls.

[edolstra]

@peti Thanks! I'll try to use those.

[edolstra]

@peti Trying to use hydra.cryp.to as binary cache gives:

Caught exception in Hydra::View::NARInfo->process "cannot open file ‘/etc/nix/hydra.cryp.to-1/secret’: Permission denied at /nix/store/5rrmjacsls56gjlgm0ydx85kbksnxfva-hydra-perl-deps/lib/perl5/site_perl/5.20.3/x86_64-linux-thread-multi/Nix/Utils.pm line 37."

[peti]

Oops. I broke access to the key file when I fixed the uid collision between hydra and nscd a few weeks ago. It's fixed now.

[domenkozar]

I'm also hitting this on 16.03, are there any workarounds until we have mirrored the old tarballs?

[michalrus]

This is really unfortunate. 😿

[peti]

@namore, please do.

[sorpaas]

@namore, can you share the commit? I'm in need of installing the texlive packages, which is apparently broken now.

[namore]

@sorpaas, @peti, I pushed the commit to my fork [1]. I'm not sure if it contains all missing hashes, but the changes allow me to build texlive.combined.scheme-full.
[1] [https://github.com/namore/nixpkgs/commit/a07b7bb2637a97facae55cf8837c2a27ce1892ed]

[peti]

I've committed a (slightly modified) patch in 7abe192. These hashes get out of sync fast! I just hope that our source tarball mirroring works now or we'll have the exact same problem again in a couple of days.

[vcunat]

I just hope that our source tarball mirroring works now

I don't think it works in this case: #10026 (comment)

BTW, over the long term, we might better use a combination of package versions as close as possible to what's released upstream, instead of a snapshot at some random moment.

[sorpaas]

Hi @namore, can you share the code you used to generate fixedHashes.nix?

[vcunat]

Note: the source files in master contain shell fragments I used to get all the generated stuff. They're in comments at the point where the generated file is included from the rest.

[namore]

@vcunat, I could not successfully use the commands in the comments and gave up trying after a while. I had the impression that they were out of date, but I may be wrong.
@sorpaas, the script I used to fix the hashes is as follows:

#!/bin/sh
set -u
set -e

function fh { sed -i "s/$1/$2/" pkgs.nix; }
function fh2 { sed -i "s/$1/$2/" fixedHashes.nix; }

TMP=`mktemp`
set +e
nix-env -f ~/nixpkgs --keep-going -i -A texlive.combined.scheme-full > $TMP 2>&1
cat $TMP >&2
set -e
grep '^output path.*md5 hash' $TMP |\
    awk '{ print $9 " " $7 }'|\
    sed -r 's/[‘’]//g' |\
    tee /dev/stderr |\
    while read f; do
        fh $f
    done
grep '^output path.*r:sha1 hash' $TMP |\
    awk '{ print $9 " " $7 }'|\
    sed -r 's/[‘’]//g' |\
    tee /dev/stderr |\
    while read f; do
        fh2 $f
    done
rm -f $TMP

I ran the script a few times until all hash related errors were resolved.

[vcunat]

Well, yes, it is possible they changed the format of tlpdb.

[vcunat]

Interested parties can pre-test texlive-2016: #16391

[vcunat]

Let's close this texlive 2015 discussion. Problems around mirroring the current 2016 version can be discussed on #24683.