Make mdBook's output reproducible/deterministic

[kzys]

It was originally reported as rust-lang/rust#24473 and it has been fixed. However Debian's tests.reproducible-builds.org still shows some nondeterminism in searchindex.js, which is generated by mdBook.

https://tests.reproducible-builds.org/debian/rb-pkg/buster/amd64/diffoscope-results/rustc.html

[Michael-F-Bryan]

Hmm... this comment makes it sound like the generated searchindex.js should be stable and just skimming through the code nothing jumps out as being non-deterministic. There are a couple HashMap and HashSets used for the ammonia HTML sanitizer, but they shouldn't affect the output in anyway. this footnote_numbers variable also uses a HashMap, but it's populated with numeric keys which are generated by incrementing a counter as we scan through the document.

Are there any easy ways to test for non-determinism without having to go through the full Debian reproducibility process?

[kzys]

Generate docs multiple times? Actually I haven't tried to reproduce the issue on my local machine yet.

[Michael-F-Bryan]

I can't seem to reproduce the issue on my machine. Using master (ced74ca) I tried to generate the mdbook user guide a couple times using git to check for any changes, but it seems to be stable... Could it perhaps be something to do with rebuilding the same thing on different machines?

I even got the same SHA1 for searchindex.js (and searchindex.json) when running master and the most recently published mdbook version,0.2.1...

$ cd ~/Documents/forks/mdBook/book-example
$ cargo run -- --version
    Finished dev [unoptimized + debuginfo] target(s) in 0.12s                                                                                     
     Running `/home/michael/Documents/forks/mdBook/target/debug/mdbook --version`
mdBook v0.2.2-alpha.0
$ cargo run -- build
    Finished dev [unoptimized + debuginfo] target(s) in 0.12s                                                                                     
     Running `/home/michael/Documents/forks/mdBook/target/debug/mdbook build`
2018-09-27 14:59:39 [INFO] (mdbook::book): Book building has started
2018-09-27 14:59:39 [INFO] (mdbook::book): Running the html backend
$ cd book
$ git init . && git add . && git commit -m "initial commit"
$ cd ..
$ cp -r book /tmp
$ cargo run -- build
$ cp -r book/* /tmp/book
$ cd /tmp/book
$ git status
On branch master
nothing to commit, working tree clean
$ sha1sum searchindex.js
192ecaa3cc22745ae87de0296d1724427c82da37  searchindex.js

$ cd ~/Documents/forks/mdBook/book-example
$ rm -r book
$ mdbook --version 
mdBook v0.2.1
$ mdbook build
2018-09-27 15:07:44 [INFO] (mdbook::book): Book building has started
2018-09-27 15:07:44 [INFO] (mdbook::book): Running the html backend
$ sha1sum book/searchindex.js
192ecaa3cc22745ae87de0296d1724427c82da37  book/searchindex.js

[mnd]

@Michael-F-Bryan Looks like Debian tests was made on 1.28.0 rust release that was delivered without #692 changes.
I trying to solve same reproducibility issue for rust 1.28.0 in package for GuixSD and I'll update rust-lang/rust#24473 if #692 will fix situation.

[kzys]

Thanks @mnd. I will close the issue then.

[pietroalbini]

Closing as requested by @XAMPPRocky