Build Input Caching

[mguentner]

Beside distributing binary files, the build inputs of a derivation could be cached in IPFS.
That would speed up CI quite a bit since the cache could move "closer" to the build instance (using an IPFS Gateway that is within the local network).

Current Status:
@knupfer created fetchIPFS: NixOS/nix#859 (comment)
@CMCDragonkai is doing something similiar @ https://github.com/MatrixAI/Forge-Package-Archiving
NAR replacement discussion: NixOS/nix#1006

There are different methods on how to approach this:

• put build inputs into IPFS and make a sha256:IPFS mapping inside nixpkgs (a huge set)
  Migration: easy (can be solved by a script that builds all /nix/store/ paths and puts them into IPFS)
  Support: every fetch derivation needs to be patched to look up the IPFS hash first, either fetch it from the local running daemon or a gateway. If no hash is found each derivation falls back to its normal
  operation (curl, git, svn etc.)
• append the IPFS hash to each package (@knupfer's idea).
  Migration: takes a lot of effort (each derivation must be touched)
  Support: already implemented, however it must be added manually to each package

Should we produce an intermediate cache format (tar? -> can be chunked efficiently in IPFS) that is used to cache each src? This can then be used within each fetch derivation. As a big plus, the build inputs are deduplicated and share blocks with the official package.
However it would be better just to reuse an already archived src (like tar.*, .zip etc.) and not to archive it again -> We need some form of manifest -> We need IPLD

[CMCDragonkai]

We are currently in the midst of going through ipfs internals (specifically libp2p) in order to create a haskell implementation and have a deeper integration of build input source hashes and ipfs architecture.

[CMCDragonkai]

MatrixAI/Forge-Package-Archiving#1

[CMCDragonkai]

The biggest benefit is the ability to version srcs that are currently not versionable. Loads of packages distributors don't keep historical versions.

[CMCDragonkai]

We're halfway through implementing the haskell multiaddress library: https://github.com/MatrixAI/haskell-multiaddr (see improved branch). Still to do - binary encoding/decoding.

Perhaps we should meetup and compare notes?

[mguentner]

This script prints all tarballs for an expression:

nix-instantiate --eval --json --strict maintainers/scripts/find-tarballs.nix --arg expr '(import <nixpkgs> {}).hello'

there is also maintainers/scripts/copy-tarballs.pl that uses it to upload to S3

[mguentner]

First implementation: https://github.com/NixIPFS/nixipfs-scripts/blob/master/nixipfs/src/mirror_tarballs.py