rewrite rustup to protect against truncation errors #19170

erickt · 2014-11-21T04:30:46Z

This closes #19168.

Please be careful reviewing this since this gets used all over the place. I've tested all the options and everything appears to be working though.

aturon · 2014-11-21T06:01:08Z

cc @brson @alexcrichton

alexcrichton · 2014-11-21T06:54:48Z

src/etc/rustup.sh

 then
-        MAYBE_PREFIX="--prefix=${CFG_PREFIX}"
+    CFG_NIGHTLY=`date "+%Y-%m-%d"`


I think we should still default to downloading from rust-nightly-... because we're not always guaranteed to have nightlies for each date (they could fail to sync).

or... I may not know how to read bash. So let me confirm instead! This doesn't download from a specific date with no flags specified, right?

Yes, when no flag is specified it downloads the current nightly, which does not include the date in the url.

I think this sets the cache dir based on the local system's date? If so, won't it fail to update if a new nightly is available on the server, but the date hasn't yet changed on my computer (e.g. if I'm in a different timezone)?

I agree with this comment. It is very likely the dates will get out of sync. I'd suggest that if a specific date isn't provided that either a) --save has no effect, b) it is stored directly in the .rustup folder and the .sha256 file (which exists for all tarballs) is d/l'd and used to determine if it is fresh before using.

Alternately, it could be stored in the dated folder, but all cached installs could be checksummed from the network before installing. Might be a slightly simpler arrangement.

Oh, hm. It looks like because of https://github.com/erickt/rust/blob/9afe5de38d279a0cc127b038c072e849adaffe18/src/etc/rustup.sh#L495, if --save is selected then the script tries to download from the archives using today's date. That solves the problem of the dates being out of sync, but adds the problem that today may not have a nightly yet. I at least like this better since --save is not the default, but it's a bit wonky.

(My original comment may have been confusing since I didn't notice this).

alexcrichton · 2014-11-21T21:17:31Z

Just confirming, but you've run this multiple times in a few flavorful configurations, to make sure it works, right?

alexcrichton · 2014-11-21T21:17:48Z

also otherwise looks good to me, r? @brson (good to have two sets of eyes on shell scripts)

alexcrichton · 2014-11-21T21:19:50Z

Hm I checked this out locally and got this error:

$ curl -O https://raw.githubusercontent.com/erickt/rust/rustup/src/etc/rustup.sh
$ chmod +x rustup.sh
$ ./rustup.sh
rustup: CFG_CURL             := /usr/bin/curl (7.35.0)
rustup: CFG_TAR              := /bin/tar (1.27.1)
rustup: 
rustup: processing ./rustup.sh args
rustup: 
rustup: CFG_PREFIX           :=  
rustup: CFG_NIGHTLY          :=  
rustup: 
rustup: validating ./rustup.sh args
rustup: 
rustup: host triple: x86_64-unknown-linux-gnu
rustup: downloading https://static.rust-lang.org/dist/rust-nightly-x86_64-unknown-linux-gnu.tar.gz to ./rustup-tmp-install/rust-nightly-x86_64-unknown-linux-gnu.tar.gz
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100  105M  100  105M    0     0  43.9M      0  0:00:02  0:00:02 --:--:-- 43.9M
rustup: Extracting ./rustup-tmp-install/rust-nightly-x86_64-unknown-linux-gnu.tar.gz
/bin/tar: ./rustup-tmp-install/rust-nightly-x86_64-unknown-linux-gnu.tar.gz: Cannot open: No such file or directory
/bin/tar: Error is not recoverable: exiting now
rustup: error: failed to unpack installer

brson · 2014-11-25T17:45:54Z

This change appears to ignore the --disable-cargo flag. This flag may not be worth supporting since in the future we're going to be installing both from the same tarball, but it should probably not just be ignored.

The old script did all downloading before all installing, which seems appropriate to me: get the fallable network business done first. I think I'd prefer to preserve that behavior.

What does it mean to 'protect against truncation'? Is it that everything is wrapped in a function call, so the entire function has to be downloaded before anything in the script can be run? If so please spell that out in comments so I don't forget.

I presume the use-case for --save is projects that are pinning to a specific nightly and expect to be switching between branches frequently.

In anticipation of release channels we might want to futureproof the name of the --nightly flag. There will probably end up being a --channel=nightly|beta|stable flag, and it will be possible to pull the archives from any of those channels. I might suggest --date as the name of this flag instead of --nightly.

erickt · 2014-12-01T06:02:58Z

shell can be confusing to read, so I refactored the code into a bunch of small, incremental changes. I've also pulled out the caching behavior for a future PR so that we can land a safer rustup.sh sooner rather than later. In addition, I fixed the problem @alexcrichton ran into, and added better documentation as requested by @brson.

alexcrichton · 2014-12-03T07:25:34Z

@erickt hm I still get an odd error locally, maybe a different one?

[~] $ curl https://raw.githubusercontent.com/erickt/rust/rustup/src/etc/rustup.sh | sh 
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 10860  100 10860    0     0   205k      0 --:--:-- --:--:-- --:--:--  207k
rustup: CFG_CURL             := /usr/bin/curl (7.35.0)
rustup: CFG_TAR              := /bin/tar (1.27.1)
rustup: 
rustup: processing sh args
rustup: 
rustup: CFG_PREFIX           :=  
rustup: 
rustup: validating sh args
rustup: 
rustup: host triple: x86_64-unknown-linux-gnu
rustup: downloading https://static.rust-lang.org/dist/rust-nightly-x86_64-unknown-linux-gnu.tar.gz to ./rustup-tmp-install/rust-nightly-x86_64-unknown-linux-gnu.tar.gz
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100  104M  100  104M    0     0  61.7M      0  0:00:01  0:00:01 --:--:-- 61.7M
rustup: downloading https://static.rust-lang.org/cargo-dist/cargo-nightly-x86_64-unknown-linux-gnu.tar.gz to ./rustup-tmp-install/cargo-nightly-x86_64-unknown-linux-gnu.tar.gz
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 4642k  100 4642k    0     0  30.4M      0 --:--:-- --:--:-- --:--:-- 30.6M
rustup: Extracting rust-nightly-x86_64-unknown-linux-gnu.tar.gz
/bin/tar: ./rustup-tmp-install/rust-nightly-x86_64-unknown-linux-gnu.tar.gz: Cannot open: No such file or directory
/bin/tar: Error is not recoverable: exiting now
rustup: error: failed to unpack installer

This closes rust-lang#19168. It's possible that if the downloading of `rustup.sh` is interrupted, bad things could happen, such as running a naked "rm -rf /" instead of "rm -rf /path/to/tmpdir". This wraps rustup.sh's functionality in a function that gets called at the last time that should protect us from these truncation errors.

nodakai · 2014-12-04T02:16:35Z

static.rust-lang.org should publicize MD5 or SHA* checksums and that should be the final solution.

thestinger · 2014-12-04T02:21:50Z

It should have GPG signatures for nightlies just as it does for the tagged releases.

thestinger · 2014-12-04T02:22:12Z

There can be a separate key for whatever automated builder is responsible for making them.

erickt · 2014-12-04T02:45:52Z

Nightlies do have a .sha256 hash file along side of the tar balls. I've got a patch queued up that checks the hash, I just had enough trouble with this PR I wanted land this first before I do the next round of patches.

This closes #19168. Please be careful reviewing this since this gets used all over the place. I've tested all the options and everything appears to be working though.

minor: Update lockfile

alexcrichton reviewed Nov 21, 2014
View reviewed changes

huonw mentioned this pull request Nov 23, 2014

Moved block of code into a function #19226

Closed

erickt force-pushed the rustup branch from 9afe5de to 105b1da Compare December 1, 2014 06:00

erickt changed the title ~~rewrite rustup to support caching nightlies, protects against truncation errors~~ rewrite rustup to protect against truncation errors Dec 1, 2014

erickt force-pushed the rustup branch 3 times, most recently from 7b02098 to 76fbf57 Compare December 1, 2014 07:49

brson mentioned this pull request Dec 3, 2014

Use mktemp for temporary download directory #19227

Merged

erickt added 11 commits December 3, 2014 14:40

whitespace cleanup

c7ae4a9

rustup: make rustup executable

3b4ad72

rustup: probe for the existance of tar

98f01f1

rustup: add a RUST_ prefix to the rust-specific variables

e688eb3

rustup: rename TMP_DIR to CFG_TMP_DIR

fcb31e8

rustup: factor out the install flags into a CFG_INSTALL_FLAGS variable

4f65d97

rustup: factor out downloading and extracting the snapshot tarballs

b388dc6

rustup: factor out installing packages into a function

8ca8e6f

rustup: extract the tarballs as part of installation

694500b

rustup: simplify downloading packages

f867379

erickt force-pushed the rustup branch from c5b6961 to f867379 Compare December 3, 2014 23:23

bors added a commit that referenced this pull request Dec 4, 2014

auto merge of #19170 : erickt/rust/rustup, r=erickt

4053e82

This closes #19168. Please be careful reviewing this since this gets used all over the place. I've tested all the options and everything appears to be working though.

bors closed this Dec 4, 2014

bors merged commit f867379 into rust-lang:master Dec 4, 2014

lnicola added a commit to lnicola/rust that referenced this pull request Feb 17, 2025

Merge pull request rust-lang#19170 from lnicola/update-lockfile

e08736d

minor: Update lockfile

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

rewrite rustup to protect against truncation errors #19170

rewrite rustup to protect against truncation errors #19170

erickt commented Nov 21, 2014

aturon commented Nov 21, 2014

alexcrichton Nov 21, 2014

alexcrichton Nov 21, 2014

erickt Nov 21, 2014

zofrex Nov 21, 2014

brson Nov 25, 2014

brson Nov 25, 2014

brson Nov 25, 2014

alexcrichton commented Nov 21, 2014

alexcrichton commented Nov 21, 2014

alexcrichton commented Nov 21, 2014

brson commented Nov 25, 2014

erickt commented Dec 1, 2014

alexcrichton commented Dec 3, 2014

nodakai commented Dec 4, 2014

thestinger commented Dec 4, 2014

thestinger commented Dec 4, 2014

erickt commented Dec 4, 2014

rewrite rustup to protect against truncation errors #19170

rewrite rustup to protect against truncation errors #19170

Conversation

erickt commented Nov 21, 2014

aturon commented Nov 21, 2014

alexcrichton Nov 21, 2014

Choose a reason for hiding this comment

alexcrichton Nov 21, 2014

Choose a reason for hiding this comment

erickt Nov 21, 2014

Choose a reason for hiding this comment

zofrex Nov 21, 2014

Choose a reason for hiding this comment

brson Nov 25, 2014

Choose a reason for hiding this comment

brson Nov 25, 2014

Choose a reason for hiding this comment

brson Nov 25, 2014

Choose a reason for hiding this comment

alexcrichton commented Nov 21, 2014

alexcrichton commented Nov 21, 2014

alexcrichton commented Nov 21, 2014

brson commented Nov 25, 2014

erickt commented Dec 1, 2014

alexcrichton commented Dec 3, 2014

nodakai commented Dec 4, 2014

thestinger commented Dec 4, 2014

thestinger commented Dec 4, 2014

erickt commented Dec 4, 2014