Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Permanent build failure: Unpacking kubernetes release ... gzip: stdin: not in gzip format #990

Closed
ghost opened this issue Nov 4, 2016 · 8 comments · Fixed by kubernetes/release#195
Closed

Permanent build failure: Unpacking kubernetes release ... gzip: stdin: not in gzip format #990

ghost opened this issue Nov 4, 2016 · 8 comments · Fixed by kubernetes/release#195
Assignees
@rmmh

Comments

@ghost
Copy link

ghost commented Nov 4, 2016

We've been seeing failures like the following in the federation e2e tests on Jenkins since around midnight last night (and on occasion yesterday too). Anyone know anything about this?

Unpacking kubernetes release v1.5.0-alpha.2.540+ccaba2ccd2231d

gzip: stdin: not in gzip format
tar: Child returned status 1
tar: Error is not recoverable: exiting now

Recent examples in gubernator:

https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/kubernetes-e2e-gce-federation/5475

https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/kubernetes-e2e-gce-federation/5465

cc @kubernetes/test-infra-admins @kubernetes/sig-testing

Q
@ghost ghost changed the title Build failure: Unpacking kubernetes release ... gzip: stdin: not in gzip format Permanent build failure: Unpacking kubernetes release ... gzip: stdin: not in gzip format Nov 4, 2016
@ixdy
Copy link
Member

ixdy commented Nov 4, 2016 

$ wget https://storage.googleapis.com/kubernetes-federation-release/ci/v1.5.0-alpha.2.540+ccaba2ccd2231d/kubernetes.tar.gz
--2016-11-04 11:31:39--  https://storage.googleapis.com/kubernetes-federation-release/ci/v1.5.0-alpha.2.540+ccaba2ccd2231d/kubernetes.tar.gz
Resolving storage.googleapis.com (storage.googleapis.com)... 2607:f8b0:4005:808::2010, 172.217.5.112
Connecting to storage.googleapis.com (storage.googleapis.com)|2607:f8b0:4005:808::2010|:443... connected.
HTTP request sent, awaiting response... 403 Forbidden
2016-11-04 11:31:39 ERROR 403: Forbidden.

Not sure why curl isn't making this failure more obvious.

@rmmh
Copy link
Contributor

rmmh commented Nov 4, 2016

https://github.com/kubernetes/kubernetes/blob/master/cluster/get-kube.sh#L206

If you run curl without the -f flag, it doesn't exit with nonzero if it talked to the server but got an error response code.

@ixdy
Copy link
Member

ixdy commented Nov 4, 2016

Right, that's what I thought I wanted, but I got confused by the manpage.

$ curl -f https://storage.googleapis.com/kubernetes-federation-release/ci/v1.5.0-alpha.2.540+ccaba2ccd2231d/kubernetes.tar.gz -o kubernetes.tar.gz
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
curl: (22) The requested URL returned error: 403 Forbidden

I'll update cluster/get-kube.sh. @quinton-hoole I'm not sure why the tarball is inaccessible. Did anyone change ACLs on the kubernetes-federation-release bucket recently?

@ixdy ixdy mentioned this issue Nov 4, 2016
Merged
@ghost
Copy link
Author

ghost commented Nov 4, 2016

Not I. Perhaps @colhom or @madhusudancs ?

@colhom
Copy link
Contributor

colhom commented Nov 4, 2016

So, interestingly enough fetching a tarball from a very recent successful build works fine- which leads me to believe it's not the ACLs on the bucket.

Investigating further.

@madhusudancs
Copy link
Contributor

@ixdy I don't think ACLs changed recently. Also, ACLs on the bucket looks very similar to the kubernetes-release-dev-jenkins

@rmmh
Copy link
Contributor

rmmh commented Nov 4, 2016 •
edited
Loading

I think the problem here is GCS's eventual consistency on bucket listing.

From https://cloud.google.com/storage/docs/consistency

If you upload a new object to a bucket, then immediately perform a list objects operation on the bucket, the new object might not appear in the returned list of objects. (The new object is still available for immediate use, even though it does not appear in the list of objects.)

If you look at federation-build 3285, you'll see the lines:

- Hashing files in gcs-stage/v1.5.0-alpha.2.540+ccaba2ccd2231d...
- Copying release artifacts to gs://kubernetes-federation-release/ci/v1.5.0-alpha.2.540+ccaba2ccd2231d: 
push-build.sh::release::gcs::copy_release_artifacts(): /usr/bin/gsutil -qm cp -r /var/lib/jenkins/workspace/kubernetes-federation-build/_output/gcs-stage/v1.5.0-alpha.2.540+ccaba2ccd2231d/bin /var/lib/jenkins/workspace/kubernetes-federation-build/_output/gcs-stage/v1.5.0-alpha.2.540+ccaba2ccd2231d/extra /var/lib/jenkins/workspace/kubernetes-federation-build/_output/gcs-stage/v1.5.0-alpha.2.540+ccaba2ccd2231d/kubernetes-client-linux-amd64.tar.gz /var/lib/jenkins/workspace/kubernetes-federation-build/_output/gcs-stage/v1.5.0-alpha.2.540+ccaba2ccd2231d/kubernetes-client-linux-amd64.tar.gz.md5 /var/lib/jenkins/workspace/kubernetes-federation-build/_output/gcs-stage/v1.5.0-alpha.2.540+ccaba2ccd2231d/kubernetes-client-linux-amd64.tar.gz.sha1 /var/lib/jenkins/workspace/kubernetes-federation-build/_output/gcs-stage/v1.5.0-alpha.2.540+ccaba2ccd2231d/kubernetes-manifests.tar.gz /var/lib/jenkins/workspace/kubernetes-federation-build/_output/gcs-stage/v1.5.0-alpha.2.540+ccaba2ccd2231d/kubernetes-manifests.tar.gz.md5 /var/lib/jenkins/workspace/kubernetes-federation-build/_output/gcs-stage/v1.5.0-alpha.2.540+ccaba2ccd2231d/kubernetes-manifests.tar.gz.sha1 /var/lib/jenkins/workspace/kubernetes-federation-build/_output/gcs-stage/v1.5.0-alpha.2.540+ccaba2ccd2231d/kubernetes-salt.tar.gz /var/lib/jenkins/workspace/kubernetes-federation-build/_output/gcs-stage/v1.5.0-alpha.2.540+ccaba2ccd2231d/kubernetes-salt.tar.gz.md5 /var/lib/jenkins/workspace/kubernetes-federation-build/_output/gcs-stage/v1.5.0-alpha.2.540+ccaba2ccd2231d/kubernetes-salt.tar.gz.sha1 /var/lib/jenkins/workspace/kubernetes-federation-build/_output/gcs-stage/v1.5.0-alpha.2.540+ccaba2ccd2231d/kubernetes-server-linux-amd64.tar.gz /var/lib/jenkins/workspace/kubernetes-federation-build/_output/gcs-stage/v1.5.0-alpha.2.540+ccaba2ccd2231d/kubernetes-server-linux-amd64.tar.gz.md5 /var/lib/jenkins/workspace/kubernetes-federation-build/_output/gcs-stage/v1.5.0-alpha.2.540+ccaba2ccd2231d/kubernetes-server-linux-amd64.tar.gz.sha1 /var/lib/jenkins/workspace/kubernetes-federation-build/_output/gcs-stage/v1.5.0-alpha.2.540+ccaba2ccd2231d/kubernetes-src.tar.gz /var/lib/jenkins/workspace/kubernetes-federation-build/_output/gcs-stage/v1.5.0-alpha.2.540+ccaba2ccd2231d/kubernetes-src.tar.gz.md5 /var/lib/jenkins/workspace/kubernetes-federation-build/_output/gcs-stage/v1.5.0-alpha.2.540+ccaba2ccd2231d/kubernetes-src.tar.gz.sha1 /var/lib/jenkins/workspace/kubernetes-federation-build/_output/gcs-stage/v1.5.0-alpha.2.540+ccaba2ccd2231d/kubernetes.tar.gz /var/lib/jenkins/workspace/kubernetes-federation-build/_output/gcs-stage/v1.5.0-alpha.2.540+ccaba2ccd2231d/kubernetes.tar.gz.md5 /var/lib/jenkins/workspace/kubernetes-federation-build/_output/gcs-stage/v1.5.0-alpha.2.540+ccaba2ccd2231d/kubernetes.tar.gz.sha1 /var/lib/jenkins/workspace/kubernetes-federation-build/_output/gcs-stage/v1.5.0-alpha.2.540+ccaba2ccd2231d/kubernetes-test.tar.gz /var/lib/jenkins/workspace/kubernetes-federation-build/_output/gcs-stage/v1.5.0-alpha.2.540+ccaba2ccd2231d/kubernetes-test.tar.gz.md5 /var/lib/jenkins/workspace/kubernetes-federation-build/_output/gcs-stage/v1.5.0-alpha.2.540+ccaba2ccd2231d/kubernetes-test.tar.gz.sha1 gs://kubernetes-federation-release/ci/v1.5.0-alpha.2.540+ccaba2ccd2231d/
OK
- Marking all uploaded objects public: 
push-build.sh::release::gcs::copy_release_artifacts(): /usr/bin/gsutil -q -m acl ch -R -g all:R gs://kubernetes-federation-release/ci/v1.5.0-alpha.2.540+ccaba2ccd2231d
OK

It uploads the artifacts, then lists them and changes the ACLs. If we're unlucky, that second listing will miss some of the artifacts. The proper fix here is to upload the files with the public_read ACL in the first place. We could also sleep for some period of time to reduce the chances of getting bitten by eventual consistency.

@rmmh rmmh self-assigned this Nov 4, 2016
rmmh pushed a commit to rmmh/k8s-release that referenced this issue Nov 4, 2016
Upload artifacts with public_read ACL to start with.
6cd1cf1 
This fixes kubernetes/test-infra#990 where a recursive acl change
immediately after an upload might miss objects, since GCS bucket listing
is eventually consistent.
@rmmh rmmh mentioned this issue Nov 4, 2016
Merged
@ghost
Copy link
Author

ghost commented Nov 4, 2016

@rmmh Rather than sleep for some arbitrary period of time, please poll until the desired consistency is reached.

rmmh pushed a commit to rmmh/k8s-release that referenced this issue Nov 4, 2016
Upload artifacts with public_read ACL to start with.
887442f 
This fixes kubernetes/test-infra#990 where a recursive acl change
immediately after an upload might miss objects, since GCS bucket listing
is eventually consistent.
k8s-github-robot pushed a commit to kubernetes/kubernetes that referenced this issue Nov 5, 2016
Merge pull request #36252 from ixdy/get-kube-binaries-script
4d7960b 
Automatic merge from submit-queue

Use curl -f in cluster/get-kube.sh

**What this PR does / why we need it**:
Make `curl` explicitly fail if there are HTTP errors. This makes things more obvious than tar mysteriously failing. x-ref kubernetes/test-infra#990

(I also added -L so HTTP redirects work, in case we eventually want to use them.)
@ixdy ixdy mentioned this issue Nov 15, 2016
Merged
ostromart pushed a commit to ostromart/test-infra that referenced this issue Jul 26, 2019
@sebastienvas @istio-testing
Remove kubeconfig (kubernetes#990)
a41fd2a
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@rmmh rmmh

Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Upload artifacts with public_read ACL to start with. rmmh/k8s-release
4 participants
@madhusudancs @rmmh @colhom @ixdy