Images created by Jib occasionally causes 400 BAD REQUEST (BLOB_UPLOAD_INVALID) in Red Hat Quay

[hbjastad]

Environment:

• Jib version: 1.6.1
• Build tool: Maven 3.5.0
• OS: Linux

Description of the issue:
Images created by Jib occasionally causes 400 Bad request in Red Hat Quay. It happens so rarely, it can hardly be categorized as reproducible, and if run again after failing then it works.

We have only seen this occur when running with -Djib.serialize=true

Expected behavior:
Upload success.

Steps to reproduce:

1. Get our project that uses Jib to upload to QUAY-REGISTRY
2. Run mvn compile jib:build

jib-maven-plugin Configuration:

<configuration>
    <to>
        <image>QUAY-REGISTRY/image-built-with-jib</image>
    </to>
</configuration>

Log output:

[2019-09-20T09:05:45.888Z] [ERROR] Failed to execute goal com.google.cloud.tools:jib-maven-plugin:1.6.1:build (build_docker_image) on project app-backend: Tried to push BLOB for QUAY-REGISTRY/group/app with digest sha256:1eebb6432678ad971dfd2f732c51595050482f3d7aad8059ed5549f789cd273c but failed because: other: blob upload invalid | If this is a bug, please file an issue at https://github.com/GoogleContainerTools/jib/issues/new: 400 BAD REQUEST
[2019-09-20T09:05:45.888Z] [ERROR] {"errors":[{"code":"BLOB_UPLOAD_INVALID","detail":{},"message":"blob upload invalid"}]}
[2019-09-20T09:05:45.888Z] [ERROR] -> [Help 1]

Additional Information:
Please see #1986, which is also an error with the Jib/Quay combo, and has the first mention of this error. We have not seen this error with the hello world app, but with another app we're developing this has now happened twice this week (out of maybe 100 runs) - so it's time to bring it up as an issue on its own.

@kurtismullins, can you please have a look at this one also?

When Quay says the blob upload is invalid, is it more likely that Jib created an invalid blob or that Quay incorrectly finds it to be invalid?

[chanseokoh]

This is when pushing either 1) layer blobs (either base image layers or application layers); or 2) a container configuration JSON. Jib did attempt pushing a blob.

A blob push in Jib consists of three HTTP requests. It is unclear what was the actual endpoint without more details. It should be one of

1. initial POST to get the upload location (may or may not be cross-repo blob mount)
2. PATCH to write a blob to the upload location
3. PUT with a digest to conclude the push

We have never had a report of BLOB_UPLOAD_INVALID until now, although I won't rule out the possibility that something can go wrong with Jib.

For this kind of troubleshooting, I think the best approach is for Red Hat to check the server code to see under what conditions it can return the BLOB_UPLOAD_INVALID and cross-check that with the server error log.

[chanseokoh]

I don't know if this is still observable, but it's worth trying Jib 2.0.0. It's unclear if this is a bug in Quay. But since we haven't heard back for a while, I'll close this just for the sake of trying to get more attention. Try Jib 2.0.0 and if you still see BLOB_UPLOAD_INVALID, please re-open.