Internal Server Error on pushing to Container Registry when storage type for packages is minio #21320
Comments
|
The Do you get this error every time? Do all seven layers fail or just some of them? |
|
@KN4CK3R all layers are failing but only for certain images, some images are being pushed without a problem. |
|
Could you test if it's the same problem as in #19586 (comment)
Is one of the failing images public so I could test it?
|
Tested it, this is not the same problem. It is still retrying, now just one layer at a time.
Yes, I've just retagged existing public images to test it.
|
|
Great, will test it tomorrow
|
|
I can't reproduce it. Which docker cli version do you use? Your client makes multiple calls to the authenticate endpoint which my client does not. That's not the reason for the error but we use different clients then. |
|
|
I have the same version. Wondering why your client behaves differently. My client does not send the mount ( Anyway, the EOF error should be a missing config layer. For |
|
Yes I checked the file, it is a valid JSON document. |
|
Can the mount and delete requests have something to do with me already having tried to push the same image multiple times unsuccessfully? |
|
Maybe, but that should not be a problem. From the log, the mount is ignored and a new upload is instantiated. Do you have the same errors if you use a (local) minio server instance instead of AWS? I have no idea how the EOF error could occur if all 4 files got uploaded. I could create a PR which adds more logging but I'm not sure if this helps in this case. |
|
Will test that as soon as I get a chance |
|
Meanwhile, can you please point me to the part of the code where the file is uploaded to minio server. Maybe I'd be able to help in debugging. |
|
The EOF error should occur here inside gitea/routers/api/packages/container/manifest.go Lines 95 to 104 in 2d2cf58 There the config file is read from the storage and parsed. configReader is the reader for the AWS file.
|
Yes :( PS: I am not able to reproduce the EOF error in UploadManifest, will try again with a new setup. |
|
Strange that works for me. Maybe we can build a Docker compose file setup where the error is reproducible.
|
|
I did some digging and from my limited understanding, the api request is returning 500 at the following place, when gitea/routers/api/packages/container/container.go Lines 339 to 342 in 2d2cf58 Just a thought: |
|
I created a minimal configuration in which I am encountering the bug. Steps:
|
|
What's the content of your |
|
My bad, I updated the compose file with inline environment variables. |
|
Thank you, I can reproduce the |
|
If I use |
|
Yeah, the error is gone. Any idea timelines for v1.18 stable release? |
|
Feature freeze is in one week. Then there will be the RC versions. My guess would be 4-5 weeks after that. There were no relevant changes to the container code since 1.17, so sadly I don't know if I could help you for 1.17. |
|
I was on 1.18 RC1 but I saw this And the problem still persists. I will try to wipe the data first |
|
Looks like it's not a valid JSON document...? |
|
@KN4CK3R I took some time to download the offending files, so it seems like it's an affirmative, because the file, it looks like this which is obviously not a JSON this coincides with the error shown here: |
|
hmmmmmmmmmmmm maybe its a bug from OpenStack Swift that the QuObjects from QNAP was using an ancient version for? But I think this kind of bug is egregious and I don't believe the bugtracker for OpenStack didn't have any mention about edit: Bingo |
|
From your first screenshot it looks like QuObjects saved the chunked upload into the file without removing the chunk parts. Maybe you can configure something to allow chunked uploads? Otherwise I don't know if we can help you. edit: Looks like you found the problem. If you can compile Gitea you may try this ( func (m *MinioStorage) Save(path string, r io.Reader, size int64) (int64, error) {
uploadInfo, err := m.client.PutObject(
m.ctx,
m.bucket,
m.buildMinioPath(path),
r,
size,
- minio.PutObjectOptions{ContentType: "application/octet-stream"},
+ minio.PutObjectOptions{ContentType: "application/octet-stream", DisableContentSha256: true},
)
if err != nil {
return 0, convertMinioErr(err)
}
return uploadInfo.Size, nil
}That may disable the signature for uploads. https://github.com/minio/minio-go/blob/36e5116001d9e05dc27e8ecc790d261bd1632906/api.go#L841 |
Description
I have configured an s3 bucket to store packages.
When I try to push a container:
When I tried to investigate the logs, I found 2 different types of errors for different containers
Type 1 (in
container.EndUploadBlob):Type 2 (in
container.UploadManifest):All these problems disappear as soon as I switch to local storage.
Gitea Version
1.17
Can you reproduce the bug on the Gitea demo site?
No
Log Gist
No response
Screenshots
No response
Git Version
No response
Operating System
No response
How are you running Gitea?
Official Gitea Docker Image
Database
SQLite
The text was updated successfully, but these errors were encountered: