HTTPError: 400 Bad Request from https://test.pypi.org/legacy/ (with no additional information) #303

jeFF0Falltrades · 2024-11-12T06:18:57Z

Hello team!

I've implemented this action into my workflow here, but for nearly every build (I'm unsure of why it sometimes passes) I am getting a failure on this job (e.g. this instance) with only the following showing in the logs:

Transparency log entry created with index: 148295038
Uploading distributions to https://test.pypi.org/legacy/
Uploading rat_king_parser-3.1.5-py3-none-any.whl
WARNING  Error during upload. Retry with the --verbose option for more details. 
ERROR    HTTPError: 400 Bad Request from https://test.pypi.org/legacy/          
         Bad Request

I'm unsure of how to increase the verbosity via Actions to learn more about the request, and strangely, the package still ends up deployed on test.pypi.org even with the failure. But the pipeline on GitHub shows it as a failure all the same.

I was wondering if you all had any insight into what can cause this issue - Please let me know if I can provide additional logging in order to help triage.

Thank you for your hard work!

The text was updated successfully, but these errors were encountered:

webknjaz · 2024-11-12T08:02:21Z

That verbose: true should make twine output more detailed. Also, restarting the job in debug mode may add extra context in other places.

The error line is printed by twine, not us. So we don't really have the context. But the HTTP response is coming from TestPyPI.

Let's see if @woodruffw or @di could corelate this with any of the logs within the Warehouse deployment...

di · 2024-11-12T16:39:57Z

@woodruffw I suspect this is https://github.com/pypi/warehouse/blob/79269a2856526fdf4e1257d021a6453b88e81022/warehouse/forklift/legacy.py#L1229-L1252. Two issues here:

I think the exception message is becoming empty here for some reason (unsure why)
This whole check needs to happen much earlier, otherwise we're going to create the file & release before we potentially raise this exception.

netomi · 2024-11-12T20:05:18Z

Upon upgrading to version 1.12.2 from 1.11.0 I encounter now a similar 400 error, see https://github.com/eclipse-csi/otterdog/actions/runs/11804531745/job/32885169579 when publishing to Test PyPI. Did not have the time yet to test with --verbose flag.

Edit: Tried to use this version again and enabled the verbose flag, though this time it succeeded:

https://github.com/eclipse-csi/otterdog/actions/runs/11817561849/job/32923330217

so maybe a similar error as for the OP with duplicate versions, though I use dynamic versioning to being able to publish to TestPyPI with every commit (inspired by urllib3) to spot publishing errors prior to do an actual release.

jeFF0Falltrades · 2024-11-12T22:21:13Z

Thank you for the great conversation here!

For my part, I was able to resolve this issue, and it turned out to be a user error: I had not restricted my publishing actions to the master branch, so every time a new tag was pushed to the repo, the workflow was triggered twice - Once for the master branch, and once for the new tag.

Adding verbose: true to my workflow YAML did indeed help, as it alerted me to the true source of the 400 error code:

Uploading rat_king_parser-3.1.7-py3-none-any.whl
INFO     Response from https://test.pypi.org/legacy/:                           
         400 Bad Request                                                        
INFO     <html>                                                                 
          <head>                                                                
           <title>400 File already exists

The package was being uploaded twice: Successful on the first push to master, unsuccessful on the push to the tag as the release already existed after the push to master.

All of that said, I appreciate the work done in the repos referenced above that seek to emit the root error of the 400 response so that even if verbose is not set, the user can see what the true cause of the issue is.

Unless you want to address any other Issues raised here, please feel free to close this Issue at your discretion.

di · 2024-11-12T23:42:07Z

@webknjaz Might be nice to include the full error message at the lowest level of verbosity to avoid situations like this in the future.

webknjaz · 2024-11-15T03:33:27Z

@di well, we can't control individual messages Twine prints out. We call Twine via CLI since there's no public API (pypa/twine#194 / pypa/twine#361).

We can only flip the default value of the verbose input but that would generate a lot of output confusing people…

Closing, since the mysteries reported above seem to have been solved by now...

acampove · 2025-01-19T03:43:32Z

Thank you for the great conversation here!

For my part, I was able to resolve this issue, and it turned out to be a user error: I had not restricted my publishing actions to the master branch, so every time a new tag was pushed to the repo, the workflow was triggered twice - Once for the master branch, and once for the new tag.

Adding verbose: true to my workflow YAML did indeed help, as it alerted me to the true source of the 400 error code:
Uploading rat_king_parser-3.1.7-py3-none-any.whl
INFO     Response from https://test.pypi.org/legacy/:                           
         400 Bad Request                                                        
INFO     <html>                                                                 
          <head>                                                                
           <title>400 File already exists                                       
The package was being uploaded twice: Successful on the first push to master, unsuccessful on the push to the tag as the release already existed after the push to master.

All of that said, I appreciate the work done in the repos referenced above that seek to emit the root error of the 400 response so that even if verbose is not set, the user can see what the true cause of the issue is.

Unless you want to address any other Issues raised here, please feel free to close this Issue at your discretion.

Hi @jeFF0Falltrades,

I have the same problem, see this. How do you specify that you want to publish to master? I have this in my publishing section:

  publish-to-testpypi:
    name: Publish Python 🐍 distribution 📦 to TestPyPI
    needs:
    - build
    runs-on: ubuntu-latest

    environment:
      name: testpypi
      url: https://test.pypi.org/p/rx_selection

    permissions:
      id-token: write  # IMPORTANT: mandatory for trusted publishing

    steps:
    - name: Download all the dists
      uses: actions/download-artifact@v4
      with:
        name: python-package-distributions
        path: dist/
    - name: Publish distribution 📦 to TestPyPI
      uses: pypa/gh-action-pypi-publish@release/v1
      with:
        repository-url: https://test.pypi.org/legacy/

jeFF0Falltrades · 2025-01-19T03:50:39Z

Hey @acampove !

I've since changed my action manifest to only run on tagged releases, but you can see the setup when I was only triggering it for pushes to master here:

Essentially you just have to add this to the manifest:

on:
  push:
    branches:
      - master

pypa/gh-action-pypi-publish#303

acampove · 2025-01-19T04:39:14Z

Hey @acampove !

I've since changed my action manifest to only run on tagged releases, but you can see the setup when I was only triggering it for pushes to master here:

Essentially you just have to add this to the manifest:

on:
push:
branches:
- master

Hi @jeFF0Falltrades

Yes, that's how this would work. We should only run in tagged releases, because they are the only ones worth publishing. I see that you also took your YAML file from here and you added the verbose:true.

In my case, I seem to be creating two pipelines at the same time too:

one for main and the other for the tag. The first pipeline has the testpypi succeeding and the pypi skipped:

while the second succeeds sending this to pypi, but fails on testpypi because it was already published:

I need a config that allows me to skip both testpypi and pypi when pushing to main. That means that both the testing and the main pypi will only run for tagged versions. Completely defeating the purpose of having a testpypi. Or maybe I am confused?

jeFF0Falltrades · 2025-01-19T04:50:03Z

Ah sorry @acampove I thought you had already looked at the current workflow version, which has the newer approach of only publishing on a tagged release; See my current workflow here.

Specifically:

  publish-to-testpypi:
    name: Publish Python 🐍 distribution 📦 to TestPyPI
    if: startsWith(github.ref, 'refs/tags/')

acampove · 2025-01-19T05:04:34Z

Ah sorry @acampove I thought you had already looked at the current workflow version, which has the newer approach of only publishing on a tagged release; See my current workflow here.

Specifically:

publish-to-testpypi:
name: Publish Python 🐍 distribution 📦 to TestPyPI
if: startsWith(github.ref, 'refs/tags/')

Hi @jeFF0Falltrades

Thank you for your reply. Yes, i kind of agree with that approach in the sense that we trigger publication only for new versions, otherwise the publication would fail. But then the test and the actual pypi publication will both run for the same commits. So I am not sure how this approach would test anything, we are just publishing the same thing to two different places. I asked about this issue here, in case you are also interested.

jeFF0Falltrades · 2025-01-19T05:11:42Z

Ah sorry @acampove I thought you had already looked at the current workflow version, which has the newer approach of only publishing on a tagged release; See my current workflow here.

Specifically:

publish-to-testpypi:
name: Publish Python 🐍 distribution 📦 to TestPyPI
if: startsWith(github.ref, 'refs/tags/')

Hi @jeFF0Falltrades

Thank you for your reply. Yes, i kind of agree with that approach in the sense that we trigger publication only for new versions, otherwise the publication would fail. But then the test and the actual pypi publication will both run for the same commits. So I am not sure how this approach would test anything, we are just publishing the same thing to two different places. I asked about this issue here, in case you are also interested.

Ahhh my apologies - I misunderstood the problem you were trying to solve. I understand now.

For separating the test vs. "prod" build to PyPi, you could probably use either:

Separate release and testing branches, so that the Test PyPi workflow triggers only on the testing/staging branch
Different tagging system for a test build vs release build, and edit the workflow triggers to work off of those tagging patterns.

You're right in that it makes more sense to break out the workflows from a singular workflow - for me, I've kept them in one simply because if the Test PyPi build fails in a significant way, it will nearly always fail for the same reason in the real PyPi, so nothing lost/nothing gained before going back and fixing the issue.

But I will probably change to separate workflow triggers for the reason you describe above, just to keep the two separated.

webknjaz · 2025-01-20T18:30:07Z

@acampove one way to publish on pushes to both the default branch and tags is migrating away from hardcoded version numbers. If you're using setuptools-scm, that would give you versions that are unique per-commit. You'd only need to make sure that you're not publishing the same version from multiple similar events.

acampove · 2025-01-21T06:54:32Z

@acampove one way to publish on pushes to both the default branch and tags is migrating away from hardcoded version numbers. If you're using setuptools-scm, that would give you versions that are unique per-commit. You'd only need to make sure that you're not publishing the same version from multiple similar events.

Dear @webknjaz

Thanks for your feedback. Yes, I agree, I have read about this and I might have to do that. What I do so far is:

Manually create a new tag by editing the pyproject.toml version entry.
Committing and running publish as defined here. This utility makes a new tag with the version name, if the version is not found among any of the tags.
It then pushes commits and tags.

Do you have by any chance any good resource to adopt this automatic tags approach and move away from my publish utility?

Cheers.

webknjaz · 2025-01-24T02:52:33Z

@acampove I don't have a simplified example, unfortunately. Here's one of my latest automations: https://github.com/ansible/awx-plugins/blob/0d569b5/.github/workflows/ci-cd.yml#L120-L587. You can inspect it if you like. But there's a number of things going on there that might confuse a casual reader.

Basically, I don't like using “tag created” events since I believe that a tag is something that should be an outcome of successful publishing, not a trigger. So releases a triggered by clicking a button on GH UI (the workflow_dispatch event) with the desired version typed in. Then, in the workflow, that version is applied as a tag temporarily and conditionally but just for "release requests".

I distinguish "merge into devel" events (pushes) and do the opposite — I remove any tags on the commit from temporary Git checkout. This allows the automatic publishing to TestPyPI to not conflict between PR merges and separate releases of the same commit (since they'd compute the same version otherwise). And I also monkey-patch the setuptools-scm config to drop local versions that are illegal on (Test)PyPI per PEP 440.

I also distinguish "PR builds" for which I don't do anything with the tags since those don't attempt any publishing.

In the past, long ago, I was using "tag created" events, but the technique for ensuring that the versions are unique was similar. Some people also rely on "GitHub Release published" but that's basically the same.

woodruffw mentioned this issue Nov 12, 2024

legacy: ensure invalid attestation error is never empty pypi/warehouse#17065

Merged

di mentioned this issue Nov 12, 2024

Capture sentry message when _exc_with_message is called without a message pypi/warehouse#17068

Merged

webknjaz closed this as completed Nov 15, 2024

acampove pushed a commit to acampove/rx_selection that referenced this issue Jan 19, 2025

Using config from:

09805b8

pypa/gh-action-pypi-publish#303

webknjaz mentioned this issue Jan 20, 2025

Cannot publish: HTTPError: 400 Bad Request from #321

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

HTTPError: 400 Bad Request from https://test.pypi.org/legacy/ (with no additional information) #303

HTTPError: 400 Bad Request from https://test.pypi.org/legacy/ (with no additional information) #303

jeFF0Falltrades commented Nov 12, 2024

webknjaz commented Nov 12, 2024

di commented Nov 12, 2024

netomi commented Nov 12, 2024 •

edited

Loading

jeFF0Falltrades commented Nov 12, 2024

di commented Nov 12, 2024

webknjaz commented Nov 15, 2024

acampove commented Jan 19, 2025 •

edited

Loading

jeFF0Falltrades commented Jan 19, 2025

acampove commented Jan 19, 2025

jeFF0Falltrades commented Jan 19, 2025

acampove commented Jan 19, 2025

jeFF0Falltrades commented Jan 19, 2025

webknjaz commented Jan 20, 2025

acampove commented Jan 21, 2025

webknjaz commented Jan 24, 2025

HTTPError: 400 Bad Request from https://test.pypi.org/legacy/ (with no additional information) #303

HTTPError: 400 Bad Request from https://test.pypi.org/legacy/ (with no additional information) #303

Comments

jeFF0Falltrades commented Nov 12, 2024

webknjaz commented Nov 12, 2024

di commented Nov 12, 2024

netomi commented Nov 12, 2024 • edited Loading

jeFF0Falltrades commented Nov 12, 2024

di commented Nov 12, 2024

webknjaz commented Nov 15, 2024

acampove commented Jan 19, 2025 • edited Loading

jeFF0Falltrades commented Jan 19, 2025

acampove commented Jan 19, 2025

jeFF0Falltrades commented Jan 19, 2025

acampove commented Jan 19, 2025

jeFF0Falltrades commented Jan 19, 2025

webknjaz commented Jan 20, 2025

acampove commented Jan 21, 2025

webknjaz commented Jan 24, 2025

netomi commented Nov 12, 2024 •

edited

Loading

acampove commented Jan 19, 2025 •

edited

Loading