Dependabot use snapshots repositories for checking released versions #5947

slawekjaranowski · 2022-10-21T16:30:29Z

Is there an existing issue for this?

I have searched the existing issues

Package ecosystem

Maven

Package manager version

No response

Language version

No response

Manifest location and content before the Dependabot update

No response

dependabot.yml content

No response

Updated dependency

No response

What you expected to see, versus what you actually saw

When Maven project in pom.xml or in parent project has defined additional additional repositories
only repositories for release version should be used.

Eg. we can have in porject - pom.xml:

<repositories>
    <repository>
      <id>ossrh-snapshots</id>
      <name>ossrh-snapshots</name>
      <releases>
        <enabled>false</enabled>
      </releases>
      <snapshots>
        <checksumPolicy>fail</checksumPolicy>
        <enabled>true</enabled>
      </snapshots>
      <url>https://oss.sonatype.org/content/repositories/snapshots</url>
    </repository>
  </repositories>

so such repository should not be used for released artifact version.

We can have two situations - metadata not present in snapshot repositories, then as fallback default release is used

updater | INFO <job_489323335> Checking if org.apache.commons:commons-lang3 3.12.0 needs updating
  proxy | 2022/10/21 13:25:06 [024] GET https://repo.maven.apache.org:443/maven2/org/codehaus/mojo/mojo-parent/70/mojo-parent-70.pom
  proxy | 2022/10/21 13:25:06 [024] 200 https://repo.maven.apache.org:443/maven2/org/codehaus/mojo/mojo-parent/70/mojo-parent-70.pom
  proxy | 2022/10/21 13:25:06 [026] GET https://oss.sonatype.org:443/content/repositories/snapshots/org/apache/commons/commons-lang3/maven-metadata.xml
  proxy | 2022/10/21 13:25:07 [026] 404 https://oss.sonatype.org:443/content/repositories/snapshots/org/apache/commons/commons-lang3/maven-metadata.xml
  proxy | 2022/10/21 13:25:07 [028] GET https://repo.maven.apache.org:443/maven2/org/apache/commons/commons-lang3/maven-metadata.xml
  proxy | 2022/10/21 13:25:07 [028] 200 https://repo.maven.apache.org:443/maven2/org/apache/commons/commons-lang3/maven-metadata.xml
  proxy | 2022/10/21 13:25:07 [030] HEAD https://oss.sonatype.org:443/content/repositories/snapshots/org/apache/commons/commons-lang3/3.12.0/commons-lang3-3.12.0.jar
  proxy | 2022/10/21 13:25:07 [030] 404 https://oss.sonatype.org:443/content/repositories/snapshots/org/apache/commons/commons-lang3/3.12.0/commons-lang3-3.12.0.jar
  proxy | 2022/10/21 13:25:07 [032] HEAD https://repo.maven.apache.org:443/maven2/org/apache/commons/commons-lang3/3.12.0/commons-lang3-3.12.0.jar
  proxy | 2022/10/21 13:25:07 [032] 200 https://repo.maven.apache.org:443/maven2/org/apache/commons/commons-lang3/3.12.0/commons-lang3-3.12.0.jar
updater | INFO <job_489323335> Latest version is 3.12.0
updater | INFO <job_489323335> No update needed for org.apache.commons:commons-lang3 3.12.0

Metadata is present - no new version is detected

updater | INFO <job_489323335> Checking if org.codehaus.mojo:mojo-parent 70 needs updating
  proxy | 2022/10/21 13:25:23 [006] GET https://repo.maven.apache.org:443/maven2/org/codehaus/mojo/mojo-parent/70/mojo-parent-70.pom
  proxy | 2022/10/21 13:25:23 [006] 200 https://repo.maven.apache.org:443/maven2/org/codehaus/mojo/mojo-parent/70/mojo-parent-70.pom
  proxy | 2022/10/21 13:25:23 [008] GET https://oss.sonatype.org:443/content/repositories/snapshots/org/codehaus/mojo/mojo-parent/maven-metadata.xml
  proxy | 2022/10/21 13:25:23 [008] 200 https://oss.sonatype.org:443/content/repositories/snapshots/org/codehaus/mojo/mojo-parent/maven-metadata.xml
updater | INFO <job_489323335> Latest version is 
updater | INFO <job_489323335> Requirements to unlock update_not_possible
updater | INFO <job_489323335> Requirements update strategy 
updater | INFO <job_489323335> No update possible for org.codehaus.mojo:mojo-parent 70

Native package manager behavior

No response

Images of the diff or a link to the PR, issue, or logs

No response

Smallest manifest that reproduces the issue

No response

The text was updated successfully, but these errors were encountered:

nscuro · 2022-12-02T13:49:33Z

This is affecting us as well.

We too have Sonatype's snapshot repository enabled in our pom.xml:

<repositories>
    <repository>
        <id>ossrh-snapshot</id>
        <url>https://oss.sonatype.org/content/repositories/snapshots</url>
        <snapshots>
            <updatePolicy>always</updatePolicy>
            <enabled>true</enabled>
        </snapshots>
    </repository>
</repositories>

For example, we're using org.postgresql:postgresql 42.5.0. Version 42.5.1 is available in central. Dependabot however only checks the snapshot repository, and not central:

updater | INFO <job_529643220> Checking if org.postgresql:postgresql 42.5.0 needs updating
  proxy | 2022/12/02 08:00:44 [114] GET https://oss.sonatype.org:443/content/repositories/snapshots/org/postgresql/postgresql/maven-metadata.xml
  proxy | 2022/12/02 08:00:44 [114] 200 https://oss.sonatype.org:443/content/repositories/snapshots/org/postgresql/postgresql/maven-metadata.xml
updater | INFO <job_529643220> No update possible for org.postgresql:postgresql 42.5.0

This behavior must be caused by a recent change, because the ossrh-snapshot repository has been present in our project since forever, and Dependabot worked up until a few weeks ago.

pzygielo · 2023-01-03T12:10:14Z

This behavior must be caused by a recent change, because the ossrh-snapshot repository has been present in our project since forever, and Dependabot worked up until a few weeks ago.

Observing the same. (updater_version=8ef74b531cd4ac93a3cbdab7efdb0f1712141eaa)

dependabot/dependabot-core#5947

Needed for dependabot: dependabot/dependabot-core#5947

stefan-it · 2023-04-18T13:24:37Z

Hi,

issue is also affecting us and the only "workaround" seems to: remove snapshot section and also do not use snapshot version in dependencies (which is not really great).

@jeffwidman do you accidentally know how to fix or workaround this 🤔 Many thanks!

jeffwidman · 2023-04-20T05:41:33Z

This behavior must be caused by a recent change, because the ossrh-snapshot repository has been present in our project since forever, and Dependabot worked up until a few weeks ago.

Looking through the history of the maven folder, any of these three seem like they might have this as an unexpected side-effect:

maven: implement replaces-base to avoid calling central #5908
Fix Maven inability to overwrite repository urls by ID #5878 👈 seems unlikely to be the culprit
maven: stop querying repositories once one returns a result #5872

The easiest way to narrow this down is for someone affected by this issue to use one of our debugging tools to simulate running the update job locally: https://github.com/dependabot/dependabot-core#debugging-problems

And then try checking out the dependabot-core repo at the commits right before each of the above changes and see if one of them fixes this. If you can narrow down the issue to a more specific commit, that'd be very helpful for us.

Alternatively, you can simply run the latest version of dependabot-core with the dry-run tooling and add debugger statements to figure out where the logic is going astray and using the wrong snapshots repositories: https://github.com/dependabot/dependabot-core#adding-debug-breakpoints

All the code in our Maven updater is ruby, so the debugger should be all you need, there's no native Maven tooling right now unfortunately.

slawekjaranowski · 2023-05-18T11:30:08Z

All the code in our Maven updater is ruby, so the debugger should be all you need, there's no native Maven tooling right now unfortunately.

@jeffwidman - There is versions plugin - what do you need from native tooling that will be possible to using by dependabot?

slawekjaranowski · 2023-05-18T11:41:23Z

I see that from sometime also dependency check in ASF project which use another artifacts from AFS stop working - it is probably connected with it.

Eg: apache/maven-war-plugin#47

pull request was created at 5th May
today (18 May) add asked about rebase ... but PR was closed with message:

Looks like org.apache.maven.shared:maven-shared-utils is no longer updatable, so this is no longer needed.

maven-shared-utils - is still used in project and should be updated

Since we don't publish SNAPSHOT releases of Graylog for a long time, we don't need the custom configuration anymore. The configuration also breaks GitHub's dependabot checks due to a bug in the dependabot-core maven module: dependabot/dependabot-core#5947

Since we haven't published SNAPSHOT releases of Graylog for a long time, we don't need the custom configuration anymore. The configuration also breaks GitHub's dependabot checks due to a bug in the dependabot-core maven module: dependabot/dependabot-core#5947

JackPGreen · 2023-12-02T13:55:58Z

Encountered this as well, have found a consistent local reproducer using dependabot cli.

Take the following pom.xml (central declaration is implied and already inherited from super-POM but helps illustrate the issue):

<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
         xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/maven-v4_0_0.xsd">
    <modelVersion>4.0.0</modelVersion>
    <groupId>com.me</groupId>
    <artifactId>dependabot-reproducer</artifactId>
    <version>0.0.1-SNAPSHOT</version>
    <repositories>
        <repository>
            <id>snapshot-repository</id>
            <url>https://oss.sonatype.org/content/repositories/snapshots/</url>
        </repository>
        <repository>
            <id>central</id>
            <url>https://repo.maven.apache.org/maven2/</url>
        </repository>
    </repositories>
    <dependencies>
        <dependency>
            <groupId>io.grpc</groupId>
            <artifactId>grpc-protobuf</artifactId>
            <version>1.58.0</version>
        </dependency>
    </dependencies>
</project>

The latest version of grpc-protobuf is 1.59.1.

Run:

dependabot update maven dependabot-reproducer --local .

Gives output (failure):

updater | 2023/12/02 13:40:40 INFO Checking if io.grpc:grpc-protobuf 1.58.0 needs updating
  proxy | 2023/12/02 13:40:40 [004] GET https://oss.sonatype.org:443/content/repositories/snapshots/io/grpc/grpc-protobuf/maven-metadata.xml
  proxy | 2023/12/02 13:40:40 [004] 200 https://oss.sonatype.org:443/content/repositories/snapshots/io/grpc/grpc-protobuf/maven-metadata.xml
updater | 2023/12/02 13:40:40 INFO Latest version is 
updater | 2023/12/02 13:40:40 INFO Requirements to unlock update_not_possible
updater | 2023/12/02 13:40:40 INFO Requirements update strategy 
updater | 2023/12/02 13:40:40 INFO No update possible for io.grpc:grpc-protobuf 1.58.0

But if you re-order pom.xml so the central declaration is first:

<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
         xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/maven-v4_0_0.xsd">
    <modelVersion>4.0.0</modelVersion>
    <groupId>com.me</groupId>
    <artifactId>dependabot-reproducer</artifactId>
    <version>0.0.1-SNAPSHOT</version>
    <repositories>
        <repository>
            <id>central</id>
            <url>https://repo.maven.apache.org/maven2/</url>
        </repository>
        <repository>
            <id>snapshot-repository</id>
            <url>https://oss.sonatype.org/content/repositories/snapshots/</url>
        </repository>
    </repositories>
    <dependencies>
        <dependency>
            <groupId>io.grpc</groupId>
            <artifactId>grpc-protobuf</artifactId>
            <version>1.58.0</version>
        </dependency>
    </dependencies>
</project>

You get (success):

updater | 2023/12/02 13:43:56 INFO Checking if io.grpc:grpc-protobuf 1.58.0 needs updating
  proxy | 2023/12/02 13:43:56 [004] GET https://repo.maven.apache.org:443/maven2/io/grpc/grpc-protobuf/maven-metadata.xml
  proxy | 2023/12/02 13:43:56 [004] 200 https://repo.maven.apache.org:443/maven2/io/grpc/grpc-protobuf/maven-metadata.xml
  proxy | 2023/12/02 13:43:56 [006] HEAD https://repo.maven.apache.org:443/maven2/io/grpc/grpc-protobuf/1.59.1/grpc-protobuf-1.59.1.jar
  proxy | 2023/12/02 13:43:57 [006] 200 https://repo.maven.apache.org:443/maven2/io/grpc/grpc-protobuf/1.59.1/grpc-protobuf-1.59.1.jar
updater | 2023/12/02 13:43:57 INFO Latest version is 1.59.1

This shows there's some kind of ordering issue which means only the first repository is being checked.

@stefan-it

issue is also affecting us and the only "workaround" seems to: remove snapshot section and also do not use snapshot version in dependencies (which is not really great).

I have another workaround:
Copy the central declaration from the super-POM to the top of the repositories section:

<repository>
    <id>central</id>
    <name>Central Repository</name>
    <url>https://repo.maven.apache.org/maven2/</url>
    <releases>
        <enabled>true</enabled>
    </releases>
    <snapshots>
       <enabled>false</enabled>
    </snapshots>
</repository>

- snapshots repositories will be used for parent pom resolving - for available versions checking snapshots repositories will be excluded fix dependabot#5947

Items repository -> snapshots,releases -> enabled has a default value as true, so it is enough to defined releases as false for snapshots repositories fix dependabot#5947

Items repository -> snapshots,releases -> enabled has a default value as true, so it is enough to defined releases as false for snapshots repositories fix #5947

slawekjaranowski added the T: bug 🐞 Something isn't working label Oct 21, 2022

deivid-rodriguez added the L: java:maven Maven packages via Maven label Oct 21, 2022

nscuro mentioned this issue Dec 6, 2022

Bump dependencies DependencyTrack/dependency-track#2244

Merged

5 tasks

pzygielo mentioned this issue Jan 3, 2023

Block inherited repository pzygielo/idlj-maven-plugin#8

Merged

jeffwidman mentioned this issue Feb 11, 2023

Add full support for SNAPSHOT packages in Maven #6083

Open

3 tasks

slawekjaranowski added a commit to mojohaus/mojo-parent that referenced this issue Feb 12, 2023

Workaround for dependabot with snapshots repositories

9b8ee24

dependabot/dependabot-core#5947

slawekjaranowski mentioned this issue Feb 12, 2023

Workaround for dependabot with snapshots repositories mojohaus/mojo-parent#345

Merged

slawekjaranowski added a commit to mojohaus/mojo-parent that referenced this issue Feb 12, 2023

Workaround for dependabot with snapshots repositories

ebda315

dependabot/dependabot-core#5947

stefan-it added a commit to dbmdz/iiif-server-hymir that referenced this issue Apr 18, 2023

pom: add updatePolicy in repository settings

2a44dfc

Needed for dependabot: dependabot/dependabot-core#5947

bernd mentioned this issue Sep 26, 2023

Remove "<repositories>" from pom.xml Graylog2/graylog2-server#16672

Merged

slawekjaranowski mentioned this issue Oct 26, 2023

Remove snapshot repositories from project mojohaus/wagon-maven-plugin#69

Merged

DimitrisStaratzis mentioned this issue Nov 13, 2023

group trino-root and trino-testing in dependabot TileDB-Inc/TileDB-Trino#99

Merged

slawekjaranowski mentioned this issue Dec 2, 2023

Skip Maven snapshots repositories from versions checking #8514

Merged

jakecoffman closed this as completed in #8514 Dec 4, 2023

slawekjaranowski mentioned this issue Dec 6, 2023

Skip Maven snapshots repositories from versions checking - fix 2 #8542

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Dependabot use snapshots repositories for checking released versions #5947

Dependabot use snapshots repositories for checking released versions #5947

slawekjaranowski commented Oct 21, 2022

nscuro commented Dec 2, 2022 •

edited

Loading

pzygielo commented Jan 3, 2023 •

edited

Loading

stefan-it commented Apr 18, 2023

jeffwidman commented Apr 20, 2023 •

edited

Loading

slawekjaranowski commented May 18, 2023

slawekjaranowski commented May 18, 2023

JackPGreen commented Dec 2, 2023 •

edited

Loading

Dependabot use snapshots repositories for checking released versions #5947

Dependabot use snapshots repositories for checking released versions #5947

Comments

slawekjaranowski commented Oct 21, 2022

Is there an existing issue for this?

Package ecosystem

Package manager version

Language version

Manifest location and content before the Dependabot update

dependabot.yml content

Updated dependency

What you expected to see, versus what you actually saw

Native package manager behavior

Images of the diff or a link to the PR, issue, or logs

Smallest manifest that reproduces the issue

nscuro commented Dec 2, 2022 • edited Loading

pzygielo commented Jan 3, 2023 • edited Loading

stefan-it commented Apr 18, 2023

jeffwidman commented Apr 20, 2023 • edited Loading

slawekjaranowski commented May 18, 2023

slawekjaranowski commented May 18, 2023

JackPGreen commented Dec 2, 2023 • edited Loading

nscuro commented Dec 2, 2022 •

edited

Loading

pzygielo commented Jan 3, 2023 •

edited

Loading

jeffwidman commented Apr 20, 2023 •

edited

Loading

JackPGreen commented Dec 2, 2023 •

edited

Loading