-
-
Notifications
You must be signed in to change notification settings - Fork 11
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Remove jcenter
, and oss.sonatype.org-releases
repositories from public
virtual repository; reconfigure Atlassian remote repositories
#3842
Comments
I built over 240 of the top 250 plugins with the settings.xml file listed later (using repo.jenkins-ci.org for the releases repository, the jgit repository, and the netbeans repository). I had to download the jbcrypt library separately but other than that, I was able to build without running tests. The repositories that did not build with those settings are:
One plugin (gradle-plugin) does not build with Maven, so I can't check it. <settings xmlns="http://maven.apache.org/SETTINGS/1.0.0"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://maven.apache.org/SETTINGS/1.0.0
http://maven.apache.org/xsd/settings-1.0.0.xsd">
<pluginGroups>
<pluginGroup>org.jenkins-ci.tools</pluginGroup>
</pluginGroups>
<profiles>
<profile>
<id>jenkins</id>
<activation>
<activeByDefault>true</activeByDefault>
</activation>
<repositories>
<repository>
<id>repo.jenkins-ci.org</id>
<url>https://repo.jenkins-ci.org/releases/</url>
</repository>
<repository>
<id>jgit.repo.jenkins-ci.org</id>
<url>https://repo.jenkins-ci.org/jgit/</url>
</repository>
<repository>
<id>netbeans.repo.jenkins-ci.org</id>
<url>https://repo.jenkins-ci.org/netbeans/</url>
</repository>
</repositories>
<pluginRepositories>
<pluginRepository>
<id>repo.jenkins-ci.org</id>
<url>https://repo.jenkins-ci.org/releases/</url>
</pluginRepository>
</pluginRepositories>
</profile>
</profiles>
</settings> |
@daniel-beck do you have a recommendation or a preference for the location where the org.connectbot.jbcrypt:jbcrypt:1.0.0 artifact from jcenter-cache should be placed? My first thought was to place it in the Jenkins releases repository because it is required for a Jenkins release and not available in our other repositories. According to mvnrepository.com, that library is available from the Spring plugins repository and the Spring Lib M repository. The 1.0.1 and 1.0.2 versions of that library are available from repo.maven.apache.org, but the 1.0.0 version is not. |
or update whatever is using 1.0.0 to a newer version? |
That's an approach we've used in the past but would consider it mostly legacy from when just everything went into |
That would break builds of previous versions of Jenkins core, since it requires jbcrypt 1.0.0. I failed the last time that I tried to update the jbcrypt 1.0.0 dependency to a newer version. I don't have notes on the failure, but I remember that the attempt did not succeed. I think it is a good idea to update the version of jbcrypt to a newer release if possible, but I think that should be handled separately from this effort. |
Thanks. I've updated the text of the issue to propose the creation of a new repository, addition of that new repository to the public virtual repository, and the move of the jbcrypt 1.0.0 artifact into the new repository. |
This plan looks good to me, I'm gonna proceed with the first step. proposal for the new repository name: |
jenkins-infra/helpdesk#3842 is the issue that outlines the plan for this brownout.
See: https://repo.jenkins-ci.org/artifactory/orphans/org/connectbot/jbcrypt/jbcrypt/ cc @MarkEWaite so you can continue your tests. |
* Announce Artifactory brownout Wed Dec 6, 2023 jenkins-infra/helpdesk#3842 is the issue that outlines the plan for this brownout. * Link to developer list announcement * Fix spelling errors
Analyzing artifactory plugin build failureThe artifactory plugin has a dependency on the token macro plugin 2.12. The token macro plugin 2.12 has a dependency on com.jayway.jsonpath:json-path:2.4.0. Our cached copy of the JGit repository includes an outdated copy of the json-path 2.4.0 pom file for the 2.4.0 release of json-path. Based on comments in openhab/openhab-addons#10102 (comment) , we need to replace our outdated copy of 2.4.0 with an updated pom file and jar file. If we don't want to replace the file in our cached copy of the JGit repository (because the outdated file still exists in the Eclipse JGit repository), we could place a fresh copy into the orphans repository and move the orphans repository before the JGit repository in our virtual repository search order. Is there a way to safely remove or exclude the outdated / incorrect json-path 2.4.0 that we are getting from the Eclipse repository? |
Analyzing configuration as code plugin build failureThe configuration as code plugin depends on the crowd2 plugin. The crowd2 plugin depends on the crowd-integration-client-common 5.1.5 library that appears to only be available from our jcenter-cache. Since crowd2 is an Atlassian product, I expected it to be in the atlassian-cache repository, but it does not seem to be there. We could add the Atlassian repository to the configuration as code plugin like this:
That won't solve the case for other projects that depend on the Atlassian libraries. It also does not explain why the Atlassian libraries are not in our atlassian-cache repository but are in our jcenter-cache repository. |
Analyzing Jira plugin build failureThe Jira plugin depends on Atlassian libraries that are available from our jcenter-cache but are not found unless jcenter-cache is included. The build passed after I added the Atlassian repository to the jira-plugin pom file:
|
Analyzing robot plugin build failureThe robot plugin requires Java 11. It won't build with Java 21. The robot plugin has a dependency on com.jayway.jsonpath:json-path:2.4.0. Our cached copy of the JGit repository includes an outdated copy of the json-path 2.4.0 pom file for the 2.4.0 release of json-path. Based on comments in openhab/openhab-addons#10102 (comment) , we need to replace our outdated copy of 2.4.0 with an updated pom file and jar file. If we don't want to replace the file in our cached copy of the JGit repository (because the outdated file still exists in the Eclipse JGit repository), we could place a fresh copy into the orphans repository and move the orphans repository before the JGit repository in our virtual repository search order. Is there a way to safely remove or exclude the outdated / incorrect json-path 2.4.0 that we are getting from the Eclipse repository? |
Analyzing promoted builds plugin build failureThe promoted builds plugin has an optional dependency on the project inheritance plugin 21.04.03. The project inheritance plugin 21.04.03 pom file uses http instead of https in its repository URL references. Maven rejects attempts to from an http repository. There are hints like this on several pages that http blocking can be bypassed with a |
Any particular reason why? |
I could not find the artifact org.connectbot.jbcrypt:jbcrypt:1.0.0 on repo.jenkins-ci.org in any repository other than jcenter-cache. Since the goal is to remove jcenter-cache from the definition of the public virtual repository, it seemed like we need to copy that artifact to a new repository that can be included in the public virtual repository without including all of jcenter-cache. Did that address your question or is there something more that I'm not understanding in the question? |
From my proposal of the new repo holding these dependencies:
That trivially ensures identical artifacts. Per Damien,
That looks very manual and results in differences. For example, we don't have If my proposed approach is not possible, or has drawbacks I'm unaware of, and that's the reason for the different approach, I'd like to know. |
Googling brings up https://jfrog.com/help/r/jfrog-artifactory-documentation/basic-settings-for-remote-repositories which shows that you can configure exclude patterns on a remote repository (i.e., a cache). Based on this documentation I think we should delete the |
Confluence Publisher is also impacted: #3856 |
* Suspend Crowd2 plugin distribution - uses closed source dependency jenkins-infra/helpdesk#3854 explains that the Crowd2 integration plugin uses a dependency that is not open source licensed. The Crowd2 integration library is Atlassian licensed as described in jenkins-infra/helpdesk#3842 (comment) The Atlassian license is not an open source license. Refer to https://www.atlassian.com/legal/software-license-agreement for the details of the license. https://www.jenkins.io/project/governance/#license says that the Jenkins project requires plugins that it distributes to be open source, including their dependencies. When a closed source dependency is detected in a plugin, we suspend distribution of that plugin. If maintainers update the plugin to remove the closed source dependency, distribution can begin for the new release that removes the closed source dependency. Fixes jenkins-infra/helpdesk#3854 * Better comment phrase * Reduce diffs
Pairing operation with @MarkEWaite today: the repository https://repo.jenkins-ci.org/artifactory/atlassian-maven-public is now available publicly (added to the permissions scheme "Any Remote" which adds Read and Cache to the |
Start of the 2nd brownout on repo.jenkins-ci.org with @MarkEWaite: We've updated the curl --silent https://repo.jenkins-ci.org/api/repositories/public | jq -r '.'
{
"key": "public",
"packageType": "maven",
"description": "All public repositories aggregated for the purpose of building Jenkins and its plugins",
"repositories": [
"releases",
"orphans",
"atlassian-maven-public",
"shibboleth",
"maven.jenkins-ci.org",
"maven.twttr.com",
"javanet2",
"tmate",
"cloudbees.public.release",
"jgit",
"netbeans",
"fusesource",
"snapshots"
],
"rclass": "virtual"
}
|
After implementing the production change, we found that the artifactory plugin depends on a 3.1.7 release of a platform pom that is only available from the Spring plugins repository at https://repo.spring.io/plugins-release/ . Other versions of that artifact are available from Atlassian public, but not that version. @dduportal created a jcenter-orphans remote repository that includes a specific item com.atlassian.pom:public-pom:pom:4.0.7 was then missing. That was fixed by adding another entry to jcenter-orphans com.atlassian.sal:sal-api:jar:3.0.7 was missing. Items that were missing have been added to the jcenter-orphans remote repository with a narrow inclusion definition to avoid including any more of jcenter than those few items that are not available elsewhere. The jcenter-orphans repository is intentionally placed near the end of the search list so that the official Atlassian repository cache is used first., |
The crawler job that detects updated tools is failing on ci.jenkins.io with a Grape error Resolved by adding Maven central to the Grape repository list in: |
Thanks @basil for the batch of PRs to fix and improve the crawler (and @MarkEWaite @timja @NotMyFault for the revies/deploys). |
Update:
|
Thanks very much. Should the same change be applied to |
The Artifactory bandwidth reduction project described at jenkins-infra/helpdesk#3842 has shown that some outdated JIRA API jar files are only available from the jcenter cache. Let's test with a newer version of the Jira plugin to avoid that older JIRA API jar file dependency. JIRA plugin 3.12 fixes a security vulnerability. It was released 4 weeks ago.
* Require Jenkins 2.401.3 or newer https://stats.jenkins.io/pluginversions/declarative-pipeline-migration-assistant.html shows: * 88% of the installations of the most recent release are already running Jenkins 2.401.3 or newer. The most recent release was 4 months ago. The most recent release is installed on 460 / 1388 total installations. * 48% of the installations of the next most recent release are already running Jenkins 2.401.3 or newer. That release was 6 months ago. That release is installed on 104 / 1388 total installations. Updating to require Jenkins 2.401.3 or newer allows us to rely on the plugin bill of materials for two more version declarations: * GitHub plugin * Ansicolor plugin Updating to require Jenkins 2.401.3 replaces the following two pull requests: * #233 #233 * #234 #234 * Test with JIRA plugin 3.12 to avoid outdated API jar dependency The Artifactory bandwidth reduction project described at jenkins-infra/helpdesk#3842 has shown that some outdated JIRA API jar files are only available from the jcenter cache. Let's test with a newer version of the Jira plugin to avoid that older JIRA API jar file dependency. JIRA plugin 3.12 fixes a security vulnerability. It was released 4 weeks ago. * Depend on build timeout plugin 1.31 Build timeout plugin 1.31 includes the removal of Prototype.js that is required for Jenkins 2.426.1 and newer. It was released 7 months ago and is already installed on 40% of all build timeout plugin installations. * Use parent pom 4.76 * Test with Java 21 and Java 17 Java 21 was released Sep 19, 2023. Full support for Java 21 was announced in November 2023. The acceptance test harness and plugin bill of materials tests are already passing with Java 21. This is a further step to improve plugin readiness for use with Java 21 and for development with Java 21. The change intentionally tests only two Java configurations, Java 17 and Java 21 because we believe that the risk of a regression that only affects Java 11 is low. We generate Java 11 byte code with the Java 17 and the Java 21 builds, so we're already testing Java 11 byte code. Java 11 is also tested for this plugin in the plugin bill of materials each week.
Jenkins core 2.439 and later now depend on a version of the jbcrypt library that is available from Apache Maven Central. Thanks to @andham for that improvement. Refer to the merged pull request: |
The repository |
Update:
|
Service(s)
Artifactory
Summary
The data transfer volume from https://repo.jenkins-ci.org is still unexpectedly high. Log file analysis for a recent 5 day period shows that the top repositories based on the sum of the sizes of the requested files are:
JFrog has explained that even though we removed Apache Maven Central from the public repository, the contents of Apache Maven Central still exist as copies in the jcenter repository. We need to also remove the jcenter repository and the jcenter-cache repository from the definition of the public repository so that binaries from Apache Maven Central will be resolved from Apache Maven Central instead of being resolved from our repository.
When I had tried some initial experiments removing jcenter, several months ago, there were some cases where it appeared that we depend on artifacts that are only in jcenter and possibly only in our cache of jcenter. Unfortunately, my memory is weak and I cannot find any notes that reference that experiment.
I think that we should take the following steps:
com.jayway.jsonpath
groupId and all its packages from thejgit-cache
. The artifactory plugin build failure and the robot plugin build failure will be resolved by that exclusion. The comment from @basil includes more detailsweekly-test
configuration of the plugin bill of materialsweekly-test
configuration of the plugin bill of materialsReproduction steps
Steps that I took to analyze the requests:
select repo,sum(size_bytes)/1024/1024/1024 as SUM_GB from logs group by repo order by SUM_GB desc limit 10;
The text was updated successfully, but these errors were encountered: