Reduce Artifactory storage and bandwidth use

[MarkEWaite]

Service(s)

Artifactory

Summary

Jenkins artifact storage use has increased to greater than 10 TB and bandwidth use has recently increased significantly, almost to the bandwidth use prior to the work done in:

• Remove jcenter, and oss.sonatype.org-releases repositories from public virtual repository; reconfigure Atlassian remote repositories #3842

JFrog sponsors open source without charging the open source project. They have asked us to reduce our storage to less than their 5 TB open source sponsorship threshold. We also want to reduce our bandwidth use to be much closer to the bandwidth we were using in early 2024.

Reproduction steps

1. Review the storage use and reduce storage use to less than 5 TB
2. Review the bandwidth use and reduce to be much closer to the January 2024 usage

[dduportal]

Let's start with a snapshot of metrics provided by JFrog so we can watch the effect of any change.

• Outbound Bandwidth (e.g. amount of data downloaded) can be seen in JFrog's portal:

• Storage (e.g. amount of data stored on disk) can be found in Artifactory administration:

=> We are clearly pointing fingers to the Atlassian Mirror `` for both storage and outbound bandwidth as the top-level item to work on
=> Cleaning up incrementals repository is useful BUT should be secondary

[timja]

I mean the atlassian one should be fairly easy to resolve, just swap it to includes only.

[dduportal]

Update: we've granted access to @darinpope to Artifactory so he can drive this topic:

• Ref. https://groups.google.com/g/jenkinsci-dev/c/6_X0ABdLAgE/m/3XwzSn1xEAAJ for the decision process
• I've created him a secondary (non LDAP) account darinpope-admin with a permission scheme named darinpope-helpdesk-4533 for easier tracking (and reverting in the future)
  • He already has accessed the account and set up a strong password
  • @darinpope do you mind enabling 2FA on this account (since it is not an LDAP)?
  • Despite the account name (darinpope-admin) , he does not have administration permission: the permission scheme allows him to Manage only 5 repositories: all *atlassian* mirrors and the incrementals. That should be sufficient enough: we'll see if he need more permissions.

[timja]

I wonder if we still need the jcenter-cache as well?

[dduportal]

I mean the atlassian one should be fairly easy to resolve, just swap it to includes only.

Let's now track work for each subject on associated issues:

• Work related to atlassian mirror: [repo.jenkins-ci.org] Atlassian Mirror Repository: Artifactory outdated maven-metadata.xml for public/com/github/jnr/jnr-posix/ #4472
• Work related to incrementals: [INFRA-2821] Remove unused incrementals #2386

=> there might be subsqiuent follows up issue.

[dduportal]

I wonder if we still need the jcenter-cache as well?

Yes, we do. It has artifacts which have been deleted from remotes as far as I remember. Worth checking this subject but let's not get carried away: if @darinpope is able to solve the atlassian and incremental, we'll see immediate results to share with JFrog

[darinpope]

@dduportal I've dug around and cannot find a way to implement MFA on my admin account. Is that another switch you need to set on my account?

[dduportal]

@dduportal I've dug around and cannot find a way to implement MFA on my admin account. Is that another switch you need to set on my account?

My mistake: MFA is only available on the JFrog portal, not on Artifactory. Sorry for wasting your time on this one.

Ref. #4472 (comment):

@dduportal with my current permissions with the admin user, I don't have access to look at the configuratio page for repos.

@darinpope I've set your account as admin, you should have access now

[darinpope]

applied com/atlassian/**/* as an include on atlassian-maven-public at 10:15am Central/1615 UTC on 2025-02-11

Now waiting to see if the repo cleans itself up or not. If not by this evening my time, I'll start purging.

[darinpope]

Well, that was fast. It appears that it's already cleaned up.

but not really. It appears it still exists on disk:

so now to figure out how/if the HDD gets cleaned up

[darinpope]

It might be because of "Store Artifacts Locally":

I want to wait on unchecking that box until we meet with JFrog on 2025-02-12.

[MarkEWaite]

Thanks very much! I've started jobs on ci.jenkins.io that I expect will require Atlassian libraries. In case others want to check those jobs, here they are with embeddable build status links:

[timja]

I think what dduportal normally does is move all artifacts out to a separate archived repository that isn't mirrored.

Then once we are confident we can delete?

We can likely just repopulate the atlassian one by running builds like Mark has.

[dduportal]

I think what dduportal normally does is move all artifacts out to a separate archived repository that isn't mirrored.

Then once we are confident we can delete?

Usually, we let the Artifactory's internal Garbage Collector to kick. But we never really checked artifacts were really deleted (unless exceptional cases with less than 10 occurrences) from disk as the storage always has been advertised by JFrog as "not an issue" until nowadays (focus was on bandwidth and builds safety).
Additionally, there are artifacts only present on mirror of upstream which are now gone so we used to be really careful.

The "Store Artifacts Locally" checkbox (to unselect) seems like a good idea for Atlassian public specifically.

We can likely just repopulate the atlassian one by running builds like Mark has.

+1 with @timja : it make sense for this specific repository. @darinpope I believe you can go for this

[darinpope]

We implemented the new artifactory-pkg-maven-cache today (2025-02-13) into public and remove the old atlassian entry around 1545 UTC.

At ~1910 UTC, I changed the include pattern from com/atlassian/**/* to just com/atlassian/** to see if that will make the pull through work as we expect.

[MarkEWaite]

@yonarbel could you post the most recent usage status report? We have our weekly infra team meeting tomorrow and we want to be sure that storage use is still less than 5 TB. @darinpope is out of the office at the moment, so the next reduction in storage use may be delayed until he returns, but we have a plan for the next reduction.

[MarkEWaite]

Latest usage is above 5 TB of storage:

[MarkEWaite]

@dduportal will investigate the increased storage use tomorrow. @darinpope is out of the office with a family emergency.

[darinpope]

I'm still out, but I did change the include com/atlassian/connect/** to com/atlassian/connect/ac-play-java_2.10/** because the only plugin using that path is zephyr-for-jira-test-management-plugin. It looks like the last release of that plugin was 7+ years ago, so we might be able to completely drop the connect include. I'm just trying to see if that properly cleans up the connect path in Artifactory.

[darinpope]

The repo cleans up, but the cache doesn't, so I'm going through and deleting the leftover directories in cache.

[MarkEWaite]

the only plugin using that path is zephyr-for-jira-test-management-plugin. It looks like the last release of that plugin was 7+ years ago, so we might be able to completely drop the connect include.

I agree that we can drop the connect include.

The plugin has less than 200 installations and has 2 unresolved security vulnerabilities. As far as I can tell from the Atlassian documentation, that plugin has been superseded by one of these two plugins:

• Zephyr enterprise test management
• Zephyr Scale

[darinpope]

I'm going to do the same variation for the plugins include. There's a lot we can get rid of. Looking at:

https://github.com/search?q=org%3Ajenkinsci+%22%3CgroupId%3Ecom.atlassian.plugins%3C%2FgroupId%3E%22&type=code

we can add two include paths for plugins (atlassian-plugins-core and atlassian-plugins-osgi) and really trim that down too. I'm starting on that now.

[darinpope]

Same with crowd:

https://github.com/search?q=org%3Ajenkinsci+%22%3CgroupId%3Ecom.atlassian.crowd%3C%2FgroupId%3E%22&type=code

headed towards:

• com/atlassian/crowd/crowd-integration-client/**
• com/atlassian/crowd/crowd-core/**
• com/atlassian/crowd/crowd-integration-client-common/**
• com/atlassian/crowd/crowd-integration-api/**

[darinpope]

Same with confluence:

https://github.com/search?q=org%3Ajenkinsci+%22%3CgroupId%3Ecom.atlassian.confluence%3C%2FgroupId%3E%22&type=code

headed towards:

• com/atlassian/confluence/confluence-rest-client/**

[darinpope]

Here's the rough before I started deleting connect, plugins, crowd and confluence (I forgot to do a real capture but this is close):

and after:

so we are headed in the right direction. I need to dig through the other top levels, mainly bitbucket and jira, to see how to tighten them up.

[darinpope]

Same for fugue:

https://github.com/search?q=org%3Ajenkinsci+%22%3CgroupId%3Ecom.atlassian.fugue%3C%2FgroupId%3E%22&type=code

• com/atlassian/fugue/fugue/**

[darinpope]

Same for sal:

https://github.com/search?q=org%3Ajenkinsci+%22%3CgroupId%3Ecom.atlassian.sal%3C%2FgroupId%3E%22&type=code

• com/atlassian/sal/sal-api/**

[darinpope]

yes, jira is the problem child:

it will probably be tomorrow before I can work through that. I need to make sure I don't break any plugin builds after I put in the includes.

[darinpope]

https://github.com/search?q=org%3Ajenkinsci+%22%3CgroupId%3Ecom.atlassian.jira%3C%2FgroupId%3E%22&type=code

• com/atlassian/jira/atlassian-jira/**
• com/atlassian/jira/jira-rest-java-client-api/**
• com/atlassian/jira/jira-rest-java-client-core/**
• com/atlassian/jira/jira-rest-java-client/**
• com/atlassian/jira/jira-rest-plugin/**

Again, I'm holding on cleaning up the cache for jira until I get back to a more stable internet connection later this evening Central time. It'll probably run a good part of my night and then I can verify in the morning.

[darinpope]

The cleanup ran pretty fast, so I tested buildling jira-plugin. I had to add two more includes:

• com/atlassian/jira/jira-rest-java-client-parent/**
• com/atlassian/sal/sal-parent/**

I'm not calling this done yet, but definitely better that earlier today.

[darinpope]

When building JiraTestResultReporter-plugin, I had to add a handful more includes:

• com/atlassian/jira/jira-internal-bom/**
• com/atlassian/jira/jira-bom/**
• com/atlassian/jira/jira-api-bom/**
• com/atlassian/jira/jira-deprecated-api-bom/**
• com/atlassian/plugins/rest/atlassian-rest-exported-libraries/**
• com/atlassian/plugins/rest/atlassian-rest-parent/**
• com/atlassian/plugins/rest/atlassian-rest-v2-exported-libraries/**

[darinpope]

here's the current usage (timestamp in shot)

so I think its closer. If a plugin fails, we'll need to determine the correct include to add.

However, we're right on the edge of 4TB, so the sooner we can get jcenter-cache archived, the better.

[darinpope]

Looks like we are overall holding reasonably steady after about 9 hours:

[jglick]

lots of "weird" paths (activeobject, adf, alternator, etc) that none of the plugins use

Can the POMs of the Jira-related plugin(s) be improved to exclude large unused transitive deps so the problem does not recur, rather than trying to hack around in repository configuration? (disregard if I failed to follow the discussion correctly)

[darinpope]

@jglick I agree that's the right mid/long term solution, but we needed to get back quickly below the storage limit that JFrog requested.

FTR, here's the plugins I tested with mvn clean verify using Maven 3.9.9 and Temurin 17.0.14:

• JiraTestResultReporter-plugin
• atlassian-bitbucket-server-integration-plugin
• atlassian-jira-software-cloud-plugin
• confluence-publisher-plugin
• jira-plugin

There are a few others that import com.atlassian or io.atlassian (crowd, artifactory, etc) but I didn't test them right now.

[dduportal]

@jglick I agree that's the right mid/long term solution, but we needed to get back quickly below the storage limit that JFrog requested.

FTR, here's the plugins I tested with mvn clean verify using Maven 3.9.9 and Temurin 17.0.14:

* `JiraTestResultReporter-plugin`

* `atlassian-bitbucket-server-integration-plugin`

* `atlassian-jira-software-cloud-plugin`

* `confluence-publisher-plugin`

* `jira-plugin`

There are a few others that import com.atlassian or io.atlassian (crowd, artifactory, etc) but I didn't test them right now.

Do you mind if we add https://github.com/jenkins-infra/repository-permissions-updater/blob/c2cd952d3afb2e808ec0e647221c4ce15d4d9c01/pom.xml#L120-L124 to the list of "sanity checks" (to avoid reproducing #4578)?

[darinpope]

yeah, that shouldn't be any problem. I see that its already been added. I didn't think about searching through jenkins-infra for usages 🤦

[dduportal]

yeah, that shouldn't be any problem. I see that its already been added. I didn't think about searching through jenkins-infra for usages 🤦

No worries, we also did not check either and it is an healthy reminder! Thanks for the huge work @darinpope !

[darinpope]

@dduportal since we seem to have atlassian-pkg-maven-public pretty well dialed in and it's the only one active right now:

would it make sense to delete the other 3 remote repos:

• atlassian = 109GB
• atlassian-maven-atlassian-external = 0 bytes
• private-atlassian = 0 bytes

We're only talking 109GB, but if we don't need it, why keep it around?

[MarkEWaite]

Makes sense to me to delete them