Fetching list of changed files fails with "rate limit exceeded for installation ID"

[lhotari]

Problem

I'm working on refactoring the existing Apache Pulsar (apache/pulsar) build in my fork lhotari/pulsar.

This is the way how I'm using paths-filter:
https://github.com/lhotari/pulsar/blob/ef23acb30548a37de648cc89965f854668975e95/.github/workflows/pulsar-ci.yaml#L32-L50

However this fails after a few builds with this type of errors

Fetching list of changed files for PR#11 from Github API
Number of changed_files is 157
Invoking listFiles(pull_number: 11, page: 1, per_page: 100)
Error: API rate limit exceeded for installation ID 6407194.

failed workflow run.

Full logs

2021-03-05T09:10:57.7553228Z ##[section]Starting: Request a runner to run this job
2021-03-05T09:10:58.4697507Z Can't find any online and idle self-hosted runner in current repository that matches the required labels: 'ubuntu-20.04'
2021-03-05T09:10:58.4697604Z Can't find any online and idle self-hosted runner in current repository's account/organization that matches the required labels: 'ubuntu-20.04'
2021-03-05T09:10:58.4697805Z Found online and idle hosted runner in current repository's account/organization that matches the required labels: 'ubuntu-20.04'
2021-03-05T09:10:58.7015644Z ##[section]Finishing: Request a runner to run this job
2021-03-05T09:11:05.2736491Z Current runner version: '2.277.1'
2021-03-05T09:11:05.2761183Z ##[group]Operating System
2021-03-05T09:11:05.2761887Z Ubuntu
2021-03-05T09:11:05.2762255Z 20.04.2
2021-03-05T09:11:05.2762584Z LTS
2021-03-05T09:11:05.2762925Z ##[endgroup]
2021-03-05T09:11:05.2763374Z ##[group]Virtual Environment
2021-03-05T09:11:05.2763851Z Environment: ubuntu-20.04
2021-03-05T09:11:05.2764295Z Version: 20210302.0
2021-03-05T09:11:05.2765039Z Included Software: https://github.com/actions/virtual-environments/blob/ubuntu20/20210302.0/images/linux/Ubuntu2004-README.md
2021-03-05T09:11:05.2766380Z Image Release: https://github.com/actions/virtual-environments/releases/tag/ubuntu20%2F
2021-03-05T09:11:05.2767122Z ##[endgroup]
2021-03-05T09:11:05.2768917Z ##[group]GITHUB_TOKEN Permissions
2021-03-05T09:11:05.2770175Z Actions: write
2021-03-05T09:11:05.2770932Z Checks: write
2021-03-05T09:11:05.2771460Z Contents: write
2021-03-05T09:11:05.2772083Z Deployments: write
2021-03-05T09:11:05.2772752Z Issues: write
2021-03-05T09:11:05.2773287Z Metadata: read
2021-03-05T09:11:05.2774019Z OrganizationPackages: write
2021-03-05T09:11:05.2774678Z Packages: write
2021-03-05T09:11:05.2775315Z PullRequests: write
2021-03-05T09:11:05.2775974Z RepositoryProjects: write
2021-03-05T09:11:05.2776698Z SecurityEvents: write
2021-03-05T09:11:05.2777277Z Statuses: write
2021-03-05T09:11:05.2777911Z ##[endgroup]
2021-03-05T09:11:05.2781103Z Prepare workflow directory
2021-03-05T09:11:05.3421642Z Prepare all required actions
2021-03-05T09:11:05.3430093Z Getting action download info
2021-03-05T09:11:05.7040658Z Download action repository 'dorny/paths-filter@v2'
2021-03-05T09:11:08.1104090Z ##[group]Run dorny/paths-filter@v2
2021-03-05T09:11:08.1104621Z with:
2021-03-05T09:11:08.1105303Z   token: ***
2021-03-05T09:11:08.1106287Z   filters: src:
  # pattern syntax uses https://github.com/micromatch/picomatch
  - '!((site2|deployment)/**)'
2021-03-05T09:11:08.1107127Z   list-files: none

2021-03-05T09:11:08.1107678Z   initial-fetch-depth: 10

2021-03-05T09:11:08.1108218Z env:

2021-03-05T09:11:08.1110259Z   MAVEN_OPTS: -Dmaven.test.failure.ignore=true -DtestRetryCount=0 -Xmx1024m -Dhttp.keepAlive=false -Dmaven.wagon.http.pool=false -Dmaven.wagon.http.retryHandler.class=standard -Dmaven.wagon.http.retryHandler.count=3

2021-03-05T09:11:08.1112038Z ##[endgroup]

2021-03-05T09:11:08.5138995Z ##[group]Fetching list of changed files for PR#11 from Github API

2021-03-05T09:11:08.5140857Z Number of changed_files is 157

2021-03-05T09:11:08.5331126Z Invoking listFiles(pull_number: 11, page: 1, per_page: 100)

2021-03-05T09:11:08.6419876Z ##[error]API rate limit exceeded for installation ID 6407194.

2021-03-05T09:11:08.6544385Z Evaluate and set job outputs

2021-03-05T09:11:08.6564810Z Cleaning up orphan processes

It seems that paths-filters exceeds the GitHub API rate limits when using a large repository like apache/pulsar.

Expected behavior

paths-filter would use GitHub API in a way that it doesn't exceed the rate limiting.

Is there any workaround for this problem?

Other

I have reported a similar issue about test-reporter, dorny/test-reporter#68

[dorny]

Hello,

I don't see much possibilities how paths-filter could erroneously make too many API requests.
I already commented on dorny/test-reporter#68 - there is clearly inefficient implementation which makes too many API requests.

Wouldn't be possible that test-reporter was a root cause and paths-filter failed simply because API rate limit was already exceeded?

I would suggest to wait till test-reporter is fixed and then see if there is still some issue with paths-filter.

[lhotari]

Yes, the problem was most likely triggered by test-reporter. I'll close this issue. Thanks for the 2 great GH Actions. I like the design and feature set.