-
Notifications
You must be signed in to change notification settings - Fork 130
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[Dependency Cache] Enable Dependency Cache on CI #13303
Merged
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
ParaskP7
added
status: do not merge
Dependent on another PR, ready for review but not ready for merge.
category: tooling
Anything that involves building & maintaining the project, including scripts, `Fastfile`, etc.
Core
labels
Jan 14, 2025
ParaskP7
force-pushed
the
ci/enable-dependency-cache-on-ci-final
branch
from
January 14, 2025 12:09
6216579
to
af97b81
Compare
📲 You can test the changes from this Pull Request in WooCommerce-Wear Android by scanning the QR code below to install the corresponding build.
|
📲 You can test the changes from this Pull Request in WooCommerce Android by scanning the QR code below to install the corresponding build.
|
ParaskP7
force-pushed
the
ci/enable-dependency-cache-on-ci-final
branch
from
January 14, 2025 12:29
af97b81
to
6ba4e7c
Compare
ParaskP7
changed the title
[Dependency Cache] Dependency Cache on CI
[Dependency Cache] Enable Dependency Cache on CI
Jan 14, 2025
ParaskP7
force-pushed
the
ci/enable-dependency-cache-on-ci-final
branch
from
January 14, 2025 15:21
6ba4e7c
to
4475a6a
Compare
ParaskP7
removed
the
status: do not merge
Dependent on another PR, ready for review but not ready for merge.
label
Jan 15, 2025
FYI: This change is done for testing purposes and until the below 'A8C CI Toolkit' #135 PR gets merged to 'trunk'. When that's done, the 'a8c-ci-toolkit' will be updated to '3.9.0' instead. A8C CI Toolkit PR: [Dependency Updates] [Dependency Cache] Dependency Cache on CI per Project [without GRADLE_RO_DEP_CACHE] #135 - Automattic/a8c-ci-toolkit-buildkite-plugin#135
FYI: This job will be then used by 'buildkite-ci' and configured as a 'buildkite_pipeline_schedule' with a weekly frequency. PS: The targeted 'pipeline' related jobs are: - Mobile App - Wear App - Lint - Unit Tests - Android tests
PS: The targeted 'pipeline' related jobs are: - Mobile App - Wear App - Lint - Unit Tests - Android tests PS: The 'detekt' job isn't targeted because it only takes about 20+ seconds of network request to download all the dependencies needed for this specific job, which is about the same time it take for the dependency cache to actually be restored (10+ seconds). Thus, this optimization is not really worth it for this specific job. For all those other jobs targeted, it take more than a minute of network requests to download all the dependencies needed, and as such worth the diff is worth it.
Release Notes: https://github.com/Automattic/ a8c-ci-toolkit-buildkite-plugin/releases/tag/3.9.0 Related: 4475a6a
ParaskP7
force-pushed
the
ci/enable-dependency-cache-on-ci-final
branch
from
January 15, 2025 09:07
9820220
to
3547e66
Compare
FYI: This change is done for testing purposes and until the below 'A8C CI Toolkit' #138 PR gets merged to 'trunk'. When that's done, the 'a8c-ci-toolkit' will be updated to '3.9.1' instead. A8C CI Toolkit PR: [Dependency Cache] Fix Dependency Cache #138 - Automattic/a8c-ci-toolkit-buildkite-plugin#138
1 task
Release Notes: https://github.com/Automattic/ a8c-ci-toolkit-buildkite-plugin/releases/tag/3.9.1 Related: 49d771a
wzieba
approved these changes
Jan 15, 2025
Co-authored-by: Wojciech Zięba <wojtek.zieba@automattic.com>
PR Comment: https://github.com/woocommerce/woocommerce-android/pull/ 13303#discussion_r1916752265
…to ci/enable-dependency-cache-on-ci-final
When using 'single-cmd' and not using this double quotes structure it is is producing 'did not find expected key' type of errors.
…to ci/enable-dependency-cache-on-ci-final
This was referenced Jan 17, 2025
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Labels
category: tooling
Anything that involves building & maintaining the project, including scripts, `Fastfile`, etc.
Core
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Project Thread: paaHJt-7CE-p2
Related: #13170 + #13122 + #13288
Depends On: a8c-ci-toolkit#135 -> a8c-ci-toolkit#138
Required By: buildkite-ci#576
Description
This PR enables dependency cache on CI for this project. 💾
The dependency cache mechanism is split between:
WooCommerce:assembleJalapenoDebug
)WooCommerce-Wear:assembleJalapenoDebug
)WooCommerce:lintJalapenoDebug
)ssembleJalapenoDebugUnitTest assembleDebugUnitTest
)assembleJalapenoDebugAndroidTest
)prototype-build.sh
)prototype-build.sh
)lint.sh
)run-unit-tests.sh
)run-instrumented-tests.sh
)PS.1: The
detekt
job isn't targeted because it only takes about 20+ seconds of network request to download all the dependencies needed for this specific job (build + scan), which is about the same time it takes for the dependency cache to actually be restored, which is more or less about 10+ seconds (build). Thus, this optimization is not really worth it for this specific job. For all those other jobs targeted, it takes more than a minute of network requests to download all the dependencies needed (build + scan), and as such the diff is worth it.PS.2: Dependency cache isn't enabled on the release-builds.yml pipeline just yet, but just temporarily, and until we become comfortable with running this new process on the main pipeline.yml pipeline.
Future Improvements
As per this comment, me and @wzieba has been discussing on DMs about some possible future improvement, which are including but are not limited to the below list:
assemble android test
task, which depends onassemble
task anyway, and would most likely resolve most of the dependencies, for all other tasks to benefit from. However, this needs further testing to make sure that running other tasks such uslint
andassemble unit test
are providing little to no help.init.d
). Using the init script, go through all available modules for a project, and then, using that custom Gradle task, resolve all dependencies for all tasks that are available per module. We could take some inspiration from plugins such as this.dependency cache
build to run weekly (buildkite-ci#576), make it run more often, possibly scheduling it to even run daily. However, since dependencies are not being updated too often, and, even when they do on that specific week, it would only be for a few of them. This will indeed cause some extra network requests, but most of the dependencies cached would still benefit from the existing and weekly refreshed dependency cache.Testing information
This dependency cache solution was thoroughly tested via this #13170 test-only PR, but to help you asserting that this change is indeed making an impact, plus not causing any trouble for the project, try and follow the below:
Save Cache
:dependency cache
job to complete: 7m 42s#android-core-notifs
):The dependency cache has been updated.
(p1736943241087289-slack-C0787KRDYDC)Failure to update the dependency cache.
(not tested, but should behave similarly to ☝️)Restore Cache
:- 1m 45s
0s
1m 3s
- 1m 47s
0s
44s
- 2m 3s
0s
1m 35s
- 2m 3s
0s
1m 21s
+ 3m 34s
0s
1m 23s
Note
Without
: Without Dependency CacheWith
: With Dependency CacheBT
: Build TimeNR
: Network RequestsRC
: Restore CacheWin
: NR (Without) - RC (With)Warning
(*): The
BT (Diff)
presented above is relative, so don't read too much into it, it might not match with theWIN
, which is a much more solid number. But still, it is nice to have and compare those two number together. To to be clear about that, although I tried to pick the most representative builds, without and with the dependency cache mechanism in place, there are other factors that can make any build faster, or slower for that matter (likebuild cache
,firebase test lab
, etc), so again, don't read too much into it.Merge Instructions
status: do not merge
label.RELEASE-NOTES.txt
if necessary. Use the "[Internal]" label for non-user-facing changes.Reviewer (or Author, in the case of optional code reviews):
Please make sure these conditions are met before approving the PR, or request changes if the PR needs improvement: