Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Create initial workflow for publishing to Maven Central #1600

Merged
merged 77 commits into from
Jun 30, 2024
Merged
Show file tree
Hide file tree
Changes from 63 commits
Commits
Show all changes
77 commits
Select commit Hold shift + click to select a range
eaf4796
WIP Create initial workflow for publishing to Maven Central (#1594)
jonathanl-bq Jun 18, 2024
cde6106
Add AWS_ACTIONS option
jonathanl-bq Jun 18, 2024
94262f1
Sign JAR file
jonathanl-bq Jun 19, 2024
cccdd96
Fix signing issue
jonathanl-bq Jun 19, 2024
f8954bb
Try to fix issue with generating secring.gpg file
jonathanl-bq Jun 20, 2024
b452845
Fix path to secring.gpg
jonathanl-bq Jun 20, 2024
cc3b9db
Try to fix secring.gpg retrieval issue
jonathanl-bq Jun 20, 2024
443d7d8
Remove base64 decode
jonathanl-bq Jun 20, 2024
5cf24ff
Try to fix multi-line issue with GPG key secret
jonathanl-bq Jun 20, 2024
696668e
Go back to echo approach
jonathanl-bq Jun 21, 2024
93bb466
Decode base64 properly this time
jonathanl-bq Jun 21, 2024
003c819
Use GPG_KEY_ID
jonathanl-bq Jun 21, 2024
d748048
Surround password in quotes
jonathanl-bq Jun 21, 2024
f8be434
Publish JAR to local Maven and upload
jonathanl-bq Jun 21, 2024
b79ae55
Update examples build.gradle
jonathanl-bq Jun 21, 2024
6f0a585
Sign publishToMavenLocal build
jonathanl-bq Jun 21, 2024
9dfaa46
Update version of Java JAR
jonathanl-bq Jun 21, 2024
30e3a75
Properly fetch src_folder variable contents
jonathanl-bq Jun 21, 2024
83fcac2
Reorganize JAR contents
jonathanl-bq Jun 21, 2024
8debc66
Update path of uploaded JAR
jonathanl-bq Jun 21, 2024
4ce2317
Update artifact ID
jonathanl-bq Jun 21, 2024
31a8c27
Add missing comma
jonathanl-bq Jun 21, 2024
c981b67
Replace placeholders in build.gradle
jonathanl-bq Jun 21, 2024
469fe3a
Update examples build.gradle
jonathanl-bq Jun 21, 2024
b838801
Remove test runs from java.yml workflow
jonathanl-bq Jun 22, 2024
5912796
Add debugging info to workflow
jonathanl-bq Jun 22, 2024
1e04158
Adjust debug info
jonathanl-bq Jun 22, 2024
9e0b003
Readd placeholder text in build.gradle
jonathanl-bq Jun 22, 2024
d46e768
Add more debug info
jonathanl-bq Jun 22, 2024
35394f2
Change how the JAR is copied
jonathanl-bq Jun 22, 2024
eeaf346
Add configurations for ARM linux and x86 macos
jonathanl-bq Jun 24, 2024
0696057
Prevent output artifacts from being swallowed
jonathanl-bq Jun 24, 2024
ddf4538
Update build matrix to use proper RUNNERs
jonathanl-bq Jun 26, 2024
efda7c9
Try to use self-hosted runner for ARM Linux builds
jonathanl-bq Jun 26, 2024
125e0b1
Delete gradle-cd workflow
jonathanl-bq Jun 26, 2024
8d7377d
Add id-token permissions
jonathanl-bq Jun 26, 2024
d31304b
Add step to setup self-hosted runner access
jonathanl-bq Jun 26, 2024
faab1b3
Add CONTAINER property to java.yml workflow
jonathanl-bq Jun 26, 2024
727119d
Remove install Redis step from java.yml workflow
jonathanl-bq Jun 26, 2024
d502a6d
Remove test-benchmark step from java.yml workflow
jonathanl-bq Jun 26, 2024
d06ba10
Fix issue with Java classifier
jonathanl-bq Jun 26, 2024
e3e48a2
Update java.yml to use classifier
jonathanl-bq Jun 26, 2024
2d9084f
Bump version and add archiveClassifier
jonathanl-bq Jun 26, 2024
4b6dfe8
Change groupId to valkey-client
jonathanl-bq Jun 27, 2024
c9607dd
Update example and base archive name
jonathanl-bq Jun 27, 2024
8b2616c
Update workflow
jonathanl-bq Jun 27, 2024
aeeccf1
Rename to glide-for-redis
jonathanl-bq Jun 27, 2024
faff846
Extracting Java Deployment to a different workflow
affonsov Jun 27, 2024
143818a
trying to make the workflow to build
affonsov Jun 28, 2024
8864434
testing action to prepare build
affonsov Jun 28, 2024
d097a1f
forcing new action to trigger
affonsov Jun 28, 2024
1b5fce9
Revert "forcing new action to trigger"
affonsov Jun 28, 2024
b3d247e
Revert "testing action to prepare build"
affonsov Jun 28, 2024
e4083f4
Revert "trying to make the workflow to build"
affonsov Jun 28, 2024
11f8470
Revert "Extracting Java Deployment to a different workflow"
affonsov Jun 28, 2024
c74ddbb
Revert "Revert "Extracting Java Deployment to a different workflow""
affonsov Jun 28, 2024
7bd96a5
fixing workflow
affonsov Jun 28, 2024
1eab353
fixed path for the local maven
affonsov Jun 28, 2024
26dca80
removing bundle from the tests
affonsov Jun 28, 2024
b112082
fix java workflow
affonsov Jun 28, 2024
379ef60
removing classifier from the pom
affonsov Jun 28, 2024
434b681
fixing concurrency
affonsov Jun 28, 2024
ea98c08
Remove publishToMavenLocal line in examples build.gradle
jonathanl-bq Jun 28, 2024
77cf7b7
fix examples
affonsov Jun 29, 2024
644b0a9
Merge remote-tracking branch 'origin/java/dev_lotjonat_pkg_maven_cent…
affonsov Jun 29, 2024
e7394d7
cleaning up java.yml
affonsov Jun 29, 2024
d0e06b7
testing removing test dependency
affonsov Jun 29, 2024
e448788
adding skip signing
affonsov Jun 29, 2024
35f8c80
Revert "adding skip signing"
affonsov Jun 29, 2024
ae7b23f
Revert "testing removing test dependency"
affonsov Jun 29, 2024
3f02add
Revert "cleaning up java.yml"
affonsov Jun 29, 2024
2742c38
removing dependency of singing in the local build
affonsov Jun 29, 2024
3071d68
java.yml clean up
affonsov Jun 29, 2024
3970193
removing steps from java.yml
affonsov Jun 29, 2024
b0fb61f
added comments
affonsov Jun 29, 2024
8239e1c
Merge branch 'refs/heads/main' into java/dev_lotjonat_pkg_maven_central
affonsov Jun 29, 2024
71e5350
removed step on sed the examples and removed if always from the uploa…
affonsov Jun 30, 2024
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
178 changes: 178 additions & 0 deletions .github/workflows/java-cd.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,178 @@
name: Java Prepare Deployment

on:
pull_request:
paths:
- .github/workflows/java-cd.yml
- .github/workflows/install-shared-dependencies/action.yml
- .github/workflows/start-self-hosted-runner/action.yml
push:
tags:
- "v*.*"

concurrency:
group: java-cd-${{ github.head_ref || github.ref }}
cancel-in-progress: true

permissions:
id-token: write

jobs:
start-self-hosted-runner:
if: github.repository_owner == 'aws'
runs-on: ubuntu-latest
environment: AWS_ACTIONS
steps:
- name: Checkout
uses: actions/checkout@v4
with:
fetch-depth: 0

- name: Start self hosted EC2 runner
uses: ./.github/workflows/start-self-hosted-runner
with:
role-to-assume: ${{ secrets.ROLE_TO_ASSUME }}
aws-region: ${{ secrets.AWS_REGION }}
ec2-instance-id: ${{ secrets.AWS_EC2_INSTANCE_ID }}

create-binaries-to-publish:
needs: start-self-hosted-runner
if: github.repository_owner == 'aws'
timeout-minutes: 35
strategy:
# Run all jobs
fail-fast: false
matrix:
java:
- 11
- 17
redis:
- 6.2.14
- 7.2.3
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

do we need to generate binaries for different versions of redis?

host:
- {
OS: ubuntu,
RUNNER: ubuntu-latest,
TARGET: x86_64-unknown-linux-gnu,
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

why unknown?

CLASSIFIER: linux-x86_64
}
- {
OS: ubuntu,
RUNNER: ["self-hosted", "Linux", "ARM64"],
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is it one runner or 3 different on the same OS/arch?

TARGET: aarch64-unknown-linux-gnu,
CLASSIFIER: linux-aarch_64,
CONTAINER: "2_28"
}
- {
OS: macos,
RUNNER: macos-12,
TARGET: x86_64-apple-darwin,
CLASSIFIER: osx-x86_64
}
- {
OS: macos,
RUNNER: macos-13-xlarge,
TARGET: aarch64-apple-darwin,
CLASSIFIER: osx-aarch_64
}

runs-on: ${{ matrix.host.RUNNER }}

steps:
- name: Setup self-hosted runner access
if: ${{ contains(matrix.build.RUNNER, 'self-hosted') && matrix.build.TARGET != 'aarch64-unknown-linux-musl' }}
run: sudo chown -R $USER:$USER /home/ubuntu/actions-runner/_work/glide-for-redis

- uses: actions/checkout@v4
with:
submodules: recursive

- name: Set the release version
shell: bash
run: |
export version=`if ${{ github.event_name == 'pull_request' }}; then echo '255.255.255'; else echo ${GITHUB_REF:11}; fi`
echo "RELEASE_VERSION=${version}" >> $GITHUB_ENV

- name: Set up JDK ${{ matrix.java }}
uses: actions/setup-java@v4
with:
distribution: "temurin"
java-version: ${{ matrix.java }}

- name: Install shared software dependencies
uses: ./.github/workflows/install-shared-dependencies
with:
os: ${{ matrix.host.OS }}
target: ${{ matrix.host.TARGET }}
github-token: ${{ secrets.GITHUB_TOKEN }}

- name: Install protoc (protobuf)
uses: arduino/setup-protoc@v3
with:
version: "26.1"
repo-token: ${{ secrets.GITHUB_TOKEN }}

- name: Create secret key ring file
working-directory: java/client
run: |
echo "$SECRING_GPG" | base64 --decode > ./secring.gpg
ls -ltr
env:
SECRING_GPG: ${{ secrets.SECRING_GPG }}

- name: Replace placeholders and version in build.gradle
shell: bash
working-directory: ./java/client
run: |
SED_FOR_MACOS=`if [[ "${{ matrix.host.os }}" =~ .*"macos".* ]]; then echo "''"; fi`
sed -i $SED_FOR_MACOS 's/placeholder/${{ matrix.host.CLASSIFIER }}/g' build.gradle
sed -i $SED_FOR_MACOS "s/255.255.255/${{ env.RELEASE_VERSION }}/g" build.gradle

- name: Replace placeholders and version in build.gradle for examples
shell: bash
working-directory: ./java/examples
run: |
SED_FOR_MACOS=`if [[ "${{ matrix.host.os }}" =~ .*"macos".* ]]; then echo "''"; fi`
sed -i $SED_FOR_MACOS 's/placeholder/${{ matrix.host.CLASSIFIER }}/g' build.gradle
sed -i $SED_FOR_MACOS "s/255.255.255/${{ env.RELEASE_VERSION }}/g" build.gradle


- name: Build java client
working-directory: java
run: |
./gradlew :client:publishToMavenLocal -Psigning.secretKeyRingFile=secring.gpg -Psigning.password="${{ secrets.GPG_PASSWORD }}" -Psigning.keyId=${{ secrets.GPG_KEY_ID }}

- name: Ensure no skipped files by linter
working-directory: java
run: ./gradlew spotlessDiagnose | grep 'All formatters are well behaved for all files'
affonsov marked this conversation as resolved.
Show resolved Hide resolved

- name: Bundle JAR
working-directory: java
run: |
src_folder=~/.m2/repository/software/amazon/glide/glide-for-redis/${{ env.RELEASE_VERSION }}
cd $src_folder
jar -cvf bundle.jar *
ls -ltr
cd -
cp $src_folder/bundle.jar .

- name: Upload test reports
affonsov marked this conversation as resolved.
Show resolved Hide resolved
if: always()
continue-on-error: true
uses: actions/upload-artifact@v4
with:
name: test-reports-java-${{ matrix.java }}-redis-${{ matrix.redis }}-${{ matrix.host.TARGET }}
path: |
java/client/build/reports/**
java/integTest/build/reports/**
utils/clusters/**
benchmarks/results/**

- name: Upload artifacts to publish
if: always()
Yury-Fridlyand marked this conversation as resolved.
Show resolved Hide resolved
continue-on-error: true
uses: actions/upload-artifact@v4
with:
name: java-bundle-${{ matrix.java }}-redis-${{ matrix.redis }}-${{ matrix.host.TARGET }}
path: |
java/bundle.jar
42 changes: 29 additions & 13 deletions .github/workflows/java.yml
Original file line number Diff line number Diff line change
Expand Up @@ -41,16 +41,16 @@ jobs:
- 6.2.14
- 7.2.3
host:
- {
OS: ubuntu,
RUNNER: ubuntu-latest,
TARGET: x86_64-unknown-linux-gnu
}
- {
OS: macos,
RUNNER: macos-latest,
TARGET: aarch64-apple-darwin
}
- {
OS: ubuntu,
RUNNER: ubuntu-latest,
TARGET: x86_64-unknown-linux-gnu
}
- {
OS: macos,
RUNNER: macos-latest,
TARGET: aarch64-apple-darwin
}

runs-on: ${{ matrix.host.RUNNER }}

Expand Down Expand Up @@ -79,15 +79,23 @@ jobs:
repo-token: ${{ secrets.GITHUB_TOKEN }}

- name: Install redis
# TODO: make this step macos compatible: https://github.com/aws/glide-for-redis/issues/781
# TODO: make this step macos compatible: https://github.com/aws/glide-for-redis/issues/781
if: ${{ matrix.host.OS == 'ubuntu' }}
uses: ./.github/workflows/install-redis
with:
redis-version: ${{ matrix.redis }}

- name: Create secret key ring file
working-directory: java/client
run: |
echo "$SECRING_GPG" | base64 --decode > ./secring.gpg
ls -ltr
env:
SECRING_GPG: ${{ secrets.SECRING_GPG }}

- name: Build java client
working-directory: java
run: ./gradlew --continue build
run: ./gradlew --continue build :client:publishToMavenLocal -Psigning.secretKeyRingFile=secring.gpg -Psigning.password="${{ secrets.GPG_PASSWORD }}" -Psigning.keyId=${{ secrets.GPG_KEY_ID }}
Yury-Fridlyand marked this conversation as resolved.
Show resolved Hide resolved

- name: Ensure no skipped files by linter
working-directory: java
Expand Down Expand Up @@ -164,9 +172,17 @@ jobs:
working-directory: java
run: cargo build --release

- name: Create secret key ring file
working-directory: java/client
run: |
echo "$SECRING_GPG" | base64 --decode > ./secring.gpg
ls -ltr
env:
SECRING_GPG: ${{ secrets.SECRING_GPG }}

- name: Build java part
working-directory: java
run: ./gradlew --continue build
run: ./gradlew --continue build :client:publishToMavenLocal -Psigning.secretKeyRingFile=secring.gpg -Psigning.password="${{ secrets.GPG_PASSWORD }}" -Psigning.keyId=${{ secrets.GPG_KEY_ID }}

- name: Upload test reports
if: always()
Expand Down
10 changes: 8 additions & 2 deletions java/client/build.gradle
Original file line number Diff line number Diff line change
Expand Up @@ -3,6 +3,7 @@ import java.nio.file.Paths
plugins {
id 'java-library'
id 'maven-publish'
id 'signing'
}

repositories {
Expand Down Expand Up @@ -180,7 +181,7 @@ publishing {
mavenJava(MavenPublication) {
from components.java
groupId = 'software.amazon.glide'
artifactId = 'glide'
artifactId = 'glide-for-redis'
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

glide-for-valkey?

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This name was given by Andrew

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@acarbonetto - is this right or should it get rebranded now?

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

0.4.3 - is branded as glide for redis (so suggesting to keep it for consistency), 1.0.0 - will be branded differently.

version = "255.255.255"
Yury-Fridlyand marked this conversation as resolved.
Show resolved Hide resolved
pom {
name = 'glide-for-valkey'
Expand Down Expand Up @@ -213,6 +214,10 @@ publishing {
}
}

signing {
sign publishing.publications
}

tasks.withType(Test) {
testLogging {
exceptionFormat "full"
Expand All @@ -222,5 +227,6 @@ tasks.withType(Test) {
}

jar {
archiveBaseName = "glide"
archiveBaseName = "glide-for-redis"
archiveClassifier = "placeholder"
Yury-Fridlyand marked this conversation as resolved.
Show resolved Hide resolved
}
4 changes: 1 addition & 3 deletions java/examples/build.gradle
Original file line number Diff line number Diff line change
Expand Up @@ -10,15 +10,13 @@ repositories {
}

dependencies {
implementation 'software.amazon.glide:glide:255.255.255'
implementation group: 'software.amazon.glide', name: 'glide-for-redis', version: '255.255.255', classifier: 'placeholder'

implementation 'redis.clients:jedis:4.4.3'
implementation 'io.lettuce:lettuce-core:6.2.6.RELEASE'
implementation 'commons-cli:commons-cli:1.5.0'
}

run.dependsOn ':client:publishToMavenLocal'

application {
// Define the main class for the application.
mainClass = 'glide.examples.ExamplesApp'
Expand Down
Loading