Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

ci(java): build JNI release package #2516

Merged
merged 36 commits into from
Jul 13, 2024
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
36 commits
Select commit Hold shift + click to select a range
f011a96
build
eddyxu Jun 24, 2024
12303fd
use ubuntu 22.04
eddyxu Jun 24, 2024
36181b0
show pwd
eddyxu Jun 24, 2024
69b7655
copy lib
eddyxu Jun 24, 2024
32b6f29
related path
eddyxu Jun 24, 2024
5001ff9
Merge branch 'main' into lei/jni_multiplatform
eddyxu Jun 24, 2024
6a6c0ff
Merge branch 'main' into lei/jni_multiplatform
eddyxu Jun 24, 2024
2c11b61
Merge branch 'main' into lei/jni_multiplatform
eddyxu Jun 25, 2024
c9bc525
Merge branch 'main' into lei/jni_multiplatform
eddyxu Jul 10, 2024
cf6b2d4
fix: propagate storage_options to v2 writer (#2578)
jiachengdb Jul 10, 2024
85bab4d
fix: be compatible with v1 index format (#2582)
BubbleCal Jul 11, 2024
a8603a1
feat: do flat search if too many rows are filtered out (#2583)
BubbleCal Jul 11, 2024
7a841dc
feat: add label list index (#2581)
westonpace Jul 11, 2024
0e65568
fix pom
eddyxu Jul 11, 2024
2399d43
Merge branch 'main' into lei/jni_multiplatform
eddyxu Jul 11, 2024
b611b53
two dry runs
eddyxu Jul 11, 2024
2a388c8
use token
eddyxu Jul 11, 2024
e7dda34
setup pgp
eddyxu Jul 11, 2024
7fc46bc
release 0.0.4 for now
eddyxu Jul 11, 2024
0cffc78
not dry run
eddyxu Jul 11, 2024
6df0979
rm snapshot
eddyxu Jul 11, 2024
cf941a5
use snapshot
eddyxu Jul 11, 2024
2345a1f
release in batch mode
eddyxu Jul 11, 2024
3cef449
do not push change for now
eddyxu Jul 11, 2024
bbadc42
skip
eddyxu Jul 11, 2024
4f0f2a9
set github config
eddyxu Jul 12, 2024
699b2d7
Merge branch 'main' into lei/jni_multiplatform
eddyxu Jul 12, 2024
5eb1771
pass
eddyxu Jul 12, 2024
2416983
setup gdp tty
eddyxu Jul 12, 2024
2e95685
Merge branch 'main' into lei/jni_multiplatform
eddyxu Jul 12, 2024
718df33
expose tty
eddyxu Jul 12, 2024
5df953b
Merge branch 'main' into lei/jni_multiplatform
eddyxu Jul 12, 2024
41cda57
export
eddyxu Jul 12, 2024
36ca1a3
setup java env
eddyxu Jul 12, 2024
41e07c3
Merge branch 'main' into lei/jni_multiplatform
eddyxu Jul 12, 2024
8357129
dont run prepare
eddyxu Jul 13, 2024
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
110 changes: 110 additions & 0 deletions .github/workflows/java-publish.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,110 @@
name: Build and publish Java packages
on:
release:
types: [released]
pull_request:
paths:
- .github/workflows/java-publish.yml

jobs:
macos-arm64:

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

i felt like it should not be 3 jobs. but instead we should have 3 run matrix and at the very end we need to copy all 3 share lib (linux-x86, linux-arm, macos-arm) into the same fat jar?

this will still release 3 different JAR IIUC?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yes, if we call mvn deploy in three different job, we have to release 3 different jars (with different names?).
I am not sure that mvn will resolve dependency based on the system, like what pypi does.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

we should have 3 run matrix and at the very end we need to copy all 3 share lib (linux-x86, linux-arm, macos-arm) into the same fat jar?

Put them into separate jobs, so that i can pass upload-artifcat/download-artifcat between the jobs. If we just run 3 parallel metrics, we need the 4th one to download the artifact?

name: Build on MacOS Arm64
runs-on: macos-14
timeout-minutes: 30
defaults:
run:
working-directory: ./java
steps:
- name: Checkout repository
uses: actions/checkout@v4
- uses: Swatinem/rust-cache@v2
- name: Install dependencies
run: |
brew install protobuf
- name: Build release
run: |
cargo build --release
- uses: actions/upload-artifact@v4
with:
name: liblance_jni_darwin_aarch64.zip
path: target/release/liblance_jni.dylib
retention-days: 1
if-no-files-found: error
linux-arm64:
name: Build on Linux Arm64
runs-on: warp-ubuntu-2204-arm64-8x
timeout-minutes: 30
defaults:
run:
working-directory: ./java
steps:
- name: Checkout repository
uses: actions/checkout@v4
- uses: Swatinem/rust-cache@v2
- uses: actions-rust-lang/setup-rust-toolchain@v1
with:
toolchain: "1.79.0"
cache-workspaces: "src/rust"
# Disable full debug symbol generation to speed up CI build and keep memory down
# "1" means line tables only, which is useful for panic tracebacks.
rustflags: "-C debuginfo=1"
- name: Install dependencies
run: |
sudo apt -y -qq update
sudo apt install -y protobuf-compiler libssl-dev pkg-config
- name: Build release
run: |
cargo build --release
cp ../target/release/liblance_jni.so liblance_jni.so
- uses: actions/upload-artifact@v4
with:
name: liblance_jni_linux_aarch64.zip
path: target/release/liblance_jni.so
retention-days: 1
if-no-files-found: error
linux-x86:
runs-on: warp-ubuntu-2204-x64-8x
timeout-minutes: 30
needs: [macos-arm64, linux-arm64]
defaults:
run:
working-directory: ./java
steps:
- name: Checkout repository
uses: actions/checkout@v4
- uses: Swatinem/rust-cache@v2
- name: Set up Java 8
uses: actions/setup-java@v4
with:
distribution: temurin
java-version: 8
cache: "maven"
server-id: ossrh
server-username: SONATYPE_USER
server-password: SONATYPE_TOKEN
gpg-private-key: ${{ secrets.GPG_PRIVATE_KEY }}
gpg-passphrase: ${{ secrets.GPG_PASSPHRASE }}
- name: Install dependencies
run: |
sudo apt -y -qq update
sudo apt install -y protobuf-compiler libssl-dev pkg-config
- name: Download artifact
uses: actions/download-artifact@v4
- name: Copy native libs
run: |
mkdir -p ./core/target/classes/nativelib/darwin-aarch64 ./core/target/classes/nativelib/linux-aarch64
cp ../liblance_jni_darwin_aarch64.zip/liblance_jni.dylib ./core/target/classes/nativelib/darwin-aarch64/liblance_jni.dylib
cp ../liblance_jni_linux_aarch64.zip/liblance_jni.so ./core/target/classes/nativelib/linux-aarch64/liblance_jni.so
- name: Set github
run: |
git config --global user.email "Lance Github Runner"
git config --global user.name "dev+gha@lancedb.com"
- name: Publish with Java 8
run: |
echo "use-agent" >> ~/.gnupg/gpg.conf
echo "pinentry-mode loopback" >> ~/.gnupg/gpg.conf
export GPG_TTY=$(tty)
mvn --batch-mode -DskipTests -DpushChanges=false -Dgpg.passphrase=${{ secrets.GPG_PASSPHRASE }} deploy -P deploy-to-ossrh
env:
SONATYPE_USER: ${{ secrets.SONATYPE_USER }}
SONATYPE_TOKEN: ${{ secrets.SONATYPE_TOKEN }}
4 changes: 2 additions & 2 deletions java/core/pom.xml
Original file line number Diff line number Diff line change
Expand Up @@ -8,7 +8,7 @@
<parent>
<groupId>com.lancedb</groupId>
<artifactId>lance-parent</artifactId>
<version>0.11.0-SNAPSHOT</version>
<version>0.0.4</version>
<relativePath>../pom.xml</relativePath>
</parent>

Expand Down Expand Up @@ -68,7 +68,7 @@
</goals>
<configuration>
<path>lance-jni</path>
<!--<release>true</release>-->
<release>true</release>
<!-- Copy native libraries to target/classes for runtime access -->
<copyTo>${project.build.directory}/classes/nativelib</copyTo>
<copyWithPlatformDir>true</copyWithPlatformDir>
Expand Down
26 changes: 16 additions & 10 deletions java/pom.xml
Original file line number Diff line number Diff line change
@@ -1,12 +1,10 @@
<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
<modelVersion>4.0.0</modelVersion>

<groupId>com.lancedb</groupId>
<artifactId>lance-parent</artifactId>
<version>0.11.0-SNAPSHOT</version>
<version>0.0.4</version>
<packaging>pom</packaging>

<name>Lance Parent</name>
Expand All @@ -26,12 +24,6 @@
</license>
</licenses>

<scm>
<developerConnection>scm:git:git@github.com:lancedb/lance.git</developerConnection>
<tag>HEAD</tag>
<url>scm:git:git@github.com:lancedb/lance.git</url>
</scm>

<properties>
<project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
<arrow.version>15.0.0</arrow.version>
Expand All @@ -43,6 +35,12 @@
<module>spark</module>
</modules>

<scm>
<connection>scm:git:https://github.com/lancedb/lance.git</connection>
<developerConnection>scm:git:ssh://git@github.com/lancedb/lance.git</developerConnection>
<url>https://github.com/lancedb/lance</url>
</scm>

<dependencyManagement>
<dependencies>
<dependency>
Expand Down Expand Up @@ -143,6 +141,14 @@
</execution>
</executions>
</plugin>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-release-plugin</artifactId>
<version>3.1.0</version>
<configuration>
<autoVersionSubmodules>true</autoVersionSubmodules>
</configuration>
</plugin>
</plugins>
<pluginManagement>
<plugins>
Expand Down
4 changes: 2 additions & 2 deletions java/spark/pom.xml
Original file line number Diff line number Diff line change
Expand Up @@ -8,7 +8,7 @@
<parent>
<groupId>com.lancedb</groupId>
<artifactId>lance-parent</artifactId>
<version>0.11.0-SNAPSHOT</version>
<version>0.0.4</version>
<relativePath>../pom.xml</relativePath>
</parent>

Expand Down Expand Up @@ -40,7 +40,7 @@
<dependency>
<groupId>com.lancedb</groupId>
<artifactId>lance-core</artifactId>
<version>0.11.0-SNAPSHOT</version>
<version>0.0.4</version>
</dependency>
<dependency>
<groupId>org.apache.spark</groupId>
Expand Down
Loading