GH-13: Set up JNI build (dataset, etc.) #449

lidavidm · 2024-12-05T03:04:00Z

Fixes #13.

lidavidm · 2024-12-05T05:25:23Z

The docker image takes so long to build that we really need to cache it.

lidavidm · 2024-12-09T01:14:04Z

It's going to be a while before I can get back to this.

lidavidm · 2024-12-30T01:55:23Z

Hmm, this problem doesn't happen on a local build if I change to a debug build...

lidavidm · 2024-12-30T02:03:13Z

Ah, interesting, LD_PRELOADING ASan (even if the binaries are built without it) "fixes" the issue. That's rather unfortunate, since ASan was how I was trying to figure out the corruption issue...

lidavidm · 2024-12-30T02:14:20Z

Valgrind also seems to "fix" the issue :/

lidavidm · 2024-12-30T02:30:20Z

I may disable the ORC test for now to get CI passing...we'll have to revisit a lot of the JNI work.

wgtmac · 2024-12-30T03:00:32Z

What is the error message for ORC test?

lidavidm · 2024-12-30T03:15:53Z

It is crashing in CI. When I debug locally it appears that a malloc assertion fails. However it no longer reproduces for me after fiddling around.

lidavidm · 2024-12-30T03:16:40Z

I tried to use ASan and Valgrind (separately) to identify possible memory corruption but it turns out under these tools, the crash no longer reproduces. Also after using the tools, even with them turned off now, I can't reproduce the crash locally anymore.

wgtmac · 2024-12-30T03:21:51Z

I suspect ORC crash is due to missing the timezone database: apache/arrow#36026.

lidavidm · 2024-12-30T03:27:53Z

It crashes in malloc, though. I do have the core dump:

#0  __pthread_kill_implementation (threadid=<optimized out>, signo=signo@entry=6, no_tid=no_tid@entry=0) at ./nptl/pthread_kill.c:44
#1  0x00007f6268d37f1f in __pthread_kill_internal (signo=6, threadid=<optimized out>) at ./nptl/pthread_kill.c:78
#2  0x00007f6268ce8fb2 in __GI_raise (sig=sig@entry=6) at ../sysdeps/posix/raise.c:26
#3  0x00007f6268cd3472 in __GI_abort () at ./stdlib/abort.c:79
#4  0x00007f6268d2c430 in __libc_message (action=action@entry=do_abort, fmt=fmt@entry=0x7f6268e48b80 "Fatal glibc error: malloc assertion failure in %s: %s\n")
    at ../sysdeps/posix/libc_fatal.c:155
#5  0x00007f6268d442bc in __malloc_assert (function=0x7f6268e49ba0 <__PRETTY_FUNCTION__.8> "sysmalloc", line=2611, file=<synthetic pointer>, 
    assertion=0x7f6268e49330 "(old_top == initial_top (av) && old_size == 0) || ((unsigned long) (old_size) >= MINSIZE && prev_inuse (old_top) && ((unsigned long) old_end & (pagesize - 1)) == 0)") at ./malloc/malloc.c:299
#6  sysmalloc (nb=nb@entry=2624, av=av@entry=0x7f6260000030) at ./malloc/malloc.c:2611
#7  0x00007f6268d4508e in _int_malloc (av=av@entry=0x7f6260000030, bytes=bytes@entry=2614) at ./malloc/malloc.c:4403
#8  0x00007f6268d45989 in __GI___libc_malloc (bytes=2614) at ./malloc/malloc.c:3323
#9  0x00007f61321523bf in orc::DataBuffer<char>::reserve(unsigned long) () from /tmp/target7371369939985176724arrow_orc_jni
#10 0x00007f6132152426 in orc::DataBuffer<char>::DataBuffer(orc::MemoryPool&, unsigned long) () from /tmp/target7371369939985176724arrow_orc_jni
#11 0x00007f613214c9dc in orc::createReader(std::unique_ptr<orc::InputStream, std::default_delete<orc::InputStream> >, orc::ReaderOptions const&) ()
   from /tmp/target7371369939985176724arrow_orc_jni
#12 0x00007f613160f6cd in arrow::adapters::orc::ORCFileReader::Impl::Open(std::shared_ptr<arrow::io::RandomAccessFile> const&, arrow::MemoryPool*) ()
   from /tmp/target7371369939985176724arrow_orc_jni
#13 0x00007f6131609bd2 in arrow::adapters::orc::ORCFileReader::Open(std::shared_ptr<arrow::io::RandomAccessFile> const&, arrow::MemoryPool*) ()
   from /tmp/target7371369939985176724arrow_orc_jni
#14 0x00007f6131170270 in Java_org_apache_arrow_adapter_orc_OrcReaderJniWrapper_open () from /tmp/target7371369939985176724arrow_orc_jni
#15 0x00007f624fc6b9c0 in ?? ()
#16 0x0000000000000000 in ?? ()

lidavidm · 2024-12-30T03:47:30Z

And interestingly now ORC passes in CI.

Fixes apache#13.

lidavidm · 2024-12-31T00:09:53Z

Anyways, for now I've disabled the test so that the JNI build can pass. It turns out one of the fixes here is needed to fix the Arrow CI (apache/arrow#45128).

kou

+1

kou · 2024-12-31T01:07:41Z

ci/scripts/java_full_build.sh

Can we remove java_ prefix because this is the apache/arrow-java repository?

kou · 2024-12-31T01:11:45Z

.github/workflows/test_jni.yml

+      ARCH_ALIAS: ${{ matrix.platform.archery_arch_alias }}
+      ARCH_SHORT: ${{ matrix.platform.archery_arch_short }}


We can remove them.

kou · 2024-12-31T01:15:30Z

.github/workflows/test_jni.yml

+          password: ${{ secrets.GITHUB_TOKEN }}
+      - name: Build C++ libraries
+        env:
+          VCPKG_BINARY_SOURCES: "clear;nuget,GitHub,readwrite"


This may not work because we need more codes for this.
For example, https://github.com/apache/arrow/pull/44644/files#diff-e45e45baeda1c1e73482975a664062aa56f20c03dd9d64a827aba57775bed0d3R2135 and so on are needed.

But we can work on this as a follow-up task.

wgtmac · 2024-12-31T01:27:06Z

I found that the JNI libraries built on ubuntu has linked with both jemalloc and mimalloc. The coredump indicates an invalid initial state in the sysmalloc. I'm not sure if it is an undefined behavior if we have enabled both jemalloc and mimalloc. Should we consider disabling jemalloc by default?

wgtmac · 2024-12-31T01:29:37Z

ci/scripts/java_jni_manylinux_build.sh

+: "${ARROW_GANDIVA:=ON}"
+export ARROW_GANDIVA
+: "${ARROW_GCS:=ON}"
+: "${ARROW_JEMALLOC:=ON}"


Suggested change

: "${ARROW_JEMALLOC:=ON}"

: "${ARROW_JEMALLOC:=OFF}"

: "${ARROW_MIMALLOC:=ON}"

lidavidm · 2024-12-31T01:53:03Z

I can try that. But I thought we've shipped multiple allocators in one binary before. (Arrow doesn't use jemalloc or mimalloc to replace system malloc.)

This reverts commit f0bcf4d.

lidavidm · 2024-12-31T02:54:00Z

Huh, that passed @wgtmac. Let's hope it stays that way 😅

I addressed Kou's feedback too.

kou

+1

.github/workflows/test_jni.yml

Co-authored-by: Sutou Kouhei <kou@cozmixng.org>

kou · 2025-01-02T02:37:52Z

Hmm. The ORC crash is reproduced...?
https://github.com/apache/arrow-java/actions/runs/12557315120/job/35010522318?pr=449#step:7:19745

Fatal glibc error: malloc.c:2599 (sysmalloc): assertion failed: (old_top == initial_top (av) && old_size == 0) || ((unsigned long) (old_size) >= MINSIZE && prev_inuse (old_top) && ((unsigned long) old_end & (pagesize - 1)) == 0)
Aborted (core dumped)

lidavidm · 2025-01-02T02:44:55Z

I think it's flaky then. Are we ok with disabling it for now?

wgtmac · 2025-01-02T03:12:20Z

Yes, we need to disable ORC test in this PR. Is it helpful to enable ASAN in a separate PR to build JNI libraries and use it for the ORC test?

lidavidm · 2025-01-02T03:13:45Z

I can try again in another PR. I think I found that ASAN hid the problem (Valgrind too). (Also both tools are finicky with the JVM.)

lidavidm · 2025-01-02T04:32:20Z

Filed #473

lidavidm · 2025-01-02T04:33:08Z

Now that we have some CI I'm going to start merging Dependabot updates again

wgtmac · 2025-01-02T05:14:58Z

I tried to enable ASAN on Apache ORC and no issue has been found: https://github.com/apache/orc/actions/runs/12577822275/job/35055736586?pr=2097

lidavidm · 2025-01-02T05:33:21Z

ASan replaces the malloc implementation, so the error may get masked. (Though if it is indeed memory corruption presumably ASan would find that instead.)

lidavidm · 2025-01-02T05:34:41Z

One other thing we could do is try various combinations of MALLOC_CHECK and MALLOC_PERTURB

wgtmac · 2025-01-02T05:45:59Z

I still don't understand why we cannot see symbols of mimalloc from the coredump backtrace if we have linked mimalloc.

lidavidm · 2025-01-02T06:07:08Z

We don't use mimalloc to replace malloc. It's only used by the Arrow memory pool. So we are still using glibc malloc for regular allocations.

…ipts (#45165) ### Rationale for this change apache/arrow-java removed `java_` prefix from scripts by apache/arrow-java#449 . ### What changes are included in this PR? Follow the script name change. ### Are these changes tested? Yes. ### Are there any user-facing changes? No. * GitHub Issue: #45164 Authored-by: Sutou Kouhei <kou@clear-code.com> Signed-off-by: Sutou Kouhei <kou@clear-code.com>

lidavidm force-pushed the gh-13 branch 2 times, most recently from ac6bbb9 to d98309e Compare December 5, 2024 05:12

lidavidm force-pushed the gh-13 branch 6 times, most recently from 0598a43 to c8eb06b Compare December 5, 2024 08:29

lidavidm force-pushed the gh-13 branch 8 times, most recently from fc17d5e to dcef949 Compare December 27, 2024 01:22

lidavidm force-pushed the gh-13 branch from dcef949 to 32c6e1a Compare December 30, 2024 02:09

apacheGH-13: Set up JNI build (dataset, etc.)

f3d74e8

Fixes apache#13.

kou approved these changes Dec 31, 2024

View reviewed changes

wgtmac reviewed Dec 31, 2024

View reviewed changes

lidavidm added 2 commits December 30, 2024 20:57

feedback

c1ab159

Revert "disable ORC test"

610fc45

This reverts commit f0bcf4d.

kou approved these changes Dec 31, 2024

View reviewed changes

.github/workflows/test_jni.yml Outdated Show resolved Hide resolved

Update .github/workflows/test_jni.yml

3236805

Co-authored-by: Sutou Kouhei <kou@cozmixng.org>

kou mentioned this pull request Jan 2, 2025

MINOR: [Docs] Update docs to point to separate Java codebase apache/arrow#45134

Merged

disable ORC test

b132d23

lidavidm merged commit ad9bad9 into apache:main Jan 2, 2025
16 checks passed

lidavidm deleted the gh-13 branch January 2, 2025 04:31

lidavidm mentioned this pull request Jan 2, 2025

[CI] Try to re-enable ORC JNI tests with Valgrind or ASan for debugging #473

Open

This was referenced Jan 4, 2025

[CI][Integration] Follow build script name change in apache/arrow-java apache/arrow#45164

Closed

GH-45164: [CI][Integration] Remove "java_" prefix from Java build scripts apache/arrow#45165

Merged

lidavidm added this to the 18.2.0 milestone Jan 30, 2025

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

GH-13: Set up JNI build (dataset, etc.) #449

GH-13: Set up JNI build (dataset, etc.) #449

lidavidm commented Dec 5, 2024

lidavidm commented Dec 5, 2024

lidavidm commented Dec 9, 2024

lidavidm commented Dec 30, 2024

lidavidm commented Dec 30, 2024

lidavidm commented Dec 30, 2024

lidavidm commented Dec 30, 2024

wgtmac commented Dec 30, 2024

lidavidm commented Dec 30, 2024

lidavidm commented Dec 30, 2024

wgtmac commented Dec 30, 2024

lidavidm commented Dec 30, 2024

lidavidm commented Dec 30, 2024

lidavidm commented Dec 31, 2024

kou left a comment

kou Dec 31, 2024

kou Dec 31, 2024

kou Dec 31, 2024

wgtmac commented Dec 31, 2024 •

edited

Loading

wgtmac Dec 31, 2024

lidavidm commented Dec 31, 2024

lidavidm commented Dec 31, 2024

kou left a comment

kou commented Jan 2, 2025

lidavidm commented Jan 2, 2025

wgtmac commented Jan 2, 2025

lidavidm commented Jan 2, 2025

lidavidm commented Jan 2, 2025

lidavidm commented Jan 2, 2025

wgtmac commented Jan 2, 2025

lidavidm commented Jan 2, 2025

lidavidm commented Jan 2, 2025

wgtmac commented Jan 2, 2025

lidavidm commented Jan 2, 2025

		ARCH_ALIAS: ${{ matrix.platform.archery_arch_alias }}
		ARCH_SHORT: ${{ matrix.platform.archery_arch_short }}

	: "${ARROW_JEMALLOC:=ON}"
	: "${ARROW_JEMALLOC:=OFF}"
	: "${ARROW_MIMALLOC:=ON}"

GH-13: Set up JNI build (dataset, etc.) #449

GH-13: Set up JNI build (dataset, etc.) #449

Conversation

lidavidm commented Dec 5, 2024

lidavidm commented Dec 5, 2024

lidavidm commented Dec 9, 2024

lidavidm commented Dec 30, 2024

lidavidm commented Dec 30, 2024

lidavidm commented Dec 30, 2024

lidavidm commented Dec 30, 2024

wgtmac commented Dec 30, 2024

lidavidm commented Dec 30, 2024

lidavidm commented Dec 30, 2024

wgtmac commented Dec 30, 2024

lidavidm commented Dec 30, 2024

lidavidm commented Dec 30, 2024

lidavidm commented Dec 31, 2024

kou left a comment

Choose a reason for hiding this comment

kou Dec 31, 2024

Choose a reason for hiding this comment

kou Dec 31, 2024

Choose a reason for hiding this comment

kou Dec 31, 2024

Choose a reason for hiding this comment

wgtmac commented Dec 31, 2024 • edited Loading

wgtmac Dec 31, 2024

Choose a reason for hiding this comment

lidavidm commented Dec 31, 2024

lidavidm commented Dec 31, 2024

kou left a comment

Choose a reason for hiding this comment

kou commented Jan 2, 2025

lidavidm commented Jan 2, 2025

wgtmac commented Jan 2, 2025

lidavidm commented Jan 2, 2025

lidavidm commented Jan 2, 2025

lidavidm commented Jan 2, 2025

wgtmac commented Jan 2, 2025

lidavidm commented Jan 2, 2025

lidavidm commented Jan 2, 2025

wgtmac commented Jan 2, 2025

lidavidm commented Jan 2, 2025

wgtmac commented Dec 31, 2024 •

edited

Loading