Speed-up partiql-lang-kotlin build time

[am357]

Currently Gradle build for partiql-lang-kotli takes around 5 minutes to complete; more details:

% time ./gradlew build
...
BUILD SUCCESSFUL in 5m 19s
43 actionable tasks: 43 executed
./gradlew build  3.21s user 0.53s system 1% cpu 5:20.35 total

# Memory
% free -g
              total        used        free      shared  buff/cache   available
Mem:             15           3           4           0           7          11
Swap:             0           0

# CPU
% lscpu
Architecture:        x86_64
CPU op-mode(s):      32-bit, 64-bit
Byte Order:          Little Endian
CPU(s):              8
On-line CPU(s) list: 0-7
Thread(s) per core:  2
Core(s) per socket:  4
Socket(s):           1
NUMA node(s):        1
Vendor ID:           GenuineIntel
CPU family:          6
Model:               85
Model name:          Intel(R) Xeon(R) Platinum 8124M CPU @ 3.00GHz
Stepping:            4
CPU MHz:             1268.513
BogoMIPS:            5999.99
Hypervisor vendor:   KVM
Virtualization type: full
L1d cache:           32K
L1i cache:           32K
L2 cache:            1024K
L3 cache:            25344K
NUMA node0 CPU(s):   0-7
Flags:               fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ss ht syscall nx pdpe1gb rdtscp lm constant_tsc rep_good nopl xtopology nonstop_tsc cpuid aperfmperf tsc_known_freq pni pclmulqdq ssse3 fma cx16 pcid sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand hypervisor lahf_lm abm 3dnowprefetch invpcid_single pti fsgsbase tsc_adjust bmi1 hle avx2 smep bmi2 erms invpcid rtm mpx avx512f avx512dq rdseed adx smap clflushopt clwb avx512cd avx512bw avx512vl xsaveopt xsavec xgetbv1 xsaves ida arat pku ospke

This issue is for tracking the effort that aims to reduce the build time.

Acceptance

• Successful or not the effort is documented in this issue along with the reason for success and failure.

[dlurton]

Agreed. The 1.4 Kotlin compiler is considerably slower than 1.3.

I have some solutions to improve this for me locally, let me see if I can get a couple PRs together. One of them is to increase the -Xmx JVM argument to 4G (it's laughably low by default for some reason) and to finally fix the generatePigDomains gradle task to properly recognize its inputs and outputs so gradle can avoid running it if the partiql.ion file hasn't changed. This avoids needing to needing recompile them with every build.

A third option is to release and use the most recent PIG, which splits its output into multiple files.

[am357]

Agreed. The 1.4 Kotlin compiler is considerably slower than 1.3.

I have some solutions to improve this for this locally, let me see if I can get a couple PRs together. One of them is to increase the -Xmx JVM argument to 4G (it's laughably low by default for some reason) and to finally fix the generatePigDomains gradle task to properly recognize its inputs and outputs so gradle can avoid running it if the partiql.ion file hasn't changed. This avoids needing to recompile them later.

A third option is to release and use the most recent PIG, which splits its output into multiple files.

That's great, we've also got some noticeable gain after merging this PR, it'd be nice to make it even faster!

[am357]

Had an offline discussion with David, having the PIG out of the scope, going ahead with closing this issue. David has the PIG related changes to be PR'ed.