RJS-2101: "Build optimizations" for iOS and Android #6650

kraenhansen · 2024-05-03T10:10:29Z

What, How & Why?

Important

With this change, our SDK is no longer pinned to a specific ABI of JSI and we expect the next release of our SDK (after merging this PR) to be backwards and forwards compatible with many past versions of React Native.

This is my friendly takeover of #6396, by reverting the compilation of Realm Core from source and instead distributing prebuilds via our NPM package:

For React Native Apple (npm run prebuild-apple --workspace realm):
- Prebuild format is an XCFramework of Realm Core
- Deferred compilation on the developers machine is our binding (between Realm Core and React Native Core & JSI) driven by CocoaPods & XCode.
For React Native Android (npm run prebuild-android --workspace realm):
- Prebuilds format is a CPack install directory (containing CMake config files)
- Deferred compilation on the developers machine is our binding (between Realm Core and React Native Core & JSI) driven by Gradle calling into CMake.
For Node.js we're able to rely on the ABI stability of N-API and we produce prebuilds containing both Realm Core and our binding (between Realm Core and N-API) (npm run prebuild-node --workspace realm).

This also moves all binding related code (prebuilt or not), including that previously in packages/realm/react-native into packages/realm/binding in platform specific directories.

We won't support compiling our prebuilds from from the NPM archive as users still need to git clone our mono-repository to produce a build from source.

It's worth mentioning that this PR introduce an internal CLI incapsulating the complexity of invoking the CMake scripts for Android and Apple platforms. I imagine we could extend this to wrap cmake-js to provide a single cohesive experience for building our prebuilds in the future: This is a replacement for the packages/realm/scripts for Android and iOS that we currently use.

This is the resulting size of our tarball:

npm notice === Tarball Details === 
npm notice name:          realm                                   
npm notice version:       12.9.0                                  
npm notice filename:      realm-12.9.0.tgz                        
npm notice package size:  212.5 MB                                
npm notice unpacked size: 1.1 GB                                  
npm notice shasum:        f2bc7cffc60ee97a6c00161fefe3bac05b7becae
npm notice integrity:     sha512-V8wGrC2KNZe3O[...]Q5r/SEfraxqdg==
npm notice total files:   2895

Here's a bit more context on this change and why we choose the approach we did:

As we started implementing out "React Native SDK Build Optimizations" project we realized a few things:

Although we are generating 11K lines of C++ from the binding generator, it’s very fast to compile - most likely because it’s a single translation unit.
This generated binding layer is the only code dependent on the React Native headers / ABI.
We need Cmake to generate the Xcode project to compile Realm Core, which might not be installed on an app developers machine (in the correct version) but we can compile the binding layer without it.
If we choose to leave out the binding layer from the prebuilt binary (and we can because of ☝️) we don’t need to produce separate prebuilds for each React Native version.
Because of ☝️ we actually don’t have to delay the download of the prebuilt until “pod install” nor build-time. We could actually include it as an xcframework into the archive that we upload to NPM, like we do now. The main drawback of this approach is the download time for users as they “npm install” our package, but that won’t be a regression in DX to what we currently have and it doesn’t seem to be a deal-breaker to anyone as-is.
For Android, the situation is slightly more complicated, since Realm Core is dependent on ABI of the C++ Standard Library brought in by the NDK. Since the NDK version is configurable by the developer, this becomes a bit more complicated. Luckily the React Native template app (and Expo) declares a default for this NDK version, which we could rely on for the prebuild we upload to NPM, as long as we take great care to error if developers try using this prebuild with an unexpected NDK version and provide an easy way to build Realm Core from source, for those developers choosing a different NDK than the default.

In the context of the above, if we choose to include prebuild binaries for iOS and Android of Realm Core in the NPM package, which will be ABI independent on the React Native version (except for the caveat of NDK version, which has a default that I'd expect 95+% to leave unchanged) ... do we actually need to support compiling from source when installing the realm package via NPM as the original project scope suggests? I'd expect the need for that to be very rare (as it is right now) and it does come with drawbacks:

We'll have to avoid hoisting shared dev-dependencies to the root package of our mono-repo
Depending on a decision of forcing developers that wants to use this to have to npm install in their node_modules/realm we might need to include these dev-dependencies as actual dependencies of realm. Which would add to the install time of our package.
We'll most likely have to run tests to ensure all relevant dependencies are actually available to build from source when the realm package is installed outside of the mono-repo.
We'll have to include all of the Realm Core source-code into the repository, not a huge deal - but it all adds up.

The alternative is to ask developers that want to consume Realm JS and Realm Core from source to perform a checkout of the repository, run the relevant build script and npm pack the SDK package themselves.

Another (now closed) PR has some more context on the problem this PR is solving: #4027

Notes for the reviewer:

We deleted the packages/realm/binding/CMakeLists.txt and instead either refer to the CMakeLists.txt in bindgen/vendor/realm-core directly or CMakeLists.txt files for the individual platforms, instead of having an shared CMake project used by all bindings. This, because it felt the requirements for each platform was too different across platforms to play for the complexity of having a shared project.

☑️ ToDos

Check the package.json files to ensure all files needed to compile is included.
📝 Changelog entry
Include before / after size of the NPM package.
📝 Compatibility label is updated or copied from previous entry
📝 Update COMPATIBILITY.md
🚦 Tests
📦 Updated internal package version in consuming package.jsons (if updating internal packages)
📱 Check the React Native/other sample apps work if necessary
💥 Breaking label has been applied or is not necessary

kraenhansen · 2024-05-24T19:30:55Z

.github/workflows/pr-realm-js.yml

-      - uses: actions/setup-java@v3
-        if: ${{ matrix.variant.os == 'android' }}
-        with:
-          distribution: 'zulu' # See 'Supported distributions' for available options
-          java-version: '17'
-


This was moved up: https://github.com/realm/realm-js/pull/6650/files#diff-63a513d26453362aa63e7f2a51f5e83c112a18561d1de82473faa5cf2bcbf0d8R448-R453

kraenhansen · 2024-05-24T19:32:56Z

.github/workflows/pr-realm-js.yml

-      - name: Setup Java
-        if: ${{ matrix.variant.os == 'android' }}
-        uses: actions/setup-java@v3
-        with:
-          distribution: 'zulu' # See 'Supported distributions' for available options
-          java-version: '11'
-
-      - name: Setup Android SDK
-        if: ${{ matrix.variant.os == 'android' }}
-        uses: android-actions/setup-android@v2
-
-      - name: Install NDK
-        if: ${{ matrix.variant.os == 'android' }}
-        run: sdkmanager --install "ndk;${{ env.NDK_VERSION }}"


These were moved into a separate Android specific prebuild-android job.

kraenhansen · 2024-05-27T18:53:07Z

integration-tests/environments/react-native-test-app/package.json

    "pod-install:ci": {
      "command": "pod-install",
+      "clean": "if-file-deleted",
+      "files": [
+        "ios/Podfile",
+        "../../../packages/realm/RealmJS.podspec"
+      ],
+      "output": [
+        "ios/Pods",
+        "ios/Podfile.lock",
+        "ios/RealmTests.xcworkspace"
+      ],


No need for dependencies on other scripts when testing on CI, since that's already handled by other jobs uploading artifacts.

kraenhansen · 2024-05-27T21:53:11Z

contrib/debugging-react-native.md

@@ -1,111 +0,0 @@
-


Deleting this entirely, since almost all the information was outdated and would have to be re-tested anyway.

kraenhansen · 2024-05-27T21:53:29Z

packages/realm/RealmJS.podspec


  s.frameworks             = uses_frameworks ? ['React'] : []

  s.library                = 'c++', 'z', 'compression'

  s.pod_target_xcconfig    = {
                                # Setting up clang
-                                'CLANG_CXX_LANGUAGE_STANDARD' => 'c++17',
+                                'CLANG_CXX_LANGUAGE_STANDARD' => 'c++20',


We're using C++20 features in the binding.

Should we merge or close #5053?

Perhaps 🤔 I haven't double checked the current state of the Node.js binding.

kraenhansen · 2024-05-27T21:54:13Z

packages/realm/bindgen/src/templates/jsi.ts

+          #pragma GCC diagnostic push
+          #pragma GCC diagnostic ignored "-Wswitch"


Adding this to silence warnings that might be confusing to our users.

But will we accidentally overlook something?

I'd say very unlikely, since it's pop'ed just after the code that relies on it.

packages/realm/binding/apple/RealmReact.mm

kraenhansen · 2024-05-27T21:57:03Z

packages/realm/scripts/submit-analytics.js

+  const { submitAnalytics } = require("../dist/scripts/submit-analytics");
+  submitAnalytics().catch(console.error);
+} catch (err) {
+  console.error(err instanceof Error ? err.message : err);


This only prints if loading the analytics file fails, since submitAnalytics returns a rejected promise if failing. This is mostly to avoid any uncaught exceptions resulting in an unclean exit code.

Your comment could be added to the code as well if you'd like 🙂

kraenhansen · 2024-05-27T21:58:25Z

packages/realm/src/scripts/build/android.ts

@@ -0,0 +1,172 @@
+////////////////////////////////////////////////////////////////////////////


This is the build script for the Android prebuild.

kraenhansen · 2024-05-27T21:58:40Z

packages/realm/src/scripts/build/apple.ts

@@ -0,0 +1,181 @@
+////////////////////////////////////////////////////////////////////////////


This is the build script for the Apple / iOS prebuild.

kraenhansen · 2024-05-27T22:00:29Z