install and link native modules from npm (as alternative to cocoapods/carthage)

[mauron85]

Introduction

Following is mostly valid for iOS.
Currently react-native-cli doesn't provide any support for plugin developers to install and link native modules like frameworks, static libraries, projects.

One can write custom link script and register it in rnpm's postlink phase and "misuse" not very well documented local-cli link scripts. However even with this approach, plugin needs to carry it's native dependencies. With more and more packages, this is slowly becoming a problem. I'm already aware of clang's linker error "IOS build error duplicate symbols for architecture x86_64" caused by two plugins linking same library.

Another popular option is to manage plugin dependencies via cocoapods.
However IMO cocapods/carthage are kind of alien in world of react-native. User need to know, how to install it, how to create podfile, etc. Linking modules should be implementation detail and users should be able to link all modules with just single react-native link command.

The Core of It

For all above reasons I decided to work on project I internally call native-modules. It will:

• augment common native libraries (like fmdb) with package.json with instructions how it should be linked in react-native project. Such a modules will be published to npm and could be used by any rn plugin.
• provide new react-native linker (and new clean xcode project parser) which will install and link all dependencies registered in app package.json

Discussion points

I understand I'm pretty brief here, but before going to deep into describing specification I would like know, if there is broad interest in such a thing.

[kelset]

cc @empyrical @grabbou

[matt-oakes]

This is something which has been on my mind recently as well. The experience as it stands at the minute only partly automates the process and leaves a lot of room for errors, especially with users who are not experienced with the native build systems.

Android is a fairly solved issue with the Gradle dependency system and most libraries making use of the rootProject.googlePlayServicesVersion convention.

iOS, however, needs some work.

For the Apple provided iOS frameworks, we could simply allow the React Native libraries to specify which ones they require in a standard way (for example react-native-location needs CoreLocation and react-native-device-info requires SystemConfiguration). The react-native link command can then ensure the correct ones are included in the targets linked libraries.

For third-party libraries, I think there are two possible directions this could be taken:

Create a native-module like system

As you mentioned, we could create a system where the required libraries are built and distributed over the npm registry. This would allow the libraries to specify their dependency on them and react-native link to correctly link them.

Advantages

• No need to learn about or use another dependency manager.

Disadvantages

• Requires each native library to be packaged, built, and distributed on NPM to be used. This would be fine for common libraries but would discourage use of smaller libraries.

Make use of an existing dependency manager

We could also leverage an existing iOS dependency manager in the same way that we do with Gradle for the Android apps. Cocoapods or Carthage would be the most obvious choices. RN libraries should be able to specify that they require a third-party library in a standard way (in a custom section of the package.json maybe?) and then the react-native link system should be able to send that information to the dependency manager to download and link the dependencies.

Advantages

• Most libraries for iOS are already packaged for one or both of these systems so any iOS library can easily be used.

Disadvantages

• The user might need to be aware of how the dependency system works, in the same way which they do right now with Gradle on Android.

---

Personally, my vote would be to build on top of Carthage. This avoids the need to reinvent an entire dependency manager for iOS, but it is also a lot simpler than Cocoapods. It can work by specifying the dependencies in a Cartfile (could be generated from the dependency requirements of the RN libraries). It then builds the *.a libraries for each dependency and then react-native link could automatically ensure that these files are linked in the XCode target.

To me, this strikes a good balance of being each for users and easy for RN library and iOS library developers.

---

Edit: Looking at how NatveScript plugin are specified they seem to have it set up very nicely (https://docs.nativescript.org/plugins/plugin-reference). They allow the library to include the library directly as a .framework or .a but they support and recommend specifying dependencies using Cocoapods which are then linked and installed by their CLI. They also use .xconfig files to specify build settings and allow merging of Info.plist entries. It's a very nice setup overall and leaves end users with basically nothing to do (if it all works correctly).

[mauron85]

I was considering same approaches, with exactly same reasoning (which is very encouraging).
Still not decided yet which one is the best, so we can discuss.

Requires each native library to be packaged, built, and distributed on NPM to be used. This would be fine for common libraries but would discourage use of smaller libraries.

Well, one of my goals is to provide tooling for third party developers to create (and publish) such a package easily. Most common packages (modules) could be published in @native_modules (or similar) scope to npm. Third party under their npm's "private" (not so) scope.

In mean time I've started work on react-native's local-cli linker/xcode project parser replacement. Even if decision will be made to go with Carthage, I think it's worth to be working on. The current one uses cordova xcode project and have some issues:

• doesn't allow to create project from scratch (only edit existing) - not big issue, just space for improvement
• modules can be added only to specific groups (workarounded in local-cli tool)
• links all targets (if project has multiple targets like static lib, framework - all will be linked into app project, which will throw clang's linker error) Because of this, I already see some 3-rd party developers distribute npm packages like CocoaLumberjack stripped to contain single target only - static lib.
• no documentation
• written to suit cordova needs

New linker/project parser will also add features like:

• printing dependency graph
• exporting/importing project from/to multiple formats (JSON, Graphviz? ... )
• written in typescript (with strict types)
• tested with unit tests

The linker/project parser will work nicely with carthage too, but doesn't make sense with cocoapods.

EDIT: edited multiple times

[mauron85]

They allow the library to include the library directly as a .framework or .a but they support and recommend specifying dependencies using Cocoapods which are then linked and installed by their CLI.

Reacting only to the first part of the sentence (cocoapods are different story).
The problem with bundling frameworks into package is that it will necessarily result to multiple packages bundling same framework (just matter of time). This will lead to clang's linker error "IOS build error duplicate symbols for architecture".

You really need some global repository of native modules, either it will be carthage or npm (or cocoapods). Even then there will be some conflicts coming from 3-rd party packages (but in much smaller scale - let's hope).

EDIT: edited multiple times.

[matt-oakes]

I edited my comment above with links to how NativeScript handles plugins with dependencies and custom build settings. They have some nice idea which we could borrow.

Well, one of my goals is to provide tooling for third party developers to create (and publish) such a package easily. Most common packages (modules) could be published in @native_modules (or similar) scope to npm. Third party under their npm scope.

That makes sense. Having them in NPM would still mean that new versions would need to be published after the native library itself publishes an update. We'd need to be careful to make sure that this isn't reliant on a small group of people to do it or for there to be a single point of failure where someone is the only person with access but they are no longer actively maintaining the package.

But probably main reason, I started work on npm variant is that I'm not very happy with react-native's local-cli linker anyway. Even if decision will be made to go with cocoapods/carthage, I think it's worth to be working on. The current one uses cordova xcode project and have some issues:

...

XCodeGen looks like a good solution to this. It takes a YAML or JSON file and generates XCode project files from it. The file can contain dependencies and supports Carthage out of the box.

I agree that the local-cli link is pretty bad at the minute. I actually manually link most of the time as it's broken often enough that I've found it's easier to just do it myself. My background is in native app development though, which isn't typical of the React Native community at large.

The problem with bundling frameworks into package is it will necessarily result to multiple packages bundling same framework. This will lead to clang's linker error "IOS build error duplicate symbols for architecture x86_64".

You really need some global repository of native modules, either it will be carthage or npm.

I've hit that issue as well. I even had the fun of one of the native libraries which React Native itself embeds clashing with an embedded one from a non-opensource Google library. That was very hacky to solve.

I believe this is why they strongly recommend using Cocoapods instead.

[mauron85]

XCodeGen looks like a good solution to this. It takes a YAML or JSON file and generates XCode project files from it.

Looks interesting. But don't see having it any sort of API. It looks like it only supports create project from initial configuration file. But might be wrong. Missing some API's that will update that file with configuration from package.json. Something like:

// read package json and link all native-modules (frameworks, libs,...)
const proj = XcodeProject.parse('project.pbxproject');
proj.getGroup('ReactNative').addFramework('New Framework');
proj.save('project.pbxproject');

EDIT: but it could be added easily, as it's matter of updating json file (much easier than write whole parser from scratch).

[matt-oakes]

Looks interesting. But don't see having any sort of API. It looks like it only supports create project from initial configuration file. But might be wrong. You still need some tool that it will update that file with some API, something like:

The idea would be to generate the xcodeproj files from the YAML or JSON using the XcodeGen tool. If you needed to make a change you would edit the YAML or JSON and then regenerate the xcodeproj file.

This has the advantage that the edits are very simple to make; you're just editing a JSON file with a known structure. However, it does mean that we're requiring another binary and completely deleting and regenerating the users XCode project file, which won't work if the user has made changes to it manually or if they're using React Native in an existing native app.

I agree that altering the project file directly is likely the best solution. The generator uses this Swift library to do it and there is also the Ruby library which Cocoapods uses. Neither of these are easy to integrate directly with a node script, but they seem to have much better support than the Cordova one which you mentioned.

[elicwhite]

Cc @grabbou. I think he said he wrote a parser and transformer for xcodeproj files for React-Native link.

[mauron85]

Combination of XcodeGen, Carthage and xcodeproj transformer seems like viable solution to me.

But how it will work, without any changes to core react-native? How will third party plugins link their native modules? So far I can think of, following steps would be necessary:

1. Transform xcode project to XcodeGen format
2. Read package.json native modules declaration (yet to be defined) and update some main carthage Cartfile
3. Run carthage update to update and install native dependencies
4. Also add dependencies (same as in step 2.) into XcodeGen project spec file
5. Transform and save XcodeGen project spec file back to xcode project

IMO without the react-native support every package would have to do all above steps (in postlink phase). 2. - 4. are fine, but what worries me are repeating steps 1. and 5.

We could get rid of those transformations steps (1. and 5.), but it would require some support from react-native itself. React-native would have to replace the pbxproject with XcodeGenFormat and wrote some scripts that will run step 5. as part of build step. But not sure how real it is (to ask rn team to implement/merge something like this).

More questions:

1. How will be XcodeGen, Carthage distributed? Or is there any chance, they become part of react-native itself? That would need strong cooperation with rn team. Or wil be packaged and published as npm packages?
2. How it will be different from cocoapods?

EDIT: My former proposal assumed no cooperation (no changes to core react-native) with rn team whatsoever. However if such cooperation is possible, then what we really need is this native module declaration, so every plugin can declare it's native dependencies in package.json and rn linker install and links them automatically (no matter if with cocoapods or carthage/XcodeGen), mitigating the "IOS build error duplicate symbols for architecture" issue.

[mauron85]

We'd need to be careful to make sure that this isn't reliant on a small group of people to do it or for there to be a single point of failure where someone is the only person with access but they are no longer actively maintaining the package.

100% agree. Less a problem when carthage is used. In matter of fact, I already commited some PRs to add carthage support for libraries that didn't have it.

matghazaryan/SOMotionDetector#35
ziminji/objective-c-sql-query-builder#34

The process of adding carthage support can be pretty much reduced to single command with https://github.com/mauron85/XcodeToolSource (uses XcodeTool and modified template)

[axe-fb]

Relevant discussion - #18

[grabbou]

Sorry I haven't taken part in this discussion earlier - I was a bit busy working on the CLI related things. This is a very important topic and something we need to figure out - not just the "best" way to handle native dependencies of 3rd party native modules, but native modules itself too.

Let me read through this issue and come up with some observations soon.

[grabbou]

Hey,

Thanks for opening up this issue and providing so many helpful informations (including few proposals and alternative solutions - this is very encouraging). We are also discussing this matter on "React Native Community Discord". Will keep you updated (or someone else as I am going to be on a leave till the end of the year).

As you already noted, "link" hasn't been working much reliably in recent months. This is due to numerous changes that affected native iOS and Android projects as well as lack of the proper maintenance for the past year. It is a good indicator how expensive it is to keep such abstraction up to date.

Having that in mind, I think we should circle back and design this all over, not just for the "native dependencies", but for whole React Native packages too. I would really like the process of linking and dependency management to be as easy on iOS as it is already on Android, thanks to Gradle.

Thanks @mauron85 and @matt-oakes for starting this thread and listing down few options you have already explored.

Out of all the proposals mentioned here so far, I like the one that uses "native dependency manager" (e.g. CocoaPods / Carthage) the most.

We could also leverage an existing iOS dependency manager in the same way that we do with Gradle for the Android apps. Cocoapods or Carthage would be the most obvious choices.

I believe we should just do what other native developers already do on the platform. It has the benefit of being already familiar to the native developers and makes React Native much easier to co-exist in a brownfield environment (that is, environment where there's both native and React Native codebase running at the same time).

When you look at the current state of things, it turns out that we already have this kinda supported. There's React pod in the main repository that allows you to consume React Native via CocoaPods. It is also quite popular amongst the community, with many 3rd party packages already shipping their pods too.

Expo is also already using CocoaPods to support "ejected" applications, which might be an additional driver for promoting this pattern throughout the community.

However, I don't think we need this part:

RN libraries should be able to specify that they require a third-party library in a standard way (in a custom section of the package.json maybe?) and then the react-native link system should be able to send that information to the dependency manager to download and link the dependencies.

As "link" already taught us, it is quite expensive abstraction to maintain. Also, in my opinion, providing another abstraction for handling native dependencies will never satisfy all of the use-cases that direct use of CocoaPods already does. Not to mention we would have to work on a lowest common denominator of features between both CocoaPods and Gradle in order to treat "package.json" as a source of truth for both platforms.

That said, it looks that we are already there. CocoaPods support is there, community modules support it. However, since we are having this discussion, it does not work as intended.

In my opinion, here's why:

• Lack of standard. It is hard for the developers to choose the best way to handle native dependencies and suggest easiest way to link them when there's no standard. What happens is that one format gets repeated over in other packages as developers tend to do what "popular packages" already do. It's usually a mix of few different approaches, tailored to the use-cases implied by particular dependency.

• Poor tooling support. The reason "link" doesn't work reliably is that it has been implemented on top of missing standard. First, it has to search the dependency for all the native files and guess what is the format it supports. Then, it needs to figure out what is the project format. At the end, it "links" them together by doing the steps you would have to do manually if you were to do it yourself. It's not doing anything more than just automating things for your convenience.

In my opinion, here's what we need to do to make it better:

• Work on a standard. We should describe how native modules should be created and what package managers should be used for the dependencies. We could say, that a package should have a Gradle/CocoaPods file, in order to be considered compatible with React Native.

• Improve tooling. We should create tools to: (i) create a native library that conforms to the standard, (ii) validate that a package is a valid library that conforms to a standard, (iii) install a native library that conforms to a standard. First two are already missing, so that would mean we implement two "helper" commands for the package authors. From my point of view, that would be super helpful. Creating packages for React Native's been always pretty complex "manual" process that can lead to many annoying issues down the road.

• Implement better "link" command. We should use "link" to validate that package is a valid React Native library and if yes, link it to the project. No hacks and workarounds. If it doesn't follow the spec, we don't support it. That would already allow us to drop a lot of "hacky" code. All we would do is to add a dependency to CocoaPods' Podfile or Gradle dependencies. If the dependency is not a "CocoaPods" project, we could attempt generating Podfile on the go and indicate that it's workaround by printing a warning. Users should examine such file and decided to keep/drop it. Also, we could encourage to send a PR to a library authors to add an official file. In case the process fails for some reason, we would refer them to a manual guide which would be just a few easy steps. This would support the simplest form of a React Native app (ejected from Expo and "react-native init") for the majority of users that just want to add a native dependency and have no knowledge about native code. For the rest, that ship brownfield and mess around with the structure of a native project, we would automatically indicate that this tool is not the right fit.

Curious to hear your thoughts! Please mind if it takes me longer to answer due to holidays period.

[brentvatne]

RN libraries should be able to specify that they require a third-party library in a standard way (in a custom section of the package.json maybe?) and then the react-native link system should be able to send that information to the dependency manager to download and link the dependencies.

Cocoapods is helpful for this: https://github.com/expo/expo/blob/1bdf36dcd5d4e4abc8ea7d0439f3a8c582968ee0/packages/expo-gl/ios/EXGL.podspec#L19-L22

[patrickkempff]

@grabbou I like your approach.

We have a fairly large react-native brownfield app (top 3 themepark in europe) and we are using cocoapods for our ios dependencies (both 'vanilla' native and react-native) already because it works best for us.

Some of the benefits are that we are able to specify dependencies of dependencies as cocoapods already supports it (see comment of @brentvatne above), it allows us to manage react-native and 'vanilla' native dependencies the same way and native developers already know the ecosystem + it is fairly easy to learn for our RN'devs. Lots of dependencies (react-native and 'vanilla' native) already support it.

We don't have any intention of migration our whole app to react-native as we are using it only where it has clear benefits. At the moment +/- 40% of our codebase compared to +/- 30% Android and +/- 30% iOS.

[matt-oakes]

Cocoapods is helpful for this: https://github.com/expo/expo/blob/1bdf36dcd5d4e4abc8ea7d0439f3a8c582968ee0/packages/expo-gl/ios/EXGL.podspec#L19-L22

The full specification for a *.podspec file is here:

https://guides.cocoapods.org/syntax/podspec.html#specification

It supports:

• Dependencies on other cocoapods
• Vendoring frameworks and libraries
• Including system frameworks and libraries

---

As much as I dislike Cocoapods for the amount of "magic" is has inside it, it is the main tool which iOS developers will use for their dependencies and it's very uncommon to come across a library which doesn't support it. If we mandated that all React Native libraries are linked using Cocoapods then novice users would easily be able to link iOS in the same way that Gradle handles Android dependencies.

The disadvantage is that Cocoapods is an "all or nothing" style dependency manager and once it's integrated it's very hard to remove from a project. For most users, this will be absolutely fine as they'll be using React Native with very little custom native code, however, it's a much bigger issue for apps where React Native is used for only a small part of the app.

---

Open questions:

• What is the standard pattern we will ask libraries to adopt?
• How can we adopt this standard without making it incompatible with what we have currently? Requiring a compatible version fo React Native and all libraries to be compatible would keep users from upgrading for a long time.
• Do we install cocoapods in the project when running react-native init or just when the first native library is linked?
• How do we deal with the additional dependency on Ruby which Cocoapods come with?

[mauron85]

I share same distaste for cocoapods, but if react-native migrate its init template to cocoapods format, I can live with that and adjust all my plugins accordingly.

But apart from that, I can only agree with you guys and also thanks for new ideas and explanations (and bit context and history).

I'll probably continue to put effort on my former proposal anyway (as I have already started). Not saying it'll result in something useful (but hopefully it will bring at least some new helper tools).

[rhdeck]

One way to foster a marketplace of ideas approach to discovering a good answer here would be to have react-native allow the link command to be overridden by a plugin.

Then one could react native init myproject ; cd myproject ; yarn add react-native-newlink ; yarn add react-native-interesting-binary; react-native link and off to the races we go.

Right now one cannot (which is the reason I implemented react-native-linknopod in the way I did)

[grabbou]

I am going to close this discussion in favour of #96 - I wrapped up all we have talked so far (thank you so much for opening this) into a broader problem that covers all native modules in general.

Let's have a discussion over there!