Attachments Support

[jurassix]

This issue is to act as a placeholder for work remaining to fully support Attachments.

The remaining issues are as follows:

1. On replication to remote CouchDB where the local doc contains an attachment it should successfully sync the doc and attachment data. If doc contains an attachment sync fails.
2. On replication from remote CouchDB were the remote doc contain an attachment it should successfully sync the doc and attachment data. If doc contains an attachment sync fails.

Considerations unique to ReactNative:

(Note: these following information is based on my observations, some accuracy may be missing, please correct where appropriate. I'm still learning how the ReactNative packager works.)

1. ReactNative has 2 runtimes V8 in Debug and JSC in Release.
   1. During Debug (simulator build in Debug) the application is packaged targeting the browser (Chrome V8) and we have access to the window and DOM and other browser native implementations e.g. Blob
   2. During JSC Release (simulator build in Release) the application is packaged targeting JSC and we do not have access to the window and DOM. JSC isn't at feature parity with V8. Blob is replaced with node's Buffer.
2. Most polyfills fallback to native browser when available, which causes some interesting results between Debug and Release.
3. ReactNative only uses fetch
4. PouchDB has many dependencies - adapters, plugins, utils - each of these are packages for node or browser. However, ReactNative packager always chooses the browser version during package resolution (based on my testing)

Research:

1. This library currently has a hack during the postinstal phase, where the pouchdb-binary-utils/package.json is patched to force the ReactNative packager to chose the node versions of the library.
2. If I remove the above hack locally:
   1. I can successfully get sync to work for all use cases, on iOS simulator in V8 Debug. This is due to the reliance on both this library and all Pouchdb dependencies choosing browser installs, and and native browser lib implementations, all the way down.
   2. During JSC release mode, on iOS simulator, I receive Blobs from the pouchdb-adapter-http, which in turn delegated to pouchdb-ajax. Since ReactNative runtime JSC has no support for FileReader/FileReaderSync we do not have a utility to transform a Blob to Buffer or Base64. more context here
      1. I believe we can get an ArrayBuffer in all cases instead of returning a Blob and be in a better situation. More research needed here.
      2. We also cannot opt for the the node version of pouchdb-ajax because it relies on request which will not work in ReactNative without hacks to expose additional core node modules.

Next Steps:

1. I've been building out test cases for Attachments. All scenarios pass. This is due to our test suite being run in a fully node.js context and doesn't represent the ReactNative runtime. In the future it would be beneficial to extend coverage to more environment.
2. I've currently forked both pouchdb-adapter-http and pouchdb-ajax and have created ReactNative versions of these and am experimenting with getting full support for both Debug and Release. For these versions I'm supporting only fetch and always returning ArrayBuffer never Blob. This is getting close. However, many tests are not passing locally.
3. I believe if we can patch the above utilities then we will have full - or very close - support for Attachments in this library, for both the Debug and Release contexts.
4. I am worried that this effort could snowball into to essentially forking PouchDB, but right now I'm not seeing the need for this.
5. I could use some additional direction in terms of patching pouchdb-adapter-http. My current build hasn't made any changes except to the new pouchdb-ajax delegation library. There is currently no way to provide strategies without having direct access to the PouchDB instance. Long term this will not be maintainable without a strategy pattern in place at plugin registry. Also, there may be much better ways of injecting a different implementation, will need some direction from @nolanlawson e.g.

const db = new PouchDB('local');
db._ajax = fetchCore;

[jurassix]

Any additional approaches, solutions, consideration, corrections etc. is greatly appreciated. I'll keep the issue up-to-date will all progress.

[jurassix]

Update:

I've got both scenarios working: V8 Debug and JSC Release by using react-native-fetch-blob. react-native-fetch-blob polyfills both XMLHttpRequest, Fetch, and Blob. The Blob implementation defers to a Native implementation and relies on the devices file system to store the blob. With this library we can easily fetch blobs and transform them as we need. But it creates a new set of dependencies on this library etc.

I'm trying to get my fork up-to-date with my current prototype - I'll post further updates when that is completed.

Direction

After working through some use-cases I'm starting to realize that Blob support, on a Native Device, would not be realistic to do in-memory. Moving data in-out of AsyncStorage into Device memory probably only work for small files, and probably lead to instability at any real scale. My longterm approach would be to keep Attachments stored on the filesystem and stream to CouchDB on sync. Ideally we would try to remove any in-memory loading of Attachments in this library.

This could be a large shift in the direction of this library.

More discussions needed. Once I get my fork updated I'll go over the code and talk about options and tradeoffs.

[jurassix]

I finally have a working branch that supports Attachment replication/sync in both V8 (Debug) and JSC (Release) modes. Currently tested in iOS. There are 8 failing regression tests (from the original pouchdb regression set) that I'll be fixing soon.

Code is in a branch on my fork here: https://github.com/jurassix/pouchdb-react-native/tree/attachments

Note:
I've updated the package.json so we use the GitHub repo as a npm dependency. Add the following to your package.json to test out the fork above (works with npm = 3):

{
  "dependencies": {
    "pouchdb-react-native": "git://github.com/jurassix/pouchdb-react-native.git#4a44854ffa5566126d023669218dd2ee8801025e"
  }
}

The main enhancements are as follows:

1. packages/pouchdb-adapter-http-react-native - this package is a copy of pouchdb-adapter-http that injects pouchdb-fetch-react-native to replace the pouchdb-ajax module by default. There are also a couple of enhancements to blob handling for react-native-fetch-blob
2. packages/pouchdb-fetch-react-native - this packages is a copy of the pouchdb-ajax library but refactored to always use fetch
3. packages/pouchdb-adapter-asyncstorage - this package has extra blob handling for react-native-fetch-blob

To fully support Attachments your library must use react-native-fetch-blob. Code changes made to pouchdb-react-native inspect at runtime to determine if it should delegate to react-native-fetch-blob. This allows users to not have this dependency if they do not require Attachment support.

To use the above fork, you need to follow the install instructions from react-native-fetch-blob and then add the following code to you app:

import RNFetchBlob from 'react-native-fetch-blob'

global.Blob = RNFetchBlob.polyfill.Blob
const Fetch = RNFetchBlob.polyfill.Fetch
// replace built-in fetch
global.fetch = new Fetch({
  auto : true,
  // now, response with Content-Type contains `image/`, `video/`, and `video/`
  // will downloaded into file system directly.
  binaryContentTypes : ['image/', 'video/', 'audio/']
}).build()

The above code will polyfill both Fetch and Blob support in your application and pouchdb-react-native will leverage this to transform Blobs to Base64 and Base64 to Blobs etc.

Next steps:

1. Fix 8 failing tests from regression suite
2. Document all the changes made to support Blobs in RN
3. Determine if there is a simpler or smaller footprint solution that the one presented in the fork
4. Determine if there is a way to never load an Attachment into memory in any intermediary steps e.g. only if the developer specifically needs the Blob content - Perf optimization
5. Brainstorm with @nolanlawson about how to leverage more of the existing pouchdb code base e.g. how will we maintain and support packages/pouchdb-adapter-http-react-native, is there a better pattern for injecting behavior etc
6. Test Android support
7. Formalize the solution with @stockulus and release

CC @jaredly @ddolheguy @cmrichards @tseck

[stockulus]

@jurassix Thanks for the great work you did!

If you like to chat/talk with me about the integration just ping me via Twitter. In the evening Hours Central European Time works best for me.

[kkbhav]

@stockulus @jurassix Any new developments on this?

[jurassix]

@kkbhav I'm still using the fork from above for my needs. I think we should be able to move this work into an official branch and release under a tagged npm version as a first next step.

Something like pouchdb-react-native@attachment?

@stockulus I'll reach out to you via Twitter this weekend to talk about options. Moving this into a mainline branch and releasing it will go along way in getting people to help verify and find issues before we move to master. Any initial thoughts?

[stockulus]

@jurassix thats fine for me.

Is there additional setup work needed like react-native link...? If so, I'd write some documentation.

[jurassix]

@stockulus Nope the only thing that is needed for attachment support is for users to follow the install instructions from react-native-fetch-blob and then add the following code to you app:

import RNFetchBlob from 'react-native-fetch-blob'

global.Blob = RNFetchBlob.polyfill.Blob
const Fetch = RNFetchBlob.polyfill.Fetch
// replace built-in fetch
global.fetch = new Fetch({
  auto : true,
  // now, response with Content-Type contains `image/`, `video/`, and `video/`
  // will downloaded into file system directly.
  binaryContentTypes : ['image/', 'video/', 'audio/']
}).build()

If you don't do the above pouchdb-react-native will work exactly as it does today, users are who need attachment support have to do the above dependency and setup, otherwise the library should just match current master functionality.

[jurassix]

Note, a project to watch out for in the future will be react-native-node which essentially allows us to run a node.js instance in the background of our apps (currently only supports Android.) With this project we should have many more options for handling attachments in the future.

[stockulus]

I am already watching this Project, would be so great, not only for this Project!

[stockulus]

@jurassix if you create a Pull Request agains
https://github.com/stockulus/pouchdb-react-native/tree/attachments
I'd take care of the Documentation and publish it on npm with @ attachment

[jurassix]

@stockulus thanks for setting this up. I'm swamped with deadlines through Friday. Hopefully I'll have some this weekend to port over.

Can I assume that each of the existing packages pouchdb-adapter-asyncstorage, pouchdb-react-native (plus the two I've added) will be released as @attachment version?

Right now I just have relative paths between the packages in my branch, would be best to require these from npm.

[stockulus]

@jurassix no stress!!

I'd publish them on npm with @attachment version and write few line in the readme about it

[Franklin62120]

Hi,

I'have problem when I use "npm install pouchdb-react-native@attachment" command.
Is that supposed to work ?

[stockulus]

Sorry did not manage it yet, still waiting for the merged code. Try install via git link from @jurassix

as soon the https://github.com/stockulus/pouchdb-react-native/tree/attachments is up to date, I'll publish it.

[jurassix]

@Franklin62120 @stockulus Hopefully I'll start working on the Attachment Release soon, until then let me know if you have any issues installing via Git. The only issues I've encountered is npm versions unpack the project slightly different. If you hit any issues in the interim please raise them and I'll help where I can.

Again, apologies on the delays.

[Franklin62120]

Ok, thanks. It's not blocking for my project.
I'll just wait a future attachment release.

[jurassix]

@stockulus I've started working on the migration. Hopefully I'll have this completed this weekend.

[vinz243]

@jurassix how did it went? is attachment support stable now?

[jurassix]

Still in-progress, again apologies for the delay. The fork is stable, I've been using it production for 6+months.
Expect update soon.

[wemakeweb]

Hey, any updates?

[MrGurns]

Subscribing.

[jurassix]

I’m still, slowly, making progress. Current status is trying to get all regression tests to pass.

[craftzdog]

FYI: A PR that adds Blob support to RN has been finally merged:

facebook/react-native#11573

[stockulus]

@craftzdog that sounds great, that makes it much easier

[jurassix]

@stockulus @craftzdog I knew my procrastination was purposeful ;) . Most of the changes in this issue were to get Blob support to work with fetch, so we should scrap this work in favor of RN.

Guess we need to do a prototype with attachments and see if we are good.

[robertherber]

Any news on this? Considering PouchDB for a project but would need Attachments :)

[equesteo]

I've spent all day trying to get this to work any way possible, and am hitting a brick wall with being able to install the forked branch from github and get it working. If I edit my package.json as indicated above to have the "pouchdb-react-native": "github:jurassix/pouchdb-react-native#attachments", npm won't install anything, and if I do npm install --save github:jurassix/pouchdb-react-native#attachments from the command line I get the directory to be created, but I also get an error:

Error: Cannot find module '/home/.../node_modules/pouchdb-react-native-bundle/packages/pouchdb-react-native/setup.js'

It still creates a pouch-db-react-native-bundle folder in my node_modules folder, but it doesn't have an index.js in it and can't be referenced by the code.

Any help you can offer, @jurassix or @stockulus would be hugely appreciated.

[jurassix]

@equesteo yes I'm also experiencing this issue now. I've update the postinstall script, so the error you reached should be resolved. There are a couple other issues I'm seeing. The package is now being installed under the pouchdb-react-native-bundle module name, and the version of npm is important.

To resolve the module name issue, I've renamed it manually inside my node_modules to be pouchdb-react-native. This is not ideal, I'll look into how to properly handle.

Npm version is also important. Because this is a fork, the packages/ folder has newly added packages, that are not published to npm (pouchdb-fetch-react-native, pouchdb-adapter-http-react-native). These are npm installed via file paths, and new versions of npm create symlinks which breaks the React Native bundler in my testing. I've had to downgrade to node 6 + npm 3, I'm using nvm so it's not a pain. Again it's not ideal.

Doing the above has allowed the fork to install and work.

Based on the blob support now added to natively to RN's fetch, I don't think we need the fork going forward, assuming you can upgrade to the latest RN. I'm trying to put together a working example and flush out any open issues this library has to support Attachments.

Let me know if you struggle with the above install suggestions, I'll do what I can to help.

[rasmustrew]

@jurassix Whats the current status of attachments?
Does it work without the fork?

[hummerh2]

Any update? Would really like to use PouchDB but got to have attachments.

Thanks!

[iliazintchenko]

guys, any updates on this? Demand is clearly significant)
Cheers!

[jurassix]

I have not been able to test the native Blob support yet.

I agree that attachments is significant, for my company the forked version has been stable in production for over a year now, and because of that I haven't had the resources to move off the fork. Luckily this quarter I'll be revamping my companies mobile offering. I'll be eager to drop the fork and work through the native Blob support during that process.

[iliazintchenko]

cool! looking forward to it thanks again, Ilia

…

On 30 Sep 2018, at 20:23, Clint Ayres ***@***.***> wrote: I have not been able to test the native Blob support yet. I agree that attachments is significant, for my company the forked version has been stable in production for over a year now, and because of that I haven't had the resources to move off the fork. Luckily this quarter I'll be revamping my companies mobile offering. I'll be eager to drop the fork and work through the native Blob support during that process. — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#68 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ABoowJVGNmMdWGLjkrKyhqQyz_KAYDiPks5ugRopgaJpZM4NsMEt>.

[alexiri]

We need this feature as well. Has there been any progress? Anything we could do to help?

[jurassix]

@alexiri the only progress I have made was to release the original fork. You should be able to use Attachments with the following:

yarn add @jurassix/pouchdb-react-native

[jurassix]

@alexiri you will also need to follow this comment too:
#68 (comment)

[kkbhav]

Hi everyone, I tried to work with @jurassix/pouchdb-react-native, but was not able to get it working with the latest PouchDB.

Hence, I started working on my own patch to get the attachments support for react-native inspired by jurassix & stockulus implementation. People who are interested can give a try at pouchdb-adapters-rn

[garfieldnate]

@jurassix I would like to pick this up. The first step for me is simply running the PouchDB test suite in the RN environment. Looking at the comments here, it sounds like you were able to do that. How can I do this? I think just getting that running in travis would be an immense help for this project.

[jurassix]

I think the only special part was making sure you pull the submodule for the base project. This library used an existing test suite to validate the functionality.

I'd start with the current master branch and get those running, then trying the same for the fork. All tests are not passing on the branch, lots of issues with Standard js linting etc.

[garfieldnate]

Will do, thanks. Yes, everything breaks right now T_T. The PouchDB project itself has such extensive testing capabilities (see TESTING.md) that I am wondering if they would mind a PR to support RN, as well.

[garfieldnate]

Well, I'm putting this project down again. I tried to fix the tests in this repo, but I frankly have no idea how any of it works and was completely unable to get anywhere with development on it. My next idea was to get the PouchDB tests running on the simulator via Detox, but it turns out that Detox runs the mocha code in Node and only sends interaction signals over the network to the simulator. Running Mocha programmatically was also problematic because it requires you to input the directory of tests, and in Expo there is no dynamic require, so I would have to generate all of the require statements and then who knows what else... It's a bigger project than I can take on by myself. As other commenters have indicated that recent updates to RN fixed some issues, I am just going to hack on my app and add integration tests for the specific features I need.

[craftzdog]

Hey, I've got it to work with minimum changes to core packages.
Read my blogpost: https://dev.to/craftzdog/hacking-pouchdb-to-use-on-react-native-1gjh

[HZSamir]

Hello,
Any progress on this issue?

[claritysoftwaregroup]

Is there any progress on this or should we say the project is not maintained anymore?

[seigel]

I will admit my time has been absorbed elsewhere in the past few months. I am looking to having my team spend some time refreshing this project in the next little while. As far as attachments go, we are not consumers of this feature, and therefore would look for contributions to help it improve. Thanks

[claritysoftwaregroup]

I will admit my time has been absorbed elsewhere in the past few months. I am looking to having my team spend some time refreshing this project in the next little while. As far as attachments go, we are not consumers of this feature, and therefore would look for contributions to help it improve. Thanks

Thank you for your reply, I appreciate it. I will see if we can look into this and help, as it's a really important feature for us. I have looked at other solutions but PouchDB and CouchDB solve a various number of things for us.