Use alternative NoSQL databases?

[jashsayani]

MongoDB is a great NoSQL database choice, but I had a few concerns:

1. I don't want to rely on MongoLab or another Database-as-a-service. I learn from experiences.
2. I don't want to install MongoDB on a compute instance and update versions, feed it RAM

I want to use either Google Datastore or Amazon DynamoDB. Both are good NoSQL database choices.
Google: https://cloud.google.com/datastore/
Amazon: https://aws.amazon.com/dynamodb/details/

I think it would be nice to be able to interface with alternative NoSQL databases other than MongoDB.

[duergner]

Should be fairly easy to add if I'm right. You would just need to implement a DatabaseAdapter similar to https://github.com/ParsePlatform/parse-server/blob/master/ExportAdapter.js for your desired database

[gfosco]

See the wiki about ExportAdapter... There is still some work to do in modularizing the database layer, but it's part of the vision for sure.

[davidaik]

+1 I don't want to rely on MongoLab either. It'll be awesome if we have full control over the db.

[francocorreasosa]

What about RethinkDB?

[ghost]

thinking about adding support for Azure DocumentDB if there is interest.

[francocorreasosa]

@ryancrawcour Great

[richardkazuomiller]

Is the plan for Parse to use the ExportAdapterDatabaseAdapter class to do the migration to databases other than MongoDB? For example, if I were to implement my own ExportAdapterDatabaseAdapter for DynamoDB, will I be able to have Parse use that adapter to fill up my new DynamoDB table(s)?

[ghost]

If you look through the Parse code you will see that not all the Mongo code is contained within just DatabaseAdapter and ExportAdapter.

For instance, users.js has direct Mongo code.

This seems to be what @gfosco was talking about when he said that complete modularization hasn't been completed yet.

My question is how much work is required to totally modualrize datavase access?

[gfosco]

Not much (famous last words).. Turn transform.js into ExportTransformAdapter.js and create a TransformAdapter (like FilesAdapter) which defaults to it. Make Schema.js and any other module which accesses transform use the adapter class, and allow it to be injected at initialization.

[gfosco]

Of course, there's a lot more to it I'm sure. The same methods used in ExportAdapter may not always fit some other system, or other unforeseen differences..

[lacker]

Yeah honestly this part of the design is a bit wack. Ideally all the Mongo access should be contained within DatabaseAdapter and ExportAdapter but at some point we got to a tradeoff between feature completeness and modular design, and decided to move towards feature completeness. Especially around supporting some of the stranger $ operators. I'm not sure how hard it would be to make more adapters and it may or may not be tractable.

[richardkazuomiller]

I never used Parse, but I know people who did and it seems like the main of it was that they didn't have to provision and maintain their own servers. In my opinion, the stateless application layer is not so hard to maintain but going from no servers to maintaining a MongoDB replica set or even working with MongoLab is a big jump for people. I think complete support for adapters so we can use DBaaS like Google Cloud Datastore and DynamoDB should be a priority.

[jashsayani]

I think we need to:

1. Modularize/isolate the database code. The database adapter should have a small transport layer that is MondoDB. Optionally, you can write your own transport layer with a different database provider.
2. Clearly document the APIs for the database part, so its really easy to read docs/comments and implement custom transport layers.
3. Work on transport layers for Google Cloud Datastore and Amazon DynamoDB. If 1 and 2 are in place, I am sure the community will start writing these.

[richardkazuomiller]

@jashsayani sounds like a plan d(^_^)

[ghost]

@jashsayani I'm keen to contribute to the core and to building an adapter for Azure DocumentDB

[jashsayani]

@ryancrawcour I don't think Azure DocumentDB is the best choice since Azure is not as main stream as AWS. So if Microsoft does not see Azure bringing significant revenue in the future, they might restructure it or cancel services. AWS on the other hand is a proven success and DynamoDB is not going anywhere.

Google Cloud Datastore is also a gamble like Azure, but its a shiny new thing and looks promising. Is there a specific reason why you want to use Azure?

[ghost]

@jashsayani trust me, Azure isn't going anywhere :)

[richardkazuomiller]

@jashsayani I think the fact that @ryancrawcour is in the Azure GitHub organization might have something to do with why he'd like to use Azure lol

[jashsayani]

@richardkazuomiller Haha. Yes, I saw his profile. He's a Microsoft employee working on Azure. So it makes sense. Good to see Microsoft evangelizing on Github.

I found a really good helper library from Google, which will make it easy to build a transport with Datastore: https://googlecloudplatform.github.io/gcloud-node/

[saleehk]

I am trying to make support for Google DataStore.
Any quick advices how to do it best way .?

[TAProgramming]

I would like to fully migrate to Google Services as well (Cloud and Datastore). Mongolab is too pricey for my liking.

[ghost]

Yes, i work for Microsoft on Azure. in fact, i work on the Azure DocumentDB team.

Point is, the DocumentDB team would love to have a native adapter for Parse for our database and want to help contribute to modularizing db access for Parse. Not only will this help us build an adapter, but will also enhance the product and allow anyone to build adapters. Win win.

So, how do we start?
@jashsayani, are you planning to start with this?

[davidaik]

This appears to be a good plan, guys. Too bad I can only watch and learn.

[jmdobry]

I work in Google Cloud and have already started work on a Parse Cloud Datastore adapter. Also can help with modularizing.

[ghost]

@jmdobry how are you building an adapter without modularization done first?

[jmdobry]

Right now it's more investigation than working code. Just understanding everything Parse is doing with MongoDB and nailing down the Datastore equivalents. Obviously modularization needs to be completed before an adapter will completely work. I only just started.

[ghost]

@jmdobry then you're in the same place as I am with a DocumentDB adapter.

[saleehk]

@karthiksekarnz Here it is https://github.com/saleeh93/parse-datastore-adapter

[saleehk]

@karthiksekarnz Feel free to ask any questions

[flinn]

What's the current state of the work towards modularizing the database adapter so Parse Server can be used with other NoSQL databases? I'm interested in using RethinkDB for a current project, and I'd be willing to write an adapter for it if that's a possibility (or if it will be possible soon)...

[dv336699]

@flinn I'm not from the team, but since it has a enhancement and long-term label I don't think it'll be coming soon. Instead you might want to take a look at horizon.io which use RethinkDB as it's database. As long as you don't rely on Parse specific stuff it should get you covered.

[noder199]

I am also interested in an update on this, I like parse-server but it really needs support for other databases otherwise it is too expensive to run.

[blacha]

@flinn @drew-gross has been working on a postgres adapter, once he has sorted the adapter API to get that up and running, it should be pretty simple to add other database adapters in.

I think he is away on holiday atm, so I am not too sure on the ETA for that.

[lifeisfunny]

Does the Parse postgres database adapter ready for test ??
I read through the code , Could anyone point me a direction to test out this code ???

[flinn]

@blacha Thanks for the update -- I'll keep checking back for a status update from @drew-gross whenever he makes it back from his holiday.. I am still very interested in this and willing to dive in and work on a RethinkDB adapter for Parse Server.

@diego-vieira Thanks for that info, the Horizon project looks really interesting/promising... I'll take a closer look at it soon. Unfortunately, the project I am looking to set this stuff up for is a web application that's already being used in a production environment and it's tightly coupled on both the front and backend to the existing Parse infrastructure. This may be completely off-base and/or ill-advised (I haven't had a chance to dive into Horizon's docs yet, so please excuse my ignorance!), but would it make any sense to also consider building a horizon-adapter as another optional storage engine (instead of, or perhaps in addition to, a more direct/native rethinkdb-adapter)? It seems like that might offer an interesting plug-and-play solution that'd take care of the sharding/clustering of your storage solution and abstract those details away from the Parse Server... I guess the drawback is the added complexity and/or any additional network latency that might create.

[luisrgpt]

Hi. What's the database layer "modularization"'s current situation?

[flovilmart]

This should be modular enough to provide your own adapter. There still may be some quirks to move to the mongo adapter and the PG adapter is a work in progress that needs some effort to complete.

Contributions are welcome, i personally don't plan to work on this feature for the time being focusing on performance and stability instead.

[devmsh]

Seems that a lot of postgres was done, but I cannot figure how can I configure the parse server to use the postgres adapter for testing and early adaptation?

@drew-gross

[noder199]

Is this feature cold turkey? I never hear about progress anymore, I am guessing once the fb guys stopped working on it nobody else took the gauntlet?

[flovilmart]

The Postgres adapter lacks documentation but is covered the majority of tests. As for other database adapters, I don't believe it's the responsibility of that project to implement other of them. As always, we welcome contributions!

[noder199]

Sometimes I wish there was a bounty system on github to help reward contributors, I simply don't have the time to contribute. I would not mind putting some $ on the line for a google datastore adapter + documentation however. Not a lot but just for the sake of providing incentives. Anyway I digress, I will personally look into using the postgres adapter. If anyone can share their experiences using it, let me know.

[jmdobry]

Sometimes I wish there was a bounty system on github to help reward contributors

https://www.bountysource.com/

[mnearents]

At the risk of sounding dumb, what would the benefit of using the Postgres adapter over MongoDB be? I am on Heroku using Mlab and I've wondered if using Heroku's built in Postgres storage would be cheaper or have some other benefit.

[kulshekhar]

@flovilmart

The Postgres adapter lacks documentation but is covered the majority of tests.

Does this mean that it is ready for use in production?

If it's not, I'd like to contribute to help it get there quicker. Any pointers on where I can start?

Now that we can create and manage projects on Github, it might make sense to start one for completing Postgres support. This way people like me can see what's already done and what we can work on to help out.

[noder199]

Anyone interested in doing a fundraiser via bountysource to help promote the creation of a Google Datastore Adapter?

[noder199]

@saleeh93 Why did you halt working on Google datastore adapter? Did you come across some sort of limitation?

[saleehk]

@noder199 Yea. there is only some limited query supported by Google datastore. I stopped working on that

[noder199]

Ah alright, hmm was there ever a dynamodb effort made by anybody? Any limitations involved with that database? Dynamo has been around for awhile so it should be more capable, but I know it sucks at storing certain data types...

[noder199]

@saleeh93

I am gonna take a look at some of your code. Would you mind telling me exactly what limitations you faced when it came to querying?

[noder199]

Mongodb atlas is a cheaper aternative to mlab and I am switching to that service as my database needs the extra storage. I won't be needing datastore, but I'd still welcome any effort for an adapter if google makes changes to its api.

[flovilmart]

Closing as we added support to Postgres a while ago.