Allow ndb to run from outside of App Engine

[pcostell]

No description provided.

[kevinjdolan]

IS THIS HAPPENING FOR REAL?

[erugeri]

Would love to see it for objectify / ndb to be able to migrate to flex.
Thanks!

[gcbirzan]

The ndb interface to the datastore is vastly superior to the datastore one, at least for Python, hopefully this gets fixed.

[jredl-va]

With the deprecation of the compat-runtime for appengine-flex this functionality would be a game changer ultimately enabling us to move some of our workloads off of appengine standard into kubernetes without rewriting large chunks of our applications.

[Erfa]

This seems like one of the most important things to fix in order to make it easier to migrate from standard App Engine to flexible environments.

[jonathan-s]

Would love to see this!

[aatreya]

@pcostell Is this still under development? Do you have a rough timeframe on an alpha/beta release?

[ekampf]

Any news on this?
Google is really dropping the ball on GAE users here...

[artooro]

Just want to comment, in the hopes there will be an EAP for this. I'd love be on it.

[vanpelt]

+1000

[Bogdanp]

Excuse the shameless self-promotion but those of you on Python 3.6+ might find this project useful: https://github.com/Bogdanp/anom-py

[bsmithyman]

Would it be possible for someone from Google (e.g., @pcostell) to update this issue?

With the new flexible environment expansion and GA status, we are left trying to decide whether to pursue a Python 3 refactor of our backend or continue using NDB. At present, we can't do both.

This is pushing us to consider switching databases, purely from a business risk point of view. Likely, that would be to Cloud Datastore directly, but it is not the only one in the running. A bit of extra information here would be incredibly helpful. We certainly want to continue to use NDB, but it's starting to look pretty risky. From the outside, it seems like we may need to choose between NDB and Python 3 (which is a pretty tough sell).

Thanks!

[artooro]

@bsmithyman I'm not from Google, but have been told this week that they are working on it. Even so it would be nice to get a technical update from a google engineer here just to regain some confidence on the app engine platform.

[bsmithyman]

Thanks @artooro; appreciated.

[mikelambert]

So from what I understand, the GAE Managed VM python-compat bridge is being turned down end-of-October, in favor of GAE Flexible VMs use of python images.

Is "Make NDB work outside of GAE Standard" a blocker for "turn down the python-compat bridge" for the GAE/FlexVM/NDB teams?

Or will I need to plan for the possibility that my GAE NDB code will not run on python GAE Flexible? And if I need to rewrite...what is the recommended DB solution (gcloud.datastore, DB, and NDB) for common-code that needs to run in both GAE Standard and GAE Flexible? Currently, they all seem to have drawbacks:
gcloud.datastore in GAE Standard (super slow versus native bridge)
DB/NDB in GAE Flexible (library doesn't exist, and isn't supported)

[cpavon]

@mikelambert Same issue here.... python-compat is gonna be decommissioned in October so we have to face some important decisions.

We also need to keep code running in both environments: Standard and Flexible. So if we don't have a common way to access to datastore we will have a problem.

Someone has face the same problem? alternatives?

[kevinjdolan]

It's very clear that Google is not pushing the standard AppEngine environment anymore -- to the point of deprecation, in my opinion. I decided to just pull the bandaid and move to Container Engine.

[MrMaxRoss]

Hi Kevin,

I'm the eng director for App Engine. I saw your comment about the Standard environment and felt compelled to respond.

I hope Container Engine is working well for you, but there's actually a massive investment in the Standard environment underway, and you're going to start seeing the results of that investment very soon in the form of updated Standard runtimes. I'm not making any announcements here, but if you look at the blog post where we brought the Flex environment to GA you'll see we've been running an Alpha for Java 8 on Standard, and a recent blog post on Python for App Engine also points at big changes in the pipeline.

Thanks,
Max

[workmanw]

@MrMaxRoss I really hope you announce something soon, hinting at changes can only take us so far when we're trying to make real business decisions. Right now we use NDB on Flex via legacy (vm:true). However that is being sunset in October. So we're actively trying to figure out what parts of our application we need to rewrite.

[bsmithyman]

I would echo @workmanw, inasmuch as we would love to be using both NDB and Python 3, and the uncertainty makes it hard to bet on this environment. A Python 3 Standard Environment with NDB (as hinted in previous comms) would be very welcome. Thanks for updating as you're able, @MrMaxRoss.

[MrMaxRoss]

We are still planning to sunset vm:true in October, but you can upgrade to env:flex and request an exception to continue using the compat runtime past October. There are some devs testing NDB on env:flex using the compat runtime right now (one bug so far). I will report back when we are confident that it works.

I know this is different from a fully supported NDB on Python on GAE Flex, but I want you to know that we expect there to be an option for NDB beyond October.

[brianmhunt]

Hi @MrMaxRoss — I think a lot of folks (including myself) would very much enjoy an update, insofar as you can, on where this issue stands. Thank you, sincerely!

[wsh]

@brianmhunt hi there! We can't provide specific guidance about dates just yet, but we have engineers working on this right now. (I'm one of them.) I'll be sure to keep you posted on this thread.

[lcasartelli]

Hi @wsh! Do you have any news for us? Thanks!

[robhaswell]

Hey @wsh how are you getting on?

[jer-tx]

Any updates on this? We're looking to move from Standard to Flexible and not having an ORM to talk to datastore is a massive barrier.

[cr4zyd3v]

Any updates on this? I am using python datastore client for my app, but it would be cool to have ndb outside of app engine, since it offers a lot more.. Thanks in advance.

[artooro]

Everyone following this issue should be aware of this project https://github.com/googleapis/python-ndb which is going to replace the existing python 2.7 NDB library. Currently it's only Python 3 compatible but it appears it will support 2.7 as well.