Skip to content
This repository has been archived by the owner on Dec 13, 2018. It is now read-only.

Latest commit

 

History

History
76 lines (62 loc) · 3.63 KB

README.md

File metadata and controls

76 lines (62 loc) · 3.63 KB

google-cloud-python on Google App Engine

Experiments getting google-cloud-python working on Google App Engine (Python).

google-cloud-language

I used an old trick to actually run unit tests by collection them directly form imported unit test modules.

I wanted to gather all unit test modules the same way we do for datastore doctests (via pkgutil.iter_modules). But it turns out this only works for the imported modules.

Getting this to work on prod and dev took a number of workarounds:

Dev

  • Using a custom Python 2.7 virtualenv with the same (very old) versions provided as libraries in App Engine (set up via env-requirements.txt).
  • Had to patch the constructor for google.appengine.tools.devappserver2.python.runtime.stubs.FakeFile, it does not match the builtin file.
  • Patch builtin open so that it can handle opening /dev/null. This is because dill (a dependency of google-gax) uses open to alias the builtin file type (it's not so easy on Python 3). We will be dropping dill in the future (it is a more extreme version of pickle, it's not a great design choice to use object serialization).

Prod

  • Patch _pyio.open so that it can handle opening /dev/null.

Both

  • Having a meticulously pinned requirements.txt to set up vendored lib/ (this is not a grpc / google-cloud-language workaround, it's standard).
  • Removing grpc and grpcio-1.4.0.dist-info from vendored lib/ so that we don't conflict with the environment.
  • Adding a fake grpcio-1.0.0.dist-info to vendored lib/ so that the distribution info is available. (It is needed for unit tests.)
  • Had to "place" stubs appengine_config.py for standard library modules: subprocess (needed by ??), _multiprocessing (needed by ??) and ctypes (needed by setuptools, if not stubbed, the dev server won't even start).
  • Had to clear existing imports from sys.modules in appengine_config.py so that our vendored packages could take precedence. This is true for six, setuptools, pkg_resources (from setuptools) and google.protobuf.

There are still some frustrating issues:

  • The libraries in prod over-ride any vendored in equivalent (see e.g. google.protobuf in /info). This does not occur in dev.
  • grpc does not come with a dist-info directory.
  • Had to make sure to run python2.7 $(which dev_appserver.py) rather than just dev_appservery.py on a system where the bare python is not 2.7 (though this is in violation of PEP 394, so I deserve it).
  • Had to HTML-escape a hyphen in my app.yaml config (i.e. clean&#2D;env/ instead of clean-env/). This actually blocks the devappserver from even starting.