README.md

google-cloud-python on Google App Engine

Experiments getting google-cloud-python working on Google App Engine (Python).

google-cloud-language

I used an old trick to actually run unit tests by collection them directly form imported unit test modules.

I wanted to gather all unit test modules the same way we do for datastore doctests (via pkgutil.iter_modules). But it turns out this only works for the imported modules.

Getting this to work on prod and dev took a number of workarounds:

Dev

• Using a custom Python 2.7 virtualenv with the same (very old) versions provided as libraries in App Engine (set up via env-requirements.txt).

• Had to patch the constructor for google.appengine.tools.devappserver2.python.runtime.stubs.FakeFile, it does not match the builtin file (uses bufsize instead of buffering as keyword). As of gcloud 175.0.0:

  $ cd ~/google-cloud-sdk/platform/google_appengine/google/appengine/
  $ cd tools/devappserver2/python/runtime/
  $ cat stubs.py | grep -n -B 10 -A 13 'def __init__.*filename'
  265-      if FakeFile._skip_files.match(relative_filename):
  266-        visibility = FakeFile.Visibility.SKIP_BLOCK
  267-      elif FakeFile._static_files.match(relative_filename):
  268-        visibility = FakeFile.Visibility.STATIC_BLOCK
  269-
  270-    with FakeFile._availability_cache_lock:
  271-      FakeFile._availability_cache[fixed_filename] = (
  272-          visibility == FakeFile.Visibility.OK)
  273-    return visibility
  274-
  275:  def __init__(self, filename, mode='r', bufsize=-1, **kwargs):
  276-    """Initializer. See file built-in documentation."""
  277-    if mode not in FakeFile.ALLOWED_MODES:
  278-      raise IOError(errno.EROFS, 'Read-only file system', filename)
  279-
  280-    visible = FakeFile.is_file_accessible(filename)
  281-    if visible != FakeFile.Visibility.OK:
  282-      log_access_check_fail(filename, visible)
  283-      raise IOError(errno.EACCES, 'file not accessible', filename)
  284-
  285-    super(FakeFile, self).__init__(filename, mode, bufsize, **kwargs)
  286-
  287-
  288-class RestrictedPathFunction(object):
  

• Patch builtin open so that it can handle opening /dev/null. This is because dill (a dependency of google-gax) uses open to alias the builtin file type (it's not so easy on Python 3). We will be dropping dill in the future (it is a more extreme version of pickle, it's not a great design choice to use object serialization).

Prod

• Patch _pyio.open so that it can handle opening /dev/null.

Both

• Having a meticulously pinned requirements.txt to set up vendored lib/ (this is not a grpc / google-cloud-language workaround, it's standard).
• Removing grpc and grpcio-1.4.0.dist-info from vendored lib/ so that we don't conflict with the environment.
• Adding a fake grpcio-1.0.0.dist-info to vendored lib/ so that the distribution info is available. (It is needed for unit tests.)
• Had to "place" stubs appengine_config.py for standard library modules: subprocess (needed by ??), _multiprocessing (needed by ??) and ctypes (needed by setuptools, if not stubbed, the dev server won't even start).
• Had to clear existing imports from sys.modules in appengine_config.py so that our vendored packages could take precedence. This is true for six, setuptools, pkg_resources (from setuptools) and google.protobuf.

There are still some frustrating issues:

• Some libraries in prod over-ride any vendored in equivalent (see e.g. google.protobuf in /info). This does not occur in dev.
• grpc does not come with a dist-info directory.
• Had to make sure to run python2.7 $(which dev_appserver.py) rather than just dev_appservery.py on a system where the bare python is not 2.7 (though this is in violation of PEP 394, so I deserve it).
• Had to HTML-escape a hyphen in my app.yaml config (i.e. clean&#2D;env/ instead of clean-env/). This actually blocks the devappserver from even starting.
• Uploading the app includes 926 files! This is because lib/ is so very big.
• On App Engine (prod) gRPC stalled the entire request for 30s and the page just came back with 500, with no stacktrace in the logs. Then after an hour or so, it just magically started working. @jonparrott experienced the same heisen-bug.