Python: Auto-gen Improvements

[lukesneeringer]

This is a "catch-all ticket" of things I have identified that need doing in Python auto-gen.
Examples in this ticket will follow Vision (which I have hand-edited).

More important items are marked with ⚠️; this ticket should be demoted to P2 once those are completed.

*_client.py (e.g. image_annotator_client.py)

• ⚠️ In docstrings, references to string should be spelled str.
• ⚠️ In docstrings, references to iterator should be spelled Iterable.
• Docstrings should use 4-space indents.
• Docstring class references should use the ~. prefix if a period is present in the class name.
• In docstring examples, use ... for the prompt in hanging indents or continued statements (match the interpreter).
• ⚠️ @landrito The pkg_resources.get_distribution call needs to use the "manual" package name (e.g. google-cloud-vision), not the GAPIC package name. We are moving away from publishing stand-alone GAPIC packages.
• ⚠️ @landrito Use the new Future class from @jonparrott.
  • This has its own issue: Generated Python libraries should use google.cloud.future.operation instead of google.gax._OperationFuture #1426. See also Move google.cloud.future to google.api.core google-cloud-python#3764.

*_client.json (e.g. image_annotator_client.json)

• Rename to image_annotator_config.py, and write the value to a variable in the top line. Alter the corresponding GAPIC Python file to import this rather than using pkg_resources.resource_string to load the JSON file.

proto package

• ⚠️ @lukesneeringer Add protoc-docs-plugin to artman. (Add Python documentation artman#217)

Partial Veneer

• ⚠️ @landrito Add a versioned directory in the google.cloud namespace (e.g. google.cloud.vision_v1)
• ⚠️ @landrito Add a types.py file to the google.cloud.api_vN package which iterates over all the _pb2.py files in the proto and makes the messages available in a consistent namespace (see Vision for a reference implementation).
• ⚠️ @landrito Add an __init__.py file to the google.cloud.api_vN package which imports (1) the client and (2) the enums module from the GAPIC, and (3) the types module (above bullet). These should be exported wholesale in __all__. See Vision for a reference implementation.
• ⚠️ @landrito Add an unversioned file (e.g. vision.py) in the google.cloud namespace that imports each of the names from the latest GA versioned package and exports them wholesale in __all__. See Video Intelligence (not Vision) for a reference implementation.

Docs

• ⚠️ @landrito Create a gapic directory in the docs with an api.rst and types.rst file. See Vision for a reference implementation.
• ⚠️ Create an index.rst file with the docs overview. See Vision for a reference implementation.

setup.py and requirements.txt

• Remove requirements.txt entirely.
• ⚠️ Remove oauth2client from the dependency list in setup.py; add google-auth and requests.
  * @jonparrott -- please verify that this is correct. Done.
• Change the author from jjg+google-cloud-python@google.com to googleapis-packages@google.com.
• ⚠️ Remove dependencies on proto-google-cloud-*.
• ⚠️ Add google.cloud.gapic, google.cloud.proto, google.cloud.gapic.{api}, google.cloud.proto.{api} to namespace_packages.
• ⚠️ Add Python 3.6 to the list of tested Python versions.

Tests

• Move tests out of their deeply nested subdirectory structure into simply tests/gapic/vN. Updated now that we have system tests: See Reorganize GAPIC tests #1401.
• ⚠️ Remove tox.ini, add appropriate nox.py.
• Add tests for uncovered GAPIC lines.

Protos

• Add a ref/ directory with the raw proto files; add this to MANIFEST.in.

[lukesneeringer]

/cc @landrito (I intend to ask you to give me a hand with these.)

[theacodes]

Couple of thoughts:

1. I'm concerned about gapic (the term) being surfaced in the docs.
2. add google-auth-client and requests. - it's google-auth.
3. namespace_packages this needs to be done with some care depending on which namespace package strategy you're using, please make me a reviewer on this PR. Getting it wrong is disaster.

[lukesneeringer]

Affirmative re: the second two.

What is your concern about the first one? I am not surfacing the term in the docs themselves, only in the docs' directory structure.

The only difference is in the toctree, really:

.. toctree::
  :hidden:

  usage
  something_else
  gapic/api
  gapic/types

But no reader of the docs would interact with that.

[theacodes]

That's fine.

[landrito]

I think this has been resolved:

• Change the author from jjg+google-cloud-python@google.com to googleapis-packages@google.com.

https://github.com/googleapis/googleapis/blob/master/gapic/packaging/api_defaults.yaml#L19

[garrettjonesgoogle]

@lukesneeringer can you either mark this as resolved or create fine-grained issues for any little things that are left?

[vchudnov-g]

@lukesneeringer Could you update this with what is still to do, and add a comment that you reviewed and updated?

[lukesneeringer]

Whups, sorry. This is done. (Not going to do that last item.)