Adding ability to run doctests with datastore system tests. #2738

dhermes · 2016-11-15T04:40:17Z

Since Travis / CircleCI won't run this by default, here is some sample output:

$ tox -e system-tests -- datastore
GLOB sdist-make: .../setup.py
system-tests inst-nodeps: .../.tox/dist/google-cloud-0.21.0.zip
...
system-tests runtests: commands[1] | python 
    .../system_tests/attempt_system_tests.py datastore
test_it (datastore.TestDoctest) ... Running Sphinx v1.4.8
making output directory...
loading pickled environment... not yet created
building [mo]: targets for 0 po files that are out of date
building [doctest]: targets for 1 source files that are out of date
updating environment: 1 added, 0 changed, 0 removed
reading sources... [100%] contents
looking for now-outdated files... none found
pickling environment... done
checking consistency... done
running tests...

Document: contents
------------------
1 items passed all tests:
   9 tests in constructors
9 tests in 1 items.
9 passed and 0 failed.
Test passed.

Doctest summary
===============
    9 tests
    0 failures in tests
    0 failures in setup code
    0 failures in cleanup code
build succeeded.
ok

----------------------------------------------------------------------
Ran 1 test in 1.088s

OK
____________________________________ summary ____________________________________
  system-tests: commands succeeded
  congratulations :)

system_tests/datastore.py

+    def test_it(self):
+        from sphinx import application
+
+        docs_dir = self._make_temp_docs()


system_tests/datastore.py

+        return docs_dir
+
+    def test_it(self):
+        from sphinx import application


dhermes · 2016-11-16T21:42:22Z

@tseaver PTAL

dhermes · 2016-11-18T21:26:30Z

@tseaver PTAL

datastore/google/cloud/datastore/key.py

@@ -380,7 +380,7 @@ def parent(self):
        return self._parent

    def __repr__(self):
-        return '<Key%s, project=%s>' % (self.path, self.project)
+        return '<Key%s, project=%s>' % (self._flat_path, self.project)



datastore/google/cloud/datastore/__init__.py

+  >>> entity['answer'] = 42
+  >>> entity
+  <Entity('EntityKind', 1234) {'answer': 42}>
+  >>> query = client.query(kind='EntityKind')


datastore/google/cloud/datastore/entity.py

                                      super(Entity, self).__repr__())
        else:
-            return '<Entity %s>' % (super(Entity, self).__repr__())
+            return '<Entity %s>' % (super(Entity, self).__repr__(),)


datastore/unit_tests/test_entity.py

-        self.assertEqual(repr(entity), "<Entity/bar/baz {'foo': 'Foo'}>")
+        entity_vals = {'foo': 'Foo'}
+        entity.update(entity_vals)
+        expected = '<Entity%s %s>' % (flat_path, entity_vals)


dhermes · 2016-11-28T19:35:35Z

@tseaver Only comments on the doctest itself / associated repr() changes.

Any comments on the approach of making a fake docs directory and then running doctest there?

tseaver · 2016-11-28T19:56:47Z

Any comments on the approach of making a fake docs directory and then running doctest there?

We're going to need a way to test all the examples, not just those embedded in docstrings: do you have a plan for that?

tseaver · 2016-11-29T04:29:55Z

@dhermes Memory eludes me: what stops us from just running sphinx-build -b doctest in place in the normal docs directory, rather than synthesizing one?

dhermes · 2016-11-29T04:39:13Z

We're going to need a way to test all the examples, not just those embedded in docstrings: do you have a plan for that?

Yes, though that is "beyond" the scope of this PR. There are two possible approaches that I can see. The first is to "guess" which hand-written RST files apply to a given package (I am not "pro" on this approach). The second is as follows:

Discontinue handwritten RST files and use sphinx-apidoc to generate our docs (I wrote the docs gen for oauth2client using this and @jonparrott "copied" that approach in https://github.com/GoogleCloudPlatform/google-auth-library-python/)
- In this world, our "package usage" docs will go in google/cloud/foo/__init__.py
Add checks that literal blocks are as we expect, e.g. no python literal blocks that aren't tested

Memory eludes me: what stops us from just running sphinx-build -b doctest in place in the normal docs directory, rather than synthesizing one?

The synthesized one is an attempt to isolate on a per-package basis (just as is done with the system tests). It's not strictly necessary.

The "real" reason I did it was because our docs build takes quite a long time (since the project is so large), and I wanted to have a snappier doctest build.

dhermes · 2016-11-29T22:53:32Z

@tseaver Can we keep moving on this?

tseaver · 2016-11-30T21:37:59Z

@dhermes I'd still say that running sphinx-build -b doctest (maybe in a separate [testenv:doctest] environment w/ appropriate system-tests deps) would be the right way to move forward: it will pick up any snippets in non-autodoc files, for instance.

datastore/google/cloud/datastore/__init__.py


-  from google.cloud import datastore
+  import os
+  os.environ['GOOGLE_CLOUD_PROJECT'] = u'my-project'


dhermes · 2016-11-30T21:44:47Z

I'd still say that running sphinx-build -b doctest (maybe in a separate [testenv:doctest] environment w/ appropriate system-tests deps) would be the right way to move forward: it will pick up any snippets in non-autodoc files, for instance.

This is fine, though I see it having two issues:

We can't cherry pick which packages we run it for (this is actually really nasty because the docs build is very slow)
We have to set up and teardown system test-like behavior yet again in CI (as opposed to the scenario where doctests run with the system tests)

dhermes · 2016-12-02T22:16:04Z

Plan moving forward:

use sphinx-apidoc for most RST files
leave usage docs in handwritten RST files
make a (potentially "bit-rot"-able) script to pick up usage docs along with other package files when running isolated doctest
write TOC by hand

tseaver · 2016-12-02T22:22:34Z

LGTM.

Adding ability to run doctests with datastore system tests.

dhermes added api: datastore Issues related to the Datastore API. testing labels Nov 15, 2016

dhermes assigned daspecster and tseaver Nov 15, 2016

googlebot added the cla: yes This human has signed the Contributor License Agreement. label Nov 15, 2016

dhermes commented Nov 15, 2016

View reviewed changes

system_tests/datastore.py

def test_it(self):

from sphinx import application

docs_dir = self._make_temp_docs()

This comment was marked as spam.

Sign in to view

dhermes commented Nov 15, 2016

View reviewed changes

system_tests/datastore.py

return docs_dir

def test_it(self):

from sphinx import application

This comment was marked as spam.

Sign in to view

dhermes added the docs label Nov 15, 2016

dhermes force-pushed the sphinx-doctest-datastore branch from be9f0dc to 5a5b307 Compare November 15, 2016 17:06

Adding ability to run doctests with datastore system tests.

4e7d49f

dhermes force-pushed the sphinx-doctest-datastore branch from 5a5b307 to 4e7d49f Compare November 15, 2016 17:31

dhermes mentioned this pull request Nov 18, 2016

Add Flask Helper to Error Reporting #2751

Merged

tseaver reviewed Nov 28, 2016

View reviewed changes

tseaver reviewed Nov 30, 2016

View reviewed changes

datastore/google/cloud/datastore/__init__.py

from google.cloud import datastore

import os

os.environ['GOOGLE_CLOUD_PROJECT'] = u'my-project'

This comment was marked as spam.

Sign in to view

This comment was marked as spam.

Sign in to view

dhermes merged commit e4ce734 into googleapis:master Dec 2, 2016

dhermes deleted the sphinx-doctest-datastore branch December 2, 2016 22:23

dhermes mentioned this pull request Dec 9, 2016

Update versions for mega-release. #2846

Merged

dhermes mentioned this pull request Mar 20, 2017

Test all links and code examples in docs for GA/Beta. #3172

Closed

24 tasks

richkadel pushed a commit to richkadel/google-cloud-python that referenced this pull request May 6, 2017

Merge pull request googleapis#2738 from dhermes/sphinx-doctest-datastore

6224c08

Adding ability to run doctests with datastore system tests.

theacodes unassigned daspecster Sep 28, 2018

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Adding ability to run doctests with datastore system tests. #2738

Adding ability to run doctests with datastore system tests. #2738

dhermes commented Nov 15, 2016

This comment was marked as spam.

This comment was marked as spam.

dhermes commented Nov 16, 2016

dhermes commented Nov 18, 2016

This comment was marked as spam.

This comment was marked as spam.

This comment was marked as spam.

This comment was marked as spam.

This comment was marked as spam.

This comment was marked as spam.

This comment was marked as spam.

This comment was marked as spam.

dhermes commented Nov 28, 2016

tseaver commented Nov 28, 2016

tseaver commented Nov 29, 2016

dhermes commented Nov 29, 2016

dhermes commented Nov 29, 2016

tseaver commented Nov 30, 2016

This comment was marked as spam.

This comment was marked as spam.

dhermes commented Nov 30, 2016

dhermes commented Dec 2, 2016

tseaver commented Dec 2, 2016

Adding ability to run doctests with datastore system tests. #2738

Adding ability to run doctests with datastore system tests. #2738

Conversation

dhermes commented Nov 15, 2016

This comment was marked as spam.

This comment was marked as spam.

dhermes commented Nov 16, 2016

dhermes commented Nov 18, 2016

This comment was marked as spam.

This comment was marked as spam.

This comment was marked as spam.

This comment was marked as spam.

This comment was marked as spam.

This comment was marked as spam.

This comment was marked as spam.

This comment was marked as spam.

dhermes commented Nov 28, 2016

tseaver commented Nov 28, 2016

tseaver commented Nov 29, 2016

dhermes commented Nov 29, 2016

dhermes commented Nov 29, 2016

tseaver commented Nov 30, 2016

This comment was marked as spam.

This comment was marked as spam.

dhermes commented Nov 30, 2016

dhermes commented Dec 2, 2016

tseaver commented Dec 2, 2016