Yaml tests #5

Lowercases · 2018-05-25T22:15:55Z

Add the ability to create blueprint tests in the form of a .yaml file, with a format similar to stacker's configuration.

This adds the stacker_blueprints.test.YamlDirTestGenerator class, which looks for .yaml files in the same directory (this by default, can be changed) and generates a nosetests' test for each "test stack" inside (example: https://github.com/cloudtools/stacker_blueprints/blob/yaml-tests/tests/test_dynamodb.yaml).

In order to use it outside of stacker_blueprints, so that tests can be created outside of the tree, it can be subclassed with minimal effort (example: https://github.com/cloudtools/stacker_blueprints/blob/yaml-tests/tests/__init__.py). When subclassing, some of the behaviour can be changed (yaml filename, yaml tests locations, test baseclass -- details in the docstring).

We've been using this internally at remind with our own internal tests -- this would allow us to use this class appropiately subclassed now. LMK of any suggestions.

Update 7/26: After the implementation of YamlDirTestGenerator was migrated to stacker, import it from there. This PR is now a migration of some tests to yaml format.

Adds the YamlDirTestGenerator testing generator class, which can be used to load tests from .yaml files in a directory. Adds an example of its use in tests/. The base class has a name that doesn't trigger testing by nosetests, so it can be overloaded.

Add the option for the class extending YamlDirTestGenerator to specify differents search paths for yaml files.

Migrate these to .yaml format as an example of its use.

phobologic · 2018-05-26T19:42:59Z

Is this still a WIP @Lowercases or is it ready for review? Also, this is awesome. Is there anyway to include the framework for this in the base stacker? (I haven't looked at the code yet)

Lowercases · 2018-05-26T21:08:11Z

@phobologic I left it as WIP since it's still open to commentary/suggestions, but the code is ready for review. We've tested it internally and it's working fine.

As for if it should be included in stacker, I'll leave that at your judgement -- in theory it could go anywhere since it just needs to be extended from the target testing directory.

phobologic · 2018-05-26T21:21:24Z

@Lowercases ok cool - usually I only use WIP to indicate that I'm still actively working on it, and it's not ready for review - after all, all PRs are open to commentary/suggestions :) I'll take a look now!

phobologic

Really awesome - I think we should make this a part of the base stacker.

phobologic · 2018-05-26T21:21:47Z

stacker_blueprints/test/__init__.py

+from glob import glob
+
+
+class YamlDirTestGenerator():


This should be subclassed from object

phobologic · 2018-05-26T21:26:15Z

stacker_blueprints/test/__init__.py

+                try:
+                    config = parse_config(test.read())
+                except Exception as e:
+                    raise Exception("Error loading %s: %s" % (f, e))


Should this instead print an error and raise the original exception? I think the way you have it here won't provide the full stack trace (though i could be wrong, let me know).

I guess it just makes sense not to catch the error; since the stack trace will print what the issue is in any case. A "C"-ism from me :)

phobologic · 2018-05-26T21:28:09Z

stacker_blueprints/test/__init__.py

@@ -0,0 +1,125 @@
+import os.path


Ok, I really like this - I think this would be better if it were included in stacker itself (maybe in https://github.com/cloudtools/stacker/blob/master/stacker/blueprints/testutil.py), and then we imported it here. Also, it'd be awesome to have some docs for this there: http://stacker.readthedocs.io/en/latest/blueprints.html#testing-blueprints

What do you think?

I guess I agree -- I'll move the code into stacker then, creating a new PR. I'll leave this open in the meantime.

So awesome. Really looking forward to that!

Lowercases · 2018-06-04T17:39:50Z

I've migrated the class to stacker and created a PR there: cloudtools/stacker#606

I remain not entirely convinced this patch belongs there, though -- some reasons I can think we could benefit more from having it here:

The actual yaml tests will live here in stacker_blueprints (some examples in this same branch); so as these evolve the testing class may evolve as well -- it makes sense for them to share PRs instead of living in separate but concurrent PRs.
I believe we want to give this project visibility (blueprints should end here), so this could be a good way for "suggesting" private codebases to use stacker_blueprints. We could reference it still in the docs at stacker.

phobologic · 2018-06-16T19:18:14Z

Thanks @Lowercases - I think my thinking is that if someone wants to use stacker, but doesn't necessarily need any of the blueprints, we shouldn't force them to download the blueprints library just to get the awesome functionality that comes with your YAML tests. I'm open to going back to having them here though if folks don't see the benefit of that. We've just always made stacker_blueprints entirely optional to a good experience with the main stacker project.

Lowercases · 2018-06-18T16:12:17Z

Gotcha, after some thought these last few days I think it's positive not to make stacker_blueprints mandatory, but optional :) If you think these can be used directly from stacker, let's do it! I'll adapt stacker_blueprints to get it from stacker as well.

russellballestrini · 2018-07-25T19:12:22Z

Should we close this in favor of cloudtools/stacker#606?

Lowercases · 2018-07-25T20:07:56Z

@russellballestrini this has been superseded by stacker's merge indeed, however we still could port the tests to yaml -- I'll migrate this PR to do that.

Thanks for bringing this to attention though, it had slipped through the cracks for me!

Stale / Resolved

phobologic · 2018-08-06T02:34:52Z

stacker 1.4.0 is out, with the yaml tests, so if you want to update the required version to stacker 1.4.0, it should have everything we need :)

Upgrade stacker requirement to >= 1.4.0

Lowercases · 2018-08-06T14:34:50Z

@phobologic we're r2g!

Lowercases added 7 commits May 25, 2018 18:34

Yaml tests

4fae70a

Adds the YamlDirTestGenerator testing generator class, which can be used to load tests from .yaml files in a directory. Adds an example of its use in tests/. The base class has a name that doesn't trigger testing by nosetests, so it can be overloaded.

Allow overriding baseclass for yaml tests

4931b56

Allow for a default context

4ce7151

Multiple paths in yaml tests

ecd1d14

Add the option for the class extending YamlDirTestGenerator to specify differents search paths for yaml files.

Call setup_parameters() in tests

5ca553d

Migrate some of the tests to yaml

324abdb

Migrate these to .yaml format as an example of its use.

Add commentary to YamlDirTestGenerator

248590f

russellballestrini approved these changes May 26, 2018

View reviewed changes

phobologic previously requested changes May 26, 2018

View reviewed changes

Lowercases changed the title ~~Yaml tests [WIP]~~ Yaml tests May 26, 2018

Small code improvements in stacker_blueprints.test

ab67ad3

Lowercases mentioned this pull request Jun 4, 2018

Yaml tests cloudtools/stacker#606

Merged

Merge branch 'master' into yaml-tests

3c3f47a

Lowercases force-pushed the yaml-tests branch from 85afc98 to 7202def Compare August 6, 2018 14:28

Use YamlDirTestGenerator from stacker.blueprints.testutil

e687177

Upgrade stacker requirement to >= 1.4.0

Lowercases force-pushed the yaml-tests branch from 7202def to e687177 Compare August 6, 2018 14:29

Merge branch 'master' into yaml-tests

c76fc66

phobologic merged commit 56ae495 into master Aug 6, 2018

phobologic deleted the yaml-tests branch August 6, 2018 17:42

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Yaml tests #5

Yaml tests #5

Lowercases commented May 25, 2018 •

edited

Loading

phobologic commented May 26, 2018

Lowercases commented May 26, 2018

phobologic commented May 26, 2018

phobologic left a comment

phobologic May 26, 2018

phobologic May 26, 2018

Lowercases May 28, 2018

phobologic May 26, 2018

Lowercases May 28, 2018

russellballestrini May 29, 2018

Lowercases commented Jun 4, 2018 •

edited

Loading

phobologic commented Jun 16, 2018

Lowercases commented Jun 18, 2018

russellballestrini commented Jul 25, 2018

Lowercases commented Jul 25, 2018 •

edited

Loading

phobologic commented Aug 6, 2018

Lowercases commented Aug 6, 2018

Yaml tests #5

Yaml tests #5

Conversation

Lowercases commented May 25, 2018 • edited Loading

phobologic commented May 26, 2018

Lowercases commented May 26, 2018

phobologic commented May 26, 2018

phobologic left a comment

Choose a reason for hiding this comment

phobologic May 26, 2018

Choose a reason for hiding this comment

phobologic May 26, 2018

Choose a reason for hiding this comment

Lowercases May 28, 2018

Choose a reason for hiding this comment

phobologic May 26, 2018

Choose a reason for hiding this comment

Lowercases May 28, 2018

Choose a reason for hiding this comment

russellballestrini May 29, 2018

Choose a reason for hiding this comment

Lowercases commented Jun 4, 2018 • edited Loading

phobologic commented Jun 16, 2018

Lowercases commented Jun 18, 2018

russellballestrini commented Jul 25, 2018

Lowercases commented Jul 25, 2018 • edited Loading

phobologic commented Aug 6, 2018

Lowercases commented Aug 6, 2018

Lowercases commented May 25, 2018 •

edited

Loading

Lowercases commented Jun 4, 2018 •

edited

Loading

Lowercases commented Jul 25, 2018 •

edited

Loading