Unit test flakes

[foxish]

From #110 (comment)
This issue tracks excluding flakes from the travis build so that we can get clear signals from CI

cc @kimoonkim

[kimoonkim]

@foxish Thanks for filing this issue.

I see flaky tests in #152 and #154. Do we want to restart the builds? If new builds succeed, that will unblock these PRs for now. Travis gives an option to restart builds in the UI if one has permission. (I don't have the permission, but one of us may have)

In parallel, I'll exclude them in .travis.yml.

[kimoonkim]

Travis gives an option to restart builds in the UI if one has permission.

From a stackoverflow discussion:

If you have write access to the repo: On the build's detail screen, there is a button ↻ with the tooltip "Restart Build".

Note: Browser extensions like Ghostery may prevent the restart button from being displayed. Try disabling the extension or white-listing Travis CI.

Note2: If .travis.yml configuration has changed in the upstream, clicking rebuild button will run travis with old configuration. To apply upstream changes for travis configuration one has to add commit to PR or to close / reopen it.

If you've sent a pull request: You can close the PR then open it again. This will trigger a new build.

[foxish]

I did retry the builds a few times. It eventually passed in one of the jobs in #154.

It's not really a blocker at this point, but something to fix in the long run so that we don't miss actual error signals.

[ash211]

#156 ignores some flaky tests and is now merged -- let's leave this issue open for a week or two to see if more flakes pop up, then close to keep our issue backlog cleaned out.

[foxish]

We can probably close this soon. Not a blocker for the alpha anymore.