Update mnist tests so that we get good signal in periodic dashboards

[jlewi]

We need to update the E2E tests for the mnist example. The end goal should be that we get clear signal in our test grid for the periodic examples
https://k8s-testgrid.appspot.com/sig-big-data#kubeflow-periodic-examples&group-by-hierarchy-pattern=%5B%5Cw-%5D%2B

Here's a screen shot of the current dashboard

There's a couple different problems

• There's only one test result being reported named mnist-test-${TIMESTAMP}

  • But there are actually 3 different tests that should be running
  • tfjob_test.py
  • deploy_test.py
  • predict_test.py

• Every test run gets a unique name in test grid which makes it hard to see how its tracked over time.

Here are the changes I think we need to make

• Change tfjob_test.py to use pytest to support better reporting of test results

• Change deploy_test.py change to use pytest

• Replace the ksonnet workflow with the use of python to define the workflow

  • We could add it to the existing workflow for xgboost_synthetic create_e2e_workflow.py

    • or create a new a workflow

• We should be using get_kf_testing_cluster.py to create a KUBECONFIG entry for the cluster the test should run on

  • The workflow create_e2e_workflow.py already has a step to do this.

• Need to get kustomize into the test worker image Any one can help to release new image for test-worker testing#373

@jinchihe any interest in helping out with this?

[issue-label-bot]

Issue-Label Bot is automatically applying the label kind/bug to this issue, with a confidence of 0.61. Please mark this comment with 👍 or 👎 to give our bot feedback!

Links: app homepage, dashboard and code for this bot.

[jinchihe]

@jlewi OK, will do that after next week since I'm running on other track recent days :-)

[jinchihe]

Sorry, need push back the ticket more days, I still cannot free up from urgent roadmap work now...

[jlewi]

No problem. Thanks for the update