WIP: Resolve two problems in ci/cd testing.

[jinchihe]

The PR is going to fix two issue below:

1. In the PR pin the web-ui version of TF to 1.7-- same as training #658 and No signal about xgboost_synthetic test in periodic dashboard and its failing #665 , there is a problem below

error: SchemaError(io.k8s.api.certificates.v1beta1.CertificateSigningRequestList): invalid object doesn't have additional properties

The root cause is the kubectl version is v1.10.0 in the test-worker image, the kubectl has been upgraded in the PR kubeflow/testing#500, but the test-worker image is not refreshed. Once test-worker refreshed, we can remove the code change for upgrade kubectl in the PR.

2. For the cluster kf-vmaster-n00, there is a problem Workload identity not working TF; kubeflow-ci-deployment.svc.id.goog with additional claims does not have storage.objects.get access testing#499, @jlewi is going to investigate, but we can change to another cluster to work around this now, to avoid blocking other PR merging.

---

This change is 

[k8s-ci-robot]

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by:
To complete the pull request process, please assign jinchihe
You can assign the PR to them by writing /assign @jinchihe in a comment when ready.

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

• OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[jinchihe]

The first problem has been work around, but hit another problem.

W
   log_step_count_steps=self._config.log_step_count_steps) as mon_sess:
 File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/training/monitored_session.py", line 384, in MonitoredTrainingSession
   stop_grace_period_secs=stop_grace_period_secs)
 File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/training/monitored_session.py", line 795, in __init__
   stop_grace_period_secs=stop_grace_period_secs)
 File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/training/monitored_session.py", line 511, in __init__
   h.begin()
 File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/training/basic_session_run_hooks.py", line 424, in begin
   self._summary_writer = SummaryWriterCache.get(self._checkpoint_dir)
 File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/summary/writer/writer_cache.py", line 63, in get
   logdir, graph=ops.get_default_graph())
 File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/summary/writer/writer.py", line 352, in __init__
   filename_suffix)
 File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/summary/writer/event_file_writer.py", line 67, in __init__
   gfile.MakeDirs(self._logdir)
 File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/lib/io/file_io.py", line 374, in recursive_create_dir
   pywrap_tensorflow.RecursivelyCreateDir(compat.as_bytes(dirname), status)
 File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/framework/errors_impl.py", line 516, in __exit__
   c_api.TF_GetCode(self.status.status))
tensorflow.python.framework.errors_impl.PermissionDeniedError: Error executing an HTTP request (HTTP response code 403, error code 0, err
or message ''), response '{
 "error": {
   "code": 403,
   "message": "Primary: /namespaces/kubeflow-ci-deployment.svc.id.goog with additional claims does not have storage.objects.get access to
 kubeflow-ci-deployment_ci-temp/mnist/models/1186838838111113219.",
   "errors": [
     {
       "message": "Primary: /namespaces/kubeflow-ci-deployment.svc.id.goog with additional claims does not have storage.objects.get acces
s to kubeflow-ci-deployment_ci-temp/mnist/models/1186838838111113219.",
       "domain": "global",
'
         when reading metadata of gs://kubeflow-ci-deployment_ci-temp/mnist/models/1186838838111113219

[jinchihe]

/cc @jlewi

[jlewi]

This is causing the tests to run against a v0-6 cluster. Do we really want to do that?
Don't we want to test against a cluster that's running master?

[jinchihe]

@jlewi Updated this due to the issue kubeflow/testing#499, if the issue resovled, of course, we can still use master cluster. Thanks.

[jlewi]

Making the test pass doesn't help us if the test doesn't test what we care about. The meaning of the dashboard for periodic master should mean the tests are passing on clusters deployed on master.

So if we actually run on v0-6 clusters then we are creating misleading signal.

There are two possible paths forward

1. Fix the test so it passes on master (ideal)
2. Mark the test as expected to fail so that the dashboard correctly reflects that the test is failing on master but marking it as expected to fail doesn't block presubmits

One way to achieve that is to convert the test to use pytest. Then we can just annotate it to indicate its expected to fail.

We will need to update the workflow to invoke it using pytest.

You could either update the existing jsonnet file or wait for #666 and then add it to the pyfunc which might be easier then modifying ksonnet.

[jinchihe]

So I perfer to fix the test problem on master, I think that may be caused by master cluster problem kubeflow/testing#499 or workload identity changed. Will take a look later.

[jlewi]

@jinchihe regarding the master clusters against which we run the example tests. We need to fix
kubeflow/testing#444

The clusters named vmaster-n?? are no longer being updated. The issue has more info on the work that needs to be done to fix things. Its on my todo but I probably won't get to it until next week.

[jlewi]

#677 is tracking updating the mnist tests

The auto-deployed clusters should now be working and the script to get the credentials of the latest master cluste should be working as well.

[jinchihe]

Close the ticket, has been resolved in #684