Fix flaky test: [It] should delete job when expired time is up

[tenzen-y]

What this PR does / why we need it:
I fixed the flaky test case, [It] should delete job when expired time is up for TFJob.

Which issue(s) this PR fixes (optional, in Fixes #<issue number>, #<issue number>, ... format, will close the issue(s) when PR gets merged):
Fixes #1802

Checklist:

• Docs included if any changes are user facing

[coveralls]

Pull Request Test Coverage Report for Build 5033675281

• 0 of 0 changed or added relevant lines in 0 files are covered.
• No unchanged relevant lines lost coverage.
• Overall coverage decreased (-0.1%) to 39.584%

---

Totals
Change from base Build 4988117671:	-0.1%
Covered Lines:	2742
Relevant Lines:	6927

---

💛 - Coveralls

[tenzen-y]

We should get the latest TFJob after creating and updating TFJobs.

[johnugeorge]

Good one. Is this guaranteed to have cache synchronisation and we get the latest object?

[tenzen-y]

Suggested change

	// We need to wait for synchronizing cache.
	By("getting a created TFJob")
	var updatedTFJob kubeflowv1.TFJob
	Eventually(func() error {
	return reconciler.Get(ctx, client.ObjectKeyFromObject(tc.tfJob), &updatedTFJob)
	}, timeout, interval).Should(BeNil())
	By("getting a created TFJob")
	var updatedTFJob kubeflowv1.TFJob
	Expect(reconciler.Get(ctx, client.ObjectKeyFromObject(tc.tfJob), &updatedTFJob)).Should(Succeeded())

Actually, I tried the above code. But I couldn't obtain any TFJob. So we must wait for synchronizing cache.

ref: 825cc27

[tenzen-y]

I believe that we can avoid the following error since getting TFJob after creating one.

  Expected success, but got an error:
      <*errors.StatusError | 0xc0019a06e0>: {
          ErrStatus: {
              TypeMeta: {Kind: "", APIVersion: ""},
              ListMeta: {
                  SelfLink: "",
                  ResourceVersion: "",
                  Continue: "",
                  RemainingItemCount: nil,
              },
              Status: "Failure",
              Message: "Operation cannot be fulfilled on tfjobs.kubeflow.org \"test-bof-0\": the object has been modified; please apply your changes to the latest version and try again",
              Reason: "Conflict",
              Details: {
                  Name: "test-bof-0",
                  Group: "kubeflow.org",
                  Kind: "tfjobs",
                  UID: "",
                  Causes: nil,
                  RetryAfterSeconds: 0,
              },
              Code: 409,
          },
      }

[johnugeorge]

Thanks @tenzen-y
/lgtm
/approve

[google-oss-prow]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: johnugeorge, tenzen-y

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• OWNERS [johnugeorge]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment