[Testing] python presubmit tests failing 2021.3.1

[Bobgy]

https://oss-prow.knative.dev/view/gs/oss-prow/pr-logs/pull/kubeflow_pipelines/5196/kubeflow-pipelines-sdk-python36/1365228134722441216

/assign @chensun

[Bobgy]

Found in #5196

Why did it require docker?

[chensun]

Why did it require docker?

The test log seems to be messed up. The docker requirement is from another test which somehow didn't shown as "FAILED". That test explicitly asks for testing with docker:

pipelines/sdk/python/tests/local_runner_test.py

Line 195 in 8af15a3

execution_mode=LocalClient.ExecutionMode(mode="docker"),

Not sure if it was this error that messed up the subsequent tests.

Did we recently remove docker from the CI test environment?

[Bobgy]

I don't think we ever had docker there, in the CI test env, tests are run inside a Pod, so there shouldn't be docker there.

The test was introduced in #4983, curious why it didn't fail originally

[chensun]

I don't think we ever had docker there, in the CI test env, tests are run inside a Pod, so there shouldn't be docker there.

The test was introduced in #4983, curious why it didn't fail originally

Then I was probably mistaken -- the missing docker error likely didn't fail the test. Although I think we should eliminate such error anyway. Besides, a UT shouldn't have dependencies on docker, best practice should be using mock here.

That left one possibility: the other test was flaky.

Reopen this issue as P1, I'll spend some time debugging the flaky test some time this week.

[Ark-kun]

On first glance the test_execution_mode_exclude_op test seems to have been working as intended.
It tried to explicitly use docker and failed. Then tried excluding the image (so that it does not run on docker) and that succeeds.
The tests were passing here: https://oss-prow.knative.dev/view/gs/oss-prow/pr-logs/pull/kubeflow_pipelines/4983/kubeflow-pipelines-sdk-python39/1364431670366703616

[chensun]

On first glance the test_execution_mode_exclude_op test seems to have been working as intended.
It tried to explicitly use docker and failed. Then tried excluding the image (so that it does not run on docker) and that succeeds.
The tests were passing here: https://oss-prow.knative.dev/view/gs/oss-prow/pr-logs/pull/kubeflow_pipelines/4983/kubeflow-pipelines-sdk-python39/1364431670366703616

Right, I realized that. Missing docker didn't fail the test, it yields the same expected behavior though through a different route: "docker not found" vs "image not found".
That said, I think we should use mock to avoid actually calling docker binary.