Fix e2e failed case of visual prompting task

[sungchul2]

Summary

Related ticket: CVS-118394
Fix that e2e test for visual prompting task has sometimes failed as moving load_model after set_seed.

How to test

Tried e2e 20 times to check if there is any failed case.

import subprocess
for _ in range(20):
    subprocess.run("tox -vv -e tests-visprompt-py310 -- tests/e2e/cli/visual_prompting", shell=True)

=================== 7 passed, 1 warning in 104.51s (0:01:44) ===================

Any failed case is not detected on 20 trials.

$ tox -vv -e unittest-all-py310 -- tests/unit/algorithms/visual_prompting

====================== 177 passed, 106 warnings in 29.37s ======================

Checklist

• I have added unit tests to cover my changes.​
• I have added integration tests to cover my changes.​
• I have added e2e tests for validation.
• I have added the description of my changes into CHANGELOG in my target branch (e.g., CHANGELOG in develop).​
• I have updated the documentation in my target branch accordingly (e.g., documentation in develop).
• I have linked related issues.

License

• I submit my code changes under the same Apache License that covers the project.
  Feel free to contact the maintainers if that's a concern.
• I have updated the license header for each file (see an example below).

# Copyright (C) 2023 Intel Corporation
# SPDX-License-Identifier: Apache-2.0