Conftest: get_device_name() without device occupation

[dsmertin]

What does this PR do?

This change fixes a problem with deepspeed test_examples.
torch_hpu.get_device_name() has been moved to a separate process, because it occupies a device and doesn't release it.
For tests what work in a separate process if there's a need for all devices not all of them will be available due the occupation from current pytest process.

Before submitting

• This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
• Did you make sure to update the documentation with your changes?
• Did you write any new necessary tests?

[12010486]

@uartie, would you like to have a look? I'm reviewing as well

[uartie]

@dsmertin, @12010486, @regisss what if we use get_device_name() in the optimum/habana/utils.py module, instead? Would that prevent occupying a device?

[uartie]

return result.stdout.strip() since we check if not name by caller (i.e. empty string).

[uartie]

That is, on non-gaudi I get:

result = CompletedProcess(args="python -c 'import habana_frameworks.torch.hpu as torch_hpu\nprint(torch_hpu.get_device_name())'", returncode=0, stdout='\n', stderr='/usr/lib/python3.10/inspect.py:288: FutureWarning: `torch.distributed.reduce_op` is deprecated, please use `torch.distributed.ReduceOp` instead\n  return isinstance(object, types.FunctionType)\n/usr/local/lib/python3.10/dist-packages/habana_frameworks/torch/hpu/__init__.py:135: UserWarning: Device not available\n  warnings.warn("Device not available")\n')

[regisss]

@dsmertin, @12010486, @regisss what if we use get_device_name() in the optimum/habana/utils.py module, instead? Would that prevent occupying a device?

I think that should work yes 👍

[uartie]

@dsmertin, @12010486, @regisss what if we use get_device_name() in the optimum/habana/utils.py module, instead? Would that prevent occupying a device?

I think that should work yes 👍

@dsmertin please try this solution.

[uartie]

Meanwhile, would be good to fix the source torch_hpu.get_device_name() to actually do only what its name implies. Otherwise, rename it to torch_hpu.get_device_name_and_occupy()

[dsmertin]

@uartie @regisss
I've added get_device_name() from utils, as you suggested.

[12010486]

Ok, I do have a concern in using get_device_name(), as when used on gaudi3, it will output gaudi3, while currently all the tests are either enabled for gaudi2 (and 3 by proxy) or also covering gaudi, a subset of those. @dsmertin, could you check how a small test is running on g3?

[12010486]

Ok, I do have a concern in using get_device_name(), as when used on gaudi3, it will output gaudi3, while currently all the tests are either enabled for gaudi2 (and 3 by proxy) or also covering gaudi, a subset of those. @dsmertin, could you check how a small test is running on g3?

Hold on, it seems now we are taking care of it correctly

[regisss]

LGTM!
I think you'll have to run make style too (there should be a blank line after the import).

[HuggingFaceDocBuilderDev]

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

[uartie]

Ok, I do have a concern in using get_device_name(), as when used on gaudi3, it will output gaudi3, while currently all the tests are either enabled for gaudi2 (and 3 by proxy) or also covering gaudi, a subset of those. @dsmertin, could you check how a small test is running on g3?

Hold on, it seems now we are taking care of it correctly

On "main" branch, we are still waiting for #1807 to be merged. #1807 is necessary to effectively use --device gaudi3 or device auto-detection on G3. Without #1807, you will still want to use "--device gaudi2" or "GAUDI2_CI=1" on G3 until then.

@hsubramony has already pulled #1807 into transformers_4_49 branch via #1824

[12010486]

Ok, I do have a concern in using get_device_name(), as when used on gaudi3, it will output gaudi3, while currently all the tests are either enabled for gaudi2 (and 3 by proxy) or also covering gaudi, a subset of those. @dsmertin, could you check how a small test is running on g3?

Hold on, it seems now we are taking care of it correctly

On "main" branch, we are still waiting for #1807 to be merged. #1807 is necessary to effectively use --device gaudi3 or device auto-detection on G3. Without #1807, you will still want to use "--device gaudi2" or "GAUDI2_CI=1" on G3 until then.

@hsubramony has already pulled #1807 into transformers_4_49 branch via #1824

Thanks for the explanation! At this point, it seems the PR got merged in the wrong branch @regisss. or that should be at least merged in transformers_4_49

[regisss]

It is already in the transformers_4_49 branch