Remove check of device consistency for balanced_low_0.

[xkszltl]

Seems balanced_low_0 can leave GPU 0 empty and breaks this check. According to the discussion this check may be outdated.

Resolve #2429

[xkszltl]

CC @SunMarc @younesbelkada

[HuggingFaceDocBuilderDev]

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

[SunMarc]

Hi @xkszltl, thanks for the PR. Could you run the bitsandbytes tests to see if this PR do not break any tests ? Otherwise, I'll do that after we merge a pretty important PR to fix the quantization CI !

[xkszltl]

Feel free to sequence it after your work.
I'm busy with something else as well and local resources are all occupied.

BTW there's no CI coverage?

[xkszltl]

This won't break anything as long as it compiles, because it's just removing an assertion.

[younesbelkada]

Hmmm not sure we should remove this check as it is important to keep it to educate users about how to train quantized models using multiple GPUs. Is there a way to make sure we are under balanced_low_0 regime and relax the constraint only for that case?

[xkszltl]

I'm not familiar with the details here but design wise that's a bad choice due to additional coupling. Mem affinity can be versatile and low 0 is only 1 of the profile to generate a counter example. I believe in NPP the entire GPU array is treated as a group so it shouldn't matter which is the model device.

[xkszltl]

Any other concerns?

[github-actions]

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the contributing guidelines are likely to be ignored.

[SunMarc]

Closing this since this issue should be solved by this PR