multi-gpu fix

[casper-hansen]

Closes #667 #666 #662 #652 #638 (if not, please let me know in a new issue with traceback + code).

[davedgd]

Not sure if you’ve already fully tested this, but tomorrow I will run some tests to see how things look similar to the old PR and will let you know. Really appreciate your work on this!

[davedgd]

Just confirmed this works perfectly using the recommended approach, i.e.,

model = AutoAWQForCausalLM.from_pretrained(model_path)

Tested with both one and two GPUs (with the former being slightly faster likely due to less transfer).

[Jerryzhao-z]

I reproduced two device error while quantizing qwen2 7b lora with the latest version.

deps:

• transformers==4.47.0
• accelerate==1.1.1

I set two cuda devices available and track this error. The AlignDevicesHook.execution_device attached on model.rotary_emb is still equal to cuda:1 after awq.models.qwen2.move_embed set model.rotary_emb to cuda:0.
so this hook resets activations to device cuda:1 while forwarding rotary_embed module

after removing this [move_embed] code, quantization task got executed.

hope it helps

[Jerryzhao-z]

I reproduced two device error while quantizing qwen2 7b lora with the latest version.

deps:

• transformers==4.47.0
• accelerate==1.1.1

I set two cuda devices available and track this error. The AlignDevicesHook.execution_device attached on model.rotary_emb is still equal to cuda:1 after awq.models.qwen2.move_embed set model.rotary_emb to cuda:0. so this hook resets activations to device cuda:1 while forwarding rotary_embed module

after removing this [move_embed] code, quantization task got executed.

hope it helps

sorry, I didn't follow the recommended usage examples. It works perfectly now.
model = AutoAWQForCausalLM.from_pretrained(model_path)