[Awq] Enable the possibility to skip quantization for some target modules

[younesbelkada]

What does this PR do?

Adds the possibility to load AWQ models if some modules of the model are skipped for quantization.
E.g. for whisper, Llava, Mixtral, we respectively don't want to quantize the encoder, vision encoder and the gate layer to ensure inference stability.

Let's merge it once AWQ makes the 0.1.8 release

cc @ArthurZucker @casper-hansen @TheBloke @SunMarc

casper-hansen/AutoAWQ#248

This PR makes it also possible to run multi-modal models with AWQ:

from transformers import pipeline
from PIL import Image    
import requests

model_id = "ybelkada/llava-1.5-7b-hf-awq"
pipe = pipeline("image-to-text", model=model_id, device=0)
url = "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/diffusers/compel-neg.png"

image = Image.open(requests.get(url, stream=True).raw)
prompt = "USER: <image>\nCan you please describe this image?\nASSISTANT:"

outputs = pipe(image, prompt=prompt, generate_kwargs={"max_new_tokens": 100})
print(outputs[0]["generated_text"])

USER: \nCan you please describe this image?\nASSISTANT: The image features a brown and white cat sitting on a green surface, possibly a carpet or a grassy area. The cat is holding a red ball in its paws, seemingly playing with it. The cat appears to be focused on the ball, possibly preparing to play or just enjoying the toy.

[HuggingFaceDocBuilderDev]

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

[younesbelkada]

This PR enables compatiblity with Mixtral AWQ !
casper-hansen/AutoAWQ#251 being merged, this PR is ready for review 🙏

[amyeroberts]

Very nice! 🔥

I just have a quick q on the test. Otherwise LGTM

[amyeroberts]

Might be not spotting the difference - but the condition here look the same to me: how can it be both true and false?

[younesbelkada]

Ah nice catch yes! Actually v_proj should be a quantized linear (hence not a nn.Linear), changed the test a bit!

[amyeroberts]

Thanks for adding this capability!

[younesbelkada]

Thanks a lot for the review @amyeroberts !
I will merge this as soon as AutoAWQ makes the 0.1.8 Mixtral release cc @casper-hansen just for your information

[younesbelkada]

Release has been done! Merging!