Optimised 4bit inference kernels

[nivibilla]

Feature request

Integration of new 4bit kernels

https://github.com/IST-DASLab/marlin

Motivation

provide faster Inference than awq/exllama for batch sizes upto 32

Your contribution

Just saw this today, can try provide sample notebook.

[amyeroberts]

cc @younesbelkada @SunMarc

[younesbelkada]

Thanks !
@efrantar can confirm but looking at the code it looks like you essentially just need to replace all Linear layers with marlin.Linear ? (not 100% sure) if the interface is simple we can definitely add that support I think by passing MarlinConfig through quantization_config in from_pretrained.

We also do have HQQ in the backlog #28328 but we are waiting to finalize #26610 from @poedator before adding any new quantization scheme

cc @Titus-von-Koeller just FYI

[SunMarc]

@qwopqwop200 seems to be working on adding marlin to AutoGPTQ. If it is merged, we will also have support with transformers quite easily. https://github.com/qwopqwop200/AutoGPTQ-add-marlin

[efrantar]

Yes, replacing the layers is pretty much it. It might also be possible to write a (not too complex) kernel to convert a GPTQ format model (groupsize 128, sym, no act-order; or any other quantization method that produces such models) to Marlin format on-the-fly (when loading the model) in reasonable time, which could be useful to have only a single storage format. However, I am not sure how many of the current GPTQ models on the hub already use the required settings for Marlin.

[younesbelkada]

Thank you very much @efrantar for the precision! We will update you as soon as we merge #26610

[t3ls]

Any update for this feature?

[SunMarc]

I will have a look at it soon ! Since it is available on autogptq, the integration should be straightforward !

[blap]

https://github.com/ist-daslab/sparse-marlin

[blap]

Any update?