AttributeError: Can't get attribute 'SiLUActivation' on <module 'transformers.activations'

[Lokesh-Jatangi]

System Info

System info -

• transformers version: 4.36.2
• Platform: Linux-5.10.0-26-cloud-amd64-x86_64-with-glibc2.31
• Python version: 3.10.13
• Huggingface_hub version: 0.20.1
• Safetensors version: 0.4.0
• Accelerate version: 0.24.1
• Accelerate config: not found
• PyTorch version (GPU?): 2.1.1+cu118 (True)
• Tensorflow version (GPU?): not installed (NA)
• Flax version (CPU?/GPU?/TPU?): not installed (NA)
• Jax version: not installed
• JaxLib version: not installed
• Using GPU in script?: Yes
• Using distributed or parallel set-up in script?: No

I am using a custom script which loads LLAMA checkpoint through torch.
model_orig = torch.load(checkpoint_path)

While unpickling checkpoints in torch "SiLUActivation" class is missing from activations.py.
This PR #27136 removed the SiLUActivation class mentioning it was reduntant.

P.S :- With transformers version 4.35.0 , loading a checkpoint through torch containing SiLU activation layer was succesful.

Find the below trace :-

line 65, in load_model_from_checkpoint model_orig = torch.load(checkpoint_path) File "/opt/conda/envs/adapt/lib/python3.10/site-packages/torch/serialization.py", line 1014, in load return _load(opened_zipfile, File "/opt/conda/envs/adapt/lib/python3.10/site-packages/torch/serialization.py", line 1422, in _load result = unpickler.load() File "/opt/conda/envs/adapt/lib/python3.10/site-packages/torch/serialization.py", line 1415, in find_class return super().find_class(mod_name, name) AttributeError: Can't get attribute 'SiLUActivation' on <module 'transformers.activations' from '/opt/conda/envs/adapt/lib/python3.10/site-packages/transformers/activations.py'>

I would happy to add it the SiLU class back to activations.py file and submit it here. Please let me know if i can proceed .

Who can help?

@amyeroberts

Information

• The official example scripts
• My own modified scripts

Tasks

• An officially supported task in the examples folder (such as GLUE/SQuAD, ...)
• My own task or dataset (give details below)

Reproduction

Any model which has SILU activation function and loaded through "torch.load()" will face this issue.

Expected behavior

After adding reverting back the changes , the torch should be able identify SiLU activation class.

[amyeroberts]

Hi @Lokesh-Jatangi, thanks for raising this issue!

Is there a reason you're using torch.load here? The officially supported way to load checkpoints is through the from_pretrained method.

[Lokesh-Jatangi]

The checkpoint stores a pruned model whose structure and weights are different and hence I couldnot use from_pretrained method.

[coderchem]

@Lokesh-Jatangi Have you solved your problem?

[amyeroberts]

@Lokesh-Jatangi We can't guarantee backwards compatibility for a checkpoint which isn't a transformers architecture and isn't loaded through the officially supported API. In order to be able to maintain the repo, there will be objects which we'll move, rename and delete and so pickling in this way may cause issues.

I'd suggest loading the model on the most recent compatible version of transformers. Updating the model to use torch's silu activation implementation and then resave the model out. I think this should resolve the issue and allow you to load in the model in more recent transformers versions again.

[33answer33]

@Lokesh-Jatangi Have you solved your problem?

I add this to "site-packages/transformers/activations.py" ,it works
import torch.nn.functional as F
class SiLUActivation(nn.Module):
def forward(self, input: Tensor) -> Tensor:
return F.silu(input)