[younesbelkada]

What does this PR do?

Currently on transformers main + latest autoawq, users will face issues with fused modules + multiple calls of generate:

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer, AwqConfig

model_id = "hf-internal-testing/Mixtral-tiny-AWQ"

quantization_config = AwqConfig(bits=4, fuse_max_seq_len=128, do_fuse=True)
model = AutoModelForCausalLM.from_pretrained(model_id, quantization_config=quantization_config).to(0)

dummy_input = torch.LongTensor([[0, 1, 0, 1]]).to(0)

_ = model.generate(dummy_input, use_cache=True)

# second generate fails
_ = model.generate(dummy_input, use_cache=True)

This PR addresses that by checking the correct attributes in the FusedAttention modules

cc @casper-hansen

[casper-hansen]

Hi @younesbelkada, these are some interesting edge cases. Would it help using the attention module the same way as it is used in AutoAWQ? Currently, transformers just resets the start_pos which seems to require multiple fixes and there are probably more bugs that we are not aware of.