# Unexpected Keywork Argument

**URL:** </static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Funexpected-keywork-argument%2F91356%3E%5C
**Category:** Models\
**Created:** [June 10, 2024, 3:37pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Funexpected-keywork-argument%2F91356 "2024-06-10T15:37:13Z")\
**Posts on this page:** 4\
**Page:** 1

<div class="post-metadata">

**Author:** ![rishik1718](https://avatars.discourse-cdn.com/v4/letter/r/a587f6/32.png) [@rishik1718](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Frishik1718)%5C
**Post date:** [June 10, 2024, 3:37pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Funexpected-keywork-argument%2F91356%2F1 "2024-06-10T15:37:14Z")

</div>

import torch  
from transformers import WhisperFeatureExtractor, WhisperForConditionalGeneration, WhisperTokenizer, WhisperProcessor  
from peft import get\_peft\_model, LoraConfig, TaskType  
import os  
from torch.utils.data import Dataset, DataLoader  
import torchaudio  
import pandas as pd

# Load Feature extractor, Tokenizer, and Processor

feature\_extractor = WhisperFeatureExtractor.from\_pretrained(“openai/whisper-small”)  
tokenizer = WhisperTokenizer.from\_pretrained(“openai/whisper-small”, language=“Tamil”, task=“transcribe”)  
processor = WhisperProcessor.from\_pretrained(“openai/whisper-small”, language=“Tamil”, task=“transcribe”)

# Load the model

model = WhisperForConditionalGeneration.from\_pretrained(“openai/whisper-small”)

# Define the target modules for LoRA based on the actual model structure

target\_modules = [  
# Encoder layers  
“model.encoder.layers.0.self\_attn.k\_proj”,  
“model.encoder.layers.0.self\_attn.v\_proj”,  
“model.encoder.layers.0.self\_attn.q\_proj”,  
“model.encoder.layers.0.self\_attn.out\_proj”,  
“model.encoder.layers.0.fc1”,  
“model.encoder.layers.0.fc2”,  
“model.encoder.layers.1.self\_attn.k\_proj”,  
“model.encoder.layers.1.self\_attn.v\_proj”,  
“model.encoder.layers.1.self\_attn.q\_proj”,  
“model.encoder.layers.1.self\_attn.out\_proj”,  
“model.encoder.layers.1.fc1”,  
“model.encoder.layers.1.fc2”,  
“model.encoder.layers.2.self\_attn.k\_proj”,  
“model.encoder.layers.2.self\_attn.v\_proj”,  
“model.encoder.layers.2.self\_attn.q\_proj”,  
“model.encoder.layers.2.self\_attn.out\_proj”,  
“model.encoder.layers.2.fc1”,  
“model.encoder.layers.2.fc2”,  
“model.encoder.layers.3.self\_attn.k\_proj”,  
“model.encoder.layers.3.self\_attn.v\_proj”,  
“model.encoder.layers.3.self\_attn.q\_proj”,  
“model.encoder.layers.3.self\_attn.out\_proj”,  
“model.encoder.layers.3.fc1”,  
“model.encoder.layers.3.fc2”,  
“model.encoder.layers.4.self\_attn.k\_proj”,  
“model.encoder.layers.4.self\_attn.v\_proj”,  
“model.encoder.layers.4.self\_attn.q\_proj”,  
“model.encoder.layers.4.self\_attn.out\_proj”,  
“model.encoder.layers.4.fc1”,  
“model.encoder.layers.4.fc2”,  
“model.encoder.layers.5.self\_attn.k\_proj”,  
“model.encoder.layers.5.self\_attn.v\_proj”,  
“model.encoder.layers.5.self\_attn.q\_proj”,  
“model.encoder.layers.5.self\_attn.out\_proj”,  
“model.encoder.layers.5.fc1”,  
“model.encoder.layers.5.fc2”,  
“model.encoder.layers.6.self\_attn.k\_proj”,  
“model.encoder.layers.6.self\_attn.v\_proj”,  
“model.encoder.layers.6.self\_attn.q\_proj”,  
“model.encoder.layers.6.self\_attn.out\_proj”,  
“model.encoder.layers.6.fc1”,  
“model.encoder.layers.6.fc2”,  
“model.encoder.layers.7.self\_attn.k\_proj”,  
“model.encoder.layers.7.self\_attn.v\_proj”,  
“model.encoder.layers.7.self\_attn.q\_proj”,  
“model.encoder.layers.7.self\_attn.out\_proj”,  
“model.encoder.layers.7.fc1”,  
“model.encoder.layers.7.fc2”,  
“model.encoder.layers.8.self\_attn.k\_proj”,  
“model.encoder.layers.8.self\_attn.v\_proj”,  
“model.encoder.layers.8.self\_attn.q\_proj”,  
“model.encoder.layers.8.self\_attn.out\_proj”,  
“model.encoder.layers.8.fc1”,  
“model.encoder.layers.8.fc2”,  
“model.encoder.layers.9.self\_attn.k\_proj”,  
“model.encoder.layers.9.self\_attn.v\_proj”,  
“model.encoder.layers.9.self\_attn.q\_proj”,  
“model.encoder.layers.9.self\_attn.out\_proj”,  
“model.encoder.layers.9.fc1”,  
“model.encoder.layers.9.fc2”,  
“model.encoder.layers.10.self\_attn.k\_proj”,  
“model.encoder.layers.10.self\_attn.v\_proj”,  
“model.encoder.layers.10.self\_attn.q\_proj”,  
“model.encoder.layers.10.self\_attn.out\_proj”,  
“model.encoder.layers.10.fc1”,  
“model.encoder.layers.10.fc2”,  
“model.encoder.layers.11.self\_attn.k\_proj”,  
“model.encoder.layers.11.self\_attn.v\_proj”,  
“model.encoder.layers.11.self\_attn.q\_proj”,  
“model.encoder.layers.11.self\_attn.out\_proj”,  
“model.encoder.layers.11.fc1”,  
“model.encoder.layers.11.fc2”,

```
# Decoder layers
"model.decoder.layers.0.self_attn.k_proj",
"model.decoder.layers.0.self_attn.v_proj",
"model.decoder.layers.0.self_attn.q_proj",
"model.decoder.layers.0.self_attn.out_proj",
"model.decoder.layers.0.encoder_attn.k_proj",
"model.decoder.layers.0.encoder_attn.v_proj",
"model.decoder.layers.0.encoder_attn.q_proj",
"model.decoder.layers.0.encoder_attn.out_proj",
"model.decoder.layers.0.fc1",
"model.decoder.layers.0.fc2",
"model.decoder.layers.1.self_attn.k_proj",
"model.decoder.layers.1.self_attn.v_proj",
"model.decoder.layers.1.self_attn.q_proj",
"model.decoder.layers.1.self_attn.out_proj",
"model.decoder.layers.1.encoder_attn.k_proj",
"model.decoder.layers.1.encoder_attn.v_proj",
"model.decoder.layers.1.encoder_attn.q_proj",
"model.decoder.layers.1.encoder_attn.out_proj",
"model.decoder.layers.1.fc1",
"model.decoder.layers.1.fc2",
"model.decoder.layers.2.self_attn.k_proj",
"model.decoder.layers.2.self_attn.v_proj",
"model.decoder.layers.2.self_attn.q_proj",
"model.decoder.layers.2.self_attn.out_proj",
"model.decoder.layers.2.encoder_attn.k_proj",
"model.decoder.layers.2.encoder_attn.v_proj",
"model.decoder.layers.2.encoder_attn.q_proj",
"model.decoder.layers.2.encoder_attn.out_proj",
"model.decoder.layers.2.fc1",
"model.decoder.layers.2.fc2",
"model.decoder.layers.3.self_attn.k_proj",
"model.decoder.layers.3.self_attn.v_proj",
"model.decoder.layers.3.self_attn.q_proj",
"model.decoder.layers.3.self_attn.out_proj",
"model.decoder.layers.3.encoder_attn.k_proj",
"model.decoder.layers.3.encoder_attn.v_proj",
"model.decoder.layers.3.encoder_attn.q_proj",
"model.decoder.layers.3.encoder_attn.out_proj",
"model.decoder.layers.3.fc1",
"model.decoder.layers.3.fc2",
"model.decoder.layers.4.self_attn.k_proj",
"model.decoder.layers.4.self_attn.v_proj",
"model.decoder.layers.4.self_attn.q_proj",
"model.decoder.layers.4.self_attn.out_proj",
"model.decoder.layers.4.encoder_attn.k_proj",
"model.decoder.layers.4.encoder_attn.v_proj",
"model.decoder.layers.4.encoder_attn.q_proj",
"model.decoder.layers.4.encoder_attn.out_proj",
"model.decoder.layers.4.fc1",
"model.decoder.layers.4.fc2",
"model.decoder.layers.5.self_attn.k_proj",
"model.decoder.layers.5.self_attn.v_proj",
"model.decoder.layers.5.self_attn.q_proj",
"model.decoder.layers.5.self_attn.out_proj",
"model.decoder.layers.5.encoder_attn.k_proj",
"model.decoder.layers.5.encoder_attn.v_proj",
"model.decoder.layers.5.encoder_attn.q_proj",
"model.decoder.layers.5.encoder_attn.out_proj",
"model.decoder.layers.5.fc1",
"model.decoder.layers.5.fc2",
"model.decoder.layers.6.self_attn.k_proj",
"model.decoder.layers.6.self_attn.v_proj",
"model.decoder.layers.6.self_attn.q_proj",
"model.decoder.layers.6.self_attn.out_proj",
"model.decoder.layers.6.encoder_attn.k_proj",
"model.decoder.layers.6.encoder_attn.v_proj",
"model.decoder.layers.6.encoder_attn.q_proj",
"model.decoder.layers.6.encoder_attn.out_proj",
"model.decoder.layers.6.fc1",
"model.decoder.layers.6.fc2",
"model.decoder.layers.7.self_attn.k_proj",
"model.decoder.layers.7.self_attn.v_proj",
"model.decoder.layers.7.self_attn.q_proj",
"model.decoder.layers.7.self_attn.out_proj",
"model.decoder.layers.7.encoder_attn.k_proj",
"model.decoder.layers.7.encoder_attn.v_proj",
"model.decoder.layers.7.encoder_attn.q_proj",
"model.decoder.layers.7.encoder_attn.out_proj",
"model.decoder.layers.7.fc1",
"model.decoder.layers.7.fc2",
"model.decoder.layers.8.self_attn.k_proj",
"model.decoder.layers.8.self_attn.v_proj",
"model.decoder.layers.8.self_attn.q_proj",
"model.decoder.layers.8.self_attn.out_proj",
"model.decoder.layers.8.encoder_attn.k_proj",
"model.decoder.layers.8.encoder_attn.v_proj",
"model.decoder.layers.8.encoder_attn.q_proj",
"model.decoder.layers.8.encoder_attn.out_proj",
"model.decoder.layers.8.fc1",
"model.decoder.layers.8.fc2",
"model.decoder.layers.9.self_attn.k_proj",
"model.decoder.layers.9.self_attn.v_proj",
"model.decoder.layers.9.self_attn.q_proj",
"model.decoder.layers.9.self_attn.out_proj",
"model.decoder.layers.9.encoder_attn.k_proj",
"model.decoder.layers.9.encoder_attn.v_proj",
"model.decoder.layers.9.encoder_attn.q_proj",
"model.decoder.layers.9.encoder_attn.out_proj",
"model.decoder.layers.9.fc1",
"model.decoder.layers.9.fc2",
"model.decoder.layers.10.self_attn.k_proj",
"model.decoder.layers.10.self_attn.v_proj",
"model.decoder.layers.10.self_attn.q_proj",
"model.decoder.layers.10.self_attn.out_proj",
"model.decoder.layers.10.encoder_attn.k_proj",
"model.decoder.layers.10.encoder_attn.v_proj",
"model.decoder.layers.10.encoder_attn.q_proj",
"model.decoder.layers.10.encoder_attn.out_proj",
"model.decoder.layers.10.fc1",
"model.decoder.layers.10.fc2",
"model.decoder.layers.11.self_attn.k_proj",
"model.decoder.layers.11.self_attn.v_proj",
"model.decoder.layers.11.self_attn.q_proj",
"model.decoder.layers.11.self_attn.out_proj",
"model.decoder.layers.11.encoder_attn.k_proj",
"model.decoder.layers.11.encoder_attn.v_proj",
"model.decoder.layers.11.encoder_attn.q_proj",
"model.decoder.layers.11.encoder_attn.out_proj",
"model.decoder.layers.11.fc1",
"model.decoder.layers.11.fc2",

```

]

# Configure LoRA

config = LoraConfig(  
task\_type=TaskType.SEQ\_2\_SEQ\_LM,  
r=8,  
lora\_alpha=16,  
target\_modules=target\_modules,  
lora\_dropout=0.1,  
)

# for name, module in model.named\_modules():

# print(name)

# Apply LoRA to the model

lora\_model = get\_peft\_model(model, config)

# Set model to training mode

lora\_model.train()

class WhisperDataset(Dataset):  
def **init** (self, csv\_file, audio\_dir, processor):  
self.data = pd.read\_csv(csv\_file)  
self.audio\_dir = audio\_dir  
self.processor = processor

```
def __len__ (self):
    return len(self.data)

def __getitem__ (self, idx):
    if torch.is_tensor(idx):
        idx = idx.tolist()

    audio_path = f"{self.audio_dir}/{self.data.iloc[idx, 0]}"
    transcription = self.data.iloc[idx, 1]

    # Load audio file
    speech_array, sampling_rate = torchaudio.load(audio_path)

    # Process the audio and transcription
    input_features = self.processor.feature_extractor(speech_array.squeeze().numpy(), sampling_rate=sampling_rate, return_tensors="pt").input_features
    labels = self.processor.tokenizer(transcription, return_tensors="pt").input_ids
    # print(f"Processing index: {idx}")
    # print(f"Transcription: {transcription}")
    # print(f"Input features shape: {input_features.shape}")
    # print(f"Labels (input_ids): {labels}")

    return {
        "input_features": input_features.squeeze(),
        "labels": labels.squeeze()
    }

```

# Define a custom collate function to pad sequences

def collate\_fn(batch):  
input\_features = [item[‘input\_features’] for item in batch]  
labels = [item[‘labels’] for item in batch]

```
# Pad sequences
input_features_padded = torch.nn.utils.rnn.pad_sequence(input_features, batch_first=True, padding_value=0)
labels_padded = torch.nn.utils.rnn.pad_sequence(labels, batch_first=True, padding_value=tokenizer.pad_token_id)

return {
    'input_features': input_features_padded,
    'labels': labels_padded
}

```

# Example usage

csv\_file = “./dup.csv” # Path to your CSV file  
audio\_dir = “.” # Path to the directory containing your audio files  
processor = WhisperProcessor.from\_pretrained(“openai/whisper-small”, language=“Tamil”, task=“transcribe”)

dataset = WhisperDataset(csv\_file, audio\_dir, processor)  
dataloader = DataLoader(dataset, batch\_size=8, shuffle=True, collate\_fn=collate\_fn)

# Define the optimizer

optimizer = torch.optim.Adam(lora\_model.parameters(), lr=1e-4)

# Fine-tuning loop

num\_epochs = 3 # Define the number of epochs  
for epoch in range(num\_epochs):  
for batch in dataloader:  
# Forward pass  
input\_features = batch[‘input\_features’].to(lora\_model.device)  
labels = batch[‘labels’].to(lora\_model.device)

```
    print(f"Batch input features shape: {input_features.shape}")
    print(f"Batch labels shape: {labels.shape}")
    print(f"Batch labels: {labels}")
    # print(input_features.shape)
    outputs = lora_model(input_features=input_features, labels=labels)
    loss = outputs.loss

    # Backward pass
    optimizer.zero_grad()
    loss.backward()
    optimizer.step()

print(f"Epoch {epoch + 1}/{num_epochs}, Loss: {loss.item()}")

```

# Save the fine-tuned model

lora\_model.save\_pretrained(“/content/fine\_tuned\_whisper\_with\_lora”)

TypeError: WhisperForConditionalGeneration.forward() got an unexpected keyword argument ‘input\_ids’

Above error is shown for the code

---

<div class="post-metadata">

**Author:** ![igonzalezfs](https://avatars.discourse-cdn.com/v4/letter/i/cdc98d/32.png) [@igonzalezfs](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Figonzalezfs)%5C
**Post date:** [June 19, 2024, 6:58am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Funexpected-keywork-argument%2F91356%2F2 "2024-06-19T06:58:42Z")

</div>

Same here, it seems that Peft wrapper always calls the model with the input\_ids and input\_embeds kwargs, which is unecpected by the Whisper model

---

<div class="post-metadata">

**Author:** ![Isics](https://avatars.discourse-cdn.com/v4/letter/i/a4c791/32.png) [@Isics](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FIsics)%5C
**Post date:** [June 19, 2024, 10:30am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Funexpected-keywork-argument%2F91356%2F3 "2024-06-19T10:30:25Z")

</div>

It looks like `get_peft_model` is only used to get the more appropriate model based on argument `task_type`from LoraConfig.

Since none of the wrappers work for Whisper, you have to create your own implementation.

So instead of `lora_model = get_peft_model(model, config)`, you can use `lora_model = WhisperTuner(model, config)`.

Where WhisperTuner would be something like this:

```auto
class WhisperTuner(PeftModel):
    def __init__ (self, model: torch.nn.Module, peft_config: PeftConfig, adapter_name: str = "default") -> None:
        super(). __init__ (model, peft_config, adapter_name)
        self.base_model_prepare_inputs_for_generation = self.base_model.prepare_inputs_for_generation
        self.base_model_prepare_encoder_decoder_kwargs_for_generation = (
            self.base_model._prepare_encoder_decoder_kwargs_for_generation
        )

    def forward(
            self,
            attention_mask=None,
            decoder_input_ids=None,
            decoder_attention_mask=None,
            decoder_inputs_embeds=None,
            labels=None,
            output_attentions=None,
            output_hidden_states=None,
            return_dict=None,
            task_ids=None,
            **kwargs,
    ):
        peft_config = self.active_peft_config
        if not peft_config.is_prompt_learning:
            if peft_config.peft_type == PeftType.POLY:
                kwargs["task_ids"] = task_ids

            with self._enable_peft_forward_hooks(**kwargs):
                kwargs = {k: v for k, v in kwargs.items() if k not in self.special_peft_forward_args}
                return self.base_model(
                    attention_mask=attention_mask,
                    decoder_input_ids=decoder_input_ids,
                    decoder_attention_mask=decoder_attention_mask,
                    decoder_inputs_embeds=decoder_inputs_embeds,
                    labels=labels,
                    output_attentions=output_attentions,
                    output_hidden_states=output_hidden_states,
                    return_dict=return_dict,
                    **kwargs,
                )

    def generate(self, **kwargs):
        peft_config = self.active_peft_config
        self.base_model.prepare_inputs_for_generation = self.prepare_inputs_for_generation
        self.base_model._prepare_encoder_decoder_kwargs_for_generation = (
            self._prepare_encoder_decoder_kwargs_for_generation
        )
        try:
            if not peft_config.is_prompt_learning:
                with self._enable_peft_forward_hooks(**kwargs):
                    kwargs = {k: v for k, v in kwargs.items() if k not in self.special_peft_forward_args}
                    outputs = self.base_model.generate(**kwargs)
        except:
            self.base_model.prepare_inputs_for_generation = self.base_model_prepare_inputs_for_generation
            self.base_model._prepare_encoder_decoder_kwargs_for_generation = (
                self.base_model_prepare_encoder_decoder_kwargs_for_generation
            )
            raise
        else:
            self.base_model.prepare_inputs_for_generation = self.base_model_prepare_inputs_for_generation
            self.base_model._prepare_encoder_decoder_kwargs_for_generation = (
                self.base_model_prepare_encoder_decoder_kwargs_for_generation
            )
            return outputs

    def prepare_inputs_for_generation(self, *args, task_ids: torch.Tensor = None, **kwargs):
        peft_config = self.active_peft_config
        model_kwargs = self.base_model_prepare_inputs_for_generation(*args, **kwargs)
        if peft_config.peft_type == PeftType.POLY:
            model_kwargs["task_ids"] = task_ids
        if model_kwargs["past_key_values"] is None and peft_config.peft_type == PeftType.PREFIX_TUNING:
            batch_size = model_kwargs["decoder_input_ids"].shape[0]
            past_key_values = self.get_prompt(batch_size)
            model_kwargs["past_key_values"] = past_key_values

        return model_kwargs

```

(I just copied an implementation of PeftModel and deleted all paths and entries that are not needed for whisper, there’s probably room for improvement in this code)

---

<div class="post-metadata">

**Author:** ![CyberBoyNull](https://avatars.discourse-cdn.com/v4/letter/c/59ef9b/32.png) [@CyberBoyNull](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FCyberBoyNull)%5C
**Post date:** [March 19, 2025, 3:07am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Funexpected-keywork-argument%2F91356%2F4 "2025-03-19T03:07:00Z")

</div>

When use openai/whisper-large-v3 and set up no\_speech\_detection will also trigger the issue of unexpected keyword argument of input\_ids. The root cause of this issue is the \_setup\_no\_speech\_detection function in WhisperGenerationMixin add a input\_ids argument which is not acceptable to the forward func of Whisper. Here is a fix PR [fix unexpected kws of input\_ids when setup no speech detection of whisper by zhaozhenyu-newsbreak · Pull Request #36809 · huggingface/transformers · GitHub](https://github.com/huggingface/transformers/pull/36809), waiting for review
