Update README.md

README.md

@@ -23,8 +23,6 @@ Model can accurately recognize emotions classes- Angry,Sad,Fearful,Happy,Disgust
 
 
 - **Developed by:** [https://www.linkedin.com/in/sharmavaruncs/]
-- **Funded by [optional]:** [More Information Needed]
-- **Shared by [optional]:** [More Information Needed]
 - **Model type:** [MultiModal - Text and Audio based]
 - **Language(s) (NLP):** [NLP, Speech processing]
 - **Finetuned from model [optional]:** [https://huggingface.co/docs/transformers/model_doc/hubert]
@@ -33,7 +31,7 @@ Model can accurately recognize emotions classes- Angry,Sad,Fearful,Happy,Disgust
 
 <!-- Provide the basic links for the model. -->
 
-  - **Repository:** [https://github.com/netgvarun2012/VirtualTherapist/]
+- **Repository:** [https://github.com/netgvarun2012/VirtualTherapist/]
 - **Paper [optional]:** [https://github.com/netgvarun2012/VirtualTherapist/blob/main/documentation/Speech_and_Text_based_MultiModal_Emotion_Recognizer.pdf]
 - **Demo [optional]:** [https://huggingface.co/spaces/netgvarun2005/VirtualTherapist]
 
@@ -43,39 +41,30 @@ Model can accurately recognize emotions classes- Angry,Sad,Fearful,Happy,Disgust
 'Virtual Therapist' app - an Intelligent speech and text input based assistant that can decipher emotions and generate therapeutic messages based on the Emotional state of the user.
 
 Emotions recognized - Angry,Sad,Fearful,Happy,Disgusted,Surprised,Calm with ~80% accuracy.
-### Direct Use
 
-<!-- This section is for the model use without fine-tuning or plugging into a larger ecosystem/app. -->
-
-[More Information Needed]
-
-### Downstream Use [optional]
-
-<!-- This section is for the model use when fine-tuned for a task, or when plugged into a larger ecosystem/app -->
-
-[More Information Needed]
-
-### Out-of-Scope Use
-
-<!-- This section addresses misuse, malicious use, and uses that the model will not work well for. -->
-
-[More Information Needed]
-
-## Bias, Risks, and Limitations
-
-<!-- This section is meant to convey both technical and sociotechnical limitations. -->
-
-[More Information Needed]
-
-### Recommendations
-
-<!-- This section is meant to convey recommendations with respect to the bias, risk, and technical limitations. -->
-
-Users (both direct and downstream) should be made aware of the risks, biases and limitations of the model. More information needed for further recommendations.
-
-## How to Get Started with the Model
-
-Use the code below to get started with the model.
+Use the code below to get started with the model:
+class MultimodalModel(nn.Module):
+    '''
+    Custom PyTorch model that takes as input both the audio features and the text embeddings, and concatenates the last hidden states from the Hubert and BERT models.
+    '''
+    def __init__(self, bert_model_name, num_labels):
+        super().__init__()
+        self.hubert = HubertForSequenceClassification.from_pretrained("netgvarun2005/HubertStandaloneEmoDetector", num_labels=num_labels).hubert
+        self.bert = AutoModel.from_pretrained(bert_model_name)
+        self.classifier = nn.Linear(self.hubert.config.hidden_size + self.bert.config.hidden_size, num_labels)
+
+    def forward(self, input_values, text):
+        hubert_output = self.hubert(input_values).last_hidden_state
+
+        bert_output = self.bert(text).last_hidden_state
+
+        # Apply mean pooling along the sequence dimension
+        hubert_output = hubert_output.mean(dim=1)
+        bert_output = bert_output.mean(dim=1)
+
+        concat_output = torch.cat((hubert_output, bert_output), dim=-1)
+        logits = self.classifier(concat_output)
+        return logits
 
 [More Information Needed]