QA-DeBERTa-v3-large-threshold-SEP

This model is a fine-tuned version of microsoft/deberta-v3-large on the saiteki-kai/Beavertails-it dataset. It achieves the following results on the evaluation set:

• Loss: 0.0807
• Accuracy: 0.6740
• Macro F1: 0.6751
• Macro Precision: 0.6605
• Macro Recall: 0.6974
• Micro F1: 0.7525
• Micro Precision: 0.7407
• Micro Recall: 0.7648
• Flagged/accuracy: 0.8552
• Flagged/precision: 0.8607
• Flagged/recall: 0.8828
• Flagged/f1: 0.8716
• Flagged/aucpr: 0.9043
• Flagged/fpr: 0.1793
• Animal Abuse/accuracy: 0.9947
• Animal Abuse/precision: 0.7708
• Animal Abuse/recall: 0.7674
• Animal Abuse/f1: 0.7691
• Animal Abuse/fpr: 0.0026
• Animal Abuse/threshold: 0.4045
• Child Abuse/accuracy: 0.9966
• Child Abuse/precision: 0.7124
• Child Abuse/recall: 0.6547
• Child Abuse/f1: 0.6823
• Child Abuse/fpr: 0.0015
• Child Abuse/threshold: 0.2736
• Controversial Topics,politics/accuracy: 0.9678
• Controversial Topics,politics/precision: 0.4791
• Controversial Topics,politics/recall: 0.5651
• Controversial Topics,politics/f1: 0.5186
• Controversial Topics,politics/fpr: 0.0194
• Controversial Topics,politics/threshold: 0.2814
• Discrimination,stereotype,injustice/accuracy: 0.9565
• Discrimination,stereotype,injustice/precision: 0.7360
• Discrimination,stereotype,injustice/recall: 0.7073
• Discrimination,stereotype,injustice/f1: 0.7214
• Discrimination,stereotype,injustice/fpr: 0.0219
• Discrimination,stereotype,injustice/threshold: 0.4525
• Drug Abuse,weapons,banned Substance/accuracy: 0.9733
• Drug Abuse,weapons,banned Substance/precision: 0.7396
• Drug Abuse,weapons,banned Substance/recall: 0.8122
• Drug Abuse,weapons,banned Substance/f1: 0.7742
• Drug Abuse,weapons,banned Substance/fpr: 0.0171
• Drug Abuse,weapons,banned Substance/threshold: 0.4799
• Financial Crime,property Crime,theft/accuracy: 0.9610
• Financial Crime,property Crime,theft/precision: 0.7883
• Financial Crime,property Crime,theft/recall: 0.8187
• Financial Crime,property Crime,theft/f1: 0.8032
• Financial Crime,property Crime,theft/fpr: 0.0237
• Financial Crime,property Crime,theft/threshold: 0.4026
• Hate Speech,offensive Language/accuracy: 0.9484
• Hate Speech,offensive Language/precision: 0.7241
• Hate Speech,offensive Language/recall: 0.6857
• Hate Speech,offensive Language/f1: 0.7044
• Hate Speech,offensive Language/fpr: 0.0257
• Hate Speech,offensive Language/threshold: 0.3522
• Misinformation Regarding Ethics,laws And Safety/accuracy: 0.9770
• Misinformation Regarding Ethics,laws And Safety/precision: 0.1934
• Misinformation Regarding Ethics,laws And Safety/recall: 0.2818
• Misinformation Regarding Ethics,laws And Safety/f1: 0.2294
• Misinformation Regarding Ethics,laws And Safety/fpr: 0.0145
• Misinformation Regarding Ethics,laws And Safety/threshold: 0.1624
• Non Violent Unethical Behavior/accuracy: 0.8818
• Non Violent Unethical Behavior/precision: 0.7041
• Non Violent Unethical Behavior/recall: 0.6986
• Non Violent Unethical Behavior/f1: 0.7013
• Non Violent Unethical Behavior/fpr: 0.0728
• Non Violent Unethical Behavior/threshold: 0.3425
• Privacy Violation/accuracy: 0.9811
• Privacy Violation/precision: 0.7991
• Privacy Violation/recall: 0.8247
• Privacy Violation/f1: 0.8117
• Privacy Violation/fpr: 0.0108
• Privacy Violation/threshold: 0.5118
• Self Harm/accuracy: 0.9967
• Self Harm/precision: 0.7757
• Self Harm/recall: 0.7171
• Self Harm/f1: 0.7452
• Self Harm/fpr: 0.0014
• Self Harm/threshold: 0.3074
• Sexually Explicit,adult Content/accuracy: 0.9829
• Sexually Explicit,adult Content/precision: 0.6181
• Sexually Explicit,adult Content/recall: 0.7630
• Sexually Explicit,adult Content/f1: 0.6830
• Sexually Explicit,adult Content/fpr: 0.0116
• Sexually Explicit,adult Content/threshold: 0.4078
• Terrorism,organized Crime/accuracy: 0.9883
• Terrorism,organized Crime/precision: 0.3615
• Terrorism,organized Crime/recall: 0.5967
• Terrorism,organized Crime/f1: 0.4502
• Terrorism,organized Crime/fpr: 0.0085
• Terrorism,organized Crime/threshold: 0.1883
• Violence,aiding And Abetting,incitement/accuracy: 0.9232
• Violence,aiding And Abetting,incitement/precision: 0.8452
• Violence,aiding And Abetting,incitement/recall: 0.8707
• Violence,aiding And Abetting,incitement/f1: 0.8578
• Violence,aiding And Abetting,incitement/fpr: 0.0578
• Violence,aiding And Abetting,incitement/threshold: 0.5215

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 1e-05
• train_batch_size: 64
• eval_batch_size: 64
• seed: 42
• optimizer: Use OptimizerNames.ADAMW_TORCH with betas=(0.9,0.999) and epsilon=1e-08 and optimizer_args=No additional optimizer arguments
• lr_scheduler_type: cosine
• lr_scheduler_warmup_ratio: 0.03
• num_epochs: 10

Training results

Training Loss	Epoch	Step	Validation Loss	Accuracy	Macro F1	Macro Precision	Macro Recall	Micro F1	Micro Precision	Micro Recall	Flagged/accuracy	Flagged/precision	Flagged/recall	Flagged/f1	Flagged/aucpr	Flagged/fpr	Animal Abuse/accuracy	Animal Abuse/precision	Animal Abuse/recall	Animal Abuse/f1	Animal Abuse/fpr	Animal Abuse/threshold	Child Abuse/accuracy	Child Abuse/precision	Child Abuse/recall	Child Abuse/f1	Child Abuse/fpr	Child Abuse/threshold	Controversial Topics,politics/accuracy	Controversial Topics,politics/precision	Controversial Topics,politics/recall	Controversial Topics,politics/f1	Controversial Topics,politics/fpr	Controversial Topics,politics/threshold	Discrimination,stereotype,injustice/accuracy	Discrimination,stereotype,injustice/precision	Discrimination,stereotype,injustice/recall	Discrimination,stereotype,injustice/f1	Discrimination,stereotype,injustice/fpr	Discrimination,stereotype,injustice/threshold	Drug Abuse,weapons,banned Substance/accuracy	Drug Abuse,weapons,banned Substance/precision	Drug Abuse,weapons,banned Substance/recall	Drug Abuse,weapons,banned Substance/f1	Drug Abuse,weapons,banned Substance/fpr	Drug Abuse,weapons,banned Substance/threshold	Financial Crime,property Crime,theft/accuracy	Financial Crime,property Crime,theft/precision	Financial Crime,property Crime,theft/recall	Financial Crime,property Crime,theft/f1	Financial Crime,property Crime,theft/fpr	Financial Crime,property Crime,theft/threshold	Hate Speech,offensive Language/accuracy	Hate Speech,offensive Language/precision	Hate Speech,offensive Language/recall	Hate Speech,offensive Language/f1	Hate Speech,offensive Language/fpr	Hate Speech,offensive Language/threshold	Misinformation Regarding Ethics,laws And Safety/accuracy	Misinformation Regarding Ethics,laws And Safety/precision	Misinformation Regarding Ethics,laws And Safety/recall	Misinformation Regarding Ethics,laws And Safety/f1	Misinformation Regarding Ethics,laws And Safety/fpr	Misinformation Regarding Ethics,laws And Safety/threshold	Non Violent Unethical Behavior/accuracy	Non Violent Unethical Behavior/precision	Non Violent Unethical Behavior/recall	Non Violent Unethical Behavior/f1	Non Violent Unethical Behavior/fpr	Non Violent Unethical Behavior/threshold	Privacy Violation/accuracy	Privacy Violation/precision	Privacy Violation/recall	Privacy Violation/f1	Privacy Violation/fpr	Privacy Violation/threshold	Self Harm/accuracy	Self Harm/precision	Self Harm/recall	Self Harm/f1	Self Harm/fpr	Self Harm/threshold	Sexually Explicit,adult Content/accuracy	Sexually Explicit,adult Content/precision	Sexually Explicit,adult Content/recall	Sexually Explicit,adult Content/f1	Sexually Explicit,adult Content/fpr	Sexually Explicit,adult Content/threshold	Terrorism,organized Crime/accuracy	Terrorism,organized Crime/precision	Terrorism,organized Crime/recall	Terrorism,organized Crime/f1	Terrorism,organized Crime/fpr	Terrorism,organized Crime/threshold	Violence,aiding And Abetting,incitement/accuracy	Violence,aiding And Abetting,incitement/precision	Violence,aiding And Abetting,incitement/recall	Violence,aiding And Abetting,incitement/f1	Violence,aiding And Abetting,incitement/fpr	Violence,aiding And Abetting,incitement/threshold
0.0666	1.0	8454	0.0838	0.6680	0.6695	0.6631	0.6864	0.7477	0.7354	0.7603	0.8478	0.8495	0.8828	0.8658	0.8988	0.1962	0.9949	0.7981	0.7471	0.7718	0.0022	0.4630	0.9967	0.7279	0.6426	0.6826	0.0013	0.6252	0.9649	0.4431	0.5706	0.4988	0.0227	0.3621	0.9549	0.7130	0.7249	0.7189	0.0252	0.4026	0.9741	0.7659	0.7776	0.7717	0.0142	0.4811	0.9597	0.7745	0.8260	0.7994	0.0259	0.4898	0.9453	0.6903	0.7067	0.6984	0.0312	0.3329	0.9794	0.2044	0.2394	0.2205	0.0115	0.1813	0.8824	0.7131	0.6828	0.6977	0.0681	0.4311	0.9806	0.7875	0.8311	0.8087	0.0116	0.4121	0.9970	0.8697	0.6512	0.7448	0.0007	0.8490	0.9833	0.6274	0.7540	0.6849	0.0110	0.2121	0.9871	0.3295	0.5925	0.4235	0.0097	0.1871	0.9196	0.8394	0.8630	0.8510	0.0599	0.4078
0.0752	2.0	16908	0.0807	0.6722	0.6752	0.6578	0.7005	0.7526	0.7359	0.7700	0.8555	0.8584	0.8865	0.8722	0.9040	0.1834	0.9947	0.7708	0.7674	0.7691	0.0026	0.4045	0.9966	0.7124	0.6547	0.6823	0.0015	0.2736	0.9679	0.4793	0.5651	0.5187	0.0194	0.2814	0.9565	0.7358	0.7073	0.7213	0.0219	0.4525	0.9733	0.7396	0.8119	0.7740	0.0171	0.4799	0.9595	0.7612	0.8501	0.8032	0.0287	0.3363	0.9484	0.7238	0.6859	0.7044	0.0257	0.3522	0.9770	0.1936	0.2818	0.2295	0.0144	0.1624	0.8801	0.6946	0.7081	0.7013	0.0772	0.3285	0.9811	0.7990	0.8243	0.8115	0.0108	0.5120	0.9967	0.7743	0.7195	0.7459	0.0014	0.3074	0.9829	0.6181	0.7630	0.6830	0.0116	0.4078	0.9884	0.3619	0.5967	0.4505	0.0085	0.1883	0.9232	0.8454	0.8706	0.8578	0.0577	0.5216
0.0613	3.0	25362	0.0797	0.6683	0.6757	0.6614	0.6987	0.7511	0.7324	0.7708	0.8534	0.8529	0.8899	0.8710	0.9021	0.1925	0.9949	0.8009	0.7427	0.7707	0.0021	0.4492	0.9965	0.6990	0.6486	0.6729	0.0016	0.3460	0.9643	0.4430	0.6412	0.5240	0.0255	0.2108	0.9522	0.6738	0.7732	0.7201	0.0323	0.3693	0.9739	0.7536	0.7983	0.7753	0.0156	0.4306	0.9604	0.7787	0.8289	0.8030	0.0254	0.4559	0.9470	0.7071	0.6960	0.7015	0.0284	0.3812	0.9776	0.2084	0.2996	0.2458	0.0140	0.0995	0.8810	0.6999	0.7019	0.7009	0.0746	0.3739	0.9810	0.7921	0.8338	0.8124	0.0114	0.4144	0.9968	0.8258	0.6707	0.7402	0.0010	0.6298	0.9837	0.6431	0.7284	0.6831	0.0100	0.3074	0.9892	0.3807	0.5572	0.4523	0.0073	0.2790	0.9235	0.8529	0.8611	0.8570	0.0538	0.5018
0.0672	4.0	33816	0.0800	0.6668	0.6740	0.6573	0.7012	0.7496	0.7303	0.7699	0.8535	0.8546	0.8877	0.8708	0.9024	0.1894	0.9952	0.8347	0.7195	0.7728	0.0016	0.6352	0.9965	0.6785	0.6907	0.6845	0.0018	0.4779	0.9639	0.4371	0.6205	0.5129	0.0253	0.2451	0.9548	0.7200	0.7065	0.7132	0.0237	0.4230	0.9732	0.7437	0.7986	0.7702	0.0164	0.4995	0.9607	0.7819	0.8262	0.8035	0.0248	0.4751	0.9468	0.7075	0.6919	0.6996	0.0281	0.3975	0.9795	0.2217	0.2736	0.2449	0.0118	0.1813	0.8793	0.6948	0.7001	0.6974	0.0763	0.3863	0.9814	0.8057	0.8220	0.8138	0.0103	0.4608	0.9968	0.7929	0.7098	0.7490	0.0013	0.3840	0.9824	0.6066	0.7671	0.6774	0.0123	0.3016	0.9877	0.3459	0.6091	0.4413	0.0093	0.1767	0.9208	0.8313	0.8812	0.8555	0.0648	0.4875

Framework versions

• Transformers 4.57.1
• Pytorch 2.7.1+cu118
• Datasets 4.4.1
• Tokenizers 0.22.1