Rx-ZJU/LLMDF-EXP

[ English | 中文 ]

Introduction

The Large Language Model for Dietary Feedback - Expanded Version (LLMDF-EXP) is fine-tuned using synthetic data produced by Human Data Induced Synthetic Data Generation (HDI-SDG) approach. For specific details on the fine-tuning process, please refer to the paper: "Using Large Language Models to Generate Dietary Feedback Similar to Human Experts in Weight Management: Based on Real-World Scenario Data."

Features

We collected 20 real dietary feedback scenarios and took human experts crafting dietary feedback as reference text. We compared the performance of LLMDF-EXP with LLMDF, ChatGLM3-6B, GPT-4, and GPT-3.5 under the same settings. Among them, ChatGLM3-6B is the base model fine-tuned for LLMDF-EXP, LLMDF is the baseline model to compare HDI-SDG with directly using real-world data, GPT-3.5 is the large language model used in the HDI-SDG method, and GPT-4 is the landmark model released by OpenAI after GPT-3.5.

We conducted both machine and human evaluations. For machine evaluations, we used word overlap based evaluation metrics such as BLEU, ROUGE, METEOR, and BERTScore based on pretrained language models. For human evaluations, we assessed the empathy, completeness, professionalism, and usefulness of the generated dietary feedback using the Likert Scale, as detailed in the aforementioned paper.

Results from both machine and human evaluations showed that LLMDF-EXP generated dietary feedback most closely resembling those crafted by human experts. It avoided the limitations of insufficient information due to real-world data quality constraints (a flaw of LLMDF) and the overly apparent traces of machine generation in generalized large language models (feedback from models like ChatGLM3 and GPT was often excessively verbose and formalized). From the insights of statistical and qualitative analyses of human evaluations, LLMDF-EXP is capable of generating comprehensive, practical, and humanized dietary feedback with the best situational adaptability.

The relevant evaluation metrics are shown in the table below:

Model	BLEU-1	ROUGE-L	METEOR	PBERT	RBERT	FBERT
GPT-4	0.084	0.112	0.170	0.583	0.693	0.633
GPT-3.5	0.105	0.136	0.177	0.590	0.701	0.641
ChatGLM3-6B	0.092	0.115	0.170	0.587	0.688	0.633
LLMDF	0.074	0.158	0.076	0.705	0.619	0.658
LLMDF-EXP	0.294	0.213	0.191	0.682	0.708	0.694

	Experts	LLMDF	LLMDF-EXP	ChatGLM3-6B	GPT-3.5	GPT-4
Empathy: The feedback expresses appropriate empathy to the user.	3.63±0.51	3.73±0.45	3.79±0.44	3.79±0.41	3.85±0.36	3.90±0.30
Completeness: The feedback provides sufficiently comprehensive information.	3.81±0.50	3.05±0.79	3.99±0.43	4.11±0.63	4.22±0.45	4.26±0.49
Professionalism: The feedback is nutritionally accurate, tailored and soundly responses to the user.	3.93±0.77	2.86±0.83	3.83±0.57	3.41±0.72	3.76±0.58	3.63±0.66
Usefulness: I can use it as a template to write my feedback to this scene.	3.70±0.78	2.73±0.77	3.79±0.72	3.03±0.57	3.50±0.72	3.30±0.68

Inference

LLMDF-EXP is fine-tuned from ChatGLM3-6B. When using Hugging Face's transformers for inference, it is necessary to pay attention to constructing a prompt template consistent with the dialogue format used during training.

<|user|>
Imagine you are a dietitian and you will receive dietary information from a client who is undergoing dietary management. Your task is to give the client feedback on the dietary information, and you need to provide insightful content to give the client dietary guidance to guide him towards a healthy lifestyle. \nYou can refer to organizing the dietary feedback from the following perspectives: analyzing the attributes of the food consumed by the client, providing healthy food choices, helping the client to develop a dietary pattern, encouraging the client to maintain healthy eating habits, and advising the client to modify his or her dietary habits.<MealInformation><MealType>Breakfast</MealType><MealIntakeInformation>Soy milk 1 bowl; 1 medium egg; pork, corn, and carrot glutinous rice 200 grams;</MealIntakeInformation></MealInformation><PersonalHealthInformation><Gender>Male</Gender><Age>19</Age><Height>180cm</Height><Weight>84.0kg</Weight><Waistline>85cm</Waistline><LaborIntensity>Medium</LaborIntensity></PersonalHealthInformation>
<|assistant|>

When using the prompt of the above example, the generation result of LLMDF-EXP is:

<|user|>
Imagine you are a dietitian and you will receive dietary information from a client who is undergoing dietary management. Your task is to give the client feedback on the dietary information, and you need to provide insightful content to give the client dietary guidance to guide him towards a healthy lifestyle. \nYou can refer to organizing the dietary feedback from the following perspectives: analyzing the attributes of the food consumed by the client, providing healthy food choices, helping the client to develop a dietary pattern, encouraging the client to maintain healthy eating habits, and advising the client to modify his or her dietary habits.<MealInformation><MealType>Breakfast</MealType><MealIntakeInformation>Soy milk 1 bowl; 1 medium egg; pork, corn, and carrot glutinous rice 200 grams;</MealIntakeInformation></MealInformation><PersonalHealthInformation><Gender>Male</Gender><Age>19</Age><Height>180cm</Height><Weight>84.0kg</Weight><Waistline>85cm</Waistline><LaborIntensity>Medium</LaborIntensity></PersonalHealthInformation>
<|assistant|>
<DietFeedback>The breakfast combination is fairly reasonable, but it is recommended to reduce the intake of pork and rice, and increase the intake of vegetables, such as adding a portion of greens or tomatoes. In addition, if you like drinking soy milk, you can choose unsweetened soy milk or low-fat soy milk to replace regular soy milk.</DietFeedback>