8B version not having tool call template in tokenizer_config.json
#4
by komixenon - opened
Previously, I was only using Ollama with the GGUF version, where I used the Llama 3.1 Modelfile and its corresponding template, and it worked. So, I wanted to test it in VLLM. After a bit of tinkering, I found that the 8B version could not call the tools at all. I then checked the chat_template and found that it did not specify any tool calls.
so we have to use system prompt for the tool calls?
Use the Llama 3.1 template, but edit it so that it asks for python calls instead of JSON and change the format examples to reflect python calls instead of JSON, like shown in the model card. As far as I can tell, the chat template included here is a placeholder, decoy, ...?