# How to count input tokens in vision model?

**URL:** </static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fhow-to-count-input-tokens-in-vision-model%2F89813%3E%5C
**Category:** 🤗Transformers\
**Created:** [June 1, 2024, 10:56pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fhow-to-count-input-tokens-in-vision-model%2F89813 "2024-06-01T22:56:13Z")\
**Posts on this page:** 5\
**Page:** 1

<div class="post-metadata">

**Author:** ![maria-ai](https://avatars.discourse-cdn.com/v4/letter/m/b19c9b/32.png) [@maria-ai](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fmaria-ai)%5C
**Post date:** [June 1, 2024, 10:56pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fhow-to-count-input-tokens-in-vision-model%2F89813%2F1 "2024-06-01T22:56:13Z")

</div>

I used apply\_chat\_template for text model:

```auto
tokens = tokenizer.apply_chat_template(messages, add_generation_prompt=True)
tokens_in = len(tokens)

```

But code fails when message have content list:

```auto
{'role': 'system', 'content': [{'type': 'text', 'text': 'You are assistant'}]}, {'role': 'user', 'content': [{'type': 'text', 'text': '1'}]}]

```

I need to count input tokens for images and text.

---

<div class="post-metadata">

**Author:** ![RaushanTurganbay](https://avatars.discourse-cdn.com/v4/letter/r/848f3c/32.png) [@RaushanTurganbay](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FRaushanTurganbay)%5C
**Post date:** [June 2, 2024, 6:39am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fhow-to-count-input-tokens-in-vision-model%2F89813%2F2 "2024-06-02T06:39:53Z")

</div>

Hi! What model you’re using for the chat template?

Not all vision LLMs currently support chat template. The one you have as an example should work for Idefics models. In case you want to add a chat template for other VLMs, you can write a jinja template yourself and do as below, where `chat_template` is your own jinja template.

```python
tokens = tokenizer.apply_chat_template(messages, chat_template=chat_template, add_generation_prompt=True)

```

For example [here](https://github.com/huggingface/transformers/blob/96eb06286b63c9c93334d507e632c175d6ba8b28/src/transformers/models/idefics2/processing_idefics2.py#L332-L353) is the template used for Idefics, but you should make sure to use the chat format that your model was trained on for better performance.

Also see the [docs](https://huggingface.co/docs/transformers/main/en/chat_templating#advanced-template-writing-tips) on writing you own template 🤗

---

<div class="post-metadata">

**Author:** ![maria-ai](https://avatars.discourse-cdn.com/v4/letter/m/b19c9b/32.png) [@maria-ai](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fmaria-ai)%5C
**Post date:** [June 2, 2024, 1:49pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fhow-to-count-input-tokens-in-vision-model%2F89813%2F3 "2024-06-02T13:49:11Z")

</div>

I accidentally use wrong tokenizer, i use [https://huggingface.co/xtuner/llava-llama-3-8b-v1\_1-transformers/tree/main](https://huggingface.co/xtuner/llava-llama-3-8b-v1_1-transformers/tree/main)  
Template is working, but now i have another issue. For 280x420 and small promt I get 96000 tokens (of course it is more then context, but generation is working). I need to make pricing according amount of tokens but I don’t understand how image processing, I saw that OpenAI have pricing for tiles.

---

<div class="post-metadata">

**Author:** ![Matheo-Bolland](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/matheo-bolland/32/37655_2.png) [@Matheo-Bolland](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FMatheo-Bolland)%5C
**Post date:** [December 16, 2024, 9:10am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fhow-to-count-input-tokens-in-vision-model%2F89813%2F4 "2024-12-16T09:10:26Z")

</div>

Hello @maria-ai ,  
How did you manage to count tokens you give to your model ?  
Thank you

---

<div class="post-metadata">

**Author:** ![maria-ai](https://avatars.discourse-cdn.com/v4/letter/m/b19c9b/32.png) [@maria-ai](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fmaria-ai)%5C
**Post date:** [December 19, 2024, 10:06pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fhow-to-count-input-tokens-in-vision-model%2F89813%2F5 "2024-12-19T22:06:24Z")

</div>

I fixed it. Now i count it with:

```python
def num_tokens_from_messages_local(messages, tokenizer_model):
    visual_tokens = 0
    messages_to_count = copy.deepcopy(messages)
    for m in messages_to_count:
        if isinstance(m['content'], list):
            set_text = ''
            for m_content in m['content']:
                if m_content['type'] == 'text':
                    set_text = m_content['text']
                elif m_content['type'] == 'image_url':
                    visual_tokens += 576 # FIXME: magic number
            m['content'] = set_text
    tokenizer = transformers.AutoTokenizer.from_pretrained(tokenizer_model)
    tokens = tokenizer.apply_chat_template(messages_to_count, add_generation_prompt=True)
    tokens_len = len(tokens)
    # print("TOKENS_LEN", tokens_len)
    return tokens_len + visual_tokens

```
