图像文本到文本

图像-文本到文本模型，也称为视觉语言模型（VLM），是一种接受图像输入的文本模型。这些模型可以处理多种任务，从视觉问答到图像分割。该任务与图像到文本有许多相似之处，但也存在一些重叠的使用场景，例如图像描述。图像到文本模型仅接受图像输入，通常用于完成特定任务；而 VLM 接受开放式文本和图像输入，是更通用的模型。

在本指南中，我们将简要介绍 VLM，并展示如何使用 Transformers 进行推理。

首先，VLM 有多种类型：

用于微调的基础模型
用于对话的经过微调的聊天模型
经过指令微调的模型

本指南专注于使用经过指令微调的模型进行推理。

让我们开始安装依赖项。

pip install -q transformers accelerate 
pip install flash-attn --no-build-isolation

让我们初始化模型和处理器。

from transformers import AutoProcessor, AutoModelForImageTextToText
from accelerate import Accelerator
import torch

device = Accelerator().device
model = AutoModelForImageTextToText.from_pretrained(
    "Qwen/Qwen3-VL-4B-Instruct",
    dtype=torch.bfloat16,
    attn_implementation="flash_attention_2",
).to(device)

像 Qwen3-VL 和 Qwen3.5 这类动态分辨率模型，默认的 longest_edge 约为 16MP，这可能导致每张图像生成超过 10,000 个视觉 token，从而在消费级 GPU 上造成推理卡顿或内存溢出（OOM）。在加载 processor 时，请设置 min_pixels 和 max_pixels 以限制该值。

processor = AutoProcessor.from_pretrained(
    "Qwen/Qwen3-VL-4B-Instruct",
    min_pixels=224 * 224,
    max_pixels=1024 * 1024,
)

该模型具有一个聊天模板，可帮助用户解析聊天输出。此外，该模型还支持在单次对话或消息中接收多张图片作为输入。接下来，我们将准备输入数据。

图像输入如下所示。

按照下图所示的结构组织您的对话，以处理包含图像和文本输入的单次提示。

messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "image": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/cats.png"},
            {"type": "text", "text": "What do we see in this image?"},
        ]
    }
]

在 user 和 assistant 角色之间交替，以利用先前的上下文来引导模型，从而生成更好的回复。

messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "image": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/cats.png"},
            {"type": "text", "text": "What do we see in this image?"},
        ]
    },
    {
        "role": "assistant",
        "content": [
            {"type": "text", "text": "In this image we can see two cats on the nets."},
        ]
    },
    {
        "role": "user",
        "content": [
            {"type": "image", "image": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/bee.jpg"},
            {"type": "text", "text": "And how about this image?"},
        ]
    },
]

接下来，我们将调用处理器的 apply_chat_template() 方法，对其输出以及图像输入进行预处理。

inputs = processor.apply_chat_template(messages, add_generation_prompt=True, tokenize=True, return_dict=True, return_tensors="pt").to(device)

现在，我们可以将预处理后的输入传递给模型。

input_len = len(inputs.input_ids[0])

with torch.no_grad():
    generated_ids = model.generate(**inputs, max_new_tokens=200)
generated_texts = processor.batch_decode(generated_ids[:, input_len:], skip_special_tokens=True)

print(generated_texts)
## ['In this image we can see flowers, plants and insect.']

Pipeline

最快的入门方式是使用 Pipeline API。指定 "image-text-to-text" 任务以及您想要使用的模型。

from transformers import pipeline
pipe = pipeline("image-text-to-text", model="llava-hf/llava-interleave-qwen-0.5b-hf")

以下示例使用聊天模板来格式化文本输入。

messages = [
     {
         "role": "user",
         "content": [
             {
                 "type": "image",
                 "image": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/bee.jpg",
             },
             {"type": "text", "text": "Describe this image."},
         ],
     },
     {
         "role": "assistant",
         "content": [
             {"type": "text", "text": "There's a pink flower"},
         ],
     },
 ]

将经过聊天模板格式化的文本和图像传入 Pipeline，并设置 return_full_text=False 以从生成的输出中移除输入。

outputs = pipe(text=messages, max_new_tokens=20, return_full_text=False)
outputs[0]["generated_text"]
#  with a yellow center in the foreground. The flower is surrounded by red and white flowers with green stems

如果您愿意，也可以单独加载图像，然后按以下方式将其传递给管道：

import requests
from PIL import Image
from transformers import pipeline

pipe = pipeline("image-text-to-text", model="HuggingFaceTB/SmolVLM-256M-Instruct")

img_urls = [
    "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/cats.png",
    "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/bee.jpg",
]
images = [
    Image.open(requests.get(img_urls[0], stream=True).raw),
    Image.open(requests.get(img_urls[1], stream=True).raw),
]

messages = [
    {
        "role": "user",
        "content": [
            {"type": "image"},
            {"type": "image"},
            {"type": "text", "text": "What do you see in these images?"},
        ],
    }
]
outputs = pipe(text=messages, images=images, max_new_tokens=50, return_full_text=False)
outputs[0]["generated_text"]
" In the first image, there are two cats sitting on a plant. In the second image, there are flowers with a pinkish hue."

图像仍会包含在输出的 "input_text" 字段中：

outputs[0]['input_text']
"""
[{'role': 'user',
  'content': [{'type': 'image',
    'image': <PIL.PngImagePlugin.PngImageFile image mode=RGBA size=622x412>},
   {'type': 'image',
    'image': <PIL.JpegImagePlugin.JpegImageFile image mode=RGB size=5184x3456>},
   {'type': 'text', 'text': 'What do you see in these images?'}]}]
"""

流式

我们可以使用文本流式传输来获得更好的生成体验。Transformers 支持通过 TextStreamer 或 TextIteratorStreamer 类进行流式传输。我们将使用 TextIteratorStreamer 配合 IDEFICS-8B。

假设我们有一个应用，它会保存聊天历史并接收新的用户输入。我们将像往常一样对输入进行预处理，并初始化 TextIteratorStreamer 以在单独的线程中处理生成。这样你就可以实时流式传输生成的文本 token。任何生成参数都可以传递给 TextIteratorStreamer。

import time
from transformers import TextIteratorStreamer
from threading import Thread

def model_inference(
    user_prompt,
    chat_history,
    max_new_tokens,
    images
):
    user_prompt = {
        "role": "user",
        "content": [
            {"type": "image"},
            {"type": "text", "text": user_prompt},
        ]
    }
    chat_history.append(user_prompt)
    streamer = TextIteratorStreamer(
        processor.tokenizer,
        skip_prompt=True,
        timeout=5.0,
    )

    generation_args = {
        "max_new_tokens": max_new_tokens,
        "streamer": streamer,
        "do_sample": False
    }

    # add_generation_prompt=True makes model generate bot response
    prompt = processor.apply_chat_template(chat_history, add_generation_prompt=True)
    inputs = processor(
        text=prompt,
        images=images,
        return_tensors="pt",
    ).to(device)
    generation_args.update(inputs)

    thread = Thread(
        target=model.generate,
        kwargs=generation_args,
    )
    thread.start()

    acc_text = ""
    for text_token in streamer:
        time.sleep(0.04)
        acc_text += text_token
        if acc_text.endswith("<end_of_utterance>"):
            acc_text = acc_text[:-18]
        yield acc_text

    thread.join()

现在，让我们调用之前创建的 model_inference 函数并流式传输这些值。

generator = model_inference(
    user_prompt="And what is in this image?",
    chat_history=messages[:2],
    max_new_tokens=100,
    images=images
)

for value in generator:
  print(value)

# In
# In this
# In this image ...

在更小的硬件上训练模型

VLM 通常体积庞大，需要进行优化才能适配较小的硬件。Transformers 支持多种模型量化库，这里我们仅演示使用 Quanto 进行 int8 量化。int8 量化最多可提供 75% 的内存优化（前提是量化所有权重）。但这并非毫无代价，由于 8 位并非 CUDA 原生精度，权重会在运行时反复进行量化与反量化，从而增加延迟。

首先，安装依赖项。

pip install -U optimum-quanto bitsandbytes

要在加载模型时进行量化，首先需要创建 QuantoConfig。然后像往常一样加载模型，但在模型初始化时传入 quantization_config。

from transformers import AutoModelForImageTextToText, QuantoConfig

model_id = "Qwen/Qwen3-VL-4B-Instruct"
quantization_config = QuantoConfig(weights="int8")
quantized_model = AutoModelForImageTextToText.from_pretrained(
    model_id, device_map="auto", quantization_config=quantization_config
)

messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "image": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/cats.png"},
            {"type": "text", "text": "What do we see in this image?"},
        ]
    },
]
inputs = processor.apply_chat_template(messages, add_generation_prompt=True, tokenize=True, return_dict=True, return_tensors="pt").to(quantized_model.device)
input_len = len(inputs.input_ids[0])

with torch.no_grad():
    generated_ids = quantized_model.generate(**inputs, cache_implementation="static", max_new_tokens=100)
generated_texts = processor.batch_decode(generated_ids[:, input_len:], skip_special_tokens=True)

print(generated_texts[0])
## ['In this image, we see two tabby cats resting on a large, tangled pile of fishing nets. The nets are a mix of brown, orange, and red colors, with some blue and green ropes visible in the background. The cats appear relaxed and comfortable, nestled into the fibers of the nets. One cat is in the foreground, looking slightly to the side, while the other is positioned further back, looking directly at the camera. The scene suggests a coastal or fishing-related setting, possibly near']

就这样，我们可以以完全相同的方式使用该模型，无需任何更改。

使用缓存进行迭代式聊天

如果你正在构建多模态聊天智能体，你很可能会在多轮对话中反复传递相同的图像或音频。通过缓存，你可以复用这些内容的编码表示，而无需每次重新处理，从而减少冗余计算。

与纯文本模型类似，多模态模型也使用 chat template 来跟踪对话历史。关键区别在于，它仅以 "text" 格式应用 chat template，并单独处理新的多模态内容。如果在应用模板时执行处理，由于 Jinja 无法区分之前已编码的输入和新的输入，整个对话会在每一轮都被重新处理。

以下示例演示了该模式。对于纯文本模型，请参阅迭代生成指南。

下面是一个简单的 Python 示例，用于演示该模式。如需查看仅使用文本模型的示例，请参阅此指南。

import torch
from transformers import AutoProcessor, Gemma4ForConditionalGeneration, TextStreamer
from transformers.cache_utils import DynamicCache
from transformers.image_utils import load_image
from transformers.audio_utils import load_audio

model_id = 'google/gemma-4-E2B-it'
processor = AutoProcessor.from_pretrained(model_id)
model = Gemma4ForConditionalGeneration.from_pretrained(model_id, dtype=torch.float32, device_map='cpu')

past_key_values = DynamicCache()
streamer = TextStreamer(processor.tokenizer, skip_prompt=True, skip_special_tokens=True)

## Turn 1: multimodal input (text + image + audio)
audio = load_audio("https://huggingface.co/datasets/Xenova/transformers.js-docs/resolve/main/jfk.wav")
image = load_image("https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/bee.jpg")
messages = [
    {
        "role": "user",
        "content": [
            {"type": "text", "text": "In detail, describe the following audio and image."},
            {"type": "audio"},
            {"type": "image"},
        ],
    },
]
input_string = processor.apply_chat_template(messages, add_generation_prompt=True, tokenize=False)
inputs = processor(text=input_string, images=[image], audio=[audio], return_tensors='pt')
output = model.generate(**inputs, past_key_values=past_key_values, max_new_tokens=1024, do_sample=False, streamer=streamer)
first_response = processor.decode(output[0][inputs['input_ids'].shape[1]:], skip_special_tokens=True)

## Turn 2: text-only follow-up (reuses KV cache, no new media)
messages.append({"role": "assistant", "content": [{"type": "text", "text": first_response}]})
cached_string = processor.apply_chat_template(messages, tokenize=False)
messages.append({"role": "user", "content": [{"type": "text", "text": "Summarize the previous descriptions in one sentence."}]})
full_string = processor.apply_chat_template(messages, add_generation_prompt=True, tokenize=False)
new_string = full_string[len(cached_string):]

new_input_ids = processor.tokenizer(new_string, add_special_tokens=False, return_tensors='pt')['input_ids']
# Build and pass an attention mask only if batched input or there is padding
# attention_mask = torch.ones(1, past_key_values.get_seq_length() + new_input_ids.shape[1], dtype=torch.long)

output2 = model.generate(input_ids=new_input_ids, past_key_values=past_key_values, max_new_tokens=1024, do_sample=False, streamer=streamer)
second_response = processor.decode(output2[0][new_input_ids.shape[1]:], skip_special_tokens=True)

## Turn 3: new image (must go through processor for image token expansion + encoding)
image2 = load_image("https://huggingface.co/datasets/Xenova/transformers.js-docs/resolve/main/artemis.jpeg")
messages.append({"role": "assistant", "content": [{"type": "text", "text": second_response}]})
cached_string2 = processor.apply_chat_template(messages, tokenize=False)
messages.append({"role": "user", "content": [{"type": "text", "text": "Describe this image."}, {"type": "image"}]})
full_string2 = processor.apply_chat_template(messages, add_generation_prompt=True, tokenize=False)
new_string2 = full_string2[len(cached_string2):]

new_inputs2 = processor(text=new_string2, images=[image2], add_special_tokens=False, return_tensors='pt')
# new_inputs2['attention_mask'] = torch.ones(1, past_key_values.get_seq_length() + new_inputs2['input_ids'].shape[1], dtype=torch.long)
new_inputs2['past_key_values'] = past_key_values

output3 = model.generate(**new_inputs2, max_new_tokens=1024, do_sample=False, streamer=streamer)

Transformers

图像文本到文本

Pipeline

流式

在更小的硬件上训练模型

使用缓存进行迭代式聊天

延伸阅读