Transformers documentation

VideoLLaMA3

Transformers

You are viewing main version, which requires installation from source. If you'd like regular pip install, checkout the latest stable version (v5.22.2).

Join the Hugging Face community

and get access to the augmented documentation experience

Collaborate on models, datasets and Spaces

Faster examples with accelerated inference

Switch between documentation themes

to get started

该模型于 2025-01-22 在 HF papers 上发布，并于 2025-10-13 贡献至 Hugging Face Transformers。

VideoLLaMA3

概述

VideoLLaMA3 模型是阿里巴巴达摩院对 VideoLLaMA2 的一次重大更新。

该论文的摘要如下：

在本文中，我们提出了 VideoLLaMA 3，一个更先进的用于图像和视频理解的多模态基础模型。VideoLLaMA3 的核心设计理念是以视觉为中心。“以视觉为中心”包含两层含义：以视觉为中心的训练范式和以视觉为中心的框架设计。我们提出的以视觉为中心的训练范式的关键洞察在于，高质量的图文数据对于图像理解和视频理解都至关重要。与其准备海量的视频-文本数据集，我们专注于构建大规模、高质量的图文数据集。VideoLLaMA3 包含四个训练阶段：1) 视觉编码器适配，使视觉编码器能够接受可变分辨率的图像作为输入；2) 视觉-语言对齐，使用涵盖多种类型（包括场景图像、文档和图表）的大规模图文数据以及纯文本数据，联合微调视觉编码器、投影器和 LLM。3) 多任务微调，引入用于下游任务的图文 SFT 数据和视频-文本数据，为视频理解奠定基础。4) 以视频为中心的微调，进一步提升模型在视频理解方面的能力。在框架设计方面，为了更好地捕捉图像中的细粒度细节，我们将预训练的视觉编码器适配为将不同大小的图像编码为相应数量的视觉 token，而不是固定数量的 token。对于视频输入，我们根据其相似性减少视觉 token 的数量，从而使视频的表示更加精确和紧凑。得益于以视觉为中心的设计，VideoLLaMA3 在图像和视频理解基准测试中均取得了令人信服的性能。

VideoLLaMA3 architecture. Taken from the technical report.

该模型由 lkhl 贡献。

使用示例

单媒体推理

该模型可以接受图像和视频作为输入。以下是一个推理示例代码。

from transformers import AutoProcessor, VideoLlama3ForConditionalGeneration


# Load the model in half-precision on the available device(s)
model = VideoLlama3ForConditionalGeneration.from_pretrained("lkhl/VideoLLaMA3-2B-Image-HF", device_map="auto")
processor = AutoProcessor.from_pretrained("lkhl/VideoLLaMA3-2B-Image-HF")


conversation = [
    {
        "role":"user",
        "content":[
            {"type": "image", "image": "https://github.com/DAMO-NLP-SG/VideoLLaMA3/raw/refs/heads/main/assets/sora.png"},
            {"type": "text", "text": "Describe this image."}
        ]
    }
]

inputs = processor.apply_chat_template(
    conversation,
    add_generation_prompt=True,
    tokenize=True,
    return_dict=True,
    return_tensors="pt"
).to(model.device)

# Inference: Generation of the output
output_ids = model.generate(**inputs, max_new_tokens=128)
generated_ids = [output_ids[len(input_ids):] for input_ids, output_ids in zip(inputs.input_ids, output_ids)]
output_text = processor.batch_decode(generated_ids, skip_special_tokens=True, clean_up_tokenization_spaces=True)
print(output_text)



# Video
conversation = [
    {
        "role": "user",
        "content": [
            {"type": "video", "video": "https://github.com/DAMO-NLP-SG/VideoLLaMA3/raw/refs/heads/main/assets/cat_and_chicken.mp4"},
            {"type": "text", "text": "What happened in the video?"},
        ],
    }
]

inputs = processor.apply_chat_template(
    conversation,
    fps=1,
    add_generation_prompt=True,
    tokenize=True,
    return_dict=True,
    return_tensors="pt"
).to(model.device)


# Inference: Generation of the output
output_ids = model.generate(**inputs, max_new_tokens=128)
generated_ids = [output_ids[len(input_ids):] for input_ids, output_ids in zip(inputs.input_ids, output_ids)]
output_text = processor.batch_decode(generated_ids, skip_special_tokens=True, clean_up_tokenization_spaces=True)
print(output_text)

批量混合媒体推理

该模型可以批量处理由图像、视频和文本等不同类型样本混合组成的输入。以下是一个示例。

# Image
conversation1 = [
    {
        "role": "user",
        "content": [
            {"type": "image", "image": "https://github.com/DAMO-NLP-SG/VideoLLaMA3/raw/refs/heads/main/assets/sora.png"},
            {"type": "text", "text": "Describe this image."}
        ]
    }
]

# Video
conversation2 = [
    {
        "role": "user",
        "content": [
            {"type": "video", "video": "https://github.com/DAMO-NLP-SG/VideoLLaMA3/raw/refs/heads/main/assets/cat_and_chicken.mp4"},
            {"type": "text", "text": "What happened in the video?"},
        ],
    }
]

# Text
conversation3 = [
    {
        "role": "user",
        "content": "What color is a banana?"
    }
]


conversations = [conversation1, conversation2, conversation3]
# Preparation for batch inference
inputs = processor.apply_chat_template(
    conversations,
    fps=1,
    add_generation_prompt=True,
    tokenize=True,
    padding=True,
    padding_side="left",
    return_dict=True,
    return_tensors="pt"
).to(model.device)


# Batch Inference
output_ids = model.generate(**inputs, max_new_tokens=128)
generated_ids = [output_ids[len(input_ids):] for input_ids, output_ids in zip(inputs.input_ids, output_ids)]
output_text = processor.batch_decode(generated_ids, skip_special_tokens=True, clean_up_tokenization_spaces=True)
print(output_text)

使用 Flash-Attention 2 加速生成

首先，请确保安装最新版本的 Flash Attention 2：

pip install -U flash-attn --no-build-isolation

此外，您的硬件必须与 Flash-Attention 2 兼容。有关详细信息，请参阅 flash attention 仓库的官方文档。只有当模型以 torch.float16 或 torch.bfloat16 加载时，才能使用 FlashAttention-2。

要使用 Flash Attention-2 加载并运行模型，只需在加载模型时添加 attn_implementation="flash_attention_2"，如下所示：

from transformers import VideoLlama3ForConditionalGeneration

model = VideoLlama3ForConditionalGeneration.from_pretrained(
    "lkhl/VideoLLaMA3-2B-Image-HF", , 
    attn_implementation="flash_attention_2",
 device_map="auto")

Transformers

VideoLLaMA3

概述

使用示例

单媒体推理

批量混合媒体推理

使用 Flash-Attention 2 加速生成

VideoLlama3Config

class transformers.VideoLlama3Config

VideoLlama3VisionConfig

类 transformers.VideoLlama3VisionConfig

VideoLlama3ImageProcessor

class transformers.VideoLlama3ImageProcessor

预处理

VideoLlama3VideoProcessor

class transformers.VideoLlama3VideoProcessor

预处理

VideoLlama3ImageProcessorPil

class transformers.VideoLlama3ImageProcessorPil

预处理

VideoLlama3Processor

class transformers.VideoLlama3Processor

__call__

VideoLlama3Model

class transformers.VideoLlama3Model

forward

get_video_features

get_image_features

VideoLlama3VisionModel

class transformers.VideoLlama3VisionModel

forward

VideoLlama3ForConditionalGeneration

类 transformers.VideoLlama3ForConditionalGeneration

forward

get_video_features

get_image_features

call