Enhance model card with metadata, abstract, overview, and usage example

This PR significantly enhances the model card by:
- Adding `pipeline_tag: text-generation` to ensure the model appears in relevant searches and filter categories.
- Adding `library_name: transformers` to indicate compatibility with the Hugging Face Transformers library, enabling the "Use in Transformers" widget.
- Including relevant `tags` such as `agent`, `tool-use`, `reinforcement-learning`, `qwen`, and `llm` for better categorization.
- Expanding the model card content with the paper's abstract, a detailed overview including key highlights and visuals from the project's GitHub, and comprehensive usage instructions for text generation with the `transformers` library.
- Consolidating existing links and adding new ones (Hugging Face collection, Hugging Face Space demo) for a richer set of resources.
- Incorporating additional valuable sections from the GitHub README, such as Citation, Acknowledgements, and Contact information.

README.md

@@ -1,11 +1,131 @@
 ---
 license: mit
+pipeline_tag: text-generation
+library_name: transformers
+tags:
+  - agent
+  - tool-use
+  - reinforcement-learning
+  - qwen
+  - llm
 ---
 
-The model checkpoint of ARPO: 
+# ARPO: Agentic Reinforced Policy Optimization
 
-Arxiv: https://arxiv.org/abs/2507.19849
+The model checkpoint of ARPO is released for the paper [**Agentic Reinforced Policy Optimization**](https://huggingface.co/papers/2507.19849).
 
-HF paper: https://huggingface.co/papers/2507.19849
+<div align="center">
+  <img src="https://github.com/dongguanting/ARPO/blob/main/logo1.png" width="150px">
+</div>
 
-Github: https://github.com/dongguanting/ARPO
+<div align="center"> 
+[![Paper](https://img.shields.io/badge/Paper-arXiv-b5212f.svg?logo=arxiv)](https://arxiv.org/abs/2507.19849)
+[![Paper](https://img.shields.io/badge/Paper-Hugging%20Face-yellow?logo=huggingface)](https://huggingface.co/papers/2507.19849)
+[![Model Collection](https://img.shields.io/badge/Model-Hugging%20Face-blue?logo=huggingface)](https://huggingface.co/collections/dongguanting/arpo-688229ff8a6143fe5b4ad8ae)
+[![Dataset Collection](https://img.shields.io/badge/Dataset-Hugging%20Face-blue?logo=huggingface)](https://huggingface.co/collections/dongguanting/arpo-688229ff8a6143fe5b4ad8ae)
+[![GitHub](https://img.shields.io/badge/GitHub-Code-blue?logo=github)](https://github.com/dongguanting/ARPO)
+[![Demo](https://img.shields.io/badge/Demo-Hugging%20Face%20Space-orange?logo=huggingface)](https://huggingface.co/spaces/dongguanting/ARPO-DeepSearch-Viewer)
+</div>
+
+## Abstract
+Large-scale reinforcement learning with verifiable rewards (RLVR) has demonstrated its effectiveness in harnessing the potential of large language models (LLMs) for single-turn reasoning tasks. In realistic reasoning scenarios, LLMs can often utilize external tools to assist in task-solving processes. However, current RL algorithms inadequately balance the models' intrinsic long-horizon reasoning capabilities and their proficiency in multi-turn tool interactions. To bridge this gap, we propose Agentic Reinforced Policy Optimization (ARPO), a novel agentic RL algorithm tailored for training multi-turn LLM-based agents. Through preliminary experiments, we observe that LLMs tend to exhibit highly uncertain behavior, characterized by an increase in the entropy distribution of generated tokens, immediately following interactions with external tools. Motivated by this observation, ARPO incorporates an entropy-based adaptive rollout mechanism, dynamically balancing global trajectory sampling and step-level sampling, thereby promoting exploration at steps with high uncertainty after tool usage. By integrating an advantage attribution estimation, ARPO enables LLMs to internalize advantage differences in stepwise tool-use interactions. Our experiments across 13 challenging benchmarks in computational reasoning, knowledge reasoning, and deep search domains demonstrate ARPO's superiority over trajectory-level RL algorithms. Remarkably, ARPO achieves improved performance using only half of the tool-use budget required by existing methods, offering a scalable solution for aligning LLM-based agents with real-time dynamic environments. Our code and datasets are released at this https URL
+
+## 💡 Overview
+
+We propose **Agentic Reinforced Policy Optimization (ARPO)**, **an agentic RL algorithm tailored for training multi-turn LLM-based agent**. The core principle of ARPO is to encourage the policy model to adaptively branch sampling during high-entropy tool-call rounds, thereby efficiently aligning step-level tool-use behaviors.
+
+<img width="1686" height="866" alt="intro" src="https://github.com/user-attachments/assets/8b9daf54-c4ba-4e79-bf79-f98b5a893edd" />
+
+### Key Highlights
+- **Entropy-based Adaptive Rollout**: ARPO incorporates an entropy-based adaptive rollout mechanism that dynamically balances global trajectory sampling and step-level sampling, promoting exploration at steps with high uncertainty after tool usage.
+- **Advantage Attribution Estimation**: By integrating an advantage attribution estimation, ARPO enables LLMs to internalize advantage differences in stepwise tool-use interactions.
+- **Superior Performance**: Achieves improved performance across 13 challenging benchmarks in computational reasoning, knowledge reasoning, and deep search domains.
+- **Efficient Tool Usage**: Remarkably, ARPO achieves improved performance using only half of the tool-use budget required by existing methods.
+
+## 🏃 Quick Start: Text Generation with Transformers
+
+You can use ARPO models for text generation or chat completion tasks via the Hugging Face `transformers` library.
+
+First, ensure you have the necessary dependencies installed, including `transformers` and `torch`. You may also need `flash-attn` for optimized performance:
+```bash
+pip install transformers torch flash-attn --no-build-isolation
+```
+
+### Text Generation Example
+
+```python
+from transformers import AutoModelForCausalLM, AutoTokenizer
+import torch
+
+# Load the model and tokenizer
+# This specific model is Qwen3-based. For other ARPO models, check the ARPO collection.
+model_id = "dongguanting/Qwen3-8B-ARPO-DeepSearch" 
+tokenizer = AutoTokenizer.from_pretrained(model_id, trust_remote_code=True)
+model = AutoModelForCausalLM.from_pretrained(
+    model_id,
+    torch_dtype=torch.bfloat16, # Use torch.float16 for smaller GPUs
+    device_map="auto",
+    trust_remote_code=True,
+)
+
+# Prepare your prompt using the chat template (recommended for Qwen-based models)
+messages = [
+    {"role": "system", "content": "You are a helpful AI assistant specialized in reasoning tasks."},
+    {"role": "user", "content": "What is the capital of France?"}
+]
+text = tokenizer.apply_chat_template(
+    messages,
+    tokenize=False,
+    add_generation_prompt=True
+)
+
+# Generate response
+input_ids = tokenizer(text, return_tensors="pt").input_ids.to(model.device)
+generated_ids = model.generate(
+    input_ids,
+    max_new_tokens=100,
+    temperature=0.7,
+    do_sample=True,
+    eos_token_id=tokenizer.eos_token_id
+)
+
+# Decode and print the output
+response = tokenizer.decode(generated_ids[0][input_ids.shape[1]:], skip_special_tokens=True)
+print(response)
+
+# Example for direct text generation (without chat template):
+# prompt = "The capital of Germany is"
+# input_ids = tokenizer.encode(prompt, return_tensors="pt").to(model.device)
+# output_ids = model.generate(input_ids, max_new_tokens=50, temperature=0.7, top_p=0.9, do_sample=True)
+# generated_text = tokenizer.decode(output_ids[0], skip_special_tokens=True)
+# print(generated_text)
+```
+
+For more detailed usage, including specific tool-use agentic setups and evaluation, please refer to the [official GitHub repository](https://github.com/dongguanting/ARPO).
+
+## 📄 Citation
+
+If you find this work helpful, please cite our paper:
+```bibtex
+@misc{dong2025arpo,
+      title={Agentic Reinforced Policy Optimization}, 
+      author={Guanting Dong and Hangyu Mao and Kai Ma and Licheng Bao and Yifei Chen and Zhongyuan Wang and Zhongxia Chen and Jiazhen Du and Huiyang Wang and Fuzheng Zhang and Guorui Zhou and Yutao Zhu and Ji-Rong Wen and Zhicheng Dou},
+      year={2025},
+      eprint={2507.19849},
+      archivePrefix={arXiv},
+      primaryClass={cs.LG},
+      url={https://arxiv.org/abs/2507.19849}, 
+}
+```
+
+## 🤝 Acknowledge
+
+This training implementation builds upon [Tool-Star](https://github.com/dongguanting/Tool-Star), [Llama Factory](https://github.com/hiyouga/LLaMA-Factory), [verl](https://github.com/volcengine/verl) and [ReCall](https://github.com/Agent-RL/ReCall). For evaluation, we rely on [WebThinker](https://github.com/RUC-NLPIR/WebThinker), [HIRA](https://github.com/RUC-NLPIR/HiRA), [WebSailor](https://github.com/Alibaba-NLP/WebAgent), [Search-o1](https://github.com/sunnynexus/Search-o1), and [FlashRAG](https://github.com/RUC-NLPIR/FlashRAG). The Python interpreter design references [ToRA](https://github.com/microsoft/ToRA) and [ToRL](https://github.com/GAIR-NLP/ToRL), while our models are trained using [Qwen2.5](https://qwenlm.github.io/blog/qwen2.5/). We express our sincere gratitude to these projects for their invaluable contributions to the open-source community. 
+
+## 📄 License
+
+This project is released under the [MIT License](LICENSE).
+
+## 📞 Contact
+
+For any questions or feedback, please reach out to us at [dongguanting@ruc.edu.cn](dongguanting@ruc.edu.cn).