README.md

---
license: mit
datasets:
- Somayeh-h/Nordland
- OPR-Project/OxfordRobotCar_OpenPlaceRecognition
language:
- en
metrics:
- recall_at_1
- recall_at_5
pipeline_tag: image-feature-extraction
tags:
- place-recognition
- visual-place-recognition
- computer-vision
- transformer
- 3d-vision
library:
- pytorch
- lightning
---

# Model Card for UniPR-3D

UniPR-3D is a universal visual place recognition (VPR) framework that supports both **single-frame** and **sequence-to-sequence** matching. It leverages **3D visual geometry grounded tokens** within a transformer architecture to produce robust, viewpoint-invariant descriptors for long-term place recognition under challenging environmental variations (e.g., seasonal, weather, lighting, and viewpoint changes).

## Model Details

### Model Description

- **Developed by:** Tianchen Deng, Xun Chen, Ziming Li, Hongming Shen, Danwei Wang, Javier Civera, Hesheng Wang  
- **Shared by:** Tianchen Deng  
- **Model type:** Vision Transformer with 3D-aware token aggregation for visual place recognition  
- **Language(s):** English (dataset metadata); model is vision-only  
- **License:** MIT  

### Model Sources

- **Repository:** [repo](https://github.com/dtc111111/UniPR-3D)  
- **Paper:** [UniPR-3D: Towards Universal Visual Place Recognition with 3D Visual Geometry Grounded Transformer](https://arxiv.org/abs/2512.21078) (arXiv:2512.21078, 2025)  
- **Demo:** No demo available

## Uses

### Direct Use

This model can be used **out-of-the-box** to extract compact, discriminative global descriptors from:
- Single RGB images (for frame-to-frame VPR)
- Sequences of images (for sequence-to-sequence VPR)

These descriptors are suitable for large-scale localization, robot navigation, and SLAM systems requiring robustness to appearance changes.

### Downstream Use

- Integration into **visual SLAM** or **long-term autonomous navigation** pipelines  
- Replacement for traditional VPR backbones (e.g., NetVLAD, MixVPR, EigenPlaces)  
- Fine-tuning on domain-specific datasets (e.g., underground, aerial, or underwater environments)

### Out-of-Scope Use

- **Not intended** for real-time inference on low-power embedded devices without optimization (latency ~8.23 ms on RTX 4090)  
- **Not designed** for non-visual modalities (e.g., LiDAR, audio, text)  
- Performance may degrade in **extreme occlusion**, **textureless scenes**, or **indoor environments not seen during training**

## Bias, Risks, and Limitations

- Trained primarily on **urban street-level imagery** (GSV-Cities, Mapillary MSLS), so generalization to rural, indoor, or non-Western cities may be limited  
- Inherits biases from training data (e.g., geographic overrepresentation of North America/Europe)  
- No explicit fairness or demographic considerations (as it is a geometric vision model)

### Recommendations

- Evaluate on target domain before deployment  
- Monitor recall performance on your specific dataset using standard VPR metrics (R@1, R@5)

## How to Get Started with the Model

The exact inference script is provided in the GitHub repo (`eval_lora.py`, `main_ft.py`). Pretrained weights are available on Hugging Face or via the repo release.

## Training Details

### Training Data

- **Single-frame model**: Trained on [GSV-Cities](https://github.com/amaralibey/gsv-cities)  
- **Multi-frame model**: Trained on [Mapillary Street-Level Sequences (MSLS)](https://www.mapillary.com/dataset/places)  
- Both datasets contain millions of geo-tagged urban street-view images across diverse cities, seasons, and conditions.

### Training Procedure

#### Preprocessing
- Images resized to 518×518  
- Sequences sampled with spatial proximity for multi-frame training

#### Training Hyperparameters
- **Backbone**: DINOv2 (ViT-large)  
- **Optimization**: AdamW, learning rate scheduling  
- **Loss**: Multi-similarity loss with pair weighting  
- **Training regime**: Mixed-precision (fp16) on NVIDIA GPUs

#### Speeds, Sizes, Times
- **Inference latency**: Single frame - 8.23 ms per image (RTX 4090)  
- **Descriptor dimension**: 17152 (for UniPR-3D)  
- Training time: Not disclosed (multi-day runs on multi-GPU setup)

## Evaluation

### Testing Data, Factors & Metrics

#### Testing Data
- Single frame evaluation:
  - <a href="https://codalab.lisn.upsaclay.fr/competitions/865">MSLS Challenge</a>, where you upload your predictions to their server for evaluation.
  - Single-frame <a href="https://www.mapillary.com/dataset/places">MSLS</a> Validation set
  - Nordland dataset, <a href="https://data.ciirc.cvut.cz/public/projects/2015netVLAD/Pittsburgh250k/">Pittsburgh</a> dataset and SPED dataset, you may download them from <a href="https://surfdrive.surf.nl/index.php/s/sbZRXzYe3l0v67W">here</a>, aligned with DINOv2 SALAD.
- Multi-frame evaluation:
  - Multi-frame <a href="https://www.mapillary.com/dataset/places">MSLS</a> Validation set
  - Two sequence from <a href="https://robotcar-dataset.robots.ox.ac.uk/datasets/">Oxford RobotCar</a>, you may download them <a href="https://entuedu-my.sharepoint.com/personal/heshan001_e_ntu_edu_sg/_layouts/15/onedrive.aspx?id=%2Fpersonal%2Fheshan001%5Fe%5Fntu%5Fedu%5Fsg%2FDocuments%2Fcasevpr%5Fdatasets%2Foxford%5Frobotcar&viewid=e5dcb0e9%2Db23f%2D44cf%2Da843%2D7837d3064c2e&ga=1">here</a>.
    - 2014-12-16-18-44-24 (winter night) query to 2014-11-18-13-20-12 (fall day) db
    - 2014-11-14-16-34-33 (fall night) query to 2015-11-13-10-28-08 (fall day) db
  - <a href="https://github.com/gmberton/VPR-datasets-downloader/blob/main/download_nordland.py">Nordland (filtered) dataset</a>

#### Factors
- Seasonal variation (summer ↔ winter)
- Day vs. night
- Weather (sunny, rainy, snowy)
- Viewpoint change (lateral shift, orientation)

#### Metrics
- **Recall@K (R@1, R@5, R@10)**: Standard metric for VPR – fraction of queries with correct match in top-K retrieved database images

### Results

#### Summary

Our method achieves significantly higher recall than competing approaches, achieving new state-of-the-art performance on both single and multiple frame benchmarks.
##### Single-frame matching results

<style>
  table, th, td {
    border-collapse: collapse;
    text-align: center;
  }
</style>
<table>
  <tr>
    <th colspan="2"></th>
    <th colspan="2">MSLS Challenge</th>
    <th colspan="2">MSLS Val</th>
    <th colspan="2">NordLand</th>
    <th colspan="2">Pitts250k-test</th>
    <th colspan="2">SPED</th>
  </tr>
  <tr>
    <th>Method</th>
    <th>Latency (ms)</th>
    <th>R@1</th>
    <th>R@5</th>
    <th>R@1</th>
    <th>R@5</th>
    <th>R@1</th>
    <th>R@5</th>
    <th>R@1</th>
    <th>R@5</th>
    <th>R@1</th>
    <th>R@5</th>
  </tr>
  <tr>
    <td>MixVPR</td>
    <td>1.37</td>
    <td>64.0</td>
    <td>75.9</td>
    <td>88.0</td>
    <td>92.7</td>
    <td>58.4</td>
    <td>74.6</td>
    <td>94.6</td>
    <td><u>98.3</u></td>
    <td>85.2</td>
    <td>92.1</td>
  </tr>
  <tr>
    <td>EigenPlaces</td>
    <td>2.65</td>
    <td>67.4</td>
    <td>77.1</td>
    <td>89.3</td>
    <td>93.7</td>
    <td>54.4</td>
    <td>68.8</td>
    <td>94.1</td>
    <td>98.0</td>
    <td>69.9</td>
    <td>82.9</td>
  </tr>
  <tr>
    <td>DINOv2 SALAD</td>
    <td>2.41</td>
    <td><u>73.0</u></td>
    <td><u>86.8</u></td>
    <td><u>91.2</u></td>
    <td><u>95.3</u></td>
    <td><u>69.6</u></td>
    <td><u>84.4</u></td>
    <td><u>94.5</u></td>
    <td><b>98.7</b></td>
    <td><u>89.5</u></td>
    <td><u>94.4</u></td>
  </tr>
  <tr>
    <td>UniPR-3D (ours)</td>
    <td>8.23</td>
    <td><b>74.3</b></td>
    <td><b>87.5</b></td>
    <td><b>91.4</b></td>
    <td><b>96.0</b></td>
    <td><b>76.2</b></td>
    <td><b>87.3</b></td>
    <td><b>94.9</b></td>
    <td>98.1</td>
    <td><b>89.6</b></td>
    <td><b>94.5</b></td>
  </tr>
</table>

##### Sequence matching results

<table>
  <tr>
    <th></th>
    <th colspan="3">MSLS Val</th>
    <th colspan="3">NordLand</th>
    <th colspan="3">Oxford1</th>
    <th colspan="3">Oxford2</th>
  </tr>
  <tr>
    <th>Method</th>
    <th>R@1</th>
    <th>R@5</th>
    <th>R@10</th>
    <th>R@1</th>
    <th>R@5</th>
    <th>R@10</th>
    <th>R@1</th>
    <th>R@5</th>
    <th>R@10</th>
    <th>R@1</th>
    <th>R@5</th>
    <th>R@10</th>
  </tr>
  <tr>
    <td>SeqMatchNet</td>
    <td>65.5</td>
    <td>77.5</td>
    <td>80.3</td>
    <td>56.1</td>
    <td>71.4</td>
    <td>76.9</td>
    <td>36.8</td>
    <td>43.3</td>
    <td>48.3</td>
    <td>27.9</td>
    <td>38.5</td>
    <td>45.3</td>
  </tr>
  <tr>
    <td>SeqVLAD</td>
    <td>89.9</td>
    <td>92.4</td>
    <td>94.1</td>
    <td>65.5</td>
    <td>75.2</td>
    <td>80.0</td>
    <td>58.4</td>
    <td>72.8</td>
    <td>80.8</td>
    <td>19.1</td>
    <td>29.9</td>
    <td>37.3</td>
  </tr>
  <tr>
    <td>CaseVPR</td>
    <td><u>91.2</u></td>
    <td><u>94.1</u></td>
    <td><u>95.0</u></td>
    <td><u>84.1</u></td>
    <td><u>89.9</u></td>
    <td><u>92.2</u></td>
    <td><u>90.5</u></td>
    <td><u>95.2</u></td>
    <td><u>96.5</u></td>
    <td><u>72.8</u></td>
    <td><u>85.8</u></td>
    <td><u>89.9</u></td>
  </tr>
  <tr>
    <td>UniPR-3D (ours)</td>
    <td><b>93.7</b></td>
    <td><b>95.7</b></td>
    <td><b>96.9</b></td>
    <td><b>86.8</b></td>
    <td><b>91.7</b></td>
    <td><b>93.8</b></td>
    <td><b>95.4</b></td>
    <td><b>98.1</b></td>
    <td><b>98.7</b></td>
    <td><b>80.6</b></td>
    <td><b>90.3</b></td>
    <td><b>93.9</b></td>
  </tr>
</table>


## Compute Infrastructure

### Hardware
- NVIDIA RTX 4090

### Software
- Python 3.11.10 + CUDA 12.1  
- Based on [SALAD](https://github.com/serizba/salad) and [VGGT](https://github.com/facebookresearch/vggt)

## Citation

**BibTeX:**
```bibtex
@article{deng2025unipr3d,
  title={UniPR-3D: Towards Universal Visual Place Recognition with 3D Visual Geometry Grounded Transformer},
  author={Deng, Tianchen and Chen, Xun and Li, Ziming and Shen, Hongming and Wang, Danwei and Civera, Javier and Wang, Hesheng},
  journal={arXiv preprint arXiv:2512.21078},
  year={2025}
}
```

**APA:**
Deng, T., Chen, X., Li, Z., Shen, H., Wang, D., Civera, J., & Wang, H. (2025). UniPR-3D: Towards Universal Visual Place Recognition with 3D Visual Geometry Grounded Transformer. *arXiv preprint arXiv:2512.21078*.

## Contact

For questions, pretrained model access, or qualitative comparisons, please contact:  
📧 **Tianchen Deng** – [[email protected]](mailto:[email protected])

---

> 📌 **Acknowledgement**: This implementation builds upon [SALAD](https://github.com/serizba/salad) and [VGGT](https://github.com/facebookresearch/vggt). Please cite those works if you use their components.