---
license: mit
datasets:
- Somayeh-h/Nordland
- OPR-Project/OxfordRobotCar_OpenPlaceRecognition
language:
- en
metrics:
- recall_at_1
- recall_at_5
pipeline_tag: image-feature-extraction
tags:
- place-recognition
- visual-place-recognition
- computer-vision
- transformer
- 3d-vision
library:
- pytorch
- lightning
---
# Model Card for UniPR-3D
UniPR-3D is a universal visual place recognition (VPR) framework that supports both **single-frame** and **sequence-to-sequence** matching. It leverages **3D visual geometry grounded tokens** within a transformer architecture to produce robust, viewpoint-invariant descriptors for long-term place recognition under challenging environmental variations (e.g., seasonal, weather, lighting, and viewpoint changes).
## Model Details
### Model Description
- **Developed by:** Tianchen Deng, Xun Chen, Ziming Li, Hongming Shen, Danwei Wang, Javier Civera, Hesheng Wang
- **Shared by:** Tianchen Deng
- **Model type:** Vision Transformer with 3D-aware token aggregation for visual place recognition
- **Language(s):** English (dataset metadata); model is vision-only
- **License:** MIT
### Model Sources
- **Repository:** [repo](https://github.com/dtc111111/UniPR-3D)
- **Paper:** [UniPR-3D: Towards Universal Visual Place Recognition with 3D Visual Geometry Grounded Transformer](https://arxiv.org/abs/2512.21078) (arXiv:2512.21078, 2025)
- **Demo:** No demo available
## Uses
### Direct Use
This model can be used **out-of-the-box** to extract compact, discriminative global descriptors from:
- Single RGB images (for frame-to-frame VPR)
- Sequences of images (for sequence-to-sequence VPR)
These descriptors are suitable for large-scale localization, robot navigation, and SLAM systems requiring robustness to appearance changes.
### Downstream Use
- Integration into **visual SLAM** or **long-term autonomous navigation** pipelines
- Replacement for traditional VPR backbones (e.g., NetVLAD, MixVPR, EigenPlaces)
- Fine-tuning on domain-specific datasets (e.g., underground, aerial, or underwater environments)
### Out-of-Scope Use
- **Not intended** for real-time inference on low-power embedded devices without optimization (latency ~8.23 ms on RTX 4090)
- **Not designed** for non-visual modalities (e.g., LiDAR, audio, text)
- Performance may degrade in **extreme occlusion**, **textureless scenes**, or **indoor environments not seen during training**
## Bias, Risks, and Limitations
- Trained primarily on **urban street-level imagery** (GSV-Cities, Mapillary MSLS), so generalization to rural, indoor, or non-Western cities may be limited
- Inherits biases from training data (e.g., geographic overrepresentation of North America/Europe)
- No explicit fairness or demographic considerations (as it is a geometric vision model)
### Recommendations
- Evaluate on target domain before deployment
- Monitor recall performance on your specific dataset using standard VPR metrics (R@1, R@5)
## How to Get Started with the Model
The exact inference script is provided in the GitHub repo (`eval_lora.py`, `main_ft.py`). Pretrained weights are available on Hugging Face or via the repo release.
## Training Details
### Training Data
- **Single-frame model**: Trained on [GSV-Cities](https://github.com/amaralibey/gsv-cities)
- **Multi-frame model**: Trained on [Mapillary Street-Level Sequences (MSLS)](https://www.mapillary.com/dataset/places)
- Both datasets contain millions of geo-tagged urban street-view images across diverse cities, seasons, and conditions.
### Training Procedure
#### Preprocessing
- Images resized to 518×518
- Sequences sampled with spatial proximity for multi-frame training
#### Training Hyperparameters
- **Backbone**: DINOv2 (ViT-large)
- **Optimization**: AdamW, learning rate scheduling
- **Loss**: Multi-similarity loss with pair weighting
- **Training regime**: Mixed-precision (fp16) on NVIDIA GPUs
#### Speeds, Sizes, Times
- **Inference latency**: Single frame - 8.23 ms per image (RTX 4090)
- **Descriptor dimension**: 17152 (for UniPR-3D)
- Training time: Not disclosed (multi-day runs on multi-GPU setup)
## Evaluation
### Testing Data, Factors & Metrics
#### Testing Data
- Single frame evaluation:
- MSLS Challenge, where you upload your predictions to their server for evaluation.
- Single-frame MSLS Validation set
- Nordland dataset, Pittsburgh dataset and SPED dataset, you may download them from here, aligned with DINOv2 SALAD.
- Multi-frame evaluation:
- Multi-frame MSLS Validation set
- Two sequence from Oxford RobotCar, you may download them here.
- 2014-12-16-18-44-24 (winter night) query to 2014-11-18-13-20-12 (fall day) db
- 2014-11-14-16-34-33 (fall night) query to 2015-11-13-10-28-08 (fall day) db
- Nordland (filtered) dataset
#### Factors
- Seasonal variation (summer ↔ winter)
- Day vs. night
- Weather (sunny, rainy, snowy)
- Viewpoint change (lateral shift, orientation)
#### Metrics
- **Recall@K (R@1, R@5, R@10)**: Standard metric for VPR – fraction of queries with correct match in top-K retrieved database images
### Results
#### Summary
Our method achieves significantly higher recall than competing approaches, achieving new state-of-the-art performance on both single and multiple frame benchmarks.
##### Single-frame matching results
|
MSLS Challenge |
MSLS Val |
NordLand |
Pitts250k-test |
SPED |
| Method |
Latency (ms) |
R@1 |
R@5 |
R@1 |
R@5 |
R@1 |
R@5 |
R@1 |
R@5 |
R@1 |
R@5 |
| MixVPR |
1.37 |
64.0 |
75.9 |
88.0 |
92.7 |
58.4 |
74.6 |
94.6 |
98.3 |
85.2 |
92.1 |
| EigenPlaces |
2.65 |
67.4 |
77.1 |
89.3 |
93.7 |
54.4 |
68.8 |
94.1 |
98.0 |
69.9 |
82.9 |
| DINOv2 SALAD |
2.41 |
73.0 |
86.8 |
91.2 |
95.3 |
69.6 |
84.4 |
94.5 |
98.7 |
89.5 |
94.4 |
| UniPR-3D (ours) |
8.23 |
74.3 |
87.5 |
91.4 |
96.0 |
76.2 |
87.3 |
94.9 |
98.1 |
89.6 |
94.5 |
##### Sequence matching results
|
MSLS Val |
NordLand |
Oxford1 |
Oxford2 |
| Method |
R@1 |
R@5 |
R@10 |
R@1 |
R@5 |
R@10 |
R@1 |
R@5 |
R@10 |
R@1 |
R@5 |
R@10 |
| SeqMatchNet |
65.5 |
77.5 |
80.3 |
56.1 |
71.4 |
76.9 |
36.8 |
43.3 |
48.3 |
27.9 |
38.5 |
45.3 |
| SeqVLAD |
89.9 |
92.4 |
94.1 |
65.5 |
75.2 |
80.0 |
58.4 |
72.8 |
80.8 |
19.1 |
29.9 |
37.3 |
| CaseVPR |
91.2 |
94.1 |
95.0 |
84.1 |
89.9 |
92.2 |
90.5 |
95.2 |
96.5 |
72.8 |
85.8 |
89.9 |
| UniPR-3D (ours) |
93.7 |
95.7 |
96.9 |
86.8 |
91.7 |
93.8 |
95.4 |
98.1 |
98.7 |
80.6 |
90.3 |
93.9 |
## Compute Infrastructure
### Hardware
- NVIDIA RTX 4090
### Software
- Python 3.11.10 + CUDA 12.1
- Based on [SALAD](https://github.com/serizba/salad) and [VGGT](https://github.com/facebookresearch/vggt)
## Citation
**BibTeX:**
```bibtex
@article{deng2025unipr3d,
title={UniPR-3D: Towards Universal Visual Place Recognition with 3D Visual Geometry Grounded Transformer},
author={Deng, Tianchen and Chen, Xun and Li, Ziming and Shen, Hongming and Wang, Danwei and Civera, Javier and Wang, Hesheng},
journal={arXiv preprint arXiv:2512.21078},
year={2025}
}
```
**APA:**
Deng, T., Chen, X., Li, Z., Shen, H., Wang, D., Civera, J., & Wang, H. (2025). UniPR-3D: Towards Universal Visual Place Recognition with 3D Visual Geometry Grounded Transformer. *arXiv preprint arXiv:2512.21078*.
## Contact
For questions, pretrained model access, or qualitative comparisons, please contact:
📧 **Tianchen Deng** – [dengtianchen@sjtu.edu.cn](mailto:dengtianchen@sjtu.edu.cn)
---
> 📌 **Acknowledgement**: This implementation builds upon [SALAD](https://github.com/serizba/salad) and [VGGT](https://github.com/facebookresearch/vggt). Please cite those works if you use their components.