Create README.md
Browse files
README.md
ADDED
|
@@ -0,0 +1,113 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
---
|
| 2 |
+
license: cc-by-4.0
|
| 3 |
+
tags:
|
| 4 |
+
- earth-observation
|
| 5 |
+
- remote-sensing
|
| 6 |
+
- diffusion
|
| 7 |
+
- generative
|
| 8 |
+
- copernicus
|
| 9 |
+
- sentinel
|
| 10 |
+
- major-tom
|
| 11 |
+
- multimodal
|
| 12 |
+
- latent-diffusion
|
| 13 |
+
library_name: cop-gen
|
| 14 |
+
datasets:
|
| 15 |
+
- Major-TOM/COP-GEN-Benchmark
|
| 16 |
+
---
|
| 17 |
+
|
| 18 |
+

|
| 19 |
+
|
| 20 |
+
# COP-GEN: Latent Diffusion Transformer for Copernicus Earth Observation Data
|
| 21 |
+
|
| 22 |
+
[](https://arxiv.org/abs/2603.03239)
|
| 23 |
+
[](https://github.com/miquel-espinosa/COP-GEN)
|
| 24 |
+
[](https://miquel-espinosa.github.io/cop-gen/)
|
| 25 |
+
[](https://huggingface.co/collections/mespinosami/copgen)
|
| 26 |
+
|
| 27 |
+
COP-GEN is a generative foundation model for Copernicus Earth observation data. It learns a joint distribution over all major Copernicus modalities — Sentinel-1 SAR, Sentinel-2 multispectral (L1C and L2A), DEM, and LULC — enabling both unconditional generation and cross-modal conditional synthesis (e.g. generate S2 RGB from S1 SAR, or generate all modalities jointly).
|
| 28 |
+
|
| 29 |
+
## Model Details
|
| 30 |
+
|
| 31 |
+
- **Developed by:** Miguel Espinosa, Eva Gmelich Meijling, Valerio Marsocci, Elliot J. Crowley, Mikolaj Czerkawski
|
| 32 |
+
- **Model type:** Latent Diffusion Transformer (multimodal, multi-resolution)
|
| 33 |
+
- **Modalities:** S1RTC (VV, VH), S2L1C (all bands + cloud mask), S2L2A (all bands), DEM, LULC, timestamps, lat-lon
|
| 34 |
+
- **License:** CC-BY-4.0
|
| 35 |
+
- **Paper:** [arXiv:2603.03239](https://arxiv.org/abs/2603.03239)
|
| 36 |
+
- **Repository:** [github.com/miquel-espinosa/COP-GEN](https://github.com/miquel-espinosa/COP-GEN)
|
| 37 |
+
|
| 38 |
+
### Architecture
|
| 39 |
+
|
| 40 |
+
COP-GEN operates in a shared latent space produced by a set of modality-specific KL-regularised VAEs. The diffusion backbone is a transformer trained jointly over all modalities, supporting arbitrary conditioning at inference time — any subset of modalities can be held as conditions while the rest are generated.
|
| 41 |
+
|
| 42 |
+
## Uses
|
| 43 |
+
|
| 44 |
+
### Direct Use
|
| 45 |
+
|
| 46 |
+
Generate synthetic Copernicus EO scenes, either unconditionally or conditioned on one or more input modalities. Useful for data augmentation, gap-filling missing modalities, and studying cross-sensor relationships.
|
| 47 |
+
|
| 48 |
+
### Downstream Use
|
| 49 |
+
|
| 50 |
+
The latent representations and generated samples can serve as inputs to downstream EO tasks: land cover classification, change detection, cloud removal, SAR-to-optical translation, and more.
|
| 51 |
+
|
| 52 |
+
## How to Get Started
|
| 53 |
+
|
| 54 |
+
```python
|
| 55 |
+
from libs.copgen import CopgenModel
|
| 56 |
+
|
| 57 |
+
model = CopgenModel(
|
| 58 |
+
model_path="path/to/model_checkpoint.pth",
|
| 59 |
+
config_path="path/to/model_config.py"
|
| 60 |
+
)
|
| 61 |
+
|
| 62 |
+
# Conditional generation: provide one or more modalities as conditions
|
| 63 |
+
samples = model.generate(
|
| 64 |
+
modalities=["S2L2A_B02_B03_B04_B08", "S1RTC_vh_vv"],
|
| 65 |
+
conditions={"S1RTC_vh_vv": s1_tensor},
|
| 66 |
+
n_samples=4,
|
| 67 |
+
)
|
| 68 |
+
|
| 69 |
+
# Unconditional generation
|
| 70 |
+
samples = model.generate(
|
| 71 |
+
modalities=["S2L2A_B02_B03_B04_B08", "S1RTC_vh_vv"],
|
| 72 |
+
n_samples=4,
|
| 73 |
+
)
|
| 74 |
+
```
|
| 75 |
+
|
| 76 |
+
See [examples/conditional_generation.py](https://github.com/miquel-espinosa/COP-GEN/blob/main/examples/conditional_generation.py) and [examples/unconditional_generation.py](https://github.com/miquel-espinosa/COP-GEN/blob/main/examples/unconditional_generation.py) for full worked examples.
|
| 77 |
+
|
| 78 |
+
## Training Details
|
| 79 |
+
|
| 80 |
+
### Training Data
|
| 81 |
+
|
| 82 |
+
Trained on [Major-TOM](https://huggingface.co/Major-TOM) global Copernicus data, covering Sentinel-1 RTC, Sentinel-2 L1C and L2A, DEM, and LULC. A pre-compiled Edinburgh subset is available at [mespinosami/copgen-edinburgh-subset](https://huggingface.co/datasets/mespinosami/copgen-edinburgh-subset) for local development and reproduction.
|
| 83 |
+
|
| 84 |
+
### Training Procedure
|
| 85 |
+
|
| 86 |
+
1. Modality-specific KL-VAEs are trained separately per modality and resolution.
|
| 87 |
+
2. All modalities are encoded into a shared latent space.
|
| 88 |
+
3. A diffusion transformer backbone is trained jointly over the merged latents, with random masking of modalities to enable conditional generation at inference.
|
| 89 |
+
|
| 90 |
+
See the [GitHub README](https://github.com/miquel-espinosa/COP-GEN) for full training instructions.
|
| 91 |
+
|
| 92 |
+
## Evaluation
|
| 93 |
+
|
| 94 |
+
Evaluated on the [COP-GEN-Benchmark](https://huggingface.co/datasets/Major-TOM/COP-GEN-Benchmark) test set (495 held-out global scenes). Distribution-level metrics (FID and related) are reported in Table 1 of the paper. To reproduce:
|
| 95 |
+
|
| 96 |
+
```bash
|
| 97 |
+
pip install -r benchmark/stochastic/requirements.txt
|
| 98 |
+
python -m benchmark.stochastic.run --output metrics.csv
|
| 99 |
+
```
|
| 100 |
+
|
| 101 |
+
## Citation
|
| 102 |
+
|
| 103 |
+
```bibtex
|
| 104 |
+
@article{copgen2026,
|
| 105 |
+
title = {COP-GEN: Latent Diffusion Transformer for Copernicus Earth
|
| 106 |
+
Observation Data},
|
| 107 |
+
author = {Espinosa, Miguel and Gmelich Meijling, Eva and Marsocci,
|
| 108 |
+
Valerio and Crowley, Elliot J. and Czerkawski, Mikolaj},
|
| 109 |
+
year = {2026},
|
| 110 |
+
journal = {arXiv preprint arXiv:2603.03239},
|
| 111 |
+
url = {https://arxiv.org/abs/2603.03239},
|
| 112 |
+
}
|
| 113 |
+
```
|