initial model card update

set for 50 epoch change, eval needs to be updated

README.md

@@ -5,7 +5,7 @@ language:
 - en
 library_name: open_clip
 model_name: BioCLIP
-model_description: "Foundation model for the tree of life, built using CLIP architecture as a vision model for general organismal biology. It is trained on TreeOfLife-10M, our specially-created dataset covering over 450K taxa--the most biologically diverse ML-ready dataset available to date."
+model_description: "Foundation model for the tree of life, built using CLIP architecture (ViT-B/16) as a vision model for general organismal biology. It is trained on TreeOfLife-10M, our specially-created dataset covering over 390K taxa--the most biologically diverse ML-ready dataset available at release."
 tags:
 - zero-shot-image-classification
 - clip
@@ -13,6 +13,8 @@ tags:
 - CV
 - images
 - animals
+- plants
+- fungi
 - species
 - taxonomy
 - rare species
@@ -30,11 +32,14 @@ datasets:
 
 # Model Card for BioCLIP
 
+If you are looking for the original BioCLIP model presented in the paper, please see [Revision 7b4abf1](https://huggingface.co/imageomics/bioclip/tree/7b4abf1f6ee747c15de00c7d28a5e62990b5dabc) (the most accurate documentation for the original version at BioCLIP [Revision ce901ab](https://huggingface.co/imageomics/bioclip/tree/ce901ab3c6a913f9e9ef94ce6d27761069f4f01c)), trained on TreeOfLife-10M [Revision ffa2a31](https://huggingface.co/datasets/imageomics/TreeOfLife-10M/tree/ffa2a318a1396f2f9e456ba171d3b5b5d8b4f051).
+
+
 <!-- 
 This modelcard has been generated using [this raw template](https://github.com/huggingface/huggingface_hub/blob/main/src/huggingface_hub/templates/modelcard_template.md?plain=1). And further altered to suit Imageomics Institute needs -->
 
 BioCLIP is a foundation model for the tree of life, built using CLIP architecture as a vision model for general organismal biology. 
-It is trained on [TreeOfLife-10M](https://huggingface.co/datasets/imageomics/TreeOfLife-10M), our specially-created dataset covering over 450K taxa--the most biologically diverse ML-ready dataset available to date. 
+It is trained on [TreeOfLife-10M](https://huggingface.co/datasets/imageomics/TreeOfLife-10M), our specially-created dataset covering over 390K taxa--the most biologically diverse ML-ready dataset available at its release. 
 Through rigorous benchmarking on a diverse set of fine-grained biological classification tasks, BioCLIP consistently outperformed existing baselines by 16% to 17% absolute. 
 Through intrinsic evaluation, we found that BioCLIP learned a hierarchical representation aligned to the tree of life, which demonstrates its potential for robust generalizability.
 
@@ -47,7 +52,7 @@ Through intrinsic evaluation, we found that BioCLIP learned a hierarchical repre
 BioCLIP is based on OpenAI's [CLIP](https://openai.com/research/clip). 
 We trained the model on [TreeOfLife-10M](https://huggingface.co/datasets/imageomics/TreeOfLife-10M) from OpenAI's ViT-B/16 checkpoint, using [OpenCLIP's](https://github.com/mlfoundations/open_clip) code.
 BioCLIP is trained with the standard CLIP objective to imbue the model with an understanding, not just of different species, but of the hierarchical structure that relates species across the tree of life. 
-In this way, BioCLIP offers potential to aid biologists in discovery of new and related creatures, since it does not see the 454K different taxa as distinct classes, but as part of an interconnected hierarchy. 
+In this way, BioCLIP offers potential to aid biologists in discovery of new and related creatures, since it does not see the 394K different taxa as distinct classes, but as part of an interconnected hierarchy. 
 
 
 - **Developed by:** Samuel Stevens, Jiaman Wu, Matthew J. Thompson, Elizabeth G. Campolongo, Chan Hee Song, David Edward Carlyn, Li Dong, Wasila M. Dahdul, Charles Stewart, Tanya Berger-Wolf, Wei-Lun Chao, and Yu Su
@@ -60,8 +65,9 @@ This model was developed for the benefit of the community as an open-source prod
 ### Model Sources
 
 - **Repository:** [BioCLIP](https://github.com/Imageomics/BioCLIP)
-- **Paper:** BioCLIP: A Vision Foundation Model for the Tree of Life ([arXiv](https://doi.org/10.48550/arXiv.2311.18803))
+- **Paper:** [BioCLIP: A Vision Foundation Model for the Tree of Life](https://openaccess.thecvf.com/content/CVPR2024/papers/Stevens_BioCLIP_A_Vision_Foundation_Model_for_the_Tree_of_Life_CVPR_2024_paper.pdf)
 - **Demo:** [BioCLIP Demo](https://huggingface.co/spaces/imageomics/bioclip-demo)
+<!-- [arXiv](https://doi.org/10.48550/arXiv.2311.18803) -->
 
 ## Uses
 
@@ -72,7 +78,7 @@ The ViT-B/16 vision encoder is recommended as a base model for any computer visi
 ### Direct Use
 
 See the demo [here](https://huggingface.co/spaces/imageomics/bioclip-demo) for examples of zero-shot classification.
-It can also be used in a few-shot setting with a KNN; please see [our paper](https://doi.org/10.48550/arXiv.2311.18803) for details for both few-shot and zero-shot settings without fine-tuning.
+It can also be used in a few-shot setting with a KNN; please see [our paper](https://openaccess.thecvf.com/content/CVPR2024/papers/Stevens_BioCLIP_A_Vision_Foundation_Model_for_the_Tree_of_Life_CVPR_2024_paper.pdf) for details for both few-shot and zero-shot settings without fine-tuning.
 
 
 ## Bias, Risks, and Limitations
@@ -102,9 +108,9 @@ tokenizer = open_clip.get_tokenizer('hf-hub:imageomics/bioclip')
 
 ### Compute Infrastructure
 
-Training was performed on 8 NVIDIA A100-80GB GPUs distributed over 2 nodes on [OSC's](https://www.osc.edu/) Ascend HPC Cluster with global batch size 32,768 for 4 days.
+Training was performed on 8 NVIDIA H100-100GB GPUs distributed over 2 nodes on [OSC's](https://www.osc.edu/) Cardinal HPC Cluster with global batch size 32,768 for 23 hours.
 
-Based on [Machine Learning Impact calculator](https://mlco2.github.io/impact#compute) presented in [Lacoste et al. (2019)](https://doi.org/10.48550/arXiv.1910.09700), that's 132.71 kg of CO<sub>2</sub> eq., or 536km driven by an average ICE car.
+Based on [Machine Learning Impact calculator](https://mlco2.github.io/impact#compute) presented in [Lacoste et al. (2019)](https://doi.org/10.48550/arXiv.1910.09700), that's 3.97 kg eq. CO<sub>2</sub> eq., or 16km driven by an average ICE car.
 
 ### Training Data
 
@@ -115,7 +121,7 @@ This model was trained on [TreeOfLife-10M](https://huggingface.co/datasets/image
 - **Training regime:** fp16 mixed precision.
 
 We resize images to 224 x 224 pixels.
-We use a maximum learning rate of 1e4 with 1000 linear warm-up steps, then use cosine decay to 0 over 100 epochs.
+We use a maximum learning rate of 1e4 with 500 linear warm-up steps, then use cosine decay to 0 over 50 epochs.
 We also use a weight decay of 0.2 and a batch size of 32K.
 
 ## Evaluation
@@ -127,7 +133,7 @@ We tested BioCLIP on the following collection of 10 biologically-relevant tasks.
  - [Birds 525](https://www.kaggle.com/datasets/gpiosenka/100-bird-species): We evaluated on the 2,625 test images provided with the dataset.
  - [Rare Species](https://huggingface.co/datasets/imageomics/rare-species): A new dataset we curated for the purpose of testing this model and to contribute to the ML for Conservation community. It consists of 400 species labeled Near Threatened through Extinct in the Wild by the [IUCN Red List](https://www.iucnredlist.org/), with 30 images per species. For more information, see our dataset, [Rare Species](https://huggingface.co/datasets/imageomics/rare-species).
 
-For more information about the contents of these datasets, see Table 2 and associated sections of [our paper](https://doi.org/10.48550/arXiv.2311.18803).
+For more information about the contents of these datasets, see Table 2 and associated sections of [our paper](https://openaccess.thecvf.com/content/CVPR2024/papers/Stevens_BioCLIP_A_Vision_Foundation_Model_for_the_Tree_of_Life_CVPR_2024_paper.pdf).
 
 ### Metrics
 
@@ -137,7 +143,7 @@ We use top-1 and top-5 accuracy to evaluate models, and validation loss to choos
 
 We compare BioCLIP to OpenAI's CLIP and OpenCLIP's LAION-2B checkpoint.
 Here are the zero-shot classification results on our benchmark tasks. 
-Please see [our paper](https://doi.org/10.48550/arXiv.2311.18803) for few-shot results.
+Please see [our paper](https://openaccess.thecvf.com/content/CVPR2024/papers/Stevens_BioCLIP_A_Vision_Foundation_Model_for_the_Tree_of_Life_CVPR_2024_paper.pdf) for few-shot results.
 
 <table cellpadding="0" cellspacing="0">
   <thead>
@@ -227,7 +233,7 @@ BioCLIP outperforms general-domain baselines by 17% on average for zero-shot.
 
 ### Model Examination
 
-We encourage readers to see Section 4.6 of [our paper](https://doi.org/10.48550/arXiv.2311.18803).
+We encourage readers to see Section 4.6 of [our paper](https://openaccess.thecvf.com/content/CVPR2024/papers/Stevens_BioCLIP_A_Vision_Foundation_Model_for_the_Tree_of_Life_CVPR_2024_paper.pdf).
 In short, BioCLIP forms representations that more closely align to the taxonomic hierarchy compared to general-domain baselines like CLIP or OpenCLIP.
 
 
@@ -238,14 +244,16 @@ In short, BioCLIP forms representations that more closely align to the taxonomic
 ```
 @software{bioclip2023,
   author = {Samuel Stevens and Jiaman Wu and Matthew J. Thompson and Elizabeth G. Campolongo and Chan Hee Song and David Edward Carlyn and Li Dong and Wasila M. Dahdul and Charles Stewart and Tanya Berger-Wolf and Wei-Lun Chao and Yu Su},
-  doi = {10.57967/hf/1511},
-  month = nov,
+  doi = {<update-on-generation>},
+  month = feb,
   title = {BioCLIP},
-  version = {v0.1},
-  year = {2023}
+  year = {2026}
 }
 ```
 
+Note that this version is updated from the original BioCLIP model presented in the paper, please see [Revision 7b4abf1](https://huggingface.co/imageomics/bioclip/tree/7b4abf1f6ee747c15de00c7d28a5e62990b5dabc) (the most accurate documentation for the original version at BioCLIP [Revision ce901ab](https://huggingface.co/imageomics/bioclip/tree/ce901ab3c6a913f9e9ef94ce6d27761069f4f01c)), trained on TreeOfLife-10M [Revision ffa2a31](https://huggingface.co/datasets/imageomics/TreeOfLife-10M/tree/ffa2a318a1396f2f9e456ba171d3b5b5d8b4f051). This updated version of the model uses an updated TreeOfLife-10M which resolves taxonomic alignment issues discovered in the first version. The taxonomic resolution was completed using [TaxonoPy](https://github.com/Imageomics/TaxonoPy), which was developed for [TreeOfLife-200M](https://huggingface.co/datasets/imageomics/TreeOfLife-200M).
+
+
 Please also cite our paper:
 
 ```
@@ -288,7 +296,9 @@ Please also consider citing OpenCLIP, iNat21 and BIOSCAN-1M:
 
 ## Acknowledgements
 
-The authors would like to thank Josef Uyeda, Jim Balhoff, Dan Rubenstein, Hank Bart, Hilmar Lapp, Sara Beery, and colleagues from the Imageomics Institute and the OSU NLP group for their valuable feedback. We also thank the BIOSCAN-1M team and the iNaturalist team for making their data available and easy to use, and Jennifer Hammack at EOL for her invaluable help in accessing EOL’s images.
+The authors would like to thank Josef Uyeda, Jim Balhoff, Dan Rubenstein, Hank Bart, Hilmar Lapp, Sara Beery, and colleagues from the Imageomics Institute and the OSU NLP group for their valuable feedback. We also thank the BIOSCAN-1M team and the iNaturalist team for making their data available and easy to use, and Jennifer Hammock at EOL for her invaluable help in accessing EOL’s images.
+
+Additionally, we thank Ziheng Zhang and Jianyang Gu for running the training and evaluation of the revised model (with the taxonomic fixes) while they were training [BioCAP](https://huggingface.co/imageomics/biocap).
 
 The [Imageomics Institute](https://imageomics.org) is funded by the US National Science Foundation's Harnessing the Data Revolution (HDR) program under [Award #2118240](https://www.nsf.gov/awardsearch/showAward?AWD_ID=2118240) (Imageomics: A New Frontier of Biological Information Powered by Knowledge-Guided Machine Learning). Any opinions, findings and conclusions or recommendations expressed in this material are those of the author(s) and do not necessarily reflect the views of the National Science Foundation.