madrisight/madrimed1.2-vision-embed

Medical Vision Encoder - 2048-D

Madrimed 1.2 Vision Embed

2048-D

Model Overview

This vision encoder is an isolated visual representation tower for dense feature extraction from multi-modality radiological scans other medical images. It projects dynamic-resolution medical imagery into an L2-normalized 2048-dimensional embedding space, optimized for cross-modal alignment and topological Graph Retrieval-Augmented Generation.

Key Architectural Specifications

• Model Type: Vision Transformer (ViT) with patch-merging projection
• Feature Dimension: 2048 (matches transformer language model input dimensions)
• Spatial Patch Merger: $2 \times 2$ patch compression via linear projection
• Primary Feature Structure: Modality-dominant feature geometry (groups scans primarily by imaging physics and tissue contrast)

---

Intended Uses & Scope

Primary Intended Uses

• Visual Indexing for Clinical Graph RAG: Building knowledge graph topologies based on mutual $k$-nearest neighbors ($k$-NN) and cosine similarity.
• Radiological Case Retrieval: Identifying visually and structurally similar clinical scans across patient cohorts.
• Content-Based Image Retrieval (CBIR): Case-based reasoning support in hospital Picture Archiving and Communication Systems (PACS).

Out-of-Scope & Clinical Usage Constraints

• Primary Diagnostic Tool: This model produces vector representations for retrieval and clustering. It is not intended to provide standalone diagnostic determinations or replace certified radiologist review.
• Uncalibrated Similarity Thresholding: Due to visual anisotropy, embeddings should be mean-centered or hubness-corrected prior to static cosine thresholding to prevent high-degree hub formation.

---

Quickstart & Usage

Installation

git clone  https://github.com/krrish-v/metalliger

Example Usage

import torch
from transformers import AutoProcessor, AutoTokenizer, Qwen3VLVisionModel, Qwen3VLTextModel

# Import classes directly from your new module
from metalliger.util.vison import func, GraphRAG

VISION_MODEL_DIR = "madrisight/madrimed1.2-vision-embed"
TEXT_MODEL_DIR = "madrisight/madrimed1.2-text-embed"

device = "cuda" if torch.cuda.is_available() else ("mps" if torch.backends.mps.is_available() else "cpu")
print(f"Loading models on: {device}")

# 1. Load Processors and Model Backbones
processor = AutoProcessor.from_pretrained(VISION_MODEL_DIR, trust_remote_code=True)
vision_model = Qwen3VLVisionModel.from_pretrained(
    VISION_MODEL_DIR,
    torch_dtype=torch.float32,
    trust_remote_code=True,
).to(device).eval()

tokenizer = AutoTokenizer.from_pretrained(TEXT_MODEL_DIR, trust_remote_code=True)
text_model = Qwen3VLTextModel.from_pretrained(
    TEXT_MODEL_DIR,
    torch_dtype=torch.float32,
    trust_remote_code=True,
).to(device).eval()

# 2. Ingest Images and Build the Clinical Graph
vision_helper = func(device=device)

# Provide file paths or PIL.Image objects
image_records = ["scan_01.png","scan_02.png","scan_03.png",]
metadata_labels = ["CT-Chest_lung", "CT-Chest_lung", "MRI-Brain_Tissue"]

G, image_embeddings, latencies = vision_helper.similarity_graph(
    images=image_records,
    image_processor=processor,
    vision_model=vision_model,
    texts=metadata_labels,
    similarity_threshold=0.85,
    batch_size=16,
    mutual_knn=True,            # Prevents hub nodes from dominating
    calibrate_hubness=True,     # Mean-centering + CSLS
    store_raw_pil_in_nodes=False # Keeps graph lightweight in RAM
)

print(f"Graph constructed: {G.number_of_nodes()} nodes, {G.number_of_edges()} edges.")
print(f"Average encoding latency: {sum(latencies)/len(latencies):.2f} ms/image")

# 3. Query the Graph RAG Service
rag_service = GraphRAG(device=device)

query_text = "axial chest CT scan showing pulmonary nodules"

# Available methods:
# - 'balanced_vision_boundary' (Ultra-fast, skips LLM layers)
# - 'fast_layer0_static' (Default balanced blend)
# - 'hybrid_blended_context' (80% Layer 0, 20% Layer 1)
seed_nodes, retrieved_subgraph, smoothed_sims = rag_service.query_graph_rag_pure(
    text_query=query_text,
    G=G,
    text_model=text_model,
    tokenizer=tokenizer,
    img_embeddings=image_embeddings,
    method="balanced_vision_boundary",
    top_k_seeds=1,
    k_neighbors=5,
    smoothing_lambda=0.40,
    verbose=True,
)

print("\nFinal Retrieval Summary:")
print(f"Entry Seed Node ID: {seed_nodes[0]}")
print(f"Retrieved Context Subgraph IDs: {retrieved_subgraph}")

Evaluation & Empirical Benchmarks

The visual representation quality was benchmarked using stratified multi-organ medical cohorts:

Medical Image Graph RAG

Vision Space Diagnostic

---

Graph Construction Guidelines

When deploying this model inside a Graph RAG or retrieval pipeline:

• Hubness Mitigation: Apply mean-centering ($\mathbf{e}_{\text{cal}} = \frac{\mathbf{e} - \boldsymbol{\mu}}{|\mathbf{e} - \boldsymbol{\mu}|_2}$) prior to distance calculation to mitigate the narrow-cone effect.
• Mutual $k$-NN Topology: Prefer mutual $k$-nearest neighbors ($k \in [6, 12]$) over flat static thresholding ($\tau = 0.85$) to prevent edge oversaturation across dominant modalities.
• Auditability: Generate deterministic SHA-256 digests over resized scan bytes to support PACS compliance and record provenance.

Citation

@software{madrimed1.2-vision-embed,
  title = {MadriMed 1.2: Advanced Multimodal Medical Reasoning at the 2B Scale},
  author = {Madrisight},
  year = {2026},
  url = {https://huggingface.co/madrisight/madrimed1.2-VL-2B}
}