PROJECT_CONTEXT.md

# GPT4All Service - Project Context

## Project Overview
This is a **Polish Car Description Enhancement Service** built as a FastAPI microservice that uses a Hugging Face Large Language Model to generate enhanced marketing descriptions for cars in Polish language.

## Core Functionality
The service takes basic car information (make, model, year, mileage, features, condition) and generates compelling, marketing-friendly descriptions in Polish using the `speakleash/Bielik-1.5B-v3.0-Instruct` model - a Polish language model from the Bielik series.

## Project Structure

```
gpt4all-service/
├── app/
│   ├── main.py                    # FastAPI application with endpoints
│   ├── models/
│   │   └── huggingface_service.py # Core LLM service wrapper
│   └── schemas/
│       └── schemas.py             # Pydantic data models
├── Dockerfile                     # Multi-stage Docker build
├── download_model.py             # Model download script for Docker
├── requirements.txt              # Python dependencies
├── start_container.ps1           # PowerShell startup script
├── start_container.sh            # Bash startup script
└── README.md                     # Comprehensive documentation
```

## Technical Architecture

### 1. FastAPI Application (`app/main.py`)
- **Framework**: FastAPI with CORS middleware
- **Main Endpoint**: `POST /enhance-description` - takes car data, returns enhanced description
- **Health Check**: `GET /health` - service status and model initialization check
- **CORS**: Configured for frontend on `http://localhost:5173` (likely React/Vue dev server)

### 2. LLM Service (`app/models/huggingface_service.py`)
- **Purpose**: Wrapper around Hugging Face Transformers pipeline
- **Model**: `speakleash/Bielik-1.5B-v3.0-Instruct` (Polish language model)
- **Features**:
  - Async initialization and text generation
  - Support for both GPU (CUDA) and CPU inference
  - Chat template support for conversation-style prompts
  - Configurable generation parameters (temperature, top_p, max_tokens)
  - Smart response parsing to extract only the assistant's response

### 3. Data Models (`app/schemas/schemas.py`)
- **CarData**: Input model with make, model, year, mileage, features[], condition
- **EnhancedDescriptionResponse**: Output model with generated description

### 4. Containerization
- **Docker**: Self-contained image with pre-downloaded model (~3.2GB)
- **Security**: Uses Docker BuildKit secrets for Hugging Face token handling
- **Model Storage**: Downloaded to `/app/pretrain_model` during build
- **Runtime**: Python 3.9-slim base image

## Key Technical Details

### Model Configuration
- **Model Path**: `/app/pretrain_model` (in container) or configurable for local dev
- **Device**: Currently set to CPU in main.py, but service supports GPU
- **Generation Params**: 150 max tokens, temperature 0.75, top_p 0.9

### Prompt Engineering
The service uses a carefully crafted Polish system prompt:
- Instructs the model to create marketing descriptions in Polish
- Limits output to 500 characters maximum
- Tells the model to ignore off-topic content
- Uses chat template format with system/user roles

### Dependencies
- **fastapi**: Web framework
- **uvicorn[standard]**: ASGI server
- **transformers[torch]**: Hugging Face transformers with PyTorch
- **accelerate**: Hugging Face optimization library

## Current State & Issues

### Git Status
- Modified `app/main.py` (likely recent changes)
- Deleted `app/models/gpt4all.py` (indicates migration from GPT4All to Hugging Face)

### Linter Issues in `huggingface_service.py`
1. Import issues: `pipeline` and `AutoTokenizer` imports need specific paths
2. Type annotations: `device: str = None` should be `Optional[str] = None`
3. Method parameters: Similar optional parameter typing issues

## Usage Scenarios
1. **Car Dealership Websites**: Auto-generate compelling descriptions from basic car specs
2. **Marketplace Applications**: Enhance user-provided car listings
3. **Inventory Management**: Bulk description generation for car databases

## Deployment Options
1. **Local Development**: Direct Python/uvicorn execution
2. **Docker Container**: Self-contained deployment with pre-downloaded model
3. **Production**: Containerized deployment with proper authentication

## Authentication Requirements
- Hugging Face Hub token required for model download (gated model)
- Token stored in `my_hf_token.txt` during Docker build
- Securely handled via Docker BuildKit secrets

## Performance Considerations
- Model size: ~3.2GB (significant memory footprint)
- CPU inference: Slower but more accessible
- GPU inference: Faster but requires CUDA setup
- Async design: Non-blocking text generation

This service represents a specialized AI application for the Polish automotive market, focusing on generating marketing content using state-of-the-art Polish language models.