🚀 Setup & Usage

This project uses a Docker-based development workflow. The Dockerfile creates a consistent environment with all dependencies, and the source code is mounted into the container at runtime.

Prerequisites

• Docker installed and running
• Docker installed for GPU access within Docker

Step 1: Build the Docker Image

Build the main Docker image, which contains the Python environment and all dependencies.

docker build -t gemma-trainer .

Step 2: Run the Training Pipeline

This command executes the full training and merging pipeline. The final merged model will be saved to your local ./models/finetuned directory.

docker run --gpus all -it --rm -e TORCH_COMPILE_DISABLE=1 -v "$(pwd)/src:/app/src" -v "$(pwd)/scripts:/app/scripts" -v "$(pwd)/data:/app/data:ro" -v "$(pwd)/models:/app/models" gemma-trainer python -m src.train_pipeline

Note: TORCH_COMPILE_DISABLE=1 is a required flag to prevent conflicts between PyTorch 2's compiler and Unsloth's deep optimizations.

Step 3: Run Inference with the Fine-Tuned Model

After training, use this command to get a response from your fine-tuned model.

docker run --gpus all -it --rm -e TORCH_COMPILE_DISABLE=1 -v "$(pwd)/src:/app/src" -v "$(pwd)/models:/app/models:ro" gemma-trainer python -m src.inference --prompt "What is the first aid for a burn?"

Step 4: Convert Model to GGUF (Optional)

To create the highly efficient GGUF models for broad deployment, use the separate converter environment.

1. Build the Converter Image

   docker build -t gguf-converter -f Dockerfile.convert .
   

2. Run the Conversion Script

   This command mounts your fine-tuned model as input and saves the resulting .gguf files to your local ./models/gguf directory.

   docker run -it --rm -v "$(pwd)/models/finetuned/gemma_3n_finetuned_merged:/app/model_input:ro" -v "$(pwd)/models/gguf:/app/model_output" -v "$(pwd)/scripts/convert_to_gguf.py:/app/convert_to_gguf.py" gguf-converter python /app/convert_to_gguf.py
   

Step 5: Test GGUF Models (Optional)

Run the comparative inference script within the gguf-converter container to test your newly created GGUF models.

docker run -it --rm -v "$(pwd)/models/gguf:/app/models:ro" -v "$(pwd)/scripts/inference_gguf.py:/app/inference_gguf.py" gguf-converter python /app/inference_gguf.py