LLaMA-Factory: Unified fine-tuning for 100+ LLMs
Parameter-efficient fine-tuning framework for 100+ LLMs.
Learn more about LLaMA-Factory
LLaMA-Factory is a fine-tuning framework designed to work with multiple large language model and vision-language model architectures. It implements parameter-efficient fine-tuning methods including LoRA, QLoRA, and full fine-tuning, with support for quantization techniques to reduce memory requirements. The framework provides both CLI and web UI access points, allowing users to configure and execute training runs across different hardware setups. Common deployment contexts include local machine training, cloud platforms like Google Colab and Alibaba PAI, and integration with production systems at organizations like Amazon and NVIDIA.
Multi-model support
Handles over 100 different LLM and VLM architectures including LLaMA, Qwen, Gemma, and DeepSeek through a unified configuration system, reducing the need for model-specific implementations.
Parameter-efficient methods
Implements multiple fine-tuning approaches including LoRA, QLoRA, and full fine-tuning with quantization support, enabling training on hardware with limited VRAM through adapter-based techniques.
Dual interface design
Provides both command-line and web UI (Gradio-based) access points, allowing users to choose between programmatic control and interactive configuration for training workflows.
import subprocess
import os
# Set up training arguments for LoRA fine-tuning
training_args = [
"llamafactory-cli", "train",
"--stage", "sft",
"--do_train",
"--model_name", "llama2_7b_hf",
"--dataset", "alpaca_gpt4_en",
"--template", "llama2",
"--finetuning_type", "lora",
"--lora_target", "q_proj,v_proj",
"--output_dir", "./saves/llama2-7b/lora/sft",
"--overwrite_cache",
"--per_device_train_batch_size", "4",
"--gradient_accumulation_steps", "4",
"--lr_scheduler_type", "cosine",
"--logging_steps", "10",
"--save_steps", "1000",
"--learning_rate", "5e-5",
"--num_train_epochs", "3.0",
"--plot_loss",
"--fp16"
]
# Execute the training
result = subprocess.run(training_args, capture_output=True, text=True)
print(f"Training completed with return code: {result.returncode}")
if result.stderr:
print(f"Errors: {result.stderr}")Major release with breaking changes including repository rename, Python 3.11+ requirement, and migration to uv package manager.
- –Repository name updated: LLaMA-Factory → LlamaFactory
- –Python 3.9–3.10 have been deprecated; LlamaFactory now requires Python 3.11–3.13
- –Migrated from pip to uv; use uv pip install llamafactory
- –The official LlamaFactory blog is now live: https://blog.llamafactory.net/en/
- –🔥 Support Orthogononal Fine-Tuning (OFT)
This release adds support for major new models including Llama4, Gemma3, Qwen3, InternVL3, and Qwen2.5-Omni.
- –🔥 InternVL2.5/InternVL3 model
- –🔥 Qwen2.5-Omni model
- –🔥 Llama 4 and Gemma 3 multimodal model
- –🔥 Official GPU docker image
- –🔥 SGLang inference
This release introduces major new features including APOLLO optimizer, SwanLab tracking, Ray Trainer, and extensive model support.
- –🔥 APOLLO optimizer
- –🔥 SwanLab experiment tracker
- –🔥 Ray Trainer
- –Batch inference with vLLM TP
- –QLoRA on Ascend NPU
See how people are using LLaMA-Factory
Related Repositories
Discover similar tools and frameworks used by developers
Continue
Multi-LLM coding agent with interactive and automated modes.
EasyOCR
PyTorch OCR library using CRAFT and CRNN models.
Paperless-ngx
Self-hosted OCR document archive with ML classification.
Triton
Domain-specific language and compiler for writing GPU deep learning primitives with higher productivity than CUDA.
Ultralytics YOLO
PyTorch library for YOLO-based real-time computer vision.