Introduction

Training large language models (LLMs) has traditionally been a resource-intensive process, often requiring massive GPU clusters and specialized hardware to handle the immense memory demands of optimizer states. GaLore (Gradient Low-Rank Projection) is a memory-efficient training strategy that allows for full-parameter learning while drastically reducing the VRAM footprint. With over 1.7k GitHub stars, GaLore enables the training of models like Llama 8B from scratch on a single consumer-grade RTX 4090 GPU (24GB), effectively democratizing access to high-performance LLM training.

What Is GaLore?

GaLore is a memory-efficient low-rank training strategy that projects gradients into a low-dimensional subspace to minimize the memory needed for optimizer statistics. Unlike common low-rank adaptation methods like LoRA, which freeze the base model and train only a small set of adapter weights, GaLore allows for full-parameter learning. It operates as a gradient projection method, meaning it is independent of the choice of optimizer and can be seamlessly integrated into existing training workflows with minimal code changes.

Maintained by researchers including Jiawei Zhao and developed under the Apache License 2.0, GaLore is designed to reduce the memory overhead of modern optimizers (like AdamW) without sacrificing the expressiveness of full-rank weight updates.

Why GaLore Matters

The primary bottleneck in LLM training is not just the model weights, but the optimizer states. For a standard Adam optimizer, the memory required to store first- and second-order moments is often two to four times the size of the model itself. This makes training 7B+ parameter models impossible on consumer hardware without extreme measures like offloading or heavy quantization.

GaLore fills this gap by compressing the optimizer states. By projecting gradients into a low-rank subspace, it reduces optimizer memory usage by 2–4× and can achieve up to an 82.5% reduction in optimizer memory when combined with 8-bit quantization. This allows researchers and developers to perform full-parameter pre-training and fine-tuning on hardware that was previously only capable of running inference or very limited PEFT (Parameter-Efficient Fine-Tuning) methods.

The ability to pre-train a 7B model on a single 24GB GPU is a significant milestone, as it removes the need for model parallelism, checkpointing, or offloading strategies, greatly simplifying the training pipeline for the open-source community.

Key Features

• Full-Parameter Learning: Unlike LoRA, GaLore updates all parameters of the model, ensuring that the training dynamics remain consistent with full-rank training and avoiding the limitations of a frozen base model.
• Gradient Low-Rank Projection: It projects gradients into a low-dimensional subspace, which significantly reduces the memory footprint of the optimizer states while maintaining the essential gradient directions.
• Optimizer Independence: GaLore is a projection layer that can be plugged into any standard optimizer (e.g., AdamW, Adafactor) with only a few lines of code.
• Dynamic Subspace Refresh: To prevent the model from missing important gradient directions over time, GaLore periodically refreshes the projection subspace based on recent gradient information.
• 8-bit Quantization Support: GaLore can be combined with 8-bit optimizers to further reduce total training memory by up to 63.3% compared to a BF16 baseline.
• Broad Framework Integration: It is already integrated into popular LLM training frameworks like LLaMA-Factory and Axolotl, making it accessible to those who do not want to write custom training loops.

How GaLore Compares

When choosing a training strategy, it is important to understand the trade-offs between full fine-tuning, LoRA, and GaLore. While full fine-tuning offers the highest quality, it is often computationally prohibitive. LoRA is fast and lightweight but limits the search space to a low-rank subspace.

GaLore provides a middle ground that offers the learning capacity of full fine-tuning with the memory efficiency of PEFT methods. The main trade-off is a slight increase in computational overhead due to the periodic SVD (Singular Value Decomposition) used to refresh the projection subspace. However, this overhead is minimal compared to the memory savings gained.

Getting Started: Installation

GaLore can be installed as a standalone optimizer library or from the source repository for those who wish to contribute or use the experiment scripts.

Install via pip

The fastest way to get started is to install the galore-torch package:

pip install galore-torch

Install from Source

If you need the full repository, including benchmark scripts and configuration files, clone the repo and install in editable mode:

git clone git@github.com:jiaweizzhao/GaLore.git
cd GaLore
pip install -e .

Prerequisites

GaLore is tested on Python 3.8+ and requires PyTorch 2.1.0 or higher. For optimal memory efficiency, it is recommended to use bitsandbytes for 8-bit optimizer support.

How to Use GaLore

Integrating GaLore into your training loop is straightforward. You simply replace your standard optimizer (e.g., torch.optim.AdamW) with a GaLore-compatible version. The core idea is to define which parameters should be projected (the galore_params) and which should not (the non_galore_params), such as biases and layer norms.

Once the optimizer is initialized, the training process proceeds as normal. GaLore handles the gradient projection and the update of the low-rank subspace in the background, meaning you do not need to change your loss function or backpropagation steps.

Code Examples

Below are examples of how to implement GaLore in a PyTorch training script. These examples are pulled directly from the official repository documentation.

Basic Optimizer Setup

This snippet shows how to initialize the GaLoreAdamW optimizer for a set of parameters.

from galore_torch import GaLoreAdamW

# Define parameter groups
param_groups = [{
    'params': non_galore_params, 
}, {
    'params': galore_params, 
    'rank': 128, 
    'update_proj_gap': 200, 
    'scale': 0.25, 
    'proj_type': 'std'
}]
optimizer = GaLoreAdamW(param_groups, lr=0.01)

Pre-Training LLaMA on C4 Dataset

For those using the provided experiment scripts, the following command demonstrates how to pre-train a 60M LLaMA model using GaLore-Adam:

torchrun --standalone --nproc_per_node 1 torchrun_main.py \
    --model_config configs/llama_60m.json \
    --lr 0.01 \
    --galore_scale 0.25 \
    --rank 128 \
    --update_proj_gap 200 \
    --batch_size 256 \
    --total_batch_size 512 \
    --num_training_steps 10000 \
    --warmup_steps 1000 \
    --weight_decay 0 \
    --dtype bfloat16 \
    --eval_every 1000 \
    --optimizer galore_adamw

Advanced Configuration

GaLore provides several hyperparameters that allow you to tune the memory-performance trade-off. The most critical parameters are rank and update_proj_gap.

• Rank: This determines the dimensionality of the low-rank subspace. A higher rank increases memory usage but can capture more gradient information, potentially improving convergence speed.
• Update Proj Gap: This is the number of steps between each subspace refresh (SVD). A larger gap reduces the computational overhead of SVD but may lead to the optimizer missing important gradient directions.
• Scale: This is a scaling factor applied to the projected gradients. It is often used to maintain the magnitude of the updates.
• Projection Type: The proj_type parameter (e.g., 'std') determines the method used to compute the projection matrix.

Real-World Use Cases

GaLore is particularly useful in scenarios where hardware constraints are the primary limiting factor for model training.

• Consumer GPU Pre-training: A researcher with a single RTX 4090 can now pre-train a 7B parameter model from scratch, a task that previously required an A100 cluster.
• Full-Parameter Fine-Tuning: Developers can perform full-parameter fine-tuning on domain-specific datasets (e.g., medical or legal texts) without relying on adapters, which often underperform compared to full-rank updates.
• Rapid Prototyping of LLM Architectures: Because GaLore reduces the memory footprint, developers can experiment with larger batch sizes or longer sequence lengths on the same hardware, accelerating the iteration cycle.
• Edge Device Training: While primarily focused on GPUs, the memory efficiency of GaLore makes it feasible to explore training on high-end consumer hardware at the edge.

Contributing to GaLore

GaLore is an open-source project and welcomes contributions from the community. While the repository does not have a formal CONTRIBUTING.md, the standard GitHub flow is used: developers can report bugs via the Issues tab and submit improvements via Pull Requests.

Recent activity in the repository shows a focus on improving the SVD implementation, fixing checkpoint resuming and adding support for INT4 projection matrices to further reduce memory.

Community and Support

GaLore has gained significant traction in the open-source community, with integrations into frameworks like LLaMA-Factory and Axolotl. It is also featured on the Hugging Face blog and has received broad media coverage.

For official support and discussion, the project maintains a Slack workspace (GaLore-Social) and the GitHub Discussions tab. The project is also actively discussed on platforms like Reddit and X (Twitter).

Conclusion

GaLore represents a significant leap in the democratization of LLM training. By shifting the focus from reducing the number of trainable parameters to reducing the memory footprint of the optimizer states, GaLore allows for full-parameter learning on consumer hardware. This is a fundamentally different approach than PEFT methods like LoRA, which limit the model’s expressiveness.

If you are a researcher or developer who wants the performance of full fine-tuning without the need for a datacenter, GaLore is the right choice. However, be aware that while it reduces VRAM usage, it does not eliminate the computational cost of training. You will still need a sufficient amount of compute power to process the tokens.

Star the repo, try the quickstart, and join the community to start training your own large models on a single GPU.

Resources

• GaLore GitHub Repository
• GaLore Research Paper (arXiv)
• GaLore Hugging Face Blog Post