application focus

graphics and stuff

niche info

ai stuff

The Magic of LoRA (Low-Rank Adaptation)

This document summarizes the core concepts of LoRA and why it is the industry standard for fine-tuning large language models.

1. The Core Concept

Instead of modifying the massive, pre-trained base model weights (the original $W$ matrix), LoRA completely freezes them. To teach the model something new (like your personality or tone), we create a separate "matrix of changes" called $\Delta W$.

During inference, the final math is simply:
Final Output = (Frozen Base Weights) + (Your Custom ΔW Changes)

2. The Math Trick

The Problem: Matrix addition requires identical dimensions. If the base matrix is $4000 \times 4000$, your $\Delta W$ matrix must also be $4000 \times 4000$ (which is 16 million numbers). Training and saving 16 million numbers for every single layer in the model requires massive supercomputers and Gigabytes of storage.

The LoRA Solution: We don't train the massive $\Delta W$ directly. Instead, we force the model to represent the changes by multiplying two tiny matrices together: $A \times B$.

• Matrix $A$ is $4000 \times 4$ (16,000 numbers)
• Matrix $B$ is $4 \times 4000$ (16,000 numbers)

(Notice the 4? That is the rank=4 hyperparameter used in the training script!)

3. Why We Do This (The Benefits)

[!TIP]
We trade a tiny bit of computation at startup to save massive amounts of VRAM during training and massive amounts of hard drive space during storage.

Extremely Fast Training

During training, the GPU only has to calculate gradients and update 32,000 numbers ($A$ and $B$) instead of 16,000,000 numbers (full $\Delta W$). This is a 99.8% reduction in mathematical workload, allowing you to train a 2 Billion parameter model on a cheap, free Kaggle GPU.

Tiny Storage Size (Computation over Space)

If we explicitly calculated $A \times B$ and saved the resulting $4000 \times 4000$ $\Delta W$ matrices for the whole model, the file would take up ~4 Gigabytes.

By only saving the "ingredients" ($A$ and $B$) to the hard drive, the final weights file is only 15 Megabytes.

Modularity

Because the LoRA files are tiny, you can have dozens of different personalities saved on your computer (e.g., a Coding LoRA, a Shakespeare LoRA, a Textbot LoRA). You can instantly swap them out, or even load two at the same time to create a hybrid personality!

Zero Inference Latency

When you actually load the model to chat, your script multiplies $A \times B$ exactly one time while booting up, and temporarily merges the result into the base weights in your RAM. Once that 2-second startup is done, the chat interface runs with zero extra steps and exactly the same speed as the original model!