Fine-tuning and LoRA

LoRA freezes the pretrained weight matrix W0 and learns a low-rank update ΔW = B·A, where A∈R^{r×k} and B∈R^{d×r} with rank r << min(d,k). The effective weight during the forward pass is W0 + (α/r)·B·A, so only A and B are trained. This drastically reduces trainable parameters because instead of d×k values you only learn r×(d+k).

# Effective weight: W = W0 + (alpha/r) * B @ A
# Only A (r x k) and B (d x r) are trainable; W0 is frozen

A is initialized with a random Gaussian (or Kaiming) distribution and B is initialized to zeros. This makes ΔW = B·A = 0 at the start, so training begins exactly from the pretrained model and the adapter introduces no perturbation initially, ensuring a stable warm-start. If both were random, the initial output would be corrupted.

The adapter output is scaled by α/r before being added to the frozen path. Alpha decouples the learning-rate-like magnitude of the update from the chosen rank, so increasing r doesn't automatically blow up the contribution. A common practice is setting α = r or α = 2r; some implementations (rsLoRA) instead scale by α/√r for better stability at high ranks.

QLoRA fine-tunes LoRA adapters on top of a base model quantized to 4-bit, enabling large-model fine-tuning on a single GPU. Its contributions are: (1) 4-bit NormalFloat (NF4), an information-theoretically optimal quantization for normally distributed weights; (2) double quantization, which quantizes the quantization constants to save additional memory; (3) paged optimizers using NVIDIA unified memory to avoid OOM spikes during gradient checkpointing.

No. The frozen base model weights are stored in 4-bit NF4, but they are dequantized to a compute dtype (typically bfloat16) on-the-fly during the forward/backward pass. The LoRA adapters themselves are kept and trained in higher precision (bf16/fp16). Gradients flow only through the adapters, never updating the quantized base.

Because ΔW = (α/r)·B·A can be computed once and added into W0, producing a single merged matrix W = W0 + ΔW that has the same shape as the original. The merged model runs at identical speed to the base with no extra matmuls. The trade-off: once merged you lose the ability to hot-swap adapters, and merging into a quantized base can introduce small numerical errors.

merged_W = W0 + (alpha / r) * (B @ A)  # same shape as W0

The original LoRA paper applied adapters to attention projections (q_proj, v_proj). Later work showed that adapting all linear layers, including k_proj, o_proj, and the MLP/FFN projections (gate/up/down), generally yields better quality, especially for instruction tuning. The QLoRA paper found targeting all linear layers is important to match full fine-tuning performance.

target_modules=['q_proj','k_proj','v_proj','o_proj','gate_proj','up_proj','down_proj']

DoRA decomposes the weight into magnitude and direction components, applying LoRA only to the directional update while learning the magnitude separately. This makes LoRA's learning dynamics more closely resemble full fine-tuning, improving accuracy especially at low ranks, with negligible extra inference cost since it can still be merged.

Higher r increases the expressiveness of ΔW (more trainable parameters) and can capture more complex adaptations, but raises memory cost and overfitting risk on small datasets. Empirically, very small ranks (4-16) often suffice because fine-tuning updates have low intrinsic rank; increasing r beyond a point yields diminishing returns. The right r depends on task complexity and data size.

Catastrophic forgetting is the degradation of previously learned capabilities when a model is fine-tuned on new data. LoRA mitigates it because the original weights W0 are frozen and only a small additive update is learned; the base knowledge is preserved and the adapter can even be removed to restore the original model. However, LoRA does not eliminate forgetting entirely—aggressive training can still shift behavior.

LoRA dropout is applied to the input of the low-rank path (before the A matrix) to regularize the adapter and reduce overfitting on small fine-tuning datasets. It does not affect the frozen base path. Typical values are 0.05-0.1; for large datasets it is often set to 0.

# y = W0 @ x + (alpha/r) * B @ (A @ dropout(x))

Adam-style optimizers store first and second moment estimates (2x the parameter count) plus an fp32 master copy, so optimizer memory dominates for full fine-tuning. Because LoRA trains only a tiny fraction of parameters, optimizer state memory scales with the adapter size, not the full model. QLoRA compounds this by also storing the frozen base in 4-bit.

Prefer full fine-tuning when you have a large high-quality dataset, ample compute, and need to substantially change the model's core behavior or learn knowledge that exceeds the low intrinsic rank of typical adaptations (e.g., a new domain/language at scale). LoRA may underperform full FT when the required weight change is high-rank or when extreme accuracy is critical and budget allows.

LoRA injects trainable low-rank matrices into the weight matrices, modifying the transformation itself. Prefix/prompt tuning instead prepends trainable continuous vectors (soft prompts) to the keys/values or input embeddings, leaving all weights frozen. LoRA generally achieves higher quality and can be merged for zero latency, whereas prompt tuning adds sequence-length overhead but is even more parameter-efficient.

Research (Aghajanyan et al., 2020) showed that pretrained models can be fine-tuned effectively within a very low-dimensional subspace—the task adaptation has low intrinsic dimension. LoRA operationalizes this by constraining the weight update to a low-rank subspace, hypothesizing ΔW itself has low rank. This justifies why tiny ranks recover most of full fine-tuning performance.

Keep the base model loaded once and hot-swap or batch multiple adapters at inference. Systems like S-LoRA and Punica use multi-adapter serving: adapters stay unmerged so different requests in a batch can use different adapters via batched grouped GEMMs (e.g., SGMV). This amortizes the large base model across thousands of adapters with minimal per-adapter memory.

The adapter was trained against dequantized weights, but folding ΔW back requires re-quantizing W0+ΔW, introducing rounding error not seen during training. The common fix is to dequantize the base to fp16/bf16 before merging (or keep the base in higher precision), then optionally re-quantize. Alternatively, keep the adapter unmerged at inference to avoid the issue.

It is r·(d + k): matrix A contributes r·k parameters and matrix B contributes d·r parameters. For example, a 4096×4096 layer at r=8 has 8·(4096+4096) = 65,536 trainable parameters versus ~16.8M for the full matrix—roughly a 256x reduction.

# A: r*k, B: d*r  ->  total = r*(d+k)

rsLoRA (rank-stabilized LoRA) replaces the α/r scaling with α/√r. With the standard α/r scaling, higher ranks effectively shrink the per-element update magnitude, so increasing r doesn't help and can even hurt. The √r scaling keeps the adapter's effective scale stable across ranks, allowing higher ranks to reliably improve performance.

You typically compute the language-modeling loss only on the response (completion) tokens, masking out the instruction/prompt tokens. This focuses learning on generating correct answers rather than memorizing/reproducing prompts, and prevents the model from being penalized for not predicting the given inputs. It usually improves response quality and training efficiency.

labels[:prompt_len] = -100  # ignore_index, no loss on prompt

LoRA generally uses a substantially higher learning rate (e.g., 1e-4 to 3e-4) than full fine-tuning (often 1e-5 to 5e-5). Because only a small low-rank adapter is trained and B starts at zero, larger learning rates are stable and necessary for the adapter to learn meaningful updates within limited steps.

Gradient checkpointing recomputes activations during the backward pass instead of storing them, cutting activation memory at the cost of extra compute (a second forward per checkpointed segment). Combined with QLoRA's 4-bit base and small adapters, it lets large models fit on consumer GPUs. The trade-off is roughly 20-30% slower training for large activation-memory savings.

model.gradient_checkpointing_enable()

Continued (domain-adaptive) pretraining keeps the self-supervised next-token objective on large unlabeled domain corpora to inject knowledge; SFT uses labeled instruction-response pairs to shape behavior/format. LoRA can be used for both, but knowledge-heavy continued pretraining sometimes needs higher ranks or full FT because new factual knowledge can require higher-rank updates than behavior shaping.

bf16 has the same 8-bit exponent range as fp32, so it avoids the overflow/underflow and loss-scaling complications that fp16's limited exponent (5 bits) causes during training. Although bf16 has fewer mantissa bits (less precision), the wider dynamic range yields more stable gradients for LLM fine-tuning, which is why QLoRA uses bf16 as its compute dtype.

bnb_config = BitsAndBytesConfig(load_in_4bit=True, bnb_4bit_compute_dtype=torch.bfloat16)

NF4 is a 4-bit data type whose 16 quantization levels are placed at the quantiles of a standard normal distribution, making it information-theoretically optimal for data that is zero-centered and normally distributed—which pretrained weights approximately are. Compared to uniform int4, NF4 allocates more levels where weights are dense (near zero), reducing quantization error.

After block-wise quantizing weights, each block has a quantization constant (scale) stored in fp32. Double quantization quantizes these constants themselves (e.g., to 8-bit) with a second-level scale, saving the memory of storing many fp32 scales. It saves roughly 0.4 bits per parameter on average (the QLoRA paper reports ~0.37) with negligible accuracy impact.

BitsAndBytesConfig(load_in_4bit=True, bnb_4bit_use_double_quant=True)

LoRA can be applied during preference optimization (DPO, PPO) to make alignment cheaper, training adapters on top of an SFT base while keeping the reference/base frozen. This is memory-efficient since DPO needs both a policy and a reference model—LoRA lets the reference be the frozen base and the policy be base+adapter, avoiding two full model copies.

Persistent high training loss that plateaus well above what the data should allow (underfitting), and the model failing to acquire the target behavior even after sufficient epochs, indicate the low-rank subspace can't express the needed update. Increasing r (and possibly alpha) or targeting more modules typically helps. In contrast, low train loss but poor validation suggests overfitting, not insufficient rank.

AdaLoRA adaptively allocates the parameter budget across layers by parameterizing ΔW in an SVD-like form (PΛQ) and pruning less-important singular values during training based on an importance score. Instead of a uniform fixed rank everywhere, it gives more rank to important layers and less to others, improving efficiency under a fixed total budget.

Adding ΔW1 + ΔW2 assumes the updates are roughly orthogonal/independent, but adapters trained separately can occupy overlapping subspaces, causing destructive interference and degraded performance on each task. Mitigations include weighted merging (TIES, DARE) that resolve sign conflicts and prune redundant deltas, training a joint multi-task adapter, or keeping adapters separate and routing per request.

If you add new tokens (e.g., special/control tokens) or adapt to a new domain vocabulary, the embedding and LM head rows for those tokens need training, so you make them trainable (often fully via modules_to_save, since pure LoRA on embeddings is uncommon). The catch is this significantly increases trainable parameters and memory because vocab×hidden is large, partially offsetting LoRA's efficiency.

modules_to_save=['embed_tokens','lm_head']

A large α/r scaling amplifies the adapter's contribution, effectively acting like a larger learning rate on the update path and potentially causing instability or overfitting. Because the scaling and learning rate both control update magnitude, they are partially redundant; it's common to fix α (e.g., α=2r) and tune the learning rate, rather than pushing alpha extremely high.

task_type tells PEFT how to wrap the model and prepare labels/heads (e.g., CAUSAL_LM, SEQ_CLS, SEQ_2_SEQ_LM, TOKEN_CLS). It ensures the correct forward signature, output head handling, and that modules_to_save/inference behavior match the task. Setting it wrong can lead to mismatched loss computation or untrained heads.

LoraConfig(task_type='CAUSAL_LM', r=16, lora_alpha=32, target_modules='all-linear')

Evaluate on a held-out task-specific test set (target metric) AND on general capability benchmarks/safety suites to detect catastrophic forgetting or alignment regressions. Compare against the base model and ideally a full-FT baseline, use multiple seeds, and include qualitative spot checks. Track both win-rate (e.g., LLM-as-judge or human) and objective metrics, since loss alone is insufficient.