Mastering LLM Training: Batch Size, Gradient Accumulation, and Throughput

You have a powerful cluster of GPUs. You have your dataset ready. You hit "train." And then... you wait. Or worse, you get a CUDA Out-Of-Memory (OOM) error before the first epoch finishes. This is the daily reality for anyone working with Large Language Models (AI systems trained on massive datasets to understand and generate human language). The difference between a training run that costs thousands in wasted compute and one that finishes efficiently often comes down to three specific settings: batch size, gradient accumulation, and throughput.

Getting these right isn't just about making things go faster. It’s about balancing memory constraints with optimization stability. If you set your batch size too high, your GPU crashes. Too low, and your model learns slowly or poorly. Let's break down how these pieces fit together so you can stop guessing and start tuning with confidence.

The Anatomy of Effective Batch Size

When people talk about "batch size" in LLM training, they are usually talking about two different things at once. To avoid confusion, we need to separate the effective batch size from the micro-batch size.

The effective batch size is the total number of samples used to calculate a single update to the model's weights. This is the number that matters for convergence-how well your model actually learns. Research and industry best practices suggest an optimal range for fine-tuning typically falls between 16 and 128 samples per optimizer step. Some tasks might need smaller numbers, others larger, but this is your starting target.

However, you cannot simply load 128 samples into a single GPU if it only has 40 GB of memory. That’s where the math comes in. The effective batch size is decomposed into three parts:

• Micro-batch size: The number of examples processed by a single GPU in one forward/backward pass.
• Gradient accumulation steps: How many micro-batches you process before updating the weights.
• World size: The number of GPUs or data-parallel workers in your cluster.

The formula looks like this:

Effective Batch Size = Micro-batch Size × Gradient Accumulation Steps × World Size

If you are using a framework like Megatron-LM (A library for training large-scale models using model parallelism), the plugin might handle this via a global batch size parameter, but the underlying logic remains the same. You pick the effective batch size based on what makes your model learn best, then you adjust the micro-batch size to fit your hardware.

Why We Use Gradient Accumulation

Imagine you want an effective batch size of 64, but your GPU can only hold 4 samples in memory without crashing. You could buy more GPUs, or you could use Gradient Accumulation (A technique that sums gradients over multiple small batches before updating model weights).

Here is how it works under the hood. Instead of updating the model weights after every 4 samples, you run those 4 samples, calculate the gradients, and save them. You do not update the weights yet. You repeat this 16 times (16 × 4 = 64). Only after processing all 64 samples do you average the accumulated gradients and perform a single weight update.

This trick allows you to simulate a large batch on limited hardware. Frameworks like DeepSpeed (A deep learning optimization library developed by Microsoft) and Axolotl (An open-source tool for fine-tuning large language models) support this natively. Crucially, DeepSpeed optimizes this further by averaging gradients locally during each step and performing a single collective communication (allreduce) at the end. This reduces network traffic significantly compared to communicating after every micro-batch.

But there is a catch. While activations (the intermediate values during computation) are freed after each micro-batch, the gradients themselves must be stored until the update happens. So, gradient accumulation does increase memory usage slightly, just not as much as loading the full batch would.

Throughput: The Real Metric That Matters

We don't train models for fun; we train them to solve problems. Therefore, speed matters. But "speed" in LLMs is measured in tokens per second, not just seconds per epoch. This metric tells you how efficiently you are using your expensive hardware.

Your GPU has two main bottlenecks: memory bandwidth and compute power. At very small batch sizes (like 1), your GPU spends most of its time waiting for data to move from memory to the processor. This is called being "memory-bound." As you increase the batch size, you amortize the cost of loading weights across more tokens. Eventually, the GPU becomes "compute-bound," meaning it is doing actual math as fast as it possibly can. This is the sweet spot.

Hardware architecture plays a huge role here. NVIDIA H100 Tensor Cores, for example, process operations in 16×16 matrix tiles. If your batch size is 127, the GPU sits idle while waiting for the next tile. If you bump it to 128, utilization jumps from 61% to 94%. That 1-token difference nearly doubles your throughput and cuts your cost per million tokens by over 60%.

The lesson? Don't just pick random numbers. Align your micro-batch size with your hardware's native tile sizes (often powers of 2 or multiples of 16) to maximize throughput.

Small Batches vs. Large Batches: The Debate

For years, the conventional wisdom was "bigger is better." Large batches provide smoother gradient estimates, leading to stable training. However, recent academic research challenges this. Studies from late 2025 suggest that for many language model tasks, smaller batch sizes can maintain optimization stability and generalization while offering higher throughput, provided the hardware is tuned correctly.

The argument is simple: if you can achieve the same convergence with a smaller effective batch size, why pay the memory and communication overhead of a larger one? On single devices, gradient accumulation adds overhead without proportional gains. It is primarily useful in multi-device setups where reducing the frequency of gradient synchronization saves network bandwidth.

So, should you use small or large batches? The modern heuristic is: choose the smallest effective batch size that maximizes your model's throughput. Start small, measure your tokens per second, and only increase if you see instability or slower convergence. Do not blindly chase large numbers.

Practical Tuning Strategy

How do you find the right settings for your specific setup? Here is a step-by-step approach used by experienced engineers:

1. Pick your Target Effective Batch Size: Start with a standard value like 64 or 128 for fine-tuning. If you are pretraining, you might need larger values.
2. Find the Max Micro-Batch Size: Generate some synthetic data. Run a single training step with a micro-batch size of 1. Measure the throughput. Double the size (2, 4, 8...) until you hit a CUDA OOM error or the throughput stops increasing. Pick the largest size that runs smoothly.
3. Calculate Accumulation Steps: Divide your target effective batch size by (Max Micro-Batch Size × Number of GPUs). This gives you your gradient accumulation steps.
4. Monitor and Adjust: Watch your GPU utilization. If it's below 80%, try increasing the micro-batch size. If you are getting OOM errors, decrease the micro-batch size and increase accumulation steps.

Be aware of framework quirks. In Megatron-LM pipelines, micro-batches often serve as the accumulation units, so you might need to set explicit accumulation steps to 1. In Axolotl, you have more direct control. Always check your framework's documentation for how it handles these parameters.

Common Pitfalls to Avoid

Even with a good strategy, things can go wrong. Here are the most common issues:

• Ignoring Learning Rate Scaling: When you change the effective batch size, you often need to scale your learning rate. A common rule is linear scaling: if you double the batch size, double the learning rate. Failure to do this can lead to unstable training or slow convergence.
• Misconfiguring Pipeline Parallelism: If you are using pipeline parallelism, the interaction between micro-batches and virtual stages is complex. Ensure your `microbatch_group_size` is set correctly to balance memory and throughput.
• Over-Accumulating on Single GPUs: As mentioned, gradient accumulation on a single device adds memory overhead for storing gradients. If you have enough VRAM, skip accumulation and just use a larger micro-batch. Save accumulation for when you are constrained by memory or network bandwidth.

Tuning these parameters is not a one-time task. As you scale up your model or change your dataset, you will need to revisit these settings. But by understanding the relationship between batch size, accumulation, and throughput, you turn a black box into a dial you can control.