Training a large language model is less about writing perfect code and more about choosing the right numbers. If you get the learning rate or batch size wrong, you don't just lose time; you waste millions of dollars in GPU hours watching your loss curve plateau or diverge. In 2026, with models pushing past 70 billion parameters, guessing these values is no longer viable. You need data-driven rules.
The landscape has shifted dramatically since the early days of transformer research. We moved from trial-and-error grid searches to predictive frameworks like the Step Law is a comprehensive scaling law framework derived from over 3,700 model configurations that predicts optimal hyperparameters based on model size and dataset volume. This approach, validated across hundreds of billions of tokens, tells us exactly how to scale our training settings as we grow our models. Here is what actually matters when you are setting up your pretraining pipeline today.
Why Learning Rate Is the Single Most Critical Lever
If you only have time to tune one thing, make it the learning rate. It dictates how aggressively the model updates its weights after seeing each batch of data. Too high, and the model oscillates wildly, failing to converge. Too low, and it takes years to learn basic grammar patterns. Recent empirical data shows that even a 5% deviation from the optimal rate can increase final perplexity by 0.2 to 0.5 points. That might sound small, but in competitive benchmarks, that gap is the difference between being top-3 and mid-tier.
The relationship isn't linear. According to the Step Law framework, the optimal learning rate ($\eta$) scales with both the number of model parameters ($N$) and the total dataset size ($D$). The formula follows a power-law relationship: $\eta \propto N^{-0.11} \times D^{0.05}$. This means larger models require slightly lower learning rates, while larger datasets allow for slightly higher ones. For example, if you double your dataset size, you can nudge the learning rate up by roughly 3.5%. If you quadruple your model size, you should reduce the rate by about 8%. Ignoring this interaction leads to the "convergence cliff," where training becomes unstable immediately after scaling up.
Balancing Batch Size Against Computational Efficiency
Batch size determines how many examples the model sees before updating its weights. There is a persistent myth that batch size should scale linearly with model size. The data says otherwise. Optimal batch size ($B$) depends primarily on the dataset size, following the rule $B \propto D^{0.75}$. This sub-linear scaling is crucial for cost management.
Consider a practical scenario: you are training a 7-billion parameter model on a 100-billion token corpus. The optimal batch size is around 2,048 tokens. Now, imagine scaling that same model to 70 billion parameters using the same data volume. You might intuitively think you need a much larger batch to match the increased capacity. However, the optimal batch size remains relatively stable unless you also significantly expand the dataset. For a 70B model on a 100T token dataset, the optimal batch size jumps to 8,192 tokens, but this is driven by the data volume, not just the model's weight count. Getting this balance wrong results in either under-utilized GPUs (small batches) or noisy gradients that prevent fine-tuning of complex features (large batches).
| Hyperparameter | Scaling Dependency | Formula Relationship | Risk of Deviation |
|---|---|---|---|
| Learning Rate | Model Params (N) & Dataset Size (D) | $\eta \propto N^{-0.11} \times D^{0.05}$ | Divergence or slow convergence |
| Batch Size | Dataset Size (D) | $B \propto D^{0.75}$ | Noisy gradients or wasted compute |
| Minimum Learning Rate | Peak Learning Rate | Fixed value, not peak/100 | Prevents convergence to local minima |
The Hidden Trap: Minimum Learning Rate Configuration
Most practitioners use a cosine decay schedule, starting at a peak learning rate and ending at a minimum rate. The conventional wisdom was to set the minimum rate to 1% of the peak (peak/100). This is often a mistake. When peak learning rates are high, a relative minimum creates an absolute floor that is too high for the model to settle into the best possible local minimum. Studies show that using a fixed, empirically determined minimum learning rate reduces final loss by 12-18% compared to the traditional relative approach. This is particularly critical for Mixture-of-Experts (MoE) architectures, where expert sparsity ratios can shift the optimal minimum rate by up to 37%. If you are training MoE models, do not rely on generic decay schedules; test specific minimum values.
Architectural Nuances: Dense vs. Mixture-of-Experts
Not all models behave the same way. While dense transformers follow the standard scaling laws closely, Mixture-of-Experts (MoE) variants introduce complexity due to their sparse activation patterns. In MoE models, the effective number of active parameters changes dynamically during training. This means the "optimal" learning rate is not static; it varies based on the expert sparsity ratio. A highly sparse MoE model might require a different learning rate trajectory than a denser counterpart of the same total parameter count. Recent validations across 37 distinct model shapes confirm that ignoring sparsity in your hyperparameter calculations leads to prediction errors exceeding 400% in edge cases. Always factor in your architecture's specific sparsity profile when applying scaling laws.
Practical Implementation: From Theory to Pipeline
How do you apply this in a real-world workflow? You don't need to run thousands of experiments. The modern approach involves a two-step verification process. First, use the Step Law formulas to predict your initial learning rate and batch size. Second, run a small-scale verification grid around these predictions. Specifically, test three points: the predicted optimal value, 15% above it, and 15% below it. This requires only 1,500 to 2,000 additional GPU hours for a mid-sized model, a fraction of the 15,000+ hours required for a full grid search. Practitioners who adopted this method reported reducing hyperparameter search time from weeks to days while achieving better final perplexity.
Tools like Optuna is a hyperparameter optimization framework that integrates native Step Law support in version 4.2 to automate search processes now offer built-in integrations for these scaling laws. This automation removes the manual calculation burden and ensures consistency across team members. However, be cautious with domain-specific corpora. If your training data is heavily skewed toward code or scientific text, standard scaling laws may deviate by 15-25%. In such cases, treat the predicted values as a starting point rather than a guarantee, and monitor your loss curves closely in the first 10% of training steps.
Frequently Asked Questions
Does batch size need to scale with model size?
No, batch size primarily scales with dataset size, not model parameter count. The relationship is sub-linear ($B \propto D^{0.75}$), meaning doubling your model size does not require doubling your batch size unless you also increase your data volume.
What is the safest way to determine the minimum learning rate?
Avoid setting it as a fixed percentage of the peak rate (like 1%). Instead, use a fixed absolute value determined through short validation runs. This prevents the model from getting stuck in suboptimal local minima, especially in high-peak-rate scenarios.
How accurate are scaling laws for Mixture-of-Experts models?
They are accurate if you account for expert sparsity. Standard dense-model formulas can fail by up to 400% for MoE models if sparsity is ignored. Adjust your learning rate predictions based on the specific sparsity ratio of your architecture.
Can I use these rules for non-English languages?
Yes, the Step Law framework has shown statistical invariance across 14 different data distributions, including multilingual corpora. Prediction accuracy remains within 8-12% regardless of language, provided the data distribution is reasonably homogeneous.
What tools help automate this process?
Frameworks like Optuna (version 4.2+) include native Step Law integration. These tools can automatically calculate recommended ranges and execute verification grids, reducing manual configuration errors and saving significant computational resources.