Grid search feels safe. You list every value you want to try, and the algorithm checks each one. But that safety has a price. Add a few more hyperparameters, and the checklist explodes into millions of combinations. Add a continuous range, like a learning rate, and grid search simply cannot cover it.
So what comes next? Two techniques solve this problem in very different ways: Random Search and Bayesian Optimization. Together, they form the natural next step after grid search, and they power almost every serious tuning pipeline in machine learning today.
In this article, we unpack both methods with one simple analogy, walk through the exact steps each algorithm follows, and finish with working Python code you can run today. This is the companion article for Episode 57 of the Intelevo Machine Learning series. If you prefer to watch and listen, the full video sits at the top of this page.
Why Grid Search Runs Out of Road
Let’s recap the problem first, because it explains why we need new tools.
Grid search tests every single combination of hyperparameter values from a list you define. It never skips anything. Consequently, it always finds the best combination — but only among the values you gave it.
That guarantee comes at a steep cost, though. Every extra hyperparameter multiplies your search space. Ten hyperparameters with ten values each produce ten billion combinations. No machine finishes that grid in a reasonable time.
Worse, grid search cannot handle continuous ranges. Suppose the ideal learning rate is 0.0347. If that exact number isn’t on your grid, you will never find it, no matter how carefully you built the list.
In short, grid search hits a wall in two ways: it explodes combinatorially, and it ignores everything between your chosen values. Once your search space grows, you need a different strategy.
A Simple Way to Think About Search: The Treasure Hunt
Before we dive into the mechanics, let’s set up one analogy. We’ll use it throughout this article, because it makes both algorithms much easier to picture.
Imagine an island. Every spot on that island represents one possible combination of hyperparameters. A “dig spot” is a combination you actually stop and test. Somewhere on the island, buried treasure sits waiting — that’s the combination with the best validation score.
Grid search walks the island in a fixed pattern, checking every square meter in order. It’s thorough, but painfully slow once the island grows large. Random search and Bayesian optimization both take smarter approaches to the same hunt.
Random Search: Smarter Coverage, Same Budget
Here’s a genuinely surprising result from search theory: you don’t need to check the entire island to find the treasure.
Why? First, in most real models, only two or three hyperparameters actually move your validation score in a meaningful way. Grid search doesn’t know this. It spends just as much effort testing values of hyperparameters that barely matter as it does on the ones that matter most.
Second, if you scatter your digs randomly across the island instead of following a rigid grid, you end up testing far more unique values for every important dial. And you do this within the exact same budget.
This isn’t a hunch. It’s the core finding from Bergstra and Bengio’s influential 2012 paper on random search for hyperparameter optimization. Their research showed that, for a fixed number of trials, random sampling reliably outperforms grid search whenever only a handful of hyperparameters truly matter.
Picture 25 digs allowed on our island, split across two hyperparameters. Grid search lays those 25 digs out as a neat five-by-five pattern. As a result, it only ever tests five unique values per dial. Random search, on the other hand, scatters the same 25 digs freely. Consequently, it can test up to 25 unique values per dial. Same budget, same number of training runs — but far richer coverage.
How Random Search Works, Step by Step
Random search follows four steps, and it closely mirrors the grid search workflow you may already know.
- Define. Instead of listing fixed values, you give each hyperparameter a range or a probability distribution to sample from.
- Sample. The algorithm randomly draws
n_itercombinations from those ranges. You choose this number directly, so it becomes your budget. - Train and score. Every sampled combination gets trained and cross-validated, exactly as before.
- Keep the best. The algorithm ranks every combination by its average validation score, then keeps the winner.
One formula is worth remembering here: total digs equal n_iter times folds. You control this number directly, and it stays fixed even as you add more hyperparameters to your search. That single property makes random search far more scalable than grid search.
When to Choose Random Search
A few clear signals tell you when random search is the right tool.
If you’re tuning only one or two hyperparameters, grid search still works fine. The space stays small enough to search exhaustively. However, once you’re tuning three or more hyperparameters, random search starts covering meaningfully more ground for the same budget.
Additionally, if any hyperparameter is continuous — a learning rate, a regularization strength — random search can test values that grid search simply cannot reach. And on a tight compute budget, you set n_iter directly, so you always know exactly how many runs you’re committing to.
There’s one exception, though. If every single training run is expensive, such as with large neural networks, you may want something smarter than blind random sampling. That’s exactly where Bayesian optimization enters the picture.
Bayesian Optimization: Learning From Every Attempt
Random search still digs blindly. Every dig happens independently of every other dig, so the algorithm never learns from its own history.
Here’s the natural next question, then: what if every dig taught you something about where to dig next? That’s precisely what Bayesian optimization does. It remembers every result so far — what worked, what didn’t — and it uses those clues to choose smarter, more promising digs with every step.
Think of it as upgrading from a random treasure hunter to a smart one who actually pays attention to the clues.
Exploration vs Exploitation
Every smart hunter, human or algorithm, balances two competing instincts.
Exploration means digging in a completely unknown part of the island. You might strike out. Alternatively, you might discover a rich new region you’d never have found otherwise.
Exploitation means digging again near your best find so far. You refine and confirm a spot you already trust, rather than gambling on something new.
Too much exploration wastes digs on dead ends. Too much exploitation risks missing a better region entirely. Bayesian optimization’s whole job is balancing these two instincts automatically. It always digs wherever its internal “clue score” runs highest.
How Bayesian Optimization Works, Step by Step
Bayesian optimization follows a four-step loop, and it builds directly on everything we’ve covered so far.
- Build a guess map. Using every dig so far, the algorithm fits a quick model that estimates the score everywhere on the island — even in spots you haven’t dug yet. This model has a technical name: a surrogate model.
- Pick the best spot. The algorithm uses that guess map to choose the next combination worth trying, automatically balancing exploration and exploitation.
- Dig and record. That chosen combination gets trained and validated for real, and its true score gets recorded.
- Update the map. The guess map refines itself using this new, real result. Then the loop repeats until the budget runs out.
This exact loop powers popular libraries like scikit-optimize and Optuna. Once you understand these four steps, both libraries become far easier to configure and trust.
Grid vs Random vs Bayesian: Quick Comparison
Let’s put all three side by side, so you have a quick reference for your next project.
Grid Search
- Tests every combination in your list
- Works best for one or two hyperparameters
- Guarantees the best result on your grid
- Cost explodes fast as you add more dials
Random Search
- Samples randomly from ranges or distributions
- Works well once you’re tuning three or more hyperparameters
- Lets you control the budget directly through
n_iter - May still miss the true best combination
Bayesian Optimization
- Learns from every past result
- Shines when each training run is expensive
- Typically needs the fewest runs to reach a good answer
- Requires more setup and an extra library
Python Code: RandomizedSearchCV and BayesSearchCV
Now, let’s write this. We’ll reuse the same Ridge regression pipeline from our previous episode on grid search, so you can compare the code side by side.
First, here’s random search in action:
from sklearn.model_selection import RandomizedSearchCV
from sklearn.pipeline import Pipeline
from sklearn.preprocessing import PolynomialFeatures
from sklearn.linear_model import Ridge
from scipy.stats import loguniform
pipe = Pipeline([("poly", PolynomialFeatures()),
("ridge", Ridge())])
param_dist = {
"ridge__alpha": loguniform(1e-3, 1e2),
"poly__degree": [1, 2, 3, 4]
}
search = RandomizedSearchCV(pipe, param_dist, n_iter=30,
cv=5, scoring="r2", random_state=42)
search.fit(X_train, y_train)
print("Best params:", search.best_params_)
Notice the loguniform(1e-3, 1e2) distribution for alpha. This detail matters. Regularization strength behaves on a logarithmic scale, so uniform sampling would waste most draws on large values. Log-uniform sampling fixes that, giving small and large values a fair, balanced chance.
Now, watch how little changes for Bayesian optimization:
from skopt import BayesSearchCV
from skopt.space import Real, Integer
bayes = BayesSearchCV(pipe, {
"ridge__alpha": Real(1e-3, 1e2, prior="log-uniform"),
"poly__degree": Integer(1, 4)
}, n_iter=30, cv=5, random_state=42)
bayes.fit(X_train, y_train)
print("Best params:", bayes.best_params_)
The interface barely changes. We still call .fit(), we still get best_params_, and we still reuse the same pipeline. Underneath, though, the search strategy is completely different. RandomizedSearchCV samples blindly from your distributions, while BayesSearchCV learns from every previous attempt and steers itself toward promising regions of the search space.
That similarity is the whole point. Once you’re comfortable with one search class in scikit-learn, switching to another becomes a two-line change, not a rewrite.
Common Mistakes to Avoid
A few pitfalls trip up beginners repeatedly, so let’s address them directly.
Setting n_iter too low. Random search offers no guarantees, only better odds. Too few digs, and you can genuinely miss the good region of your search space entirely.
Ignoring the shape of your distribution. If you sample a rate like 0.00001 to 1 uniformly, most of your draws land on the large end of that range. Always reach for a log-uniform distribution when tuning scale-sensitive hyperparameters, such as learning rate or regularization strength.
Expecting Bayesian optimization to be free. Building and updating the guess map takes real computation time on every single iteration. If your model already trains in a few seconds, plain random search often performs just as well — and it’s considerably simpler to set up.
Avoiding these three mistakes will save you hours of debugging and wasted compute.
Key Takeaways
Let’s bring everything together into one simple loop, because both algorithms really do reduce to four repeatable steps: guess, train, compare, and repeat.
You guess or sample the next spot to dig. You train and cross-validate that combination. You compare it against every previous dig. Then you repeat the cycle until your budget runs out.
That’s genuinely all there is to it. No more hand-tuning one value at a time, and no more baking every single cake, as grid search forces you to do. Instead, you get smarter, faster digging — matched to your specific budget and your specific problem.
Random search gives you excellent, controllable coverage once your search space grows past two or three hyperparameters. Bayesian optimization goes one step further, learning from every attempt so it needs fewer runs to reach a strong answer. Choose based on how expensive each training run is, and how many hyperparameters you’re tuning.
What’s Next
You’ve now found the best hyperparameters for your model. But finding good hyperparameters only solves half the problem. The next question is: how do you judge the model itself?
In Episode 58, we cover the ROC curve, AUC, and threshold tuning. We’ll explore how to read a ROC curve, what AUC actually measures, and how to pick a decision threshold that fits your specific problem, rather than defaulting blindly to 0.5.
Watch the full video walkthrough of random search and Bayesian optimization at the top of this page, and subscribe to the Intelevo channel so you don’t miss Episode 58.