Lasso Regression and ElasticNet: How to Cut the Anchor Line on Weak Features

Every regression model has a weakness. It trusts every feature you give it, even the useless ones. Ridge regression fixes part of this problem. It shrinks big, noisy coefficients back toward zero. But Ridge never lets go completely. Every feature stays in the model, no matter how small its contribution.

Lasso regression solves that gap. It doesn’t just shrink weak coefficients. It removes them. ElasticNet then takes this idea one step further, and it blends Lasso with Ridge for the trickiest datasets. This article walks through both techniques, step by step, with clear analogies and working Python code. It also accompanies the Intelevo YouTube episode “EP37: Regularization — Lasso & ElasticNet,” so feel free to watch the video alongside this write-up for the full explanation.

Why does this matter so much in practice? Modern datasets rarely arrive clean. They usually include dozens, sometimes thousands, of candidate features, and many of them contribute nothing but noise. A model that keeps every single one becomes harder to explain, slower to compute, and more fragile in production. Lasso regression and ElasticNet solve this exact issue, and they do it automatically, without any manual feature engineering on your part.

By the end, you will know exactly when to reach for Lasso regression, when ElasticNet works better, and how to implement both in scikit-learn.

A Quick Recap: What Ridge Regression Taught Us

Picture a small boat in choppy water. Every feature in your dataset holds a rope tied to that boat. Waves — the noise in your data — push the boat around. Without any resistance, the boat drifts wherever the last wave shoved it.

Ridge regression drops an anchor on every single rope. As a result, the boat still moves with real currents, but it resists wild, noisy swings. Mathematically, Ridge adds a penalty to the loss function:

Loss = RSS + λ · Σ βᵢ²

Here, RSS measures how wrong the model’s predictions are. Lambda controls how hard the anchor pulls. And the squared term adds up the “extremeness” of every coefficient. Bigger coefficients cost more, so the model only lets them grow when the data truly earns it.

However, Ridge has one built-in limit. Its penalty softens as a coefficient nears zero. Consequently, it always leaves a small sliver behind. No coefficient in Ridge ever becomes exactly zero. Every feature survives, just in a calmer form.

That’s fine when every feature matters, even a little. But what happens when some features don’t deserve an anchor at all?

The Problem: When Every Feature Overstays Its Welcome

Real-world datasets rarely contain only useful features. Instead, they often include hundreds of columns, and many of them add nothing but noise. Ridge regression shrinks every one of these coefficients, yet it still keeps every feature active. Therefore, the model stays just as complex as before. It simply behaves more calmly.

That’s not ideal. A simpler model is easier to explain, faster to compute, and less likely to overfit. So instead of just smaller coefficients, we want fewer of them. We want a model that drops the dead weight and keeps only the features that actually earn their place.

This is exactly the gap that Lasso regression fills.

Meet Lasso Regression

Let’s return to the boat and its ropes. Ridge simply shortens every rope evenly. Lasso regression, on the other hand, walks over to the useless ropes and cuts them completely. The boat then keeps only the anchors that genuinely hold it steady.

In plain words, Lasso regression asks a stronger question than Ridge does. Ridge asks, “how small can I make this coefficient?” Lasso instead asks, “could this coefficient just be zero?” It still tries to fit the data as closely as possible. However, it’s willing to drop a feature entirely if the data doesn’t need it.

So the goals shift like this:

• Old goal (Ridge): shrink every coefficient a little.
• New goal (Lasso): shrink some coefficients, and eliminate others entirely.

The Lasso Formula, Simplified

Lasso regression needs only one small change to the Ridge formula:

Loss = RSS + λ · Σ |βᵢ|

Notice how similar this looks to Ridge. RSS is still the usual “how wrong were we” error. Lambda is still the strictness knob. The only real difference sits in the last term. Instead of squaring each coefficient, Lasso takes its absolute value.

That single switch — from squares to absolute values — changes everything. It is the reason Lasso can push a coefficient all the way down to exactly zero.

Why Lasso Can Zero Out Coefficients

Here’s the part that makes Lasso click. Ridge’s penalty curves smoothly near zero. As a result, it always leaves a tiny sliver of every coefficient behind. Lasso’s penalty, however, stays sharp all the way down to zero. That sharp corner is mathematically significant. It creates a “kink” in the loss surface, and this kink is exactly what allows the optimizer to land precisely on zero, rather than just near it.

In other words, Lasso performs automatic feature selection. It decides, on its own, which features are worth keeping. Then it drops the rest from the model entirely. Fewer surviving features also means a simpler, more interpretable story about what actually drives your predictions.

Watching Lasso Work: A Real Example

Consider a small housing dataset with five features: Size, Rooms, Age, Zip-noise, and Sensor-X. Without any regularization, the coefficients might look wild:

Now, apply Lasso with lambda equal to one. Watch what happens:

Same data. Same features going in. Yet three coefficients drop to exactly zero and vanish from the model. Only Size and Rooms remain, and both shrink to steadier, more reasonable values. This is feature selection in action, and it happens automatically, without any manual pruning.

Ridge vs. Lasso: Two Philosophies, Side by Side

Neither Ridge nor Lasso is universally “better.” Instead, each one assumes something different about your data. The table below sums up the core differences:

If most of your features carry some genuine signal, Ridge usually performs well. On the other hand, if you suspect only a handful of features actually matter, Lasso regression tends to produce a cleaner, sparser model.

One Catch: When Correlated Features Team Up

Lasso isn’t perfect, though. When two features move almost in lockstep — highly correlated with each other — Lasso tends to arbitrarily keep one and zero out the other. Even if both features matter equally, only one survives. Worse, which one survives can depend on tiny quirks in the data, so the model becomes a little unpredictable. Run the same experiment on a slightly different sample, and you might see a completely different feature survive next time.

This limitation matters more than it first appears. Genomics data, marketing signals, and sensor networks all tend to have groups of correlated features. In genomics, for instance, thousands of genes might rise and fall together as a single biological pathway activates. Lasso, left on its own, could keep just one gene from that entire group and discard the rest, even though every gene in the group carries the same signal. Consequently, we need a technique that treats correlated anchors as a team, rather than picking one at random.

That technique is ElasticNet.

Meet ElasticNet: Two Anchor Types on the Same Boat

ElasticNet blends Ridge’s even, gentle pull with Lasso’s ability to cut a rope completely. As a result, correlated features shrink together, as a group, instead of getting picked apart at random. Think of it as equipping the boat with both types of anchor line at once.

The ElasticNet formula combines both penalties into a single expression:

Loss = RSS + λ [ r·Σ|βᵢ| + (1−r)·Σβᵢ² ]

Lambda still controls the overall strictness, just like before. The new term, r, is a mix ratio between zero and one. It decides how much Lasso behavior versus Ridge behavior to apply. When r equals one, you get pure Lasso. When r equals zero, you get pure Ridge. Anything in between blends the two smoothly.

In scikit-learn, this mix ratio goes by the name l1_ratio. You can dial it toward whichever behavior your dataset actually needs.

Choosing Alpha and the Mix Ratio

So, how do you pick good values for alpha and l1_ratio? The answer is the same technique used for Ridge: cross-validation. First, try a grid of alpha and l1_ratio combinations. Then, keep whichever pair predicts new, unseen data the best.

Picture a simple curve of test error plotted against alpha. The sweet spot sits at the lowest point of that curve. With ElasticNet, this search happens across a small grid of alpha and l1_ratio pairs together, rather than along a single line. Fortunately, scikit-learn automates the entire search for you through LassoCV and ElasticNetCV.

Lasso and ElasticNet in Python

Here’s where the theory becomes practical. Switching from Ridge to Lasso takes barely any code at all.

from sklearn.linear_model import Lasso

model = Lasso(alpha=1.0)
model.fit(X_train, y_train)

print(model.coef_)
# some entries are now exactly 0.0

Only the class name changes here. Everything else — fit, predict, even the alpha parameter — works exactly like it did with Ridge. Once fitted, some entries inside model.coef_ will show up as precisely 0.0. Those are the features Lasso has dropped from the model.

Now, for datasets with correlated features, switch to ElasticNet instead:

from sklearn.linear_model import ElasticNet

model = ElasticNet(alpha=1.0, l1_ratio=0.5)
model.fit(X_train, y_train)

Notice the pattern again. Only the class name changes, and one new parameter, l1_ratio, joins the mix. Here, l1_ratio=0.5 sets a fifty-fifty blend between Lasso and Ridge behavior. alpha still means lambda, the exact same strictness knob you’ve used since Ridge regression. Fit and predict behave identically across all three models, so switching between them costs almost nothing in your codebase.

Where This Shows Up in the Real World

Lasso regression and ElasticNet aren’t just academic tools. They solve real problems across several industries, and understanding where each one fits helps you choose correctly on your next project:

• Genomics: Datasets often include thousands of correlated genes, yet only a handful of patient samples exist. ElasticNet handles this imbalance particularly well, since it keeps correlated gene groups together instead of discarding most of them at random.
• Marketing mix modeling: Ad channels frequently overlap in their signals, since a single campaign might run across TV, social media, and search at the same time. Lasso regression keeps only the channels that genuinely move sales, which makes budget decisions much clearer.
• Text and NLP: Word-based features can number in the tens of thousands, and most carry no useful signal for a given task. Lasso zeroes out that noise automatically, leaving behind a compact, interpretable set of meaningful words.
• Sensor and IoT data: Nearby sensors often produce redundant readings, since two thermometers placed a few feet apart usually report nearly identical values. ElasticNet groups these correlated sensors together, rather than keeping one at random and discarding the rest.

In every case, the underlying idea stays the same. Cut what doesn’t help, and keep what does.

Key Takeaways

Let’s tie everything together:

• Ridge regression shrinks every coefficient, but it never lets any of them hit exactly zero.
• Lasso regression adds an absolute-value penalty, λ · Σ|βᵢ|, and this lets coefficients drop to zero completely.
• A zeroed coefficient simply means that feature has left the model. This is Lasso’s built-in, automatic feature selection.
• ElasticNet blends both penalties. Correlated features shrink together as a group, instead of getting picked apart at random.
• Cross-validation, through LassoCV and ElasticNetCV, finds the best alpha and l1_ratio values automatically.

Cutting an anchor was never complicated. It’s just a stronger pull, one that’s finally allowed to reach all the way to zero.

Watch the Full Video

This article accompanies the Intelevo YouTube episode “EP37: Regularization — Lasso & ElasticNet.” The video walks through every formula, chart, and code example visually, with a running boat-and-anchor analogy that makes the math click faster. Watch it for the complete walkthrough, and subscribe to Intelevo so you don’t miss the next episode.

Up next, in EP38, we shift from cutting anchors to actually watching a model learn. That episode covers Gradient Descent — Batch, Stochastic, and Mini-batch — the process that finds these very coefficients in the first place.