Braveness to Study ML: Demystifying L1 & L2 Regularization (half 3) | by Amy Ma | Nov, 2023

I’m glad you introduced up this query. To get straight to the purpose, we sometimes keep away from p values lower than 1 as a result of they result in non-convex optimization issues. Let me illustrate this with a picture displaying the form of Lp norms for various p values. Take an in depth have a look at when p=0.5; you’ll discover that the form is decidedly non-convex.

The shape of Lp norms for different p value. Source:
The form of Lp norms for various p worth. Supply:

This turns into even clearer once we have a look at a 3D illustration, assuming we’re optimizing three weights. On this case, it’s evident that the issue isn’t convex, with quite a few native minima showing alongside the boundaries.


The explanation why we sometimes keep away from non-convex issues in machine studying is their complexity. With a convex drawback, you’re assured a worldwide minimal — this makes it typically simpler to unravel. Alternatively, non-convex issues typically include a number of native minima and may be computationally intensive and unpredictable. It’s precisely these sorts of challenges we goal to sidestep in ML.

After we use methods like Lagrange multipliers to optimize a operate with sure constraints, it’s essential that these constraints are convex capabilities. This ensures that including them to the unique drawback doesn’t alter its basic properties, making it harder to unravel. This facet is crucial; in any other case, including constraints may add extra difficulties to the unique drawback.

You questions touches an fascinating facet of deep studying. Whereas it’s not that we want non-convex issues, it’s extra correct to say that we frequently encounter and need to take care of them within the discipline of deep studying. Right here’s why:

  1. Nature of Deep Studying Fashions results in a non-convex loss floor: Most deep studying fashions, notably neural networks with hidden layers, inherently have non-convex loss capabilities. That is as a result of advanced, non-linear transformations that happen inside these fashions. The mix of those non-linearities and the excessive dimensionality of the parameter area sometimes leads to a loss floor that’s non-convex.
  2. Native Minima are now not an issue in deep studying: In high-dimensional areas, that are typical in deep studying, native minima are usually not as problematic as they may be in lower-dimensional areas. Analysis means that lots of the native minima in deep studying are shut in worth to the worldwide minimal. Furthermore, saddle factors — factors the place the gradient is zero however are neither maxima nor minima — are extra widespread in such areas and are an even bigger problem.
  3. Superior optimization methods exist which can be more practical in coping with non-convex areas. Superior optimization methods, similar to stochastic gradient descent (SGD) and its variants, have been notably efficient find good options in these non-convex areas. Whereas these options may not be international minima, they typically are ok to attain excessive efficiency on sensible duties.

Though deep studying fashions are non-convex, they excel at capturing advanced patterns and relationships in massive datasets. Moreover, analysis into non-convex capabilities is frequently progressing, enhancing our understanding. Trying forward, there’s potential for us to deal with non-convex issues extra effectively, with fewer considerations.

Recall the picture we mentioned earlier displaying the shapes of Lp norms for varied values of p. As p will increase, the Lp norm’s form evolves. For instance, at p = 3, it resembles a sq. with rounded corners, and as p nears infinity, it types an ideal sq..

The shape of Lp norms for different p value. Source:
The form of Lp norms for various p worth. Supply:

In our optimization drawback’s context, think about larger norms like L3 or L4. Just like L2 regularization, the place the loss operate and constraint contours intersect at rounded edges, these larger norms would encourage weights to approximate zero, identical to L2 regularization. (If this half isn’t clear, be at liberty to revisit Part 2 for a extra detailed clarification.) Based mostly on this assertion, we are able to discuss concerning the two essential the reason why L3 and L4 norms aren’t generally used:

  1. L3 and L4 norms show comparable results as L2, with out providing important new benefits (make weights near 0). L1 regularization, in distinction, zeroes out weights and introduces sparsity, helpful for function choice.
  2. Computational complexity is one other important facet. Regularization impacts the optimization course of’s complexity. L3 and L4 norms are computationally heavier than L2, making them much less possible for many machine studying purposes.

To sum up, whereas L3 and L4 norms might be utilized in concept, they don’t present distinctive advantages over L1 or L2 regularization, and their computational inefficiency makes them much less sensible selection.

Sure, it’s certainly potential to mix L1 and L2 regularization, a method also known as Elastic Internet regularization. This method blends the properties of each L1 (lasso) and L2 (ridge) regularization collectively and may be helpful whereas difficult.

Elastic Internet regularization is a linear mixture of the L1 and L2 regularization phrases. It provides each the L1 and L2 norm to the loss operate. So it has two parameters to be tuned, lambda1 and lambda2

Elastic Net regularization. Source:
Elastic Internet regularization. Supply:

By combining each regularization methods, Elastic Internet can enhance the generalization functionality of the mannequin, decreasing the danger of overfitting extra successfully than utilizing both L1 or L2 alone.

Let’s break it down its benefits:

  1. Elastic Internet supplies extra stability than L1. L1 regularization can result in sparse fashions, which is helpful for function choice. However it will also be unstable in sure conditions. For instance, L1 regularization can choose options arbitrarily amongst extremely correlated variables (whereas make others’ coefficients grow to be 0). Whereas Elastic Internet can distribute the weights extra evenly amongst these variables.
  2. L2 may be extra steady than L1 regularization, but it surely doesn’t encourage sparsity. Elastic Internet goals to steadiness these two facets, probably resulting in extra strong fashions.

Nonetheless, Elastic Internet regularization introduces an additional hyperparameter that calls for meticulous tuning. Attaining the precise steadiness between L1 and L2 regularization and optimum mannequin efficiency entails elevated computational effort. This added complexity is why it’s not incessantly used.

Related Articles

Leave a Reply

Your email address will not be published. Required fields are marked *

Back to top button