Activation Features & Non-Linearity: Neural Networks 101 | by Egor Howell | Oct, 2023


Explaining why neural networks can be taught (almost) something and the whole lot

Photograph by Google DeepMind:

In my earlier article, we launched the multi-layer perceptron (MLP), which is only a set of stacked interconnected perceptrons. I extremely suggest you verify my earlier publish if you’re unfamiliar with the perceptron and MLP as will talk about it fairly a bit on this article:

An instance MLP with two hidden layers is proven beneath:

A fundamental two-hidden multi-layer perceptron. Diagram by writer.

Nevertheless, the issue with the MLP is that it may solely match a linear classifier. It’s because the person perceptrons have a step function as their activation function, which is linear:

The Perceptron, which is the only neural community. Diagram by writer.

So regardless of stacking our perceptrons might seem like a modern-day neural community, it’s nonetheless a linear classifier and never that a lot completely different from common linear regression!

One other downside is that it’s not absolutely differentiable over the entire area vary.

So, what will we do about it?

Non-Linear Activation Features!

What’s Linearity?

Let’s shortly state what linearity means to construct some context. Mathematically, a operate is taken into account linear if it satisfies the next situation:

There may be additionally one other situation:

However, we’ll work with the beforehand equation for this demonstration.

Take this quite simple case:


Related Articles

Leave a Reply

Your email address will not be published. Required fields are marked *

Back to top button