Support vector

We draw two boundary lines passing through feature vectors of one class closest to the feature vectors of another class. The center line of these boundary lines is the hyperplane we have been talking about. For example, for Class B, a boundary line is passing through p and q along the way and another boundary line through r and s because p and q are the closest to the feature vectors of Class B and so are r and s. These are called support vectors. We will understand now why these are called support vectors:

Let's say that if we try to remove one of the feature vectors that is not so close to the boundary line, we will not have an impact on the position or orientation of the hyperplane because the hyperplane's position is decided by boundary lines crossing through vectors p, q, r, and s. And, since these are the points holding (supporting) the hyperplane together, they have been named support vectors.

So, this equation y_i (w.x_i + b) = 1 holds true when x_i is p, q, r, or s.

We will go back to the equation w.x + b/ ||w|| ≥ γ; here, we are trying to maximize γ, and in order to do so either we need to maximize b or minimize ||w||.

Or we can say we have to minimize w.w. If we convert that into a function, Φ(w) = w.w has to be minimized. Φ(w) =1/2( w.w) (here 1/2 has been added for mathematical convenience).

So, the objective function of SVM becomes Φ(w) =1/2( w.w), which has to be minimized subject to constraints, as follows:

y_i (w.x_i + b) = 1

Since it is a constrained optimization problem, it can be converted into an unconstrained optimization problem using the Lagrangian multiplier.

Hence, L(w,b)= 1/2(w.w) - ∑ αi [yi(w.xi+b) - 1] where αi is the Lagrangian multiplier, L(w,b)= 1/2(w.w) - ∑ α_i y_i (w.x_i) -∑ α_i y_i b + ∑ α_i.

Let's find out w and b by using maxima and minima calculus:

δL/δb = 0 

It results in ∑ α_i y_i=0, δL/δw = 0 would result in ∑ αi yi xi = w. Now, putting these results back into the Lagrangian function yields the following:

L= ∑ α_i - 1/2 ∑ α_i α_j y_i y_j (x_j.x_i)

It means that if the value of α_i is very high then the corresponding x. There will be a lot of influence on the position of the hyperplane. Hence, for classification and for unknown feature vector z, the required equation would be the following:

D(z) = Sign( ∑ αi xi yi z + b)

If D(z) >0, then z would belong to class A and if D(z)<0, z ∈ class B. Let's try to perform a case study in Python: