Singular value decomposition

When we decompose an integer into its prime factors, we can understand useful properties about the integer. Similarly, when we decompose a matrix, we can understand many functional properties that are not directly evident. There are two types of decomposition, namely eigenvalue decomposition and singular value decomposition.

All real matrices have singular value decomposition, but the same is not true for Eigenvalue decomposition. For example, if a matrix is not square, the Eigen decomposition is not defined and we must use singular value decomposition instead.

Singular Value Decomposition (SVD) in mathematical form is the product of three matrices U, S, and V, where U is m*r, S is r*r and V is r*n:

The following example shows SVD using a TensorFlow svd operation on textual data:

import numpy as np
import tensorflow as tf
import matplotlib.pyplot as plts

path = "/neuralnetwork-programming/ch01/plots"

text = ["I", "like", "enjoy",
         "deep", "learning", "NLP", "flying", "."]
xMatrix = np.array([[0,2,1,0,0,0,0,0],
              [2,0,0,1,0,1,0,0],
              [1,0,0,0,0,0,1,0],
              [0,1,0,0,1,0,0,0],
              [0,0,0,1,0,0,0,1],
              [0,1,0,0,0,0,0,1],
              [0,0,1,0,0,0,0,1],
              [0,0,0,0,1,1,1,0]], dtype=np.float32)

X_tensor = tf.convert_to_tensor(xMatrix, dtype=tf.float32)

# tensorflow svd
with tf.Session() as sess:
    s, U, Vh = sess.run(tf.svd(X_tensor, full_matrices=False))

for i in range(len(text)):
    plts.text(U[i,0], U[i,1], text[i])

plts.ylim(-0.8,0.8)
plts.xlim(-0.8,2.0)
plts.savefig(path + '/svd_tf.png')

# numpy svd
la = np.linalg
U, s, Vh = la.svd(xMatrix, full_matrices=False)

print(U)
print(s)
print(Vh)

# write matrices to file (understand concepts)
file = open(path + "/matx.txt", 'w')
file.write(str(U))
file.write("\n")
file.write("=============")
file.write("\n")
file.write(str(s))
file.close()

for i in range(len(text)):
    plts.text(U[i,0], U[i,1], text[i])

plts.ylim(-0.8,0.8)
plts.xlim(-0.8,2.0)
plts.savefig(path + '/svd_np.png')

The output of this is shown as follows:

[[ -5.24124920e-01  -5.72859168e-01   9.54463035e-02   3.83228481e-01   -1.76963374e-01  -1.76092178e-01  -4.19185609e-01  -5.57702743e-02]
[ -5.94438076e-01   6.30120635e-01  -1.70207784e-01   3.10038358e-0
 1.84062332e-01  -2.34777853e-01   1.29535481e-01   1.36813134e-01]
[ -2.56274015e-01   2.74017543e-01   1.59810841e-01   3.73903001e-16
  -5.78984618e-01   6.36550903e-01  -3.32297325e-16  -3.05414885e-01]
[ -2.85637408e-01  -2.47912124e-01   3.54610324e-01  -7.31901303e-02
  4.45784479e-01   8.36141407e-02   5.48721075e-01  -4.68012422e-01]
[ -1.93139315e-01   3.38495038e-02  -5.00790417e-01  -4.28462476e-01
 3.47110212e-01   1.55483231e-01  -4.68663752e-01  -4.03576553e-01]
[ -3.05134684e-01  -2.93989003e-01  -2.23433599e-01  -1.91614240e-01
 1.27460942e-01   4.91219401e-01   2.09592804e-01   6.57535374e-01]
[ -1.82489842e-01  -1.61027774e-01  -3.97842437e-01  -3.83228481e-01
 -5.12923241e-01  -4.27574426e-01   4.19185609e-01  -1.18313827e-01]
[ -2.46898428e-01   1.57254755e-01   5.92991650e-01  -6.20076716e-01
 -3.21868137e-02  -2.31065080e-01  -2.59070963e-01   2.37976909e-01]]
[ 2.75726271  2.67824793  1.89221275  1.61803401  1.19154561  0.94833982
 0.61803401  0.56999218]
[[ -5.24124920e-01  -5.94438076e-01  -2.56274015e-01  -2.85637408e-01
 -1.93139315e-01  -3.05134684e-01  -1.82489842e-01  -2.46898428e-01]
[  5.72859168e-01  -6.30120635e-01  -2.74017543e-01   2.47912124e-01
 -3.38495038e-02   2.93989003e-01   1.61027774e-01  -1.57254755e-01]
[ -9.54463035e-02   1.70207784e-01  -1.59810841e-01  -3.54610324e-01
 5.00790417e-01   2.23433599e-01   3.97842437e-01  -5.92991650e-01]
[  3.83228481e-01   3.10038358e-01  -2.22044605e-16  -7.31901303e-02
 -4.28462476e-01  -1.91614240e-01  -3.83228481e-01  -6.20076716e-01]
[ -1.76963374e-01   1.84062332e-01  -5.78984618e-01   4.45784479e-01
 3.47110212e-01   1.27460942e-01  -5.12923241e-01  -3.21868137e-02]
[  1.76092178e-01   2.34777853e-01  -6.36550903e-01  -8.36141407e-02
 -1.55483231e-01  -4.91219401e-01   4.27574426e-01   2.31065080e-01]
[  4.19185609e-01  -1.29535481e-01  -3.33066907e-16  -5.48721075e-01
  4.68663752e-01  -2.09592804e-01  -4.19185609e-01   2.59070963e-01]
[ -5.57702743e-02   1.36813134e-01  -3.05414885e-01  -4.68012422e-01
 -4.03576553e-01   6.57535374e-01  -1.18313827e-01   2.37976909e-01]]

Here is the plot for the SVD of the preceding dataset: