Comparing sparse data using cosine similarity

When a data set has multiple empty fields, comparing the distance using the Manhattan or Euclidean metrics might result in skewed results. Cosine similarity measures how closely two vectors are oriented with each other. For example, the vectors (82, 86) and (86, 82) essentially point in the same direction. In fact, their cosine similarity is equivalent to the cosine similarity between (41, 43) and (43, 41). A cosine similarity of 1 corresponds to vectors that point in the exact same direction, and 0 corresponds to vectors that are completely orthogonal to each other.

As long as the angles between the two vectors are equal, their cosine similarity is equivalent. Applying a distance metric such as the Manhattan distance or Euclidean distance in this case produces a significant difference between the two sets of data.

The cosine similarity between the two vectors is the dot product of the two vectors divided by the product of their magnitudes.

How to do it...

Create a new file, which we will call Main.hs, and perform the following steps:

1. Implement main to compute the cosine similarity between two lists of numbers.

   main :: IO ()
   main = do
     let d1 = [3.5, 2, 0, 4.5, 5, 1.5, 2.5, 2]
     let d2 = [  3, 0, 0,   5, 4, 2.5,   3, 0]

2. Compute the cosine similarity.

     let similarity = dot d1 d2 / (eLen d1 * eLen d2)
     print similarity

3. Define the dot product and Euclidean length helper functions.

   dot a b = sum $ zipWith (*) a b  
   eLen a = sqrt $ dot a a

4. Run the code to print the cosine similarity.

   $ runhaskell Main.hs
   
   0.924679432210068
   

See also

If the data set is not sparse, consider using the Manhattan or Euclidean distance metrics instead, as detailed in the recipes Computing the Manhattan distance and Computing the Euclidean distance.