Dot Product

3.2. Dot Product#

draft version - do not distribute

3.2.1. Dot Product for NumPy Arrays#

The dot product (also called the scalar product) is an algebraic operation that takes two equal-length vectors and returns a single number. Consider two arrays a= [ 1, 2, 3], b =[ 4, 5, 6]. The dot product between these two arrays is

(3.5)#\[\begin{equation} a\cdot b = \begin{bmatrix} 1 & 2 & 3 \end{bmatrix} \begin{bmatrix} 4 \\ 5 \\ 6 \end{bmatrix} = 1\cdot 4 + 2 \cdot 5 + 3 \cdot 6 = 4 + 10 + 18 = 32 \end{equation} \]

Here is the same example but in code:

import numpy as np

a = np.array([1, 2, 3])
b = np.array([4, 5, 6])

dot = np.dot(a, b)
print(dot)  

The operation can be written as

(3.6)#\[\begin{equation} a \cdot b = \displaystyle \sum_i a_i b_i. \end{equation} \]

For a n-dimensional array the above can be written in a more Pythonic way as

(3.7)#\[\begin{equation} a \cdot b = \displaystyle \sum_{i=0}^{n-1} a[i] b[i]. \end{equation} \]

Note, the dot product is symmetric in it’s arguments \(a\cdot b= b\cdot a\).

import numpy as np

a = np.array([1, 2, 3])
b = np.array([4, 5, 6])

dot = np.dot(a, b)
print(dot)  

Below is a code to compute the dot product. NumPy already has this functionality in the np.dot method. But writing a from scratch code is a good way to practice manipulating array indices. We will test this function by using the np.random.rand function to generate arrays composed of random numbers from 0 to 1. Also, notice the ValueError raised if the two arrays do not have the same size.

def array_dot(a,b): 

    n1=a.size
    n2=b.size
    if n1 != n2:
        raise ValueError("The two arrays must be have the same number of elements to take the dot product." ) 

    c=0
    for i in range(n1):
        c+= a[i]*b[i]
    return c 

# example arrays to test arra_dot 
a = np.random.rand(4)
b = np.random.rand(4) 
c= np.random.rand(3)

print(np.dot(a,b))
print(array_dot(a,b))
print(array_dot(a,c))

1.0729740452427476
1.0729740452427476

---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
/var/folders/mq/8_f0y54n1g3f0ht8yb9b6by40000gn/T/ipykernel_41738/4276148387.py in <cell line: 0>()
      6 print(np.dot(a,b))
      7 print(array_dot(a,b))
----> 8 print(array_dot(a,c))

/var/folders/mq/8_f0y54n1g3f0ht8yb9b6by40000gn/T/ipykernel_41738/2637734469.py in array_dot(a, b)
      4     n2=b.size
      5     if n1 != n2:
----> 6         raise ValueError("The two arrays must be have the same number of elements to take the dot product." )
      7 
      8     c=0

ValueError: The two arrays must be have the same number of elements to take the dot product.

3.2.2. Geometric Meaning of Dot Product#

The dot product is a measure of how much two vectors overlap. There are two ways to compute the dot product:

The dot product is the sum of the multiplication of each vector component \begin{equation} a \cdot b = \displaystyle \sum_i a_i b_i. \end{equation}
The dot product is the magnitude of vector \(a\) times the magnitude of vector \(b\) multiplied by the cosine of the angle between the two vectors.

(3.8)#\[\begin{equation} a\cdot b = | a| |b| \cos \theta, \end{equation} \]

where \(|a|\) is the magnitude of \(a\) and \(|b|\) is the magnitude of \(b\). In this case, we are considering the arrays akin to physical vectors. The magnitudes are computed by the square root of the sum of each element squared (the Pythagorean Theorem)

(3.9)#\[\begin{equation} | a | = \sqrt{a_1^2 + a_2^2 + ... + a_n^2}. \end{equation} \]

You can obtain the magnitude of an array with np.linalg.norm. I will expemplify that in code below and also write a from scratch code to do the same operation.

def norm_array(a):
    norm = 0 
    for val in a:
        norm += val**2
    return np.sqrt(norm) 

a=np.array([1,2,3,4]) 

# test with NumPy and from scratch code.
print(np.linalg.norm(a) )
print(norm_array(a))

5.477225575051661
5.477225575051661

Here is a code that exemplifies the equivalence between the two dot product algorithms. To show the equivalence, the vector \(a\) is rotated by a rotation matrix by a known angle \(\theta\). The rotation matrix (in two dimensions) is

(3.10)#\[\begin{equation} R(\theta) = \begin{bmatrix} \cos \theta & - \sin \theta \\ \sin \theta & \cos \theta \end{bmatrix} . \end{equation} \]

# lambda function for rotation matrix 
R = lambda theta : np.array([[ np.cos(theta), - np.sin(theta)],[np.sin(theta), np.cos(theta)]])

# angle 
theta = np.pi/6

# rotation matrix 
R_mat=R(theta) 

a = np.array([1, 3])
# rotate a to get b 
b = np.matmul(R_mat, a) 

print("Dot product by first algorithm:", np.dot(a,b))
print("Dot product by second algorithm:", np.linalg.norm(a)*np.linalg.norm(b)*np.cos(theta))

Dot product by first algorithm: 8.660254037844387
Dot product by second algorithm: 8.660254037844389

3.2.3. Angle Between Two Vectors#

The angle \(\theta\) between two vectors \(a\) and \(b\) follows from the dot product formulas:

(3.11)#\[\begin{equation} \cos \theta = \frac{ a \cdot b}{|a| |b|} . \end{equation} \]

Here is a code that computes the angle between two vectors.

a=np.array([1,2,3,4]) 
b=np.array([5, 6, 7, 8]) 

cos_theta=np.dot(a,b)/np.linalg.norm(a)/np.linalg.norm(b) 
np.arccos(cos_theta) 

np.float64(0.2501959204225113)

# example of orthogonal arrays
a = np.array([1,0,0])
b = np.array([0, 1, 0]) 
cos_theta=np.dot(a,b)/np.linalg.norm(a)/np.linalg.norm(b) 
# convert to degrees 
np.arccos(cos_theta)*180/np.pi 

np.float64(90.0)

# example of parallel arrays
a = np.array([1,0,0])
b = np.array([100, 0, 0]) 
cos_theta=np.dot(a,b)/np.linalg.norm(a)/np.linalg.norm(b) 
# convert to degrees 
np.arccos(cos_theta)*180/np.pi 

np.float64(0.0)

# example of anit-parallel arrays
a = np.array([1,0,0])
b = np.array([-10, 0, 0]) 
cos_theta=np.dot(a,b)/np.linalg.norm(a)/np.linalg.norm(b) 
# convert to degrees 
np.arccos(cos_theta)*180/np.pi 

np.float64(180.0)

3.2.4. Intuition About the Dot Product#

3.2.4.1. The Dot Product is a Measure of How Much Two Arrays Overlap#

The dot product is a measure of how much two vectors (or arrays) point in the same direction. You can think of it as a kind of “overlap” between them.

If two vectors point in the same direction → the dot product is positive.
If they are perpendicular → there is no overlap → the dot product is 0.
If they point in opposite directions → the overlap is negative.

3.2.4.2. The Dot Product is a Projection#

Another way to think about it:

The dot product \(a \cdot b\) is “how much of \(b\) lies along \(a\)” — it’s the projection of one vector onto the other, scaled by the magnitude. Since the dot product is symmetric \(a\cdot b = b\cdot a\), it is also “how much of \(a\) lies along \(b\)”.