Dot Product

3.3. Dot Product#

draft version - do not distribute

3.3.1. Dot Product for NumPy Arrays#

The dot product (also called the scalar product) is an operation that takes two equal-length vectors and returns a single number. The dot product between two arrays is the sum of the element-wise multiplication of the two arrays. For two vectors \(\mathbf{a}\) and \(\mathbf{b}\), the dot product is

(3.5)#\[\begin{align} \mathbf{a} \cdot \mathbf{b} = \sum_{i=1}^{n} a_i b_i. \end{align} \]

The dot product is denoted by the “dot” \(\cdot\) symbol, which should not be confused with regular multiplication.

As an explicit example, consider the two arrays \(\mathbf{a} = [1, 2, 3]\) and \(\mathbf{b} = [4, 5, 6]\). The dot product between these two arrays is

(3.6)#\[\begin{equation} \mathbf{a} \cdot \mathbf{b} = \begin{bmatrix} 1 & 2 & 3 \end{bmatrix} \begin{bmatrix} 4 \\ 5 \\ 6 \end{bmatrix} = 1 \cdot 4 + 2 \cdot 5 + 3 \cdot 6 = 4 + 10 + 18 = 32. \end{equation} \]

Note that in the above, we are following matrix multiplication rules: a row vector times a column vector, each with an equal number of elements. You cannot take the dot product between two arrays of unequal size.

Here is the same example in code:

import numpy as np
a = np.array([1, 2, 3])
b = np.array([4, 5, 6])
result = np.dot(a, b)
print(result)  # Output: 32

The operation can also be written in index notation as:

(3.7)#\[\begin{equation} \mathbf{a} \cdot \mathbf{b} = \sum_{i=1}^{n} a_i b_i. \end{equation} \]

For an n-dimensional array, this can be written in a more Pythonic way using zero-based indexing:

(3.8)#\[\begin{equation} \mathbf{a} \cdot \mathbf{b} = \sum_{i=0}^{n-1} \mathbf{a}[i] \cdot \mathbf{b}[i]. \end{equation} \]

3.3.1.1. Important Properties#

Symmetry: The dot product is symmetric in its arguments: \(\mathbf{a} \cdot \mathbf{b} = \mathbf{b} \cdot \mathbf{a}\)
Linearity: \(\mathbf{a} \cdot (c\mathbf{b} + \mathbf{d}) = c(\mathbf{a} \cdot \mathbf{b}) + (\mathbf{a} \cdot \mathbf{d})\) for scalar \(c\)
Geometric interpretation: \(\mathbf{a} \cdot \mathbf{b} = |\mathbf{a}||\mathbf{b}|\cos\theta\), where \(\theta\) is the angle between the vectors

3.3.1.2. NumPy Implementation#

NumPy provides multiple ways to compute the dot product:

import numpy as np
a = np.array([1, 2, 3])
b = np.array([4, 5, 6])

# Method 1: np.dot()
result1 = np.dot(a, b)

# Method 2: @ operator (preferred in modern Python)
result2 = a @ b

# Method 3: .dot() method
result3 = a.dot(b)

print(f"All methods give the same result: {result1 == result2 == result3}")

import numpy as np
import matplotlib.pyplot as plt
from matplotlib import animation
from matplotlib.animation import FuncAnimation
from IPython.display import HTML


a = np.array([1, 2, 3])
b = np.array([4, 5, 6])

result = np.dot(a, b)
print(result) 
print(a@b)
print(a.dot(b))
print(b.dot(a))

from quiz_dot_product_utils import dot_product_I
display(dot_product_I())

Determine which of the following pairs of vectors have a non-zero dot product. (Use pencil and paper.)

Solution:

Correct Answer: [-2, -2], [2, 2]
Explanation: Use the code above to check the dot products

Below is a code to compute the dot product. NumPy already has this functionality in the np.dot method. But writing a from scratch code is a good way to practice manipulating array indices. We will test this function by using the np.random.rand function to generate arrays composed of random numbers from 0 to 1. Also, notice the ValueError raised if the two arrays do not have the same size.

def array_dot(a,b): 

    n1=a.size
    n2=b.size
    if n1 != n2:
        raise ValueError("The two arrays must be have the same number of elements to take the dot product." ) 

    c=0
    for i in range(n1):
        c+= a[i]*b[i]
    return c 

# example arrays to test arra_dot 
a = np.random.rand(4)
b = np.random.rand(4) 
c= np.random.rand(3)

print(np.dot(a,b))
print(array_dot(a,b))
print(array_dot(a,c))

0.930535586937742
0.9305355869377419

---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
Cell In[4], line 8
      6 print(np.dot(a,b))
      7 print(array_dot(a,b))
----> 8 print(array_dot(a,c))

Cell In[3], line 6, in array_dot(a, b)
      4 n2=b.size
      5 if n1 != n2:
----> 6     raise ValueError("The two arrays must be have the same number of elements to take the dot product." ) 
      8 c=0
      9 for i in range(n1):

ValueError: The two arrays must be have the same number of elements to take the dot product.

3.3.2. Geometric Meaning of Dot Product#

The dot product is a measure of how much two vectors overlap. There are two ways to compute the dot product:

The dot product is the sum of the multiplication of each vector component \begin{equation} a \cdot b = \displaystyle \sum_i a_i b_i. \end{equation}
The dot product is the magnitude of vector \(a\) times the magnitude of vector \(b\) multiplied by the cosine of the angle between the two vectors.

(3.9)#\[\begin{equation} a\cdot b = | a| |b| \cos \theta, \end{equation} \]

where \(|a|\) is the magnitude of \(a\) and \(|b|\) is the magnitude of \(b\). In this case, we are considering the arrays akin to physical vectors. The magnitudes are computed by the square root of the sum of each element squared (the Pythagorean Theorem)

(3.10)#\[\begin{equation} | a | = \sqrt{a_1^2 + a_2^2 + ... + a_n^2}. \end{equation} \]

You can obtain the magnitude of an array with np.linalg.norm. I will expemplify that in code below and also write a from scratch code to do the same operation.

def norm_array(a):
    norm = 0 
    for val in a:
        norm += val**2
    return np.sqrt(norm) 

a=np.array([1,2,3,4]) 

# test with NumPy and from scratch code.
print(np.linalg.norm(a) )
print(norm_array(a))

5.477225575051661
5.477225575051661

Here is a code that exemplifies the equivalence between the two dot product algorithms. To show the equivalence, the vector \(a\) is rotated by a rotation matrix by a known angle \(\theta\). The rotation matrix (in two dimensions) is

(3.11)#\[\begin{equation} R(\theta) = \begin{bmatrix} \cos \theta & - \sin \theta \\ \sin \theta & \cos \theta \end{bmatrix} . \end{equation} \]

# lambda function for rotation matrix 
R = lambda theta : np.array([[ np.cos(theta), - np.sin(theta)],[np.sin(theta), np.cos(theta)]])

# angle 
theta = np.pi/6

# rotation matrix 
R_mat=R(theta) 

a = np.array([1, 3])
# rotate a to get b 
b = np.matmul(R_mat, a) 

print("Dot product by first algorithm:", np.dot(a,b))
print("Dot product by second algorithm:", np.linalg.norm(a)*np.linalg.norm(b)*np.cos(theta))

Dot product by first algorithm: 8.660254037844387
Dot product by second algorithm: 8.660254037844389

Below is an animation showing the rotation of a vector and the computation of the dot product between the rotated vector and the original vector. This visualization will help you understand the geometric meaning of the angle between two vectors and how it affects their dot product.

# Parameters
theta = np.pi  # Total rotation angle (adjust as needed)
a = np.array([3, 2])  # Original vector (adjust as needed)

# Generate rotation angles and rotated vectors
theta_array = np.linspace(0, theta/2, 15) 
theta_array = np.r_[theta_array, theta_array[1:] + np.pi/2]
nt = theta_array.size


b_array = np.zeros((nt, 2))
for i in range(nt):
    R_mat = R(theta_array[i])
    b_array[i, :] = np.matmul(R_mat, a) 

# Set up the figure and axis
fig, ax = plt.subplots(figsize=(4, 4))
ax.set_xlim(-5, 5)
ax.set_ylim(-5, 5)
ax.set_aspect('equal')
ax.grid(True, alpha=0.3)
ax.set_xlabel('X')
ax.set_ylabel('Y')
ax.set_title('Vector Rotation Animation')

# Draw the original vector (static)
ax.arrow(0, 0, a[0], a[1], head_width=0.15, head_length=0.15, 
         fc='black', ec='black', label='original vector a', linewidth=2)

# Initialize the animated arrow (will be updated in animation)
animated_arrow = ax.annotate('', xy=(b_array[0, 0], b_array[0, 1]), xytext=(0, 0),
                           arrowprops=dict(arrowstyle='->', color='red', lw=2))

# Add legend and text annotations
ax.legend()
angle_text = ax.text(0.02, 0.95, '', transform=ax.transAxes, fontsize=10,
                    bbox=dict(boxstyle='round', facecolor='wheat', alpha=0.8))

# Add dot product text
dot_product_text = ax.text(0.02, 0.85, '', transform=ax.transAxes, fontsize=10,
                          bbox=dict(boxstyle='round', facecolor='lightgreen', alpha=0.8))

plt.close()

def animate(frame):
    """Animation function called for each frame"""
    # Update the arrow position
    animated_arrow.xy = (b_array[frame, 0], b_array[frame, 1])
    
    # Calculate dot product between a and current b
    current_b = b_array[frame, :]
    dot_product = np.dot(a, current_b)
    
    # Calculate magnitudes
    # mag_a = np.linalg.norm(a)
    # mag_b = np.linalg.norm(current_b)
    # cos_angle = dot_product / (mag_a * mag_b)
    
    # Update angle text
    current_angle = theta_array[frame] * 180 / np.pi  # Convert to degrees
    angle_text.set_text(f'Angle: {current_angle:.1f}°')
    
    # Update dot product text
    dot_product_text.set_text(f'a·b = {dot_product:.2f}')

    
    return animated_arrow, angle_text, dot_product_text, magnitude_text

# Create and run the animation
anim = FuncAnimation(fig, animate, frames=nt, interval=200, blit=True, repeat=True)

# Display 
HTML(anim.to_html5_video())

dot product between an array and the same array rotated by an angle

3.3.3. Angle Between Two Vectors#

The angle \(\theta\) between two vectors \(a\) and \(b\) follows from the dot product formulas

(3.12)#\[\begin{equation} \cos \theta = \frac{ a \cdot b}{|a| |b|} = \frac{1}{|a| |b|}\displaystyle \sum_i a_i b_i. \end{equation} \]

Here is a code that computes the angle between two vectors.

a=np.array([1,2,3,4]) 
b=np.array([5, 6, 7, 8]) 

cos_theta=np.dot(a,b)/np.linalg.norm(a)/np.linalg.norm(b) 
np.arccos(cos_theta) 

np.float64(0.2501959204225113)

3.3.4. Intuition About the Dot Product#

The dot product is a measure of how much two vectors (or arrays) point in the same direction. You can think of it as a kind of “overlap” between them.

If two vectors point in the same direction → the dot product is positive
If they are perpendicular (the angle between them is 90 degrees) → there is no overlap → the dot product is zero
If they point in opposite directions → the overlap is negative → the dot product is negative
If two vectors are linearly dependent (one is a scalar multiple of the other) → the dot product has maximum absolute value for their given magnitudes

Another way to think about the dot product \(\mathbf{a} \cdot \mathbf{b}\) is “how much of \(\mathbf{b}\) lies along \(\mathbf{a}\)” — it’s the projection of one vector onto the other, scaled by the magnitude of the vector being projected onto. Since the dot product is symmetric (\(\mathbf{a} \cdot \mathbf{b} = \mathbf{b} \cdot \mathbf{a}\)), it is also “how much of \(\mathbf{a}\) lies along \(\mathbf{b}\)”.

# example of orthogonal arrays
a = np.array([1,0,0])
b = np.array([0, 1, 0]) 
cos_theta=np.dot(a,b)/np.linalg.norm(a)/np.linalg.norm(b) 
# convert to degrees 
np.arccos(cos_theta)*180/np.pi 

np.float64(90.0)

# example of parallel arrays
a = np.array([1,0,0])
b = np.array([100, 0, 0]) 
cos_theta=np.dot(a,b)/np.linalg.norm(a)/np.linalg.norm(b) 
# convert to degrees 
np.arccos(cos_theta)*180/np.pi 

np.float64(0.0)

# example of anit-parallel arrays
a = np.array([1,0,0])
b = np.array([-10, 0, 0]) 
cos_theta=np.dot(a,b)/np.linalg.norm(a)/np.linalg.norm(b) 
# convert to degrees 
np.arccos(cos_theta)*180/np.pi 

np.float64(180.0)

from quiz_dot_product_utils import dot_product_II
display(dot_product_II())

Which of the conditions is not a guarantee that two vectors have a dot product of zero?