Dot Product's Role in Similarity, Projections, and Gradients

Today I was looking at a derivative and noticed the dot product appearing in the definition. It made me wonder: what does the dot product — typically thought of as "a · b = ∑ aᵢbᵢ," measuring similarity in machine learning or handling geometric projections—have to do with directional derivatives in calculus?

Geometric Meaning of the Dot Product

At its core, the dot product is about alignment between vectors:

• Formula:

$$a \cdot b = \|a\| \|b\| \cos \theta$$

• It's largest when vectors point in the same direction ($\theta = 0$), zero when perpendicular ($\theta = 90^\circ$), and negative when opposed ($\theta > 90^\circ$).

Similarity Measures in ML

In machine learning, the dot product indicates similarity:

• Netflix user ratings vectors $u$ and $v$ use $u \cdot v$ to measure agreement—high ratings aligning boosts the sum.
• Cosine similarity normalizes this, but the foundation is the dot product.

Projections in Geometry

Geometrically, projecting a vector $b$ onto $a$ yields a "shadow" length:

$$\text{Projection length} = \frac{a \cdot b}{\|a\|}$$

The dot product thus quantifies how far $b$ reaches in the direction of $a$.

Directional Derivatives and Gradients

In calculus, the dot product naturally appears in directional derivatives:

• Given a scalar field $f$, moving in direction $v$, the rate of change is $\nabla f \cdot v$.
• Gradient $\nabla f$ points to steepest ascent; the dot product assesses how closely $v$ aligns with this direction.
• If $v \parallel \nabla f$, ascent is maximized; if $v \perp \nabla f$, there's no change; if $v$ opposes $\nabla f$, you descend quickest.

Python Example: Directional Derivative Alignment

Here's a straightforward Python snippet to illustrate directional derivatives clearly:

import numpy as np

# Gradient of f(x, y) = x² + y²
def grad_f(x, y):
    return np.array([2*x, 2*y])

point = np.array([1, 1])
v_aligned = grad_f(*point)          # direction of steepest ascent
v_perp = np.array([1, -1])          # perpendicular to gradient

aligned_rate = np.dot(grad_f(*point), v_aligned)
perpendicular_rate = np.dot(grad_f(*point), v_perp)

print("Aligned rate:", aligned_rate)
print("Perpendicular rate:", perpendicular_rate)

#  Aligned rate: 8
#  Perpendicular rate: 0

Pretty cool!

Key Takeaways

• Dot product fundamentally measures vector alignment.
• This alignment principle underpins similarity measures in ML, geometric projections, and directional derivatives in calculus.
• Recognizing this shared concept of "alignment" reveals the dot product's broad relevance.