Vectors & Dot Product

The dot product of two vectors multiplies matching coordinates and adds the results: w·x = w₁x₁ + w₂x₂. That same number equals |w|·|x|·cos θ, so its sign is decided entirely by the angle between the vectors while its size scales with their lengths. This one multiply-and-add loop is the scoring operation inside linear classifiers, neural network layers, attention heads, and vector search.

Here is the strange part: the loop knows nothing about geometry. It never computes an angle, never measures a length. It just multiplies slot 1 by slot 1, slot 2 by slot 2, and adds. And yet the number it returns behaves like a protractor. Point two vectors the same way and the score is large and positive; turn them to a right angle and it collapses to exactly zero; turn further and it goes negative. A perceptron's score, a neuron's pre-activation, an attention logit, a "which document is most similar" ranking: each one is this loop wearing a different hat. Learn what the number means once, and you have pre-paid for most of the AI curriculum.

How the Dot Product Works

The algebra is a two-line recipe. Walk both vectors slot by slot, multiply the entries that share a slot, and accumulate: total += w[i] * x[i]. For 2-D vectors that is two multiplications and one addition; for the thousand-dimensional vectors inside a real model it is the same loop with a bigger i.

The geometry hides in three consequences of that loop. First, a vector dotted with itself gives its squared length, so |v| = √(v·v): the dot product can measure. Second, dividing w·x by |w|·|x| leaves cos θ, the cosine of the angle between the vectors: the dot product can compare directions. Third, |x|·cos θ is the signed length of x's shadow when you project it onto w's line (negative when the shadow points away from w), which turns the whole identity into a picture: w·x = |w| × shadow. That one you can check with a ruler on the plane, and the loop that produced the number never mentioned geometry at all.

The sign rule falls straight out of the cosine. Within 90° of w, cos θ > 0 and the score is positive. At exactly 90°, cos θ = 0 and the score is zero, however long the vectors are. Beyond 90° the score is negative. So the equation w·x = 0 carves the plane into two half-planes with a line through the origin, perpendicular to w. That line is the decision boundary of a linear classifier with no bias term; add a bias b and the same line slides off the origin to w·x + b = 0, keeping its direction. Either way, this lesson is where it comes from.

Raw Dot vs Cosine Similarity

Two ingredients go into one number, and the practical trouble starts when you only care about one of them. Ranking candidates by a raw dot product silently mixes in length: among positive matches, a long, mediocre one can outscore a short, perfect one, because doubling a vector doubles its score without moving its direction one degree. (Doubling a negative score makes it more negative; a zero score stays zero.) If what you mean is "most similar in direction", divide the score by both lengths and rank by cos θ instead. That is all cosine similarity is, and it is why cosine-based embedding search often normalizes vectors before indexing them.

Keep the raw dot product when length carries real information, which is the usual case inside a model. A perceptron score, a neuron's pre-activation, and an attention logit all want the magnitude, when magnitude is part of the learned scoring rule; length is not automatically calibrated confidence. Reach for cosine when you are comparing two things that happen to live in the same space and the lengths are an artifact of how they were produced.

Vectors & Dot Product Step by Step

The default input is w = (3, 1) and x = (1, 2), chosen so every number below can be checked in your head.

The algebra chapter (beats 1–4). Both arrows appear on the plane; the ledger on the left records the coordinates and nothing else is known yet. Then the loop runs. Dashed drop-lines fall from both arrow tips to the horizontal axis, isolating the first coordinates, and the term w₁·x₁ = 3 × 1 = 3 commits to the ledger. The drop-lines move to the vertical axis for w₂·x₂ = 1 × 2 = 2. The terms add: w·x = 3 + 2 = 5. That is the entire algorithm. The rest of the lesson is about what the 5 means.

The geometry chapter (beats 5–7). Each vector measures itself: |w| = √(w·w) = √10 ≈ 3.16 and |x| = √5 ≈ 2.24. Dividing gives cos θ = 5 / (√10 × √5) = 1/√2 ≈ 0.707, the cosine of 45°, so θ = 45°, which is why this default was picked. Then the identity beat: a perpendicular drops from x's tip onto w's line, casting a shadow of length |x|·cos θ ≈ 1.58. The foot of that perpendicular is a place on the plane, so read the shadow against the axis ticks if you want to check it. Multiply: |w| × shadow = 3.16 × 1.58 = 5.0, the same 5 the loop computed in beat 4. Algebra and geometry agree, and that agreement is the identity w·x = |w|·|x|·cos θ.

Experiment one: hold the angle, change the length (beat 8). A dashed copy of x at twice the length appears alongside it, pointed the same way. The shadow doubles with it, 1.58 becomes 3.16, so |w| × shadow doubles too: w·(2x) = 10, exactly 2 × 5. Meanwhile the ledger keeps cos θ = 0.707 lit while everything else fades, because that is the number the experiment holds fixed. Length moved the size of the score and left the direction verdict untouched.

Experiment two: hold the length, change the angle (beats 9–13). Now the reverse. A dashed ghost of x rotates around the plane at fixed length while the real x dims, and the loop re-scores it at five stations off w. At 0° the score is +7.07, the maximum possible, |w|·|x|. At 45°, where the real x sits, it scores +5. At 90° the score is exactly 0, and no choice of lengths could change that. At 135° it goes negative, −5; at 180°, opposed, it bottoms out at −7.07. Across five stations the score gauge has traced a cosine from +|w||x| to −|w||x| without ever plotting one.

The verdict (beat 14). w·x = 5 > 0, so x is within 90° of w. That is the whole reading: the sign says which way x leans, and the two experiments just showed that no length can change it.

The boundary (beat 15). The line w·x = 0 appears through the origin, perpendicular to w, and the half-plane where scores come out positive is tinted. Every vector on that side scores positive, every vector on the other side negative, and that split is what a linear classifier uses. The line passes through the origin here because there is no bias term; a classifier that learns a bias b draws the parallel line w·x + b = 0 instead.

On the canvas, the left ledger accumulates the algebra while the plane carries the geometry; each beat lights up exactly one new fact. If you change the vectors, watch whether the score's sign flips before its size changes. That is the angle/length separation happening live.

Time and Space Complexity

A dot product of d-dimensional vectors costs d multiplications and d − 1 additions: O(d) time, O(1) extra space. There is no clever variant to reach for; the interesting scaling is in how many dot products a system performs. A single transformer attention layer computes one per query–key pair, and a vector-search index may score millions of candidates per lookup, which is why this loop is a major target of numerical hardware optimization. Write the Python loop once to see the shape of it, then never write it again: np.dot, torch.matmul and everything under them hand the work to BLAS kernels that beat an interpreted loop by orders of magnitude.

Edge Cases

• Zero vector. A zero vector has no direction, so cos θ = w·x / (|w|·|x|) divides by zero. The input panel rejects it; every other integer pair in the ±5 grid is legal.
• Perpendicular vectors — try the preset. Two respectable, full-length vectors score exactly 0. "No alignment" reads as "nothing", which is precisely the property that makes w·x = 0 usable as a decision boundary.
• Opposed vectors. Beyond 90° the score is negative. Not an error, a direction verdict. A linear classifier reads it as "the other class."
• Collinear vectors. At θ = 0° the shadow is all of |x| and the score is the full |w|·|x|; at 180° it is the full negative. The cosine can't exceed these bounds, so neither can the score.
• Resizing x. Beat 8 resizes x in place with the angle pinned, and the score follows the factor: 5 becomes 10 on the default input while cos θ stays 0.707. It doubles x when the longer copy still fits the ±5 plane and halves it otherwise, since an x with a ±5 coordinate has nowhere to grow into. The double-length preset re-runs the whole trace on the longer vector if you want the ledger arithmetic in full. Doubling both vectors quadruples the score.

Common Mistakes

• Cross-multiplying slots. w₁·x₂ + w₂·x₁ is not the dot product; matching indexes multiply. The cross terms do build something in 2-D, but only in the other order: w₁x₂ − w₂x₁ is the 2-D cross product, the signed area of the parallelogram the two vectors span. It peaks where the dot product is zero.
• Adding instead of multiplying. Summing w_i + x_i produces sum(w) + sum(x), a number that no longer scales with x in general (only when sum(w) happens to be 0) and never lets w act as a per-slot weight. Multiplication is what makes w·x a weighted sum.
• Reading the score as a distance. A large w·x does not mean the vectors are close together; it means they point the same way and are long. Two distant points can have a huge dot product.
• Comparing raw scores across different lengths. Ranking candidates by w·x silently favors long vectors among positive matches. If you mean "most similar in direction", divide by the lengths and rank by cos θ.
• Treating a negative score as failure. The sign is information (which side of the boundary), not an error code. Perceptrons and margins are built on exactly this signed reading.