Tutorials on Advanced Math and Computer Science Concepts

Dot Product and Length

From what we have discussed so far, we've seen that vectors can be represented as a line traveling through the Cartesian plane. If we think in 2 dimensions ($\mathbb{R}^2$), a vector looks like a linear function, with a defined start and endpoint. We also know that this arrow represents a magnitude and a direction in the space that it occupies. In this section, we are going to expand on those ideas, and derive some further information about the vectors themselves.

We discussed that vectors have a magnitude and a direction, but we never really discussed how we can find the magnitude of the vector. The formula for magnitude or length of a vector can be intuitively derived through the geometric interpretation of it. We will start by looking at the $\mathbb{R}^2$ vector since it is the easiest to see with our existing knowledge.

Suppose that we have the vector $\vec{v} = \left[\begin{array}{cc} 2 & 3 \end{array}\right]$ and we want to find $|\vec{v}|$, the magnitude of the vector $\vec{v}$. Let's start by drawing out what the vector looks like in $\mathbb{R}^2$. To draw the vector, we simply draw a line that starts at the origin, and ends at the coordinate x = 2, y = 3, as shown below.

The magnitude of the vector is the same as the length of the line in the space the vector occupies. So, in order to determine $|\vec{v}|$, we need to find the length of it. To do this, we just need to observe that we can create a right-angled triangle that includes our vector in it.

In this picture, we know two pieces of information. We know that the base of the triangle has a length of 2, and a height of 3. If we want to find the length of $\vec{v}$ now, we just need to apply Pythagorean's theorem. This would give us: $|\vec{v}|^2 = x^2 + y^2$

Solving this shows us that $|\vec{v}| =\sqrt{x^2 + y^2}$. From this, we have now derived a general formula for the magnitude of a vector. This applies to any vector in $\mathbb{R}^2$. If we want to apply this to $\mathbb{R}^n$, we simply need to add the remaining components, so $|\vec{v}| = \sqrt{x_1^2 + x_2^2 + ... + x_n^2}$

Keeping with the theme of triangles, another important property we can learn about a vector is the angle that it meets another vector at. You can see that having right angles allows us to conclude information about vectors using trigonometry. Due to this, knowing the angle between two vectors can help us conclude other useful information about them.

Like our length example, we can derive a formula for the angle between two vectors intuitively using the geometric interpretation.

Suppose we have a vector $\vec{v} = \left[\begin{array}{cc} 2 & 3 \end{array}\right]$ and $\vec{w} = \left[\begin{array}{cc} 4 & 1\end{array}\right]$, and we wand to find the angle between them. Let's start by graphing these two vectors to see what they look like.

We want to know the value of $\theta$, and we know the values $|\vec{v}|$ and $|\vec{w}|$. This is unfortunately not enough information to find $\theta$, so we need to work a little harder to find the answer.

You'll notice that the two vectors don't make a right-angled triangle, which means that most of our trigonometry won't be applicable. There is however one concept that we can make use of, which is the cosine rule. If you recall, the cosine law allows us to determine the angle of a triangle even if it is not a right angle. Using this, we can find the angle of the vectors.

First, we need to create a triangle that we can apply cosine law to. The question is, how can we connect $\vec{v}$ and $\vec{w}$ in a way that we know the length of the connection itself? Suppose we were to connect the two vectors as shown below.

We know that the length of this connection is $\vec{w} - \vec{v}$ because we are drawing the line between the two endpoints, so we simply need to subtract the x and y coordinates to get the corresponding line connecting them. Now that we have this, we can apply the cosine law to get:

$|\vec{w} - \vec{v}|^2 = |\vec{w}|^2 + |\vec{v}|^2 - 2|\vec{w}||\vec{v}|cos(\theta)$

Now at this point, we encounter another small problem. We haven't discussed how to compute something like $\vec{w}*\vec{v}$. We know how to do scalar multiplication, but in order to proceed, we need to discuss how to multiply two vectors.

We refer to this operation as a dot product. The dot product is defined as the sum of each component together. For instance, the vectors in $\mathbb{R}^2$, the dot product is defined as $x_1y_1+x_2y_2$. Using this knowledge, we can simplify the equation further.

To start, notice that the left-hand side can be rewritten as:
$|\vec{w} - \vec{v}|^2 = (\vec{w}-\vec{v})(\vec{w}-\vec{v})$
$=|\vec{w}|^2 - 2\vec{w}\vec{v} + |\vec{v}|^2$

Substituting this back into cosine law equation gives us:

$|\vec{w}|^2-2\vec{w}\vec{v}+|\vec{v}|^2 = |\vec{w}|^2 + |\vec{v}|^2 - 2|\vec{w}||\vec{v}|cos(\theta)$
$-2\vec{w}\vec{v} = -2\vec{w}\vec{v}cos(\theta)$

This gives us a nice clean formula to use when determining the angle between two vectors. Let's try an example to see how this formula is used.

Example: Find the angle between the vectors $\vec{v} = \left[\begin{array}{cc} 1 & 2 & -1 \end{array}\right]$ and $\vec{w} = \left[\begin{array}{cc} 1 & -1 & -1 \end{array}\right]$

To do this, we need to determine $\vec{v}*\vec{w}$, as well as $|\vec{v}|$ and $|\vec{w}|$. Once we have this we can find the angle.

$\vec{v}*\vec{w} = 1*1+2*(-1)+(-1)*(-1) = 1-2+1 = 0$

From here, we have the equation $0 = |\vec{w}||\vec{v}|cos(\theta)$. Since the left hand side is 0, we don't even need to find $|\vec{v}|$ or $|\vec{w}|$, since they will result in 0 anyways. From here, we just need to take $cos^{-1}(0) = 90$ degrees.

This example reveals an important property of the dot product. If the dot product of two vectors is equal to 0, then the vectors are perpendicular to each other, meaning they meet at a 90-degree angle. This idea comes up frequently in linear algebra, so it is good to be familiar with the property to make it easy to determine if vectors are perpendicular. We will often refer to perpendicular vectors as orthogonal.

There are a few other theories that are useful to have for dot products. Let $\vec{x}$ and $\vec{y}$ be vectors in $\mathbb{R}^n$, and let t be some constant real number. The following theorems and properties hold true.

  1. $\vec{x}*\vec{x} = 0$ if and only if $\vec{x} = \vec{0}$
  2. $\vec{x}*\vec{y} = \vec{y}*\vec{x}$
  3. $\vec{x}*(\vec{y}+\vec{z}) = \vec{x}*\vec{y} + \vec{x}*\vec{z}$
  4. $(t\vec{x}) + \vec{y} = \vec{x} * (t\vec{y})$

There are additionally a few length based theorems that are important to know. Let $\vec{x}$ and $\vec{y}$ be vectors in $\mathbb{R}^n$, and let t be come constant real number. The following theorems and properties are true:

  1. $|\vec{x}| = 0$ if and only if $\vec{x}=0$
  2. $|t\vec{x}|=|t||\vec{x}|$
  3. $|\vec{x}*\vec{y}| \le |\vec{x}||\vec{y}|$ if and only if $\vec{x}$ and $\vec{y}$ are linearly dependent
  4. $|\vec{x}+\vec{y}| \le |\vec{x}|+|\vec{y}|$
  5. If $|\vec{x}| = 1$, then $\vec{x}$ is referred to as a unit vector.

Theorem 3 is often referred to as the Cauchy-Schwarz Inequality, and theorem 4 is often referred to as the triangle inequality. They both appear frequently in linear algebra, as well as other areas of mathematics.