Social science researchers usually hear about eigenvalues and eigenvectors somewhere in their training, typically in the context dimension reduction (principal component analysis) or personality survey creation (factor analysis). In my experience, eigenvalues and eigenvectors are often described minimally as “some property a matrix has” which isn’t terribly useful information as matrices have all sorts of properties. As such, I think people are often left wondering where in the world these things come from. The point of this post is to take a simple example and demonstrate how eigenvalues and eigen vectors are derived.
Definition:
In short, by way of stretching or compressing, eigenvectors are the axes along which a linear transformation acts. The factors by which this transformation occurs are eigenvalues.\(^1 \)
\(A \) is a nXn matrix (often a correlation or variance-covariance matrix), \(λ \) is an eigenvalue of \(A \), and \(x \) is a vector (eigenvector) of \(A \) in relation to \(λ \) whereby:
\(Ax\) = \(λx\)
We can reform the above equation as:
\((A − λI)x = 0 \) where \(I \) equals a nxn identity matrix
Setup
For this example, we will work with the following 2x2 matrix:
\(A =\begin{bmatrix}4 & 6\\ 3 & 1\end{bmatrix} \).
As we are working with a 2x2 matrix the \(λI \) will take the following form:
\(λI = \begin{bmatrix}\lambda & 0\\ 0 & \lambda\end{bmatrix} \)
Finding Eigenvalues
We’ll take our matrix \(A \) and plug it in to the framework above:
\(\begin{bmatrix}4 & 6\\ 3 & 1\end{bmatrix} - \begin{bmatrix}\lambda & 0\\ 0 & \lambda\end{bmatrix} = \begin{bmatrix}4-\lambda & 6\\ 3 & 1-\lambda\end{bmatrix} \)
Next, we need to find the determinant of \(A − λI \) and set it equal to 0:
\((4 − λ)(1 − λ)−18 \) =
\(λ^2 − 5λ − 14 \) =
\((λ + 2)(λ − 7) \) = 0
So, \(λ \) OR our eigenvalues = -2,7
Finding our first eigenvector
We’ll start by returning to the \(A − λI \) matrix we created early and plug in the corresponding eigenvalue
Let’s being with \(λ \) = 7
\(\begin{bmatrix}-3 & 6\\ 3 & -6\end{bmatrix} \)
We can use a bit of row reduction to reduce this matrix. It looks like we can perform the following operation:
\(1(R_1) + R_2 → R_2 \)
After Reducing we are left with the following matrix:
\(\begin{bmatrix}-3 & 6\\ 0 & 0\end{bmatrix} \)
We now have the following equation: \(−3x_1 + 6x_2 = 0 \)
From here we can set \(x_2= 1 \) and solve
\(6 = 3x_1 \) thus:
\(x_1 \) = 2
So or first eigenvector is \(\begin{bmatrix}2\\ 1\end{bmatrix} \)
We can normalize this vector \(x \) by dividing \(x \) by the square root of \(x^tx \); that is, \(x/\sqrt{x^tx} \). We can compare what we’ve calculated to base R with the “eigen” function.
v<-matrix(c(2,1),nrow=2)
print(v/as.numeric(sqrt(t(v)%*%v)))
## [,1]
## [1,] 0.8944272
## [2,] 0.4472136
A<-matrix(c(4,6,3,1),nrow=2,byrow=TRUE)
eigen(A)$vectors[1:2]
## [1] 0.8944272 0.4472136
Also, remember, \(Ax = λx \). Again, we can take what we’ve calculated and make sure these two quantities are indeed equal.
A%*%matrix(c(2,1),nrow=2)
## [,1]
## [1,] 14
## [2,] 7
7*matrix(c(2,1),nrow=2)
## [,1]
## [1,] 14
## [2,] 7
Eigenvector Number 2
We will simply repeat the steps above with our second eigenvalue. We’ll move on to \(λ \) = −2
\(\begin{bmatrix}6 & 6\\ 3 & 3\end{bmatrix} \)
Again, we will need to do some row reduction:
\(−.5(R_1) + R_2 → R_2 \)
After reduction:
\(\begin{bmatrix}6 & 6\\ 0 & 0\end{bmatrix} \)
Multiply row 1 by 1/6:
\(\begin{bmatrix}1 & 1\\ 0 & 0\end{bmatrix} \)
We now have the following equation: \(1x_1 + 1x_2 = 0 \)
From here we can set \(x_2 = 1 \) and solve
\(1x_1= −1 \) thus:
\(x_1= −1 \)
So or second eigenvector is \(\begin{bmatrix}-1\\ 1\end{bmatrix} \)
We can perform the same set of checks we did above to ensure our calculations are correct.
v<-matrix(c(-1,1),nrow=2)
print(v/as.numeric(sqrt(t(v)%*%v)))
## [,1]
## [1,] -0.7071068
## [2,] 0.7071068
eigen(A)$vectors[3:4]
## [1] -0.7071068 0.7071068
Now we can compare what we’ve done to the “eigen” function in base R.
A%*%matrix(c(-1,1),nrow=2)
## [,1]
## [1,] 2
## [2,] -2
-2*matrix(c(-1,1),nrow=2)
## [,1]
## [1,] 2
## [2,] -2
Hopefully, you now have a basic understanding of where eigenvectors and eigenvalues come from, and they no longer seem like they just pop out of thin air!
1.https://math.stackexchange.com/questions/23312/what-is-the-importance-of-eigenvalues-eigenvectors