Our aim is to show that the group of orientation-preserving isometries of $\mathbb{R}^3$, that is

$SO(3) = \{ M \in GL(3,\mathbb{R}) \mid MM^T= \mathrm{Id}, \det(M) =1 \}$,

is a simple group. Here we use a geometric point of view, mixing the proofs that can be found in Stillwell’s book, Naive Lie theory, and Perrin’s book, Cours d’algèbre. We begin with an easy structure lemma:

Lemma 1: If $M \in SO(3)$, there exists an orthonormal basis in which $M$ may be written as

$\left( \begin{matrix} 1 & 0 & 0 \\ 0 & \cos(\theta) & -\sin(\theta) \\ 0 & \sin(\theta) & \cos(\theta) \end{matrix} \right)$

for some $\theta \in [0,2 \pi)$.

Proof. First, notice that

$\det(M- \mathrm{Id})= \det( M- \ MM^T)= \det(M) \cdot \det( \mathrm{Id}- M) = (-1)^3 \det( M- \mathrm{Id})$,

hence $\det(M- \mathrm{Id})=0$ ie. there exists $v \in \mathbb{R}^3 \backslash \{ 0 \}$ such that $Mv=v$. Let $W=v^{\bot}$. Then for all $w \in W$,

$\langle Mw,v \rangle = \langle w, Mv \rangle = \langle w,v \rangle=0$,

so $W$ is stable under $M$. Therefore, $M$ may be written as $\left( \begin{matrix} 1 & 0 \\ 0 & S \end{matrix} \right)$ in an orthonormal basis; in particular, because $MM^T= \mathrm{Id}$ and $\det(M)=1$, $S \in SO(2)$.

If $S= \left( \begin{matrix} a & b \\ c & d \end{matrix} \right)$, the equality $S^T S = \mathrm{Id}$ becomes $\left\{ \begin{array}{l} a^2+c^2=1 \\ b^2+d^2=1 \\ ab+cd=0 \\ ad-bc=1 \end{array} \right.$. The two first equations allow us to set $\left\{ \begin{array}{l} a= \cos(\theta), \ c= \sin(\theta) \\ d = \cos( \varphi), \ b= \sin(\theta) \end{array} \right.$ for some $\theta, \varphi \in [0,2\pi)$. Then, the two last equations give

$\left\{ \begin{array}{l} \cos(\theta) \sin(\varphi) + \sin(\theta) \cos(\varphi)=0 \\ \cos(\theta) \cos(\varphi)- \sin(\varphi) \sin(\theta) = 1 \end{array} \right.$, that is $\left\{ \begin{array}{l} \sin(\theta+ \varphi)=0 \\ \cos( \theta+ \varphi)=1 \end{array} \right.$.

Therefore, $\varphi=- \theta ~ \text{mod} ~ 2\pi$ and $M$ may be written as the mentionned matrix. $\square$

Definition: If $M$ is as in the lemma, we say that $M$ is a rotation of axis $v$ and angle $\theta$.

Lemma 2: Let $M \in SO(3)$ and $P \in SO(3)$. Suppose that $M$ is a rotation of axis $v$. Then $PMP^{-1}$ is a rotation of axis $Pv$.

Proof. Let $W = v^{\bot}$. Then $PMP^{-1}(Pv)=Pv$ and $PW=(Pv)^{\bot}$ is stable under $PMP^{-1}$. Therefore, according to lemma 1, $PMP^{-1}$ is a rotation of axis $Pv$. $\square$

Lemma 3: Following the notations given on the figure below, the rotation around $P$ of angle $\theta$ composed with the rotation around $Q$ of angle $\varphi$ is a rotation around $R$ whose angle equals to twice the angle at $R$ of the spherical triangle $PQR$.

Proof. Let $H_1,H_2$ be two planes intersecting along a line $D$. Let $r_1,r_2$ be the reflections with respect to $H_1,H_2$ respectively.  Then $r_1r_2$ fixes $D$ and it is not difficult to see that the restriction of $r_1r_2$ to the plane normal to $D$ is a rotation of angle twice the angle $\theta$ between $H_1$ and $H_2$. Therefore, according to lemma 1, $r_1r_2$ is a rotation of axis $D$ and of angle $2\theta$.

Thus, if $R_{\varphi}, R_{\theta}$ denotes the rotations around $Q, P$ of angles $\varphi,\theta$ respectively, and if $r_{\mathcal{L}}, r_{\mathcal{M}}, r_{\mathcal{N}}$ denotes the reflections with respect to $\mathcal{L},\mathcal{M}, \mathcal{N}$ respectively, we can write

$R_{\varphi}=r_{\mathcal{N}} r_{\mathcal{M}}$ and $R_{\theta}= r_{\mathcal{M}} r_{\mathcal{L}}$,

hence $R_{\varphi} R_{\theta}=r_{\mathcal{N}} r_{\mathcal{L}}$, that is $R_{\varphi} R_{\theta}$ is a rotation around $R$ of angle twice the angle at $R$ of the spherical triangle $PQR$. $\square$

Now, we give two lemmas to characterize the natural action of $SO(3)$ on the sphere $\mathbb{S}^2$.

Lemma 4: The action $SO(3) \curvearrowright \mathbb{S}^2$ is transitive, ie. for every $x,y \in \mathbb{S}^2$ there exists $g \in SO(3)$ such that $g \cdot x =y$.

In fact, it is a special case of the following lemma, which states that the action $SO(3) \curvearrowright \mathbb{S}^2$ is “as $2$-transitive as possible”.

Lemma 5: The action $SO(3) \curvearrowright \mathbb{S}^2$ is isometrically 2-transitive, ie. for every $x_1,x_2,y_1,y_2 \in \mathbb{S}^2$ satisfying $\| x_1 -y_1 \| = \| x_2 - y_2 \|$ there exists $g \in SO(3)$ such that $g \cdot x_1=x_2$ and $g \cdot y_1 = y_2$.

Proof. Let $R_1$ be a rotation sending $x_1$ to $x_2$, and $R_2$ be a rotation of axis $x_2=Rx_1$ sending $R_1y_1$ to $y_2$. Then $R := R_2 \circ R_1 \in SO(3)$ and $\left\{ \begin{array}{l} Rx_1= R_2(R_1x_1)=R_2x_2=x_2 \\ Ry_1= R_2(R_1y_1)=y_2 \end{array} \right. . \square$

Theorem: $SO(3)$ is a simple group.

Proof. Let $N \lhd SO(3)$ be a non-trivial normal subgroup. We first want to show that the action $N \curvearrowright \mathbb{S}^2$ is transitive. Let $u \in N \backslash \{ \mathrm{Id} \}$ be a rotation of axis $\mathbb{R}a$ for some $a \in \mathbb{S}^2$ and let $x \in a^{ \bot} \cap \mathbb{S}^2$. Because $u \neq \mathrm{Id}$, $u(x) \neq x$, so $d:= \| x-u(x) \| >0$.

Notice that $u$ sends the meridian $L$ passing through $x$ to the meridian passing through $u(x)$, so $\|z-u(z) \|$ runs over $[0,d]$ when $z$ runs over $L$: $\|z -u(z) \|$ tends to $0$ (resp. $d$) when $z$ tends to the north pole (resp. to the equator). More precisely, if $0 \leq m \leq d$, it is possible to find so $\lambda \geq 0$ such that

$\displaystyle \frac{ \| u(x+ \lambda a) - (x+ \lambda a) \| }{ \| x+ \lambda a \| } =m$.

Because $u$ is linear and $x \in a^{\bot}$, the above equation is equivalent to

$\displaystyle d=m \| x+ \lambda a \| = \sqrt{ \|x\|^2 + \lambda^2 \| a \|^2} = \sqrt{1+ \lambda^2}$, hence $\displaystyle \lambda = \sqrt{ \frac{d^2}{m^2} -1 }$.

Let $v,w \in \mathbb{S}^2$ such that $\| v - w \| \leq d$. We just saw that there exists $y \in \mathbb{S}^2$ such that $\| u(y)-y \| = \|v-w \|$. According to lemma 5, there exists $g \in SO(3)$ such that $g(v)=y$ and $g(w)=u(y)$. Then $gug^{-1} \in N$ because $N$ is a normal subgroup and $gug^{-1}(v)=w$.

Therefore, we proved that the action $N \curvearrowright \mathbb{S}^2$ is transitive “for the small distances”. Now, if $v,w \in \mathbb{S}^2$ are any points, clearly there exists a sequence of points $v=p_0,p_1,\dots,p_n=w \in \mathbb{S}^2$ satisfying $\| p_i - p_{i+1} \| \leq d$ for all $0 \leq i \leq n-1$. In particular, for all $0 \leq i \leq n-1$, there exists $u_i \in N$ such that $u_i(p_i)=p_{i+1}$, and finally $u_{n-1} \cdots u_0$ is an element of $N$ sending $v$ to $w$. Thus, the action $N \curvearrowright \mathbb{S}^2$ is transitive.

In particular, there exist $x \in \mathbb{S}^2$ and $u \in N$ such that $u(x)=-x$; such an element of $SO(3)$, with $-1$ as an eigenvalue, is called a codimension-two reflection. Clearly, using lemmas 1 and 2, two codimension-two reflections are always conjugated. Therefore, since $N$ is a normal subgroup, any codimension-two reflection belongs to $N$.  We deduce, according to lemma 3 and using the notations of the figure below, that the rotation of angle $\theta$ around $R$ belongs to $N$.

Clearly, when $P$ runs over the equator, $\theta$ runs over $[0, \pi)$. Therefore, $N$ contains rotations of any angle. Because two rotations of the same angle are conjugated and $N$ is a normal subgroup, we deduce that any rotation belongs to $N$, that is $N= SO(3)$. $\square$

Nota Bene: The conclusion of our proof, based on lemma 3, may be replaced with an algebraic argument, stating that the codimension-two reflections generate $SO(3)$. Therefore, in order to show that a normal subgroup is $SO(3)$, it is sufficient to find only one codimension-two reflection belonging to it.

Another classical proof is to find a codimension-two reflection in a nontrivial normal subgroup $H$ by introducing the map

$\varphi : \left\{ \begin{array}{ccc} SO(3) & \to & \mathbb{R} \\ g & \mapsto & \mathrm{tr}(ghg^{-1}h^{-1}) \end{array} \right.$

for some fixed $h \in H \backslash \{ \mathrm{Id} \}$. By connectedness, $\mathrm{Im}(\varphi)= [a,3]$ for some $a \leq 3$, because for all $g \in SO(3)$ there exists $\theta \in [0,2\pi]$ such that $\mathrm{tr}(g)=1+2\cos(\theta)$ (lemma 1).  If $a=3$, then $[g,h]= \mathrm{Id}$ for all $g \in SO(3)$, that is $h$ is in the center of $SO(3)$, which is trivial. Therfore, $a<3$ and there exist $n \geq 1$ and $g_n \in SO(3)$ such that

$a \leq \varphi(g_n) = 1+2 \cos \left( \frac{\pi}{n} \right) < 3$.

Then $h_n:= g_nhg_n^{-1}h^{-1} \in H$ is a rotation of angle $\pi/n$ and $h_n^n$ is a codimension-two reflection.

Finally, a proof based on the isomorphism $SU(2)/ \mathbb{Z}_2 \simeq SO(3)$ exhibit in Fundamental group of SO(3) and Quaternions can be found in Artin’s book, Algebra.