Let [tex]\mathbf A[/tex] be a rectangular [tex]m\times n[/tex] matrix with column vectors [tex]\mathbf a_1,\ldots,\mathbf a_n[/tex], i.e.
[tex]\mathbf A=\begin{bmatrix}\mathbf a_1&\cdots&\mathbf a_n\end{bmatrix}[/tex]
Then we have
[tex]\mathbf A^\top=\begin{bmatrix}\mathbf a_1&\cdots&\mathbf a_n\end{bmatrix}^\top[/tex]
and the product of the two is
[tex]\mathbf A^\top\mathbf A=\begin{bmatrix}\mathbf a_1\cdot\mathbf a_1&\mathbf a_1\cdot\mathbf a_2&\cdots&\mathbf a_1\cdot\mathbf a_n\\\mathbf a_2\cdot\mathbf a_1&\mathbf a_2\cdot\mathbf a_2&\cdots&\mathbf a_2\cdot\mathbf a_n\\\vdots&\vdots&\ddots&\vdots\\\mathbf a_n\cdot\mathbf a_1&\mathbf a_n\cdot\mathbf a_2&\cdots&\mathbf a_n\cdot\mathbf a_n\end{bmatrix}[/tex]
Because the columns of [tex]\mathbf A[/tex] are orthonormal, we have
[tex]\mathbf a_i\cdot\mathbf a_j=\begin{cases}1&\text{for }i=j\\0&\text{for }i\neq j\end{cases}[/tex]
which means [tex]\mathbf A^\top\mathbf A[/tex] reduces to an [tex]n\times n[/tex] matrix with ones along the diagonal and zero everywhere else, i.e.
[tex]\mathbf A^\top\mathbf A=\begin{bmatrix}1&0&\cdots&0\\0&1&\cdots&0\\\vdots&\vdots&\ddots&\vdots\\0&0&\cdots&1\end{bmatrix}=\mathbf I_n[/tex]
where [tex]\mathbf I[/tex] denotes the identity matrix. This means the solution to [tex]\mathbf{Ax}=\mathbf b[/tex] is given by
[tex]\mathbf A^\top(\mathbf{Ax})=\mathbf A^\top\mathbf b\implies(\underbrace{\mathbf A^\top\mathbf A}_{\mathbf I})\mathbf x=\mathbf A^\top\mathbf b\implies\mathbf x=\mathbf A^\top\mathbf b[/tex]
