Vector

What is a vector?

A vector is a mathematical object that has both magnitude and direction. It is usually represented by an arrow. The length of the arrow represents the magnitude of the vector and the direction of the arrow represents the direction of the vector.

Mathematical representation of a vector is a list of numbers that represent the magnitude of the vector in each dimension. For example, a vector in 2D space can be represented as a list of two numbers like \((x,y)\) and a vector in 3D space can be represented as a list of three numbers like \((x,y,z)\).

Usually, a vector is represented by a column matrix.

\[\vec{v} = \begin{bmatrix} v_1,v_2,\dots,v_n \end{bmatrix}^T \in \mathbb{R}^n\]

Inner product

An inner product between two vectors is a scalar value that represents the similarity between the two vectors. It is also called a dot product.

Over complex number field \(\mathbb{C}\), the inner product is defined as

\[\left\langle u,v \right \rangle = (u^*)^Tv = \sum_{i=1}^{n} u_i^*v_i\]

If all elements of the vector are real numbers, then the inner product is simply the sum of the products of the corresponding elements of the two vectors.

\[\left\langle u,v \right \rangle = u^Tv = \sum_{i=1}^{n} u_iv_i\]

Inner product of continuous functions \(f\) and \(g\) is defined as

\[\left\langle f,g \right \rangle = \int_{a}^{b} f^*(x)g(x)dx\]

The properties of inner product are:

1. Commutative

\[\begin{aligned} \left\langle v,u \right \rangle &= \sum_{i=1}^{n} v_i^*u_i\\ &= \sum_{i=1}^{n} (u_i^*v_i)^*\\ &= \left\langle u,v \right \rangle^* \end{aligned}\]

1. Distributive

\[\begin{aligned} \left\langle u,v+w \right \rangle &= \sum_{i=1}^{n} u_i^*(v_i+w_i)\\ &= \sum_{i=1}^{n} u_i^*v_i + \sum_{i=1}^{n} u_i^*w_i\\ &= \left\langle u,v \right \rangle + \left\langle u,w \right \rangle \end{aligned}\]

1. Scalar multiplication

\[\begin{aligned} \left\langle \alpha u,v \right \rangle &= \sum_{i=1}^{n} (\alpha u_i)^*v_i\\ &= \sum_{i=1}^{n} \alpha^*u_i^*v_i\\ &= \alpha^*\sum_{i=1}^{n} u_i^*v_i\\ &= \alpha^*\left\langle u,v \right \rangle \end{aligned}\]

Since \(\alpha\) is usually a real number, then

\[\left\langle \alpha u,v \right \rangle = \alpha\left\langle u,v \right \rangle\]

1. Positive definite

\[\left\langle u,u \right \rangle \geq 0\]

Exactly, \(\left\langle u,u \right \rangle = 0\) if and only if \(u=\vec{0}\).More intuitively, the inner product of a vector with itself is equal to the square of the magnitude of the vector.

\[\left\langle u,u \right \rangle = \left\| u \right \|^2\]

And the angle between two vectors \(u\) and \(v\) is defined as

\[\begin{aligned} \cos \theta &= \frac{\left\langle u,v \right \rangle}{\left\| u \right \|\left\| v \right \|}\\ \theta &= \arccos \frac{\left\langle u,v \right \rangle}{\left\| u \right \|\left\| v \right \|} \end{aligned}\]

Linear combination

A vector \(v\) is a linear combination of vectors \(v_1,v_2,\dots,v_n\) if there exist scalars \(\alpha_1,\alpha_2,\dots,\alpha_n\) such that

\[v = \alpha_1v_1 + \alpha_2v_2 + \dots + \alpha_nv_n = \sum_{i=1}^{n} \alpha_iv_i\]

if \(v\) and \(v_i\) have the same dimension.

In other words, a set of vectors \(v_1,v_2,\dots,v_n\) can span a vector space \(V\) if every vector in \(V\) can be written as a linear combination of \(v_1,v_2,\dots,v_n\).

\[V = \left\{ \sum_{i=1}^{n} \alpha_iv_i \mid \alpha_i \in \mathbb{R} \text{ and } v_i \in \mathbb{R}^n \right\}\]

Linear independence

A set of vectors \(v_1,v_2,\dots,v_n\) is linearly independent if the only solution to the equation

\[\alpha_1v_1 + \alpha_2v_2 + \dots + \alpha_nv_n = \vec{0}\]

is \(\alpha_1=\alpha_2=\dots=\alpha_n=0\).

Linearly independent means that no vector in the set can be represented as a linear combination of the other vectors in the set, meaning that the set of vectors is the smallest set that can span the vector space.

Orthogonal vectors

Two vectors \(u\) and \(v\) are orthogonal if their inner product is zero.

\[\left\langle u,v \right \rangle = 0\]

A set of vectors \(v_1,v_2,\dots,v_n\) is orthogonal if every pair of vectors in the set is orthogonal.

\[\left\langle v_j, v_k \right \rangle = 0 \text{ for } j \neq k\]

The set of orthogonal vectors is linearly independent, they are the basis of the vector space. If the vectors are normalized, then they are called orthonormal vectors.

\[\left\langle v_j, v_k \right \rangle = \delta_{jk} = \begin{cases} 1 & \text{if } j=k\\ 0 & \text{if } j \neq k \end{cases}\]

If \(v_1,v_2,\dots,v_n\) are orthonormal vectors, then any vector \(v\) can be represented as a linear combination of \(v_1,v_2,\dots,v_n\).

\[v = \sum_{i=1}^{n} \left\langle v_i,v \right \rangle v_i\]

Biorthogonal vectors

Sometimes, a set of vectors \(v_1,v_2,\dots,v_n\) is not orthogonal, but there exists a set of vectors \(u_1,u_2,\dots,u_n\) such that

\[\begin{aligned} \left\langle u_j,v_k \right \rangle &= \delta_{jk}\\ \end{aligned}\]

Then \(u_1,u_2,\dots,u_n\) are called biorthogonal vectors of \(v_1,v_2,\dots,v_n\).

A vector \(v\) can be represented as a linear combination of \(v_1,v_2,\dots,v_n\).

\[v = \sum_{i=1}^{n} \left\langle u_i,v \right \rangle v_i\]