Mathematics for Machine Learning

Systems of Linear Equations – Matrices, Solving Systems of Linear Equations. Vector Spaces – Vector Spaces, Linear Independence, Basis and Rank. Linear Mappings – Matrix Representation of Linear Mappings, Basis Change, Image and Kernel.

Norms, Inner Products, Lengths and Distances, Angles and Orthogonality, Orthonormal Basis, Orthogonal Complement, Orthogonal Projections – Projection into One Dimensional Subspaces, Projection onto General Subspaces, Gram-Schmidt Orthogonalization. Determinant and Trace, Eigenvalues and Eigenvectors, Cholesky Decomposition, Eigen decomposition and Diagonalization, Singular Value Decomposition, Matrix Approximation.

Differentiation of Univariate Functions - Partial Differentiation and Gradients, Gradients of Vector Valued Functions, Gradients of Matrices, Useful Identities for Computing Gradients. Back propagation and Automatic Differentiation – Gradients in Deep Network, Automatic Differentiation. Higher Order Derivatives- Linearization and Multivariate Taylor Series.

Construction of a Probability Space - Discrete and Continuous Probabilities, Sum Rule, Product Rule, and Bayes’ Theorem. Summary Statistics and Independence – Gaussian Distribution - Conjugacy and the Exponential Family – Change of Variables/Inverse Transform.

Optimization Using Gradient Descent - Gradient Descent With Momentum, Stochastic Gradient Descent. Constrained Optimization and Lagrange Multipliers – Convex Optimization - Linear Programming - Quadratic Programming.