Numerical Methods for Machine Learning: Course Notes

Joe McEwen and Jesse Loi

version 0.3 2025/26

Help! This is an early version of the notes, if you find an error, please let us know. Thank you.

• 1. What is this?
• 2. Guidance
  • 2.1. Getting a Job
  • 2.2. Good Coding
  • 2.3. Psuedo Code Exercise
• 3. Array Basics
  • 3.1. Arrays in Numpy
  • 3.2. Broadcasting Examples
  • 3.3. Dot Product for NumPy Arrays
  • 3.4. Geometric Meaning of Dot Product
  • 3.5. Angle Between Two Vectors
  • 3.6. Intuition About the Dot Product
  • 3.7. Matrix Multiplication
  • 3.8. Matrix Inversion and Solving Systems of Equations
• 4. Mathematics
  • 4.1. Single Variable Calculus
  • 4.2. Multivariable Calculus
  • 4.3. Automatic Differentiation with Jax
  • 4.4. Derivatives, a Helpful Lesson
  • 4.5. Complex Numbers
  • 4.6. Representation of Numbers on the Computer
• 5. Matrices
  • 5.1. Transpose, Inverse, and Norm
  • 5.2. Linear Transformations
  • 5.3. Matrix Determinant
  • 5.4. Orthogonal Matrix (Unitary for Real Matrices)
  • 5.5. Matrix Rank
  • 5.6. Matrix Condition Number
  • 5.7. EigenValues and Principle Component Analysis
  • 5.8. Matrix Factorization Methods
  • 5.9. SVD and PCA
• 6. Fourier Analysis
• 7. Numerical Optimization
  • 7.1. Introduction to Optimization via Linear Regression
  • 7.2. Convex Functions and Optimization
  • 7.3. The Nelder-Mead Algorithm, Downhill Simplex, or the Amoeba
  • 7.4. Measuring the Temperature of the Universe with SciPy’s fmin
  • 7.5. Particle Swarm and the Eggholder Challenge
  • 7.6. Newton’s method for Root Finding
  • 7.7. Newton’s Method for Optimization
  • 7.8. Problems with the Hessian Matrix
  • 7.9. Introduction to Gradient Descent
  • 7.10. Gradient Descent in 1-Dimension
  • 7.11. Gradient Descent with Momentum
  • 7.12. The Levenberg-Marquardt Method
• 8. Neural Networks