Signal Processing
This course is about extracting information from noisy, imperfect measurements. Each lecture opens with a concrete engineering problem — fitting a sensor calibration curve, locating a device from noisy range measurements, tracking a moving object — and the mathematics needed to solve it is introduced along the way, when it is needed.
Topics run from linear least squares, conditioning and regularization, through multidimensional signals, random vectors and covariance, to maximum-likelihood, MAP and Bayesian estimation, Bayesian linear regression, and nonlinear least squares (Gauss–Newton, Levenberg–Marquardt). The course closes with optimal linear filtering: Wiener filters in time and frequency, state-space models, and the Kalman filter for multisensor tracking.
- Lec. 1Fitting a sensor calibration curve
- Lec. 2When least squares becomes unreliable
- Lec. 3Filtering an image / array of sensors
- Lec. 4What does correlated noise look like in 10 dimensions?
- Lec. 5How large is the signal hidden inside the noise?
- Lec. 6What if we already know something about the answer?
- Lec. 7An estimate is not enough: how uncertain are we?
- Lec. 8Bayesian Linear Regression I — a line with uncertainty
- Lec. 9Bayesian Linear Regression II — prediction with uncertainty
- Lec. 10Bayesian Linear Regression III — linear in parameters does not mean straight lines
- Lec. 11Nonlinear Least Squares I — where am I?
- Lec. 12Nonlinear Least Squares II — how do we actually solve it?
- Lec. 13Wiener Filtering I — the best linear estimate from noisy measurements
- Lec. 14Wiener Filtering II — optimal filtering in frequency
- Lec. 15State-Space Models — the signal we care about is hidden
- Lec. 16Kalman Filter I — prediction plus correction
- Lec. 17Kalman Filter II — real multisensor tracking
The notes are written per topic, so one document often covers several lectures.
Slides and notes for each lecture, grouped by teaching week.
“Given many noisy measurements, what line/model best explains them?”
- Linear least squares
- Residuals
- Design matrix
- Overdetermined systems
Mathematics: Vector notation, matrix multiplication, Euclidean norm, inner products, projection, gradient of a quadratic
“Why does the answer explode when two measurements/features say almost the same thing?”
- Ill-conditioning
- Rank deficiency
- Ridge/Tikhonov regularization (weighted LS: reading in the notes)
Mathematics: Rank, column space, condition number, QR, SVD, pseudoinverse, singular values
“Convolution worked for audio; what happens when the signal has two spatial coordinates or many channels?”
- Multidimensional signals & filters
- 2-D convolution
- Multichannel signals
- Separable filters; brief bridge to frequency-domain representations
Mathematics: Matrices as signals, vectorization, Kronecker/separable structure, matrix operators
“Why do errors from several sensors move together?”
- Random vectors
- Covariance matrices
- Correlated noise
- Whitening; preview of frequency-dependent channel correlation
Mathematics: Quadratic forms, symmetric/PSD matrices, covariance ellipses, eigendecomposition, Cholesky, whitening
“Estimate amplitude, phase, frequency-model parameters, DC offset or sensor bias from noisy observations”
- Maximum Likelihood Estimation; Gaussian noise; LS as Gaussian MLE; estimating mean/variance
Mathematics: Likelihood/log-likelihood, gradients, Hessian/curvature intuition, convexity
“A physical parameter cannot realistically be arbitrarily large.”
- MAP estimation
- Priors
- Likelihood × prior
- Regularization as prior information
Mathematics: Bayes rule in parameter space, multivariate Gaussian, precision matrices, completing the square
“Two estimates may have the same mean but radically different certainty”
- Bayesian parameter estimation
- Posterior distributions
- Posterior mean
- MAP
- MMSE
- Credible regions
- Sequential Bayes updates
Mathematics: Gaussian conditioning, covariance propagation, matrix identities, Cholesky solves
“Which line fits—and which other lines are still plausible?”
- Prior over coefficients
- Likelihood
- Posterior over coefficients
Mathematics: Gaussian multiplication, completing the square, covariance/precision geometry
“What happens outside the region where we measured data?”
- Posterior predictive distribution; epistemic vs measurement uncertainty; confidence/credible bands
Mathematics: Marginalization, propagation of Gaussian uncertainty, quadratic forms
“Fit temperature curves, periodic sensor signals, trends, sinusoids or smooth nonlinear functions”
- Basis functions
- Polynomial/Fourier/RBF regression
- Regularization
- Model complexity
- Hyperparameters; sequential BLR
Mathematics: Feature/design matrices, ridge connection, evidence intuition, Woodbury/matrix inversion lemma, recursive updates
“Estimate a 2-D position from noisy distances to several anchors”
- Nonlinear LS
- Residual vector
- Nonlinear measurement models
- Local approximation
Mathematics: Multivariate functions, partial derivatives, Jacobian, gradient, Taylor expansion
“Localization, unknown-frequency sinusoid estimation or system identification where no closed-form solution exists”
- Gradient descent briefly
- Gauss–Newton
- Levenberg–Marquardt
- FFT/grid initialization
- Local minima
- Robust/weighted residuals
Mathematics: Hessian, J^TJ, Newton steps, damping, conditioning, optimization geometry
“Recover a clean signal from correlated noisy observations”
- Linear MMSE
- Orthogonality principle
- FIR Wiener filter
- Wiener–Hopf equations
- Prediction
Mathematics: Projection in random-variable space, covariance matrices, Toeplitz systems
“Denoise colored noise / undo blur or channel distortion”
- Frequency-domain Wiener filter
- Signal/noise PSDs
- Deconvolution
- Multichannel extensions
- Inverse problems
Mathematics: Fourier diagonalization, PSD ratios, regularized inversion, matrix-valued frequency responses
“A moving object has position and velocity, but sensors measure only noisy position”
- State-space models
- Latent state
- Dynamics
- Observation model
- Process noise
- Measurement noise
Mathematics: State vectors, affine Gaussian transformations, covariance propagation, Markov structure
“Track position from noisy measurements”
- Scalar → simple vector Kalman filter
- Prediction
- Innovation
- Correction
- Kalman gain
Mathematics: Gaussian conditioning, uncertainty fusion, covariance ellipses
“Fuse GPS/IMU/range data; missing observations; different sensor accuracies”
- Full multivariate KF
- Multiple sensors
- Asynchronous/missing measurements
- Tuning Q,R
- Innovation analysis
Mathematics: Matrix covariance updates, linear solves, numerical implementation, matrix identities