<<< Previous 10 Lectures <<<

EE 503 Lectures (Fall 2020/21)
Lec. #31	00:00 - Example: f(x,y) uniform in 1x1 square in 1st and 3rd quadrants (revisited, Lec.30) 05:15 - Another proof min MSE via orthogonality (example) 17:22 - Uncorrelated check for error of min.MSE estimator with x^k, k:odd! 29:10 - Example: x = y + n, y and n zero-mean jointly Gaussian, find min. MSE yhat(x). 48:50 - Conclusion #1: For jointly gaussian rv's, the posterior density is Gaussian 50:44 - Conclusion #2: Non-zero mean case is a simple adaptation of zero mean case Document for the proof of Conclusion #1: .pdf Corrections: 12:13 - Unif([a,b+\Delta]) should be Unif([a,a+\Delta]) (Thanks Ugur Berk S.) 18:57 - For the explanations given on the board "k: arbitrary integer" should be "k: arbitary **odd** integer" (the results are trivially valid for even valued k, since f(x,y) = f(-x,-y), i.e. symmetric wrt to origin and many integrals with even valued k is equal to zero due to this symmetry) (Thanks Ugur Berk S.) 20:23 - sgn(x) x^k = |x|^k is only valid for odd valued k (Thanks Ugur Berk S.) For even valued say k = 2m, 1. E{ sgn(x) x^{2m} } = 0. Since, sgn(x) x^k = sgn(x) x^{2m} is an odd function of "x": Hence E{ sgn(x) x^{2m} } = \int f(x) sgn(x) x^{2m} dx = 0 since f(x) is an even function (i.e. symmetric wrt to origin f(x) = f(-x)) and then the integrand is an odd function. 2. E{ yx^{2m} } = 0. Since g(x,y) = yx^{2m} is odd symmetric with respect to the origin (g(x,y) = - g(-x,-y)); hence, \int \int f(x,y) g(x,y) dx dy = 0. This equality can be verified by a simple exchange of variables x2 = -x, y2=-y, then \int \int f(x,y) g(x,y) dx dy = - \int \int f(x2,y2) g(x2,y2) dx2 dy2 which gives the result of integration as 0. Hence all the results given on the board assume "k" is taken as an odd integer. I am explaining this a bit more in a later lecture, that is in 6:35 of Lec. 35a, but I should have clarified this during this lecture, indeed sorry CC)
Lec. #32	00:00 - Linear Minimum Mean Square Error (LMMSE) Estimators 02:00 - Example: LMMSE estimator in the form yhat = ax + b 05:04 - Deriving normal equations, by partial diff. (example) 10:59 - Solving normal equations (example) 16:40 - LMMSE calculation (example) 27:55 - Comment #1: LMMSE are parametric estimators not as good as min. MSE est. in general 29:40 - Comment #2: LMMSE est. = min. MSE est for jointly Gaussian observations and desired r.v. 30:56 - Comment #3: LMMSE is used in practice, since we only have moment estimates at our disposal 33:51 - 2nd derivation (more general) for normal Equations (gradient calc.) 42:35 - Normal equation: R_x w = r_xy 45:31 - 2nd derivation (more general) for LMMSE value 51:01 - Example: LMMSE estimator in the form yhat = ax + b (revisited)
Lec. #33	00:00:00 - LMMSE Estimation (summary) 00:03:32 - Example: x[n] = c + w[n], c and w[n] uncorrelated zero mean r.v.'s, n={1, ... , N} 00:05:20 - Discussing on problem set-up (example) 00:09:34 - Writing normal equations (example) 00:19:03 - Introducing SNR definition (example) 00:23:00 - Solving normal equations (example) 00:31:40 - Calculating LMMSE value (example) 00:34:30 - Comments: special case of infinite SNR (example) 00:34:48 - Comments: N observations with SNR = 1 or 1 observation with SNR = N 00:37:04 - Example: x[n] = c + w[n], c (non-zero mean) and w[n] uncorrelated r.v.'s, n={1, ... , N} 00:40:14 - Introducing SNR definition (example) 00:41:50 - Solution of normal equations (example) 00:42:57 - Is LMMSE estimator for this problem biased? (example) 00:47:17 - Removing bias with an affine estimator (example) 00:53:15 - Writing normal equations for affine estimator (example) 00:59:05 - Solving normal equations for affine estimator (example) 01:00:32 - Finalizing the estimator expression (example) Corrections: 22:33 - NxN entry of the matrix should be 1 + 1/SNR_N (not 1) (Thanks to Ugur Berk S.) 38:50 - E{ \sigma_w1^2 } should be E{ w_1^2} in E{x_1^2 } = E{c^2} + E{ \sigma_w1^2 } (Thanks to Ege. E)
Lec. #34	00:00 - Example: xvec = pvec \times c + nvec; pvec: known vector; c and nvec r.v.'s 02:33 - Derivation of Normal Equations for LMMSE est. (complex valued case) 13:02 - Derivation of LMMSE value (complex valued case) 22:57 : Link to paper: .pdf 25:03 - Discussion on the application related with example 26:45 - Solution for Identity Noise Cov. Mat. (example) 30:38 - Matrix inversion lemma 38:11 - Expression for the estimator (example) 41:27- Solution for General Noise Cov. Mat. (example) 42:42 - Solution by whitening (example) Matrix Inversion Lemma: wikipedia link
Lec. #35a	00:00 - Example: x,y unif. dist. in 1x1 square in 1st and 3 quadrants (Lec.30, revisited) 01:01 - Min. MSE Linear/Affine estimator yhat(x) (example) 06:30 - E{yx^k} and E{x^k} expressions (example) 15:05 - Using powers of the observation value in the min. MSE estimator (example) 25:28 - min. Cubic MSE estimator and its MSE value (example) 29:20 - Higher Order Estimators, Matlab Results (example) link: youtube-link 32:38 - Properties of LMMSE Estimators 32:43 - Property 1: Geometric (Vector Space) Interpretation 42:17 - Property 2: Multiple r.v. estimation (Total MSE minimization) 54:17 - Estimator expression for total MSE minimization (property 2) 55:34 - Example: xvec = Hyvec + n. Find LMMSE estimator for xvec. (property 2) Link to supplementary video for MATLAB content: youtube-link
Lec. #35b	Supplementary video for Lec. 35a
Lec. #36a	00:00 - Property 3: Linear combination of observations as input to LMMSE estimation 04:28 - Recursive estimation discussion 23:37 - Recursive estimator expression 24:04 - Innovation 25:15 - Property 4: LMMSE estimation of a linear combination of desired r.v.'s from same set of observations Corrections: 28:50 - zhat = M yhat should be zhat = N yhat (Thanks Ugur Berk S.)
Lec. #36b	00:00 - Wiener Filtering (Problem Setup) 03:08 - FIR Wiener Filtering 04:42 - Deriving Wiener-Hopf (or normal) Equations 22:06 - Min.MSE calculation 25:00 - Example: x[n] = d[n] + v[n], r_d[k] = \alpha^{|k|} , r_v[k] = \sigma_v^2 \delta[k] 32:25 - Writing Normal equations for 2-tap filter (example) 36:50 - Calculation of LMMSE value for 2-tap filter (example) 40:42 - 1-tap FIR Wiener filter derivaiton (example) 45:06 - SNR before and after Wiener filtering (example) 51:30 - SNR-after with 1-tap Wiener filter (example) 53:04 - SNR-after with 2-tap Wiener filter (example) 59:10 - What is the maximum SNR-after with 2-tap filter? (example) 59:30 - Max. SNR filter derivation (example)
Lec. #37	00:00 - FIR Wiener filtering (review) 4:30 - Categorization of Signal Processing Operations 4:47 - Filtering (categorization) 6:15 - Smoothing (categorization) 13:20 - Prediction (categorization) 19:50 - Example: Two-tap linear predictor for x[n] with r_x[k] = \alpha^|k| 32:34 - Comments: Prediction error as uncorrelated part of the observation 40:10 - Comments: Prediction error filter 42:30 - Backward prediction 47:25 - Auto-correlation of time-reversed WSS process (backward prediction) 50:20 - Remark: Auto-correlation matrix estimation with time-reversed vectors for WSS processes 54:17- Auto-correlation matrix for the samples of WSS processes (remark) 58:47 - Expression for the auto-correlation matrix estimation for WSS process samples (remark)
Lec. #38	00:00:00 - FIR Wiener Filtering (review) 00:02:10 - IIR Non-Causal Wiener Filtering 00:10:34 - IIR Non-Causal Wiener filter expression in Fourier domain 00:12:15 - MSE expression for IIR Non-Causal Wiener Filter 00:23:00 - Filtering application (noise removal) for IIR Non-Causal Wiener filtering 00:26:55 - Comment #1: Impulse response of IIR Non-Causal Wiener filter is an even sequence 00:28:20 - Comment #2: IIR Non-Causal Wiener filter processes each spectrum sample independent of other samples 00:30:26 - Connection between Comment #1 and the forward-backward time invariance of WSS processes (Lec. 37 link: youtube-link) 00:35:51 - Connection between Comment #2 and the fact that FT decorrelates WSS processes (Lec 27b link: youtube-link) 00:37:55 - Pictorial interpretation of IIR Non-Causal Wiener filtering for filtering (noise removal) application 00:41:44 - MSE calculation for IIR Non-Causal Wiener filtering for filtering (noise removal) application 00:48:20 - Example: x[n] = d[n] + v[n], r_d[k] = 0.8^|k| , r_v[k] = \delta_v[k], Find IIR NC Wiener filter to estimate d[n]. 1:07:05 - MSE calculation for IIR Non-Causal Wiener filter (example)
Lec. #39	00:00 - IIR Causal Wiener Filtering 02:13 - Deriving normal equations by differentiation 06:54 - Special case: Input is white noise 09:57 - General case: Decorrelating input to generate white noise input artificially 30:03 - IIR Causal Wiener Filter in z-domain 37:06 - Example: x[n] = d[n] + v[n], r_d[k] = 0.8^|k|, r_v[k] = \sigma_v^2\delta[k] 01:09:44 - Summary of results for 1 Tap, 2 Tap, Causal IIR, Non-Causal IIR Wiener filter result for the same example Corrections: 37:10 - r_v[k] should be r_v[k] = \sigma_v^2 \delta[k] (white noise)
Lec. #40	00:00 - Ergodicity 03:00 - Mean ergodicity 04:02 - Sample mean estimator 08:30 - Question on the convergence issues for sample mean estimator as N goes to infinity. 12:20 - Unbiasedness of sample mean estimator 14:01 - Consistency of sample mean estimator 27:10 - Necessary and Sufficient Condition for mean ergodicity 29:04 - An equivalent necessary and sufficient condition for mean ergodicity 29:40 - Sufficient condition for mean ergodicity 33:06 - Example: Mean-ergodic and not mean-ergodic r.p. example 49:00 - Ergodicity in auto-correlation 54:05 - Biased and unbiased estimators for auto-correlation 58:10 - Biased auto-correlation estimators always gives a valid auto-correlation sequence 01:06:10 - Consistency of auto-correlation estimator
Lec. #41	00:00:00 - Best Linear Unbiased Estimator (BLUE) 00:01:15 - Illustration of BLUE on a simple sample: x_k = c + w_k, k={1,2}, Find chat_BLUE. 00:07:11 - MSE expression for chat (illustration) 00:09:41 - Comment on MSE: Unrealizable estimator unless estimator is unbiased (illustration) 00:12:15- Optimization problem for minimum MSE unbiased estimator (illustration) 00:19:15 - Solution for uncorrelated noise (illustration) 00:22:34 - Precision = 1/noise-variance definition (illustration) 00:28:45 - Comparison with a similar random parameter estimation problem (Lec.33 link : youtube-link) 00:31:04 - General Case for the BLUE estimator 00:33:47 - Total MSE derivation (general case) 00:36:26 - Condition for the unbiased estimator (general case) 00:41:07 - Total MSE expression (general case) 00:44:17 - Optimization problem for BLUE estimator (general case) 00:47:23 - BLUE Estimator expression (general case) 00:49:45 - Total MSE expression for BLUE estimator (general case) 00:51:37- Special case #1: White noise (Rn = \sigma_n^2 \times Identity ) 00:54:30 - Comment on case #1: LS solution is the BLUE for white noise 00:55:55 - Special case #2: Non-white noise (Rn \neq \sigma_n^2 \times Identity ) 01:02:16 - Comment on case #2: Whitened LS solution is the BLUE estimator 01:03:30 - Revisiting earlier illustrative example 01:08:26 - Comparison with random parameter case, LMMSE estimation (general case)
Lec. #42	00:00:00 - Introduction to the end of EE 503! 00:00:30 - Karhunen Loeve (KL) Transform 00:03:57 - Desired properties for the KL transformation 00:06:28 - KL Transformation for 1-dimensional approximation (1D case) 00:14:25 - MSE expression for the problem (1D case) 00:23:23 - MSE minimizing solution (1D case) 00:25:15 - Eigenvectors of R_x matrix (reminder) 00:29:16 - Solution for optimal sub-space (1D case) 00:32:00 - min. MSE expression (1D case) 00:33:48 - KL Transformation for 2-dimensional approximation (2D case) 00:38:56 - Reduction to the 1D approximation case (2D case) 00:56:14 - Solution for optimal sub-space (2D case) 00:57:27 - min. MSE expression (2D case) 01:03:30 - Comment on the uncorrelatedness of expansion coefficients 01:05:55 - KL transformation (General case) 01:07:16 - Comment: Application Example: Signal Compression 01:14:34 - Comment: Case of WSS processes (Lec. 23b link: youtube-link) 01:32:26 - Goodbye!

<<< Previous 10 Lectures <<<

EE 503 Statistical Signal Processing and Modeling
(Fall 2019– 2020)

Short Description:

This course is the first course on statistical signal processing in the graduate curriculum of Department of Electrical and Electronics Engineering, Middle East Technical University (METU). Topics covered in this course are random vectors, random processes, stationary random processes, wide sense stationary processes and their processing with LTI systems with applications in optimal filtering, smoothing and prediction. A major goal is to introduce the concept of mean square error (MSE) optimal processing of random signals by LTI systems.

For the processing of the random signals, it is assumed that some statistical information about the signal of interest and distortion is known. By utilizing this information, MSE optimal LTI filters (Wiener filters) are designed. This forms the processing part of the course. The estimation of the statistical information to construct Wiener filters forms the modeling part of the course. In the modeling part, we examine AR, MA, ARMA models for random signals and give a brief discussion of Pade, Prony methods for the deterministic modeling. Among other topics of importance are decorrelating transforms (whitening), spectral factorization, Karhunen-Loeve transform

This course is a natural pre-requisite (not a formal one) to EE5506 Advanced Statistical Signal Processing. The estimation theory topics in EE 503 is mostly limited to the moment description of random processes which forms a special, but the most important, case of EE 5506.

Outline of Topics:

1. Review
1. Basics of Mathematical Deduction
   1. Necessary, Sufficient Conditions
   2. Proofs via contradiction, contraposition
2. Basics of Linear Algebra
   1. Linear independence of vectors (points in linear space)
   2. Range and Null space of the combination process
   3. Projection to Range/Null Space (orthogonality principle)
   4. Positive Definite Matrices
3. Basics of Probability
   1. Probability as a mapping, axioms, conditional probability
   2. Expectation, law of large numbers
   3. Moments, moment generating function


2. Random Processes
   1. Random variables, random vectors (or a sequence of random variables), moment descriptions (mean, variance, correlation), decorrelating transforms
   2. Random processes, stationarity, wide Sense Stationarity (WSS), power spectral density, spectral factorization, linear time invariant processing of WSS random processes, ergodicity
   
   Ref: Therrien, Hayes, Papoulis, Ross
   
3. Signal Modeling
   1. LS methods, Pade, Prony (Deterministic methods)
   2. AR, MA, ARMA Processes (Stochastic approach), Yule-Walker Equations, Non-linear set of equations for MA system fit
   3. Harmonic Processes
   
   Ref: Hayes, Papoulis
   
4. Estimation Theory Topics
   1. Random parameter estimation
      1. Cost function, loss function, square error, absolute error
      2. Conditional mean (regression line) as the minimum mean square error (MSE) estimator, orthogonality properties
      3. Linear minimum mean square error (LMMSE) estimators, orthogonality principle
      4. Regression line, orthogonality
      5. FIR, IIR, Causal–IIR Wiener filters
      6. Linear Prediction, backward prediction
      7. Random vector LMMSE estimation (multiple parameter)
   2. Non-random parameter estimation
      1. Maximum likelihood method
      2. Best Linear Unbiased Estimator (BLUE)
      3. Discussion of linear estimators for the linear observation model y=Ax+n
   3. Karhunen – Loeve Transform
   
   Ref: Therrien, Hayes
   

References:

[Hayes]: M. H. Hayes, Statistical Signal Processing and Modeling, Wiley, New York, NY, 1996.

[Therrien]: C. W. Therrien, Discrete random signals and statistical signal processing, Prentice Hall, c1992.

[Papoulis]: A. Papoulis, Probability, Random Variables, and Stochastic Processes, 3rd edition, McGraw Hill, 1991.

[Ross]: S. M. Ross, Introduction to probability models, 7th ed. Harcourt Academic Press, 2000.