EECE 522 Estimation Theory                                    

This Course is Offered Spring of Even Years; Next Offered Spring 2020

Instructor Information

• Prof. Mark Fowler

• Office: Second Floor of New Engineering & Science Building, 2315

• E-Mail: mfowler@binghamton.edu

• Office Hours:   TBD

Course Description

Addresses the theory and practice of estimating parameters for discrete-time signals embedded in noise.  Topics include:

• Classical Estimation (Deterministic Parameter)
  • Cramer-Rao Lower Bound
  • Minimum Variance Unbiased Estimation
  • Least Squares Estimation
  • Maximum Likelihood Estimation
• Bayesian Estimation (Random Parameter)
  • Minimum Mean Square Estimation
  • Maximum A Posteriori Estimation
• Optimal Filtering
  • Wiener Filtering
  • Kalman Filtering
• Applications
  • Radar, Sonar, and Emitter Location
  • Communication Systems

Background Assumed

This course is not for the mathematically weak!!

Must Have a Basic Understanding of:

• Linear Algebra or Matrix Theory (see textbook appendix & Reserve Books #1, 4)
  • Vectors: Orthogonality, Linear Independence, Inner Product (see Notes #1a for Matrix Review )
  • Matrices: Eigenvectors, Rank, Inverse
• Probability Theory and Random Functions (see textbook appendix & Reserve Books #1, 2, 3)
  • Probability Density Functions
    • Joint, Marginal, Conditional Versions
    • Gaussian/Normal
  • Mean and Variance of Random Variables
    • Expected Value, Etc.
  • Wide-Sense Stationary Random Processes
    • Correlation Function & Covariance Matrix
    • Power Spectral Density
• Digital Signal Processing (see Reserve Book #2)
  • Fourier Transform for Discrete-Time Signals
  • Discrete-Time Filters (Mostly FIR - not design, but operation via convolution)

Textbook

• Fundamentals of Statistical Signal Processing, Volume I: Estimation Theory
  by Steven Kay (Published by Prentice Hall)

Other Books of Interest   

1. Parameter Estimation - H. Sorenson
   • Covers same ground as textbook but in a different order; thus, provides an interesting alternative view.
   • Has appendices on Matrices and Probability Theory - a little more detailed than textbook.

2.     Signal Processing: Discrete Spectral Analysis, Detection, and Estimation - M. Schwartz and L. Schaw

o       Ch. 2 Reviews Digital Signal Processing

o       Ch. 3 reviews Random Discrete-Time Signals

o       Ch. 6 gives concise coverage of Parameter Estimation (Classical and Bayesian) as well as Wiener Filter

o       Ch. 7 covers Kalman Filters and has example of Aircraft Tracking

3. Introduction to Random Signal Analysis and Kalman Filtering - R. Brown
   • Gives a good overview of probability and random processes
   • Several Chapters on Kalman Filter
4. Estimation Theory and Applications - N. Nahi
   • An older book on estimation, but still might have useful perspectives on parameter estimation
   • BUT... mostly focused on state-estimation (e.g., Kalman Filter type stuff)
   • HOWEVER... has a good section on Matrix Algebra and Quadratic Forms
5. Applied Optimal Estimation - A. Gelb
   • "THE BIBLE" for Kalman Filters - on the bookshelf of virtually everyone working with Kalman Filters!
6. Data Analysis: A Bayesian Tutorial - D. Sivia
   • An Excellent, down-to-earth book on Bayesian estimation
   • Starts with Bayesian approach and shows how it "degenerates" into classical methods (ML & LS)
   • Mostly deals with problems of the scientific data analysis sort but still very good for signal processing types

Relevant Papers & Other Material  

For most of these you can find them in the library.

I'll try to post most of them on Blackboard.

Only by reading papers in the area can you really get a feeling for how this stuff works!

The following link gives some advice on how to read technical papers: How To Read Papers

General Papers

1. D. Torrieri, "Statistical Theory of Passive Location Systems," IEEE Transactions on Aerospace and Electronic Systems, pp. 183 - 198, March 1984
2. W. Gardner, "Likelihood Sensitivity and the Cramer-Rao Bound," IEEE Transactions on Information Theory, p. 491, July 1979
3. J. Cadzow, "Least Squares, Modeling, and Signal Processing," Digital Signal Processing, pp. 2 - 20, 1994
4. W. Press et al., "Ch. 15 Modeling of Data", in Numerical Recipes in C, 2nd Edition, Cambridge Press

Application Papers

5. S. Stein, "Differential Delay/Doppler ML Estimation with Unknown Signals," IEEE Transactions on Signal Processing, pp. 2717 - 2719, August 1993
6. T. Berger and R. Blahut, "Coherent Estimation of Differential Delay and Differential Doppler," Proceedings of the 1984 Conference on Information Sciences and Systems, Princeton University, pp. 537 - 541, 1984
7. M. Fowler, “Analysis of Passive Emitter Location using Terrain Data,” IEEE Transactions on Aerospace and Electronic Systems, pp. 495 – 507, April 2001.
8. K. Becker, "An Efficient Method of Passive Emitter Location," IEEE Transactions on Aerospace and Electronic Systems, pp. 1019 – 1104, Oct. 1992
9. P. Chestnut, "Emitter Location Accuracy using TDOA and Differential Doppler," IEEE Transactions on Aerospace and Electronic Systems, pp. 214 - 218.
10. M. Fowler, “Air‑to‑Air Passive Location,” U.S. Patent #5,870,056  Issued 2/9/1999
11. D. Rife and Boorstyn, "Single-Tone Parameter Estimation from Discrete-Time Observations," IEEE Transactions on Information Theory, pp. 591 - 598, Sept. 1974.
12.  S. Tretter, "Estimating the Frequency of a Noisy Sinusoid by Linear Regression," IEEE Transactions on Information Theory, pp. 832 - 835, Nov. 1985.
13.  S. Kay, "A Fast Accurate Single Frequency Estimator," IEEE Transactions on Acoustics, Speech, and Signal Processing , pp. 1987 - 1990, Dec. 1989.

Assorted Handouts

• Expected Values, Probability, Error Function for Gaussian RVs
• MATLAB m-file for 2-D Gaussian PDF

Lecture Notes

Please download, print out, and bring to the relevant class - see Course Schedule above

These notes are complete versions of my class notes.

    - You'll only need to fill in certain spoken information during class you deem important.

    - This will free you up for in-class thinking (come ready to do some!)

There also a few "reading notes" that supplement the textbook's coverage... these are now posted on BB.

New PDFs of PPT Charts

Notes #1a Probability Review      Notes #1b Vectors and Matrices Review  (See Reading Notes on BB)  

Notes #2: Ch 1 Intro to Est       Notes #3: Ch 2 MVUE

Notes #4: Ch 3 Cramer Rao Bound Pt. A    Notes #5: Ch 3 Cramer Rao Bound Pt. B

Notes #6: Ch 3 Cramer Rao Bound Pt. C   Notes #7: Ch 3 Cramer Rao Bound Pt. D

Notes #8: Ch 3 CRLB Examples  Notes #9: CRLB Example for Doppler Location (See Reading Notes on BB)

Notes #10 Ch_4 Linear Models      Notes #11 Ch_6 BLUE

Notes #12 Ch7A    Notes #13 Ch7B    Notes #14 Ch7C   Notes #15 ML Example - Revised

Notes #16 Ch8A    Notes #17 Ch8B    Notes #18 Ch8C    Notes #19 Ch8D

Notes #20 LS Single Platform (See Reading Notes on BB)   Notes #21 Doppler Tracking  (See Reading Notes on BB)

Notes #22 Results for 2 RVs (Pre-Ch. 10)  (See Reading Notes on BB)

Notes #23 Ch10A    Notes #24 Ch10B (See Reading Notes on BB)  Notes #24a Bayesian Example

Notes #25 Ch11A    Notes #26 Ch11B        Notes #26a Recursive Bayesian

Notes #27 Ch12A    Notes #28 Ch12B    Notes #28a Wiener Filter for Deblurring Images

Notes #29 Ch13A    Notes #30 Ch13B

Notes #31 Ch13C    Notes #32 Ch13D

Homework Assignments

• Assignments
  • Will be posted on Blackboard (if you don't know how to get access to it ask me)
   

   

• Solutions
  • Will be posted on Blackboard (if you don't know how to get access to it ask me)

Project Information

A significant portion of your grade will be based on a project.  It is important to start early. 

Things you can do early-on are:

1. Understand the Signal Model and Project Issues
2. Derive/Analyze Cramer-Rao bounds for your problem
3. Write simulation code to generate the data

Things you can do by mid-semester are:

1. Estimator derivation and analysis (most projects will use classical methods, all of which we will have studied by Spring Break)
2. Coding of estimator
3. Start your analyses of effects and/or trade-offs

Things you can do by end of semester are:

1. Complete your analyses of effects and/or trade-offs
2. Complete your simulations
3. Analyze your results
4. Write your report

Here are three files to help you get started. 

• The first gives a list of project suggestions.
• The second gives details on how to do your report.
• The third is a MSWord template that will help you format your report to professional publication standards

Project Files 

• Project Suggestions will be posted on Bb
• Project Guide
• Report Template

MATLAB Handouts & Links

• See EECE 301 Web Page

Links of Interest

DSP Tutorials and Reference Material

DSP Demos

Some interactive demos of DSP concepts (e.g., filter design)

DSP Tutorial

Some basics of DSP theory and implementation.

The Scientist and Engineer's Guide to Digital Signal Processing

A freely downloadable DSP Book!!!!  Provides coverage at the level assumed as a pre-requisite for EE522 - so it's a good place to start if you need a refresher.

Signals & Systems Demos (Johns Hopkins University)

A neat set of java applets that demonstrate continuous-time & discrete-time signal processing at the level assumed as a pre-requisite for EE522 - so it's a good place to start if you need a refresher.

Estimation Oriented Material

Frequency Estimation

An overview of many different ways to estimate frequency.

Blind SNR Estimation

Discusses how to estimate the SNR of a signal.

Back to Prof. Fowler's Homepage