Cramer-Rao Lower Bound���ڿ������޽�ĺܺõĽ���

Chapter 3 Cramer-Rao Lower Bound

What is the Cramer-Rao Lower Bound
Abbreviated: CRLB or sometimes just CRB

CRLB is a lower bound on the variance of any unbiased estimator:

If ��? is an unbiased estimator of �� , then
2 �Ҧ� ? (�� ) �� CRLB ? (�� ) ? �� ? (�� ) �� CRLB ? (�� )

��

��

��

The CRLB tells us the best we can ever expect to be able to do (w/ an unbiased estimator)

Some Uses of the CRLB
1. Feasibility studies ( e.g. Sensor usefulness, etc.) ? Can we meet our specifications? 2. Judgment of proposed estimators ? Estimators that don��t achieve CRLB are looked down upon in the technical literature 3. Can sometimes provide form for MVU est. 4. Demonstrates importance of physical and/or signal parameters to the estimation problem
e.g. We��ll see that a signal��s BW determines delay est. accuracy ? Radars should use wide BW signals

3.3 Est. Accuracy Consideration
Q: What determines how well you can estimate �� ? Recall: Data vector is x
samples from a random process that depends on an �� ? the PDF describes that dependence: p(x;�� )

Clearly if p(x;�� ) depends strongly/weakly on �� ��we should be able to estimate �� well/poorly. See surface plots vs. x & �� for 2 cases: 1. Strong dependence on �� 2. Weak dependence on �� ? Should look at p(x;�� ) as a function of �� for fixed value of observed data x

Surface Plot Examples of p(x;�� )

Ex. 3.1: PDF Dependence for DC Level in Noise
x = A + w
w ~ N(0,��2)

Then the parameter-dependent PDF of the data point x is:

p (x; A) =

1 2�Ц� 2

? ( x ? A) 2 ? exp ? ? ? 2 2�� ? ? ? ?

Say we observe x = 3�� So ��Slice�� at x = 3

p(x=3;�� )

3 A x

A

Define: Likelihood Function (LF)
The LF = the PDF p(x;�� ) ��but as a function of parameter �� w/ the data vector x fixed

We will also often need the Log Likelihood Function (LLF): LLF = ln{LF} = ln{ p(x;�� )}

LF Characteristics that Affect Accuracy
Intuitively: ��sharpness�� of the LF sets accuracy�� But How??? Sharpness is measured using curvature: ? ? 2 ln p (x ; �� )
?��
2 x = given data �� = true value

Curvature �� ? PDF concentration �� ? Accuracy �� But this is for a particular set of data�� we want ��in general��: So��Average over random vector to give the average curvature:
2 ? ? ? ln p (x ; �� ? E? 2 ? ? �� ?

)? ?

? ? ?��

��Expected sharpness of LF��
= true value

E{?} is w.r.t p(x;�� )

3.4 Cramer-Rao Lower Bound
Theorem 3.1 CRLB for Scalar Parameter
Assume ��regularity�� condition is met: E ? Then �� 2 �� ��?
1 ? ? ? 2 ln p (x;�� ) ? ? ? E? ? 2 ? ? ?�� ? ?��

? ? ln p( x;�� ) ? ? = 0 ?�� ?�� ? ?

= true value

Right-Hand Side is CRLB

E{?} is w.r.t p(x;�� )
2 ? ? 2 ln p (x;�� ) ? ? ln p (x;�� ) ? ? E? p( x;�� )dx ?=�� 2 2 ? ? ?�� ?�� ? ?

Steps to Find the CRLB
1. Write log 1ikelihood function as a function of ��: ? ln p(x;�� ) 2. Fix x and take 2nd partial of LLF: ? ?2ln p(x;�� )/?�� 2 3. If result still depends on x: ? Fix �� and take expected value w.r.t. x ? Otherwise skip this step 4. Result may still depend on ��: ? Evaluate at each specific value of �� desired. 5. Negate and form reciprocal

Example 3.3 CRLB for DC in AWGN
x[n] = A + w[n], n = 0, 1, �� , N �C 1
w[n] ~ N(0,��2) & white

Need likelihood function:
p (x ; A ) =
N ?1

��

1 2�Ц� 1
2

n =0

? ? (x [n ] ? A )2 exp ? 2 2 �� ? ?

? ? ? ? ? ? ? ? ? ?

Due to whiteness

=

(2�Ц� )

N 2 2

? N ?1 2 ? ? �� (x [n ] ? A ) exp ? n = 0 ? 2�� 2 ? ?

Property of exp

Now take ln to get LLF:
N? N ?1 ? 1 2 ( ) [ ] ln p ( x; A) = ? ln ? 2�Ц� 2 2 ? ? x n ? A 2 �� ? ? 2�� n =0 ? ? \$ !!! #!!! " \$!! #!! " ? (~~) =0 ?A ? (~~) =? ?A

(

)

Now take first partial w.r.t. A:
1 ? ln p ( x; A) = ?A ��2
N ?1 n =0

sample mean

�� (x[n] ? A) = �� 2 (x ? A)
N

N

(!)

Now take partial again:
?2 ?A
2

Doesn��t depend on x so we don��t need to do E{?}

ln p ( x; A) = ?

��2

Since the result doesn��t depend on x or A all we do is negate and form reciprocal to get CRLB:
CRLB = 1 ? ? ? ? 2 ln p (x;�� ) ? ? E? ? 2 ? ? ?�� ?�� ? =
= true value

��2
N

?} �� var{ A

��2
N

CRLB For fixed N & ��
2

? Doesn��t depend on A ? Increases linearly with �� 2 ? Decreases inversely with N
A

CRLB For fixed N

CRLB

Doubling Data Halves CRLB! For fixed �� 2

��2

N

Continuation of Theorem 3.1 on CRLB
There exists an unbiased estimator that attains the CRLB iff:
? ln p ( x;�� ) = I (�� )[g ( x ) ? �� ] ?��

(!)

for some functions I(�� ) and g(x) Furthermore, the estimator that achieves the CRLB is then given by:

��? = g ( x )
1 ? ��} = var{ = CRLB with I (�� )

Since no unbiased estimator can do better�� this is the MVU estimate!! This gives a possible way to find the MVU: ? Compute ?ln p(x;�� )/?�� (need to anyway) ? Check to see if it can be put in form like (!) ? If so�� then g(x) is the MVU esimator

Revisit Example 3.3 to Find MVU Estimate
For DC Level in AWGN we found in (!) that:
? N ln p ( x; A) = 2 (x ? A) ?A ��
N
Has form of I(A)[g(x) �C A]

I ( A) =

��

2

?} = ? var{ A

��2
N

= CRLB

1 ��? = g ( x ) = x = N

N ?1 n =0

�� x[n]

So�� for the DC Level in AWGN: the sample mean is the MVUE!!

Definition: Efficient Estimator
An estimator that is: ? unbiased and ? attains the CRLB is said to be an ��Efficient Estimator�� Notes: ? Not all estimators are efficient (see next example: Phase Est.) ? Not even all MVU estimators are efficient
So�� there are times when our ��1st partial test�� won��t work!!!!

Example 3.4: CRLB for Phase Estimation
This is related to the DSB carrier estimation problem we used for motivation in the notes for Ch. 1 Except here�� we have a pure sinusoid and we only wish to estimate only its phase Signal Model:
x[n ] = A cos(2��f o n + ��o ) + w[n ] \$!! ! #!! ! "
s[ n;��o ]

AWGN w/ zero mean & �� 2

Signal-to-Noise Ratio: Signal Power = A2/2 Noise Power = �� 2 Assumptions:
1. 0 < fo < ? ( fo is in cycles/sample)

SNR =

A2 2�� 2

2. A and fo are known (we��ll remove this assumption later)

Problem: Find the CRLB for estimating the phase. We need the PDF:
p (x ; �� ) = 1 ? N ?1 2 ? ? �� (x [n ] ? A cos( 2�� f o n + �� ) ) exp ? n = 0 ? 2�� 2 ? ? ? ? ? ? ? ?
Exploit Whiteness and Exp. Form

(2�Ц� )

N 2 2

Now taking the log gets rid of the exponential, then taking partial derivative gives (see book for details):
A ? ln p (x ; �� ) ? A N ?1? ? = 2 �� ? x [n ]sin( 2�� f o n + �� ) ? sin( 4�� f o n + 2�� ) ? 2 ?�� �� n =0 ? ?
2

Taking partial derivative again:
? 2 ln p (x ; �� ) ?��
2

=

? A

N ?1 n =0

��

2

�� (x [n ]cos( 2�� f o n + �� ) ? A cos( 4�� f o n + 2�� ) )
Still depends on random vector x�� so need E{}

Taking the expected value:
2 ? ? A ? ? ln p (x ; �� ) ? ? ? E? ? = E? 2 2 ? ? ?�� ?�� ? ? N ?1

? �� (x [n ]cos( 2�� f o n + �� ) ? A cos( 4�� f o n + 2�� ) )? n =0 ?

=

A

N ?1 n =0

��

2

�� (E {x [n ]}cos( 2�� f o n + �� ) ? A cos( 4�� f o n + 2�� ) )
E{x[n]} = A cos(2�� fon + �� )

So�� plug that in, get a cos2 term, use trig identity, and get
2 ? A2 ? ? ln p (x ; �� ) ? ? ? E? ?= 2 2 ? ? ?�� ? 2�� ?

? N ?1 ? ��1? ? ? n =0

N ?1

? NA 2 �� cos( 4�� f o n + 2�� ) ? �� 2�� 2 = N �� SNR ? n ?0 ?

=N

<< N if fo not near 0 or ?

N-1

n

Now�� invert to get CRLB:

?} �� var{��

1 N �� SNR

Non-dB

CRLB

Doubling Data Halves CRLB! For fixed SNR

N

CRLB For fixed N

Doubling SNR Halves CRLB!

Halve CRLB for every 3B in SNR

SNR (non-dB)

Does an efficient estimator exist for this problem? The CRLB theorem says there is only if ? ln p( x;�� ) = I (�� )[g ( x ) ? �� ] ?�� Our earlier result was:
A ? ln p (x ; �� ) ? A N ?1? ? = 2 �� ? x [n ]sin( 2�� f o n + �� ) ? sin( 4�� f o n + 2�� ) ? 2 ?�� �� n =0 ? ?
2

Efficient Estimator does NOT exist!!!
?} �� CRLB We��ll see later though, an estimator for which var{�� as N �� �� or as SNR �� ��

?} var{��
CRB

N

Such an estimator is called an ��asymptotically efficient�� estimator (We��ll see such a phase estimator in Ch. 7 on MLE)

����ĵ�

Chapter 3 Cramer-Rao Lower Bound
��1.7 ����Ĭ����(Cramer)
Cramer-Rao Lower Bounds for Time Delay and Doppler Shift Estimation
����Ĭ(Cramer)����
�߶�̬������Ƶ�ʹ��Ƽ���Cramer-Rao��
3.2.2-���ڿ������޲���ʽ��MVU����
1998 On the Cramer-Rao bound under parametric constraints
1.4����Ĭ(Cramer)����
THE CRAMER-RAO BOUND AND ML ESTIMATE FOR DATA-AIDED CHANNEL ESTIMATION IN KSP-OFDM
����Ĭ_Cramer_������ƹ㼰Matlabʵ��
���԰�