Illustrative examples of order statistics

Posted on 28 Jan 2018 | 1 Comment

The preceding post is an introduction to order statistics. This post gives examples demonstrating the calculation discussed in the preceding post.

We first work a basic example with a small sample size that walks through all the calculations discussed in the preceding post. More examples are to follow the basic example.

Example 1 – A Basic Example

Suppose that the population distribution is the uniform distribution on the interval (0,4). The CDF and the density function of this population are:

    ……\displaystyle F(x) = \frac{x}{4} \ \ \ \ \ \ \ 0<x<4

    ……\displaystyle f(x) = \frac{1}{4} \ \ \ \ \ \ \ 0<x<4

, respectively. Draw a random sample X_1,X_2,X_3 from this uniform distribution. The resulting order statistics are X_{(1)}<X_{(2)}<X_{(3)}, with X_{(1)} being the sample minimum, X_{(2)} being the sample median and X_{(3)} being the sample maximum. The joint density function of these 3 order statistics is

    ……\displaystyle f_{X_{(1)},X_{(2)},X_{(3)}}(x,y,z)=3! \ \frac{1}{4} \ \frac{1}{4} \ \frac{1}{4}=\frac{3}{32}

The support of this joint density is the 3-dimensional region 0<x<y<z<4.

The goal here is to derive the density functions of these order statistics and then calculate basic distributional quantities of these order statistics. This is a long example that is presented in five parts – Example 1a through Example 1e.

Example 1a
This example focuses on the minimum statistic.

  • Compute the mean and variance of the first order statistic X_{(1)}.
  • Compute the conditional probability P(X_{(1)}>2 \lvert X_{(1)}>1).

The key step is to derive the density function and the CDF for X_{(1)}. Then the mean and variance are obtained by evaluating the appropriate integrals. The conditional probability is obtained by evaluating the CDF. The following gives the density function.

    ……\displaystyle f_{X_{(1)}}(x)=\frac{3!}{1! \ 2!} \ \frac{1}{4} \ \biggl[1-\frac{x}{4} \biggr]^2=\frac{3}{64} \ (16-8x+x^2) \ \ \ \ \ \ 0<x<4

The following integrals give the mean and variance of X_{(1)}.

    ……\displaystyle E(X_{(1)})=\int_0^4 x \ \frac{3}{64} \ (16-8x+x^2) \ dx=1

    ……\displaystyle E(X_{(1)}^2)=\int_0^4 x^2 \ \frac{3}{64} \ (16-8x+x^2) \ dx=\frac{8}{5}

    ……\displaystyle Var(X_{(1)})=\frac{8}{5}-1^2=\frac{3}{5}

The following gives the CDF of X_{(1)}.

    ……\displaystyle \begin{aligned} F_{X_{(1)}}(x)&=\int_0^x \frac{3}{64} \ (16-8t+t^2) \ dt \\&=\frac{3}{64} \ (16 x-4x^2+\frac{1}{3} x^3), \ \ \ \ \ \ 0<x<4 \end{aligned}

The following gives the conditional probability.

    ……\displaystyle P(X_{(1)}>2 \lvert X_{(1)}>1)=\frac{P(X_{(1)}>2)}{P(X_{(1)}>1)}=\frac{1-F_{X_{(1)}}(2)}{1-F_{X_{(1)}}(1)} =\frac{8}{47}=0.17

Example 1b
This example focuses on the sample median.

  • Compute the mean and variance of the second order statistic X_{(2)}.
  • Compute the conditional probability P(X_{(2)}>3 \lvert X_{(2)}>2).

As in Example 1a, the key step is to find the density function and the CDF.

    ……\displaystyle f_{X_{(2)}}(x)=\frac{3!}{1!} \ \frac{x}{4} \ \frac{1}{4} \biggl[1-\frac{x}{4} \biggr]=\frac{6}{64} \ (4x - x^2) \ \ \ \ \ \ 0<x<4

    ……\displaystyle F_{X_{(2)}}(x)=\frac{3}{32} \ (2x^2-\frac{1}{3} x^3) \ \ \ \ \ \ 0<x<4

As in Example 1a, evaluating the appropriate integrals and evaluating the CDF appropriately give the desired answers.

    ……\displaystyle E(X_{(2)})=2

    ……\displaystyle E(X_{(2)}^2)=\frac{24}{5}

    ……\displaystyle Var(X_{(2)})=\frac{24}{5}-2^2=\frac{4}{5}

    ……\displaystyle P(X_{(2)}>3 \lvert X_{(2)}>2)=\frac{P(X_{(2)}>3)}{P(X_{(2)}>2)}=\frac{1-F_{X_{(2)}}(3)}{1-F_{X_{(2)}}(2)} =\frac{5}{16}=0.3125

Example 1c
This example focuses on the sample maximum.

  • Compute the mean and variance of the third order statistic X_{(3)}.
  • Compute the conditional probability P(X_{(3)}>1 \lvert X_{(3)}<2).

The following gives the density function and the CDF of the sample maximum.

    ……\displaystyle f_{X_{(3)}}(x)=\frac{3!}{2!} \ \biggl[\frac{x}{4} \biggr]^2 \ \frac{1}{4}=\frac{3}{64} \ x^2 \ \ \ \ \ \ 0<x<4

    ……\displaystyle F_{X_{(3)}}(x)=\frac{1}{64} \ x^3 \ \ \ \ \ \ 0<x<4

The following gives the desired results after evaluating the integrals and the CDF.

    ……\displaystyle E(X_{(3)})=3

    ……\displaystyle E(X_{(3)}^2)=\frac{48}{5}

    ……\displaystyle Var(X_{(3)})=\frac{48}{5}-3^2=\frac{3}{5}

    ……\displaystyle \begin{aligned} P(X_{(3)}>1 \lvert X_{(3)} \le 2)&=\frac{P(1<X_{(3)} \le 2)}{P(X_{(2)} \le 2)} \\&=\frac{F_{X_{(3)}}(2)-F_{X_{(3)}}(1)}{F_{X_{(2)}}(2)} =\frac{7}{8}=0.875 \end{aligned}

Example 1d
This example focuses on the joint behavior between the sample minimum and the sample maximum.

  • Evaluate the covariance of X_{(1)} and X_{(3)}.
  • Evaluate the correlation coefficient of X_{(1)} and X_{(3)}.
  • Compute the conditional probability P(X_{(1)}<1 \lvert X_{(3)}<2).
  • Evaluate the conditional mean E(X_{(3)} \lvert X_{(1)} >2).

All the calculations in this example are based on the joint density function of the sample minimum and sample maximum.

    ……\displaystyle \begin{aligned} f_{X_{(1)},X_{(3)}}(x,y)&=\frac{3!}{1!} \ \frac{1}{4} \ \biggl[\frac{y}{4}-\frac{x}{4} \biggr] \ \frac{1}{4} \\&=\frac{3}{32} \ (y-x) \ \ \ \ \ \ 0<x<y<4 \end{aligned}

The covariance is defined by Cov(X_{(1)},X_{(3)})=E[X_{(1)} X_{(3)}]-E[X_{(1)}] E[X_{(3)}]. The following shows the necessary calculations.

    ……\displaystyle \begin{aligned} E[X_{(1)} X_{(3)}]&=\int_0^4 \int_x^4 x \ y \ f_{X_{(1)},X_{(3)}}(x,y) \ dy \ dx \\&=\int_0^4 \int_x^4 x \ y \ \frac{3}{32} \ (y-x) \ dy \ dx \\&=\int_0^4 \int_x^4 \frac{3}{32} \ (xy^2-x^2 y) \ dy \ dx \\&=\int_0^4 \frac{3}{32} \biggl(\frac{64}{3} x-8x^2+\frac{1}{6} x^4 \biggr) \ dx=\frac{16}{5} \end{aligned}

    ……\displaystyle \begin{aligned} Cov(X_{(1)},X_{(3)})&=E[X_{(1)} X_{(3)}]-E[X_{(1)}] E[X_{(3)}] \\&=\frac{16}{5}-1 \cdot 3=\frac{1}{5} \end{aligned}

Using the mean and variances from Example 1a and Example 1c, the following shows the calculation for the correlation coefficient.

    ……\displaystyle \begin{aligned} \rho&=\frac{Cov(X_{(1)},X_{(3)})}{\sigma_{X_{(1)}} \ \sigma_{X_{(3)}}}=\frac{\frac{1}{5}}{\sqrt{\frac{3}{5}} \ \sqrt{\frac{3}{5}}}=\frac{1}{3}=0.333 \end{aligned}

There is a positive correlation between the sample minimum and the sample maximum. This confirms what the natural intuitive idea about the sample minimum and sample maximum. For example, when sample minimum is large, the sample maximum will have to be large as well. Likewise, when sample maximum is small, the sample minimum is also small. However, the correlation is moderate.

To evaluate the conditional probability P(X_{(1)}<1 \lvert X_{(3)}<2), it is a matter of integrating the joint density over an appropriate region. the following shows how.

    ……\displaystyle \begin{aligned} P(X_{(3)}<2)&=\int_0^2 \int_x^2 \frac{3}{32} \ (y-x) \ dy \ dx \\&=\int_0^2 \frac{3}{32} (2-2x+\frac{1}{2} x^2) \ dx=\frac{1}{8} \end{aligned}

    ……\displaystyle \begin{aligned} P(X_{(1)}<1 \text{ and } X_{(3)}<2)&=\int_0^1 \int_x^2 \frac{3}{32} \ (y-x) \ dy \ dx \\&=\int_0^1 \frac{3}{32} (2-2x+\frac{1}{2} x^2) \ dx=\frac{7}{64} \end{aligned}

    ……\displaystyle \begin{aligned} P(X_{(1)}<1 \lvert X_{(3)}<2)&=\frac{P(X_{(1)}<1 \text{ and } X_{(3)}<2)}{P(X_{(3)}<2)} \\&=\frac{\frac{7}{64}}{\frac{1}{8}}=\frac{7}{8}=0.875 \end{aligned}

Based on the CDF in Example 1a, P(X_{(1)}>2)=1/8. The following is the conditional density f_{X_{(1)},X_{(3)}}(x,y \lvert X_{(1)}>2).

    ……\displaystyle \begin{aligned} f_{X_{(1)},X_{(3)}}(x,y \lvert X_{(1)}>2)&=\frac{f_{X_{(1)},X_{(3)}}(x,y)}{P(X_{(1)}>2)} \\&=\frac{\frac{3}{32} \ (y-x)}{1/8}\\&=\frac{3}{4} \ (y-x) \ \ \ \ \ \ \ \ 2<x<y<4\end{aligned}

The conditional mean E(X_{(3)} \lvert X_{(1)} >2) is evaluated by using the conditional density.

    ……\displaystyle \begin{aligned} E(X_{(3)} \lvert X_{(1)} >2)&=\int_2^4 \int_x^4 y \ \frac{3}{4} \ (y-x) \ dy \ dx \\&=\int_2^4 \int_x^4 \frac{3}{4} \ (y^2-xy) \ dy \ dx \\&=\int_2^4 \frac{3}{4} \ (\frac{64}{3}-8x + \frac{1}{6} x^3 ) \ dx=\frac{7}{2}=3.5 \end{aligned}

From Example 1c, we see that the unconditional mean of X_{(3)} is E(X_{(3)})=3. With the additional information that X_{(1)} >2, the expected value of X_{(3)} is now higher at 3.5.

Example 1e
This example focuses on the sample range R=X_{(3)}-X_{(1)}.

  • Evaluate the CDF and the density function of the sample range R.
  • Determine the mean and variance of the sample range R.
  • Evaluate the conditional mean E(R \lvert R >2).

The CDF of the sample range can be derived by evaluating the following integral (see Fact 5 in the preceding post).

    ……\displaystyle F_R(r)=\int_0^4 3 \ [F(x+r)-F(x)]^2 \ f(x) \ dx

Because the support of the uniform distribution on (0,4), the CDF F(x+r) is 1 when x+r=4 or x=4-r. Thus the integral should be split into 2 integrals. The limits of the first integral are from 0 to 4-r. The second integral goes from 4-r to 4.

    ……\displaystyle \begin{aligned} F_R(r)&=\int_0^4 3 \ [F(x+r)-F(x)]^2 \ f(x) \ dx \\&=\int_0^{4-r} 3 \biggl[\frac{x+r}{4}-\frac{x}{2} \biggr]^2 \ \frac{1}{4} \ dx+\int_{4-r}^4 3 \biggl[1-\frac{x}{4} \biggr]^2 \ \frac{1}{4} \ dx \\&=\int_0^{4-r} \frac{3}{64} r^2 \ dx+\int_{4-r}^4 3 \biggl[1-\frac{x}{4} \biggr]^2 \ \frac{1}{4} \ dx\\&=\frac{3}{64} r^2 (4-r) - \biggl[1-\frac{x}{4} \biggr]^3 \ \biggl \lvert_{4-r}^4 \\&=\frac{3}{16} r^2-\frac{2}{64} r^3 \ \ \ \ \ \ \ 0<r<4 \end{aligned}

The density function f_R(r) is obtained by differentiating the CDF. The following shows the remaining calculations.

    ……\displaystyle f_R(r)=\frac{6}{16} \ r-\frac{6}{64} \ r^2 \ \ \ \ \ \ 0<r<4

    ……\displaystyle E(R)=\int_0^4 r \ \biggl(\frac{6}{16} \ r-\frac{6}{64} \ r^2 \biggr) \ dr=2

    ……\displaystyle E(R^2)=\int_0^4 r^2 \ \biggl(\frac{6}{16} \ r-\frac{6}{64} \ r^2 \biggr) \ dr=\frac{24}{5}

    ……\displaystyle Var(R)=\frac{24}{5}-2^2=\frac{4}{5}

    ……\displaystyle \begin{aligned} E(R \lvert R>2)&=\frac{\displaystyle \int_2^4 r \ f_R(r) \ dr}{1-F_R(2)} \\&=\frac{\displaystyle \int_2^4 r \biggl(\frac{6}{16} \ r-\frac{6}{64} \ r^2 \biggr) \ dr}{1/2} \\&=\frac{11}{4}=2.75 \end{aligned}

Observe that the unconditional mean of the sample range is 2 while the mean is 2.75 given that the sample range is larger than 2.

More Examples

Example 1 is a long example demonstrating all the basic calculations with sampling from a uniform distribution. The next several examples involves sampling from an exponential distribution.

Example 2
Let X_1,X_2,X_3 be a random sample drawn from a population that has an exponential distribution with mean \theta where \theta>0. Consider the order statistics X_{(1)}<X_{(2)}<X_{(3)}. Determine the mean and variance for X_{(j)} where j=1,2,3.

The density function and the CDF of the exponential population are:

    ……\displaystyle f(x)=\frac{1}{\theta} \ e^{-x/\theta} \ \ \ \ \ \ x>0

    ……\displaystyle F(x)=1- e^{-x/\theta} \ \ \ \ \ \ x>0

The following gives the density functions of the three order statistics.

    ……\displaystyle f_{X_{(1)}}(x)=\frac{3!}{2!} \ \frac{1}{\theta} \ e^{-x/\theta} \ (e^{-x/\theta})^2=\frac{3}{\theta} \ e^{-3x/\theta} \ \ \ \ \ \ \ x>0

    ……\displaystyle \begin{aligned} f_{X_{(2)}}(x)&=\frac{3!}{1!} \ (1-e^{-x/\theta}) \ \frac{1}{\theta} \ e^{-x/\theta} \ (e^{-x/\theta}) \\&=3 \ \frac{2}{\theta} \ e^{-2x/\theta}-2 \ \frac{3}{\theta} \ e^{-3x/\theta} \ \ \ \ \ \ \ x>0 \end{aligned}

    ……\displaystyle \begin{aligned} f_{X_{(3)}}(x)&=\frac{3!}{2!} \ (1-e^{-x/\theta})^2 \ \frac{1}{\theta} \ e^{-x/\theta} \\&=3 \ \frac{1}{\theta} \ e^{-x/\theta}-3 \ \frac{2}{\theta} \ e^{-2x/\theta}+\frac{3}{\theta} \ e^{-3x/\theta} \ \ \ \ \ \ \ x>0 \end{aligned}

Note that the density f_{X_{(1)}}(x) is that of the exponential distribution with mean \theta/3. Since the variance of an exponential distribution is the square of its mean, we have the following information.

    ……\displaystyle E(X_{(1)})=\frac{\theta}{3}

    ……\displaystyle Var(X_{(1)})=\frac{\theta^2}{9}

Note that both f_{X_{(2)}}(x) and f_{X_{(3)}}(x) consist of sums of exponential densities times constants. For example, f_{X_{(3)}}(x) is three times an exponential density minus two times another exponential density. Thus its mean and second moment can be obtained by multiplying the same constants with exponential mean and second moment.

    ……\displaystyle E(X_{(2)})=3 \ \frac{\theta}{2}-2 \ \frac{\theta}{3}=\frac{5}{6} \ \theta

    ……\displaystyle E(X_{(2)}^2)=3 \ \biggl(2! \ \frac{\theta^2}{4} \biggr)-2 \ \biggl(2! \ \frac{\theta^2}{9} \biggr)=\frac{38}{36} \ \theta^2

    ……\displaystyle Var(X_{(2)})=\frac{38}{36} \ \theta^2- \biggl(\frac{5}{6} \ \theta \biggr)^2=\frac{13}{36} \ \theta^2

Using the same thought process, the following gives the mean and variance of X_{(3)}.

    ……\displaystyle E(X_{(3)})=3 \ \theta-3 \ \frac{\theta}{2}+\frac{\theta}{3}=\frac{11}{6} \ \theta

    ……\displaystyle E(X_{(3)}^2)=3 \ \biggl(2! \ \theta^2 \biggr) -3 \ \biggl(2! \ \frac{\theta^2}{4} \biggr)+ \ \biggl(2! \ \frac{\theta^2}{9} \biggr)=\frac{170}{36} \ \theta^2

    ……\displaystyle Var(X_{(3)})=\frac{170}{36} \ \theta^2- \biggl(\frac{11}{6} \ \theta \biggr)^2=\frac{49}{36} \ \theta^2

We make additional comments about these order statistics in the section below called “A Comparison of Estimators”.

Example 3
This is a continuation of Example 2. Consider the sample range R=X_{(3)}-X_{(1)}. Evaluate the mean and variance of the sample range.

The CDF of the sample range can be derived by evaluating the following integral (see Fact 5 in the preceding post).

    ……\displaystyle F_R(r)=\int_0^\infty 3 \ [F(x+r)-F(x)]^2 \ f(x) \ dx

Because the support of the exponential distribution is unbounded above, there is no need to split the integral.

    ……\displaystyle \begin{aligned} F_R(r)&=\int_0^\infty 3 \ [e^{-x/\theta}-e^{-(x+r)/\theta}]^2 \ \frac{1}{\theta} \ e^{-x/\theta} \ dx \\&=\int_0^\infty 3 \ e^{-2x/\theta} \ (1-e^{-r/\theta})^2 \ \frac{1}{\theta} e^{-x/\theta} \ dx \\&=\int_0^\infty \frac{3}{\theta} \ e^{-3x/\theta} \ (1-e^{-r/\theta})^2 \ dx \\&=(1-e^{-r/\theta})^2 \ \ \ \ \ \ \ r>0 \end{aligned}

Differentiating the CDF producing the density function. The following also shows the mean and variance of the sample range.

    ……\displaystyle \begin{aligned} f_R(r)&=2 \ (1-e^{-r/\theta}) \ \frac{1}{\theta} \ e^{-r/\theta} \\&=2 \ \frac{1}{\theta} \ e^{-r/\theta}-\frac{2}{\theta} \ e^{-2r/\theta} \ \ \ \ \ \ \ r>0 \end{aligned}

    ……\displaystyle E(R)=2 \ \theta-\frac{\theta}{2}=\frac{3}{2} \ \theta

    ……\displaystyle E(R^2)=2 \biggl(2! \ \theta^2 \biggr)-(2! \ \frac{\theta^2}{4} \biggr)=\frac{7}{2} \ \theta^2

    ……\displaystyle Var(R)=\frac{7}{2} \ \theta^2-\biggl( \frac{3}{2} \ \theta \biggr)^2=\frac{5}{4} \ \theta^2

Example 4
This is a continuation of Example 2 and Example 3. Use the sample range R=X_{(3)}-X_{(1)} to evaluate the the covariance Cov(X_{1},X_{(3)}) and the correlation coefficient of X_{(1)} and X_{(3)}.

The covariance of X_{(1)} and X_{(2)} is defined by

    ……Cov(X_{(1)},X_{(2)})=E(X_{(1)} X_{(2)})-E(X_{(1)}) \ E(X_{(2)})

Computing E(X_{(1)} X_{(2)}) requires the joint distribution of X_{(1)} and X_{(2)}. Because R=X_{(3)}-X_{(1)}, the following gives an alternative way.

    ……Var(R)=Var(X_{(1)})+Var(X_{(3)})-2 \ Cov(X_{(1)},X_{(2)})

All the variance terms are easily accessible (and have been calculated). Thus we can solve for the covariance term.

    ……\displaystyle \begin{aligned} Cov(X_{(1)},X_{(2)})&=\frac{1}{2} \ \biggl(Var(X_{(1)})+Var(X_{(3)}) -Var(R) \biggr) \\&=\frac{1}{2} \ \biggl(\frac{1}{9}\ \theta^2+\frac{49}{36} \ \theta^2 -\frac{5}{4} \ \theta^2 \biggr) \\&=\frac{1}{9} \ \theta^2 \end{aligned}

The following gives the correlation coefficient.

    ……\displaystyle \rho=\frac{Cov(X_{(1)},X_{(2)})}{\sigma_{X_{(1)}} \ \sigma_{X_{(3)}}} =\frac{\frac{1}{9} \ \theta^2}{\frac{2}{6} \ \theta \ \frac{7}{6} \ \theta} =\frac{2}{7}=0.2857

The correlation between the sample minimum and sample maximum is quite moderate.

Example 5
Examples 2, 3 and 4 are based on a small sample size sampling from the exponential population. We now look at sampling from exponential distribution in general, i.e. the sample size is arbitrary. Let X_1,X_2,\cdots,X_n be a sample drawn from an exponential population with mean \theta. Let’s examine the sample range R=X_{(n)}-X_{(1)}.

As shown in Example 3, the CDF of the sample range is obtained by evaluating the following integral.

    ……\displaystyle F_R(r)=\int_0^\infty n \ [F(x+r)-F(x)]^{n-1} \ f(x) \ dx

This integral is quite similar to the one in Example 3 and is evaluated in the same manner.

    ……\displaystyle \begin{aligned} F_R(r)&=\int_0^\infty n \ [e^{-x/\theta}-e^{-(x+r)/\theta}]^{n-1} \ \frac{1}{\theta} \ e^{-x/\theta} \ dx \\&=\int_0^\infty n \ e^{-(n-1)x/\theta} \ (1-e^{-r/\theta})^{n-1} \ \frac{1}{\theta} e^{-x/\theta} \ dx \\&=\int_0^\infty \frac{n}{\theta} \ e^{-nx/\theta} \ (1-e^{-r/\theta})^{n-1} \ dx \\&=(1-e^{-r/\theta})^{n-1} \ \ \ \ \ \ \ r>0 \end{aligned}

Thus the CDF of the sample range when sampling from an exponential population is simply the CDF of the exponential population raised to the sample size less one. Interestingly the CDF of the sample maximum when sampling from exponential distribution looks similar, except that the power is the sample size n.

    ……\displaystyle F_{X_{(n)}}(x)=(1-e^{-x/\theta})^n \ \ \ \ \ \ \ x>0

Of course, the distributional form may look the same, the sample range and the sample maximum have different distributions.

A Comparison of Estimators

We close out the post by commenting on Example 2. Recall that Example 2 focuses on the means and variances of the three order statistics when sampling from an exponential distribution with the sample size being 3. We now compare these three order statistics with the sample mean, which in this case is the sum of the three sample items divided by 3.

    ……\displaystyle \overline{X}=\frac{X_1+X_2+X_3}{3}

Of course, the sample mean \overline{X} is a natural candidate for an estimator of the exponential population mean \theta. The three order statistics X_{(1)}, X_{(2)} and X_{(3)} can also be used as estimators of the parameter \theta. For example, a case can be made that we use the sample median as an estimator of the population mean. How good are the order statistics in comparison with the sample mean? Note that the mean of \overline{X} is \theta, the parameter that the estimator \overline{X} is trying to estimate. Here’s the mean and variance of \overline{X}.

    ……\displaystyle E(\overline{X})=\theta

    ……\displaystyle Var(\overline{X})=\frac{\theta^2}{3}

The sample mean as an estimator of \theta has a very desirable property, i.e. on average the estimator is correct. There is no systematic bias. The sample mean \overline{X} never systematically over estimates \theta or under estimates \theta. The estimator \overline{X} is said to be an unbiased estimator of the parameter \theta. How about the order statistics? Are these three order statistics unbiased estimators of \theta?

The following table lists out the means and variances of \overline{X}, X_{(1)}, X_{(2)} and X_{(3)}.

Statistic …… …… …… …… Mean …… …… …… …… Variance
\overline{X} …… …… …… …… \theta …… …… …… …… \displaystyle \frac{12}{36} \ \theta^2
…… …… …… …… …… …… …… …… …… …… ……
X_{(1)} …… …… …… …… \displaystyle \frac{1}{3} \ \theta …… …… …… …… \displaystyle \frac{4}{36} \ \theta^2
…… …… …… …… …… …… …… …… …… …… ……
X_{(2)} …… …… …… …… \displaystyle \frac{5}{6} \ \theta …… …… …… …… \displaystyle \frac{13}{36} \ \theta^2
…… …… …… …… …… …… …… …… …… …… ……
X_{(3)} …… …… …… …… \displaystyle \frac{11}{6} \ \theta …… …… …… …… \displaystyle \frac{49}{36} \ \theta^2

The sample minimum X_{(1)} systematically under estimates the population mean. On average it is one third of the target. The sample median X_{(2)} also under estimates the population mean, though the under estimation is smaller. On average it is 5/6 of the target. The sample maximum X_{(3)} over estimates the population mean. On average it is 11/6 = 1.83 of the target. So it over estimates by about 83%.

Thus the three order statistics are not so desirable as estimators of the population mean of an exponential population as they are biased estimators of the population mean \theta. However, they can be remedied by multiplying with appropriate constants so that the expected value is \theta.

Statistic …… …… …… …… Mean …… …… …… …… Variance
\overline{X} …… …… …… …… \theta …… …… …… …… \displaystyle \frac{12}{36} \ \theta^2=0.33 \ \theta^2
…… …… …… …… …… …… …… …… …… …… ……
3 \ X_{(1)} …… …… …… …… \displaystyle \theta …… …… …… …… \displaystyle \theta^2
…… …… …… …… …… …… …… …… …… …… ……
\displaystyle \frac{6}{5} \ X_{(2)} …… …… …… …… \displaystyle \theta …… …… …… …… \displaystyle \frac{13}{25} \ \theta^2=0.52 \ \theta^2
…… …… …… …… …… …… …… …… …… …… ……
\displaystyle \frac{6}{11} \ X_{(3)} …… …… …… …… \displaystyle \theta …… …… …… …… \displaystyle \frac{49}{121} \ \theta^2=0.40 \ \theta^2

The above table shows that all 4 statistics (or estimators) have the same expected value. On average they are correct. Now that we have four unbiased estimators of the same parameter \theta, how do we choose the best one?

When the competing estimators are all unbiased estimators of the same target parameter, what is another property that can distinguish among the seemingly similar estimators? When the estimators are all unbiased, we would prefer the estimator that has the smallest variance. This is because using an estimator with a smaller variance guarantees that in repeated sampling a higher proportion of the values of the estimator will be close to the target parameter. Thus, the estimator with the smallest variance is the one that will more likely to produce good estimates. In addition to unbiasedness, we would prefer the variance of the distribution of the estimator to have a variance that is as small as possible.

Let’s look at the variance of these four estimators. The last table shows that the sample mean \overline{X} is the one with the smallest variance (among these four). The one with the largest variance is 3 X_{(1)}. So among these 4 estimators of the exponential population mean \theta, the sample mean is the one that is likely to be closer to the true mean even though they are all unbiased.

Even though the order statistics are not good candidate of the exponential population mean, there are situations where order statistics are good candidates for statistical estimation. The examples of order statistics in this post are also convenient examples for demonstrating the notions of unbiased estimators and the notion that smaller variance is better in comparing two estimators that are otherwise similar.

Practice problems on order statistics are found in this companion blog.

Dan Ma math

Daniel Ma mathematics

Dan Ma stats

Daniel Ma statistics

Dan Ma statistical

Daniel Ma statistical

\copyright 2018 – Dan Ma

This entry was posted in Probability, Statistics and tagged , . Bookmark the permalink.

One response to “Illustrative examples of order statistics”

  1. Pingback: Estimating percentiles using order statistics | Mathematical Statistics

Leave a reply to Estimating percentiles using order statistics | Mathematical Statistics Cancel reply