## The correlation coefficient of two random variables

2 12 2018

This post discusses the correlation coefficient of two random variables $X$ and $Y$.

Suppose that the joint behavior of the random variables $X$ and $Y$ is known and is described by the joint density function $f(x,y)$ where $(x,y)$ belongs to some appropriate region in the xy-plane. We are interested in knowing how one variable varies with respect to the other. For example, do $X$ and $Y$ move together? In other words, if $X$ increases, does $Y$ tend to increase? If $X$ increases, does $Y$ tend to decrease? If so, how strong is the dependence between $X$ and $Y$? If the two random variables do not move together, is there a measure that describes the lack of association? The correlation coefficient $\rho(X,Y)$ is a measure of the linear relationship between the random variables $X$ and $Y$.

We also discussion of concept of regression function or regression curve as well as the concept of least squares regression line. The correlation coefficient plays a central role in both concepts.

Practice problems to reinforce the concepts discussed here are found in this companion blog.

Covariance

To define the correlation coefficient $\rho(X,Y)$, we first define the covariance $\text{Cov}(X,Y)$.

.
(1)……..$\text{Cov}(X,Y)=E[(X-\mu_X)(Y-\mu_Y)]$
.

where $\mu_X=E(X)$ and $\mu_Y=E(Y)$. Thus the covariance measure is the average value of the product of the two deviations $X-\mu_X$ and $Y-\mu_Y$. A little algebra can show that the following is an equivalent formulation.

.
(2)……..$\text{Cov}(X,Y)=E[X Y]-\mu_X \ \mu_Y$
.

Based on (1), if more probabilities are assigned to the points $(X,Y)$ where the two deviations $X-\mu_X$ and $Y-\mu_Y$ have the same sign (both positive or both negative), then the covariance measure $\text{Cov}(X,Y)$ is positive. If more probabilities are assigned to the points $(X,Y)$ where the two deviations have the opposite signs (one is positive and the other is negative), then the covariance measure $\text{Cov}(X,Y)$ is negative. If positive and negative values of $(X-\mu_X)(Y-\mu_Y)$ cancel each other out, then $\text{Cov}(X,Y)=0$.

Furthermore, if the joint probabilities cluster in a positive direction (e.g. cluster around a straight line of positive slope), then the covariance measure $\text{Cov}(X,Y)$ is positive, in which case higher values of $X$ associates with higher values of $Y$. If the joint probabilities cluster in a negative direction (e.g. cluster around a straight line of negative slope), then the covariance measure $\text{Cov}(X,Y)$ is negative, in which case higher values of $X$ associates with lower values of $Y$. Thus the covariance is a measure of the dependence of two random variables. It gives the direction of the dependence (or association). It also indicates the strength of the association – the larger the measure in absolute value, the stronger the association.

The covariance of $X$ and $Y$ reflects the units of both random variable. As a result, it may be difficult to determine at a glance whether a covariance measure is large or small. The problem is eliminated by standardizing the covariance measure.

Correlation

The correlation coefficient is defined as follows:

.
(3)……..$\displaystyle \rho(X,Y)=E\biggl[\biggl(\frac{X-\mu_X}{\sigma_X}\biggr) \biggl(\frac{Y-\mu_Y}{\sigma_Y} \biggr)\biggr]$
.

where $\sigma_X$ is the standard deviation of $X$ and $\sigma_Y$ is the standard deviation of $Y$. If the underlying random variables are understood, we drop the $X$ and $Y$ and denote the correlation coefficient by $\rho$. Note that $\rho$ is the covariance of the two standardized variables $(X-\mu_X)/\sigma_X$ and $(Y-\mu_Y)/\sigma_Y$. Thus it is a dimensionless measure of dependence of two random variables, allowing for easy comparison across joint distributions. The following are equivalent expressions of $\rho$.

.
(4)……..$\displaystyle \rho(X,Y)=\frac{\text{Cov}(X,Y)}{\sigma_X \sigma_Y}$
.

.
(5)……..$\displaystyle \rho(X,Y)=\frac{E[X Y]-\mu_X \ \mu_Y}{\sigma_X \ \sigma_Y}$
.

The formulations (4) and (5) are more calculation friendly than (3). To calculate the correlation coefficient, calculate the covariance, preferably using (2). Then divide the convariance by the two standard deviations. The covariance and correlation coefficient are applicable for both continuous and discrete joint distributions of $X$ and $Y$. The examples given here are continuous joint distributions. For discrete examples, just replace integrals with summations.

.

Example 1
Suppose that the joint density function of $X$ and $Y$ is given by $f(x,y)=\frac{3}{2} xy$ where $x>0$, $y>0$ and $x+y<2$. Calculate the covariance and the correlation coefficient.

Using (2) and (4), we need to have $\mu_X$, $\sigma_X$, $\mu_Y$, $\sigma_Y$ and $E[X Y]$. We have the option of calculating these quantities using the joint density $f(x,y)$. Another option is to use the marginal density $f_X(x)$ to calculate $\mu_X$ and $\sigma_X$ and the marginal density $f_Y(y)$ to calculate $\mu_Y$ and $\sigma_Y$. We take this approach. Of course, the calculation of the quantity $E[X Y]$ must utilize $f(x,y)$. The following gives the marginal density functions.

.

……..\displaystyle \begin{aligned} f_X(x)&=\int_0^{2-x} \frac{3}{2} \ xy \ dy \\&=\frac{3}{4} \ x \ (2-x)^2 \\&=\frac{3}{4} \ x \ (4-4x+x^2)\\&=\frac{3}{4} \ (4x-4x^2+x^3) \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ 0

……..$\displaystyle f_Y(y)=\int_0^{2-y} \frac{3}{2} \ xy \ dx=\frac{3}{4} \ (4y-4y^2+y^3) \ \ \ \ \ \ \ 0

.

Use the marginal density functions to calculate the means and variances of $X$ and $Y$.

.
……..$\displaystyle E[X]=\int_0^2 x \ \frac{3}{4} \ (4x-4x^2+x^3) \ dx=\frac{4}{5}$

……..$\displaystyle E[X^2]=\int_0^2 x^2 \ \frac{3}{4} \ (4x-4x^2+x^3) \ dx=\frac{4}{5}$

……..$\displaystyle Var[X]=\frac{4}{5}- \biggl(\frac{4}{5} \biggr)^2=\frac{4}{25}=\sigma_X^2$

……..$\displaystyle E[Y]=\int_0^2 y \ \frac{3}{4} \ (4y-4y^2+y^3) \ dx=\frac{4}{5}$

……..$\displaystyle E[Y^2]=\int_0^2 y^2 \ \frac{3}{4} \ (4y-4y^2+y^3) \ dx=\frac{4}{5}$

……..$\displaystyle Var[Y]=\frac{4}{5}- \biggl(\frac{4}{5} \biggr)^2=\frac{4}{25}=\sigma_Y^2$
.

Use the joint density $f(x,y)$ to calculate $E[XY]$.

.

……..\displaystyle \begin{aligned} E[XY]&=\int_0^2 \int_0^{2-x} xy \ \frac{3}{2} xy \ dy \ dx \\&=\int_0^2 \int_0^{2-x} \frac{3}{2} \ x^2 y^2 \ dy \ dx \\&=\int_0^2 \frac{1}{2} \ x^2 (2-x)^3 \ dx =\frac{8}{15} \end{aligned}

.

The covariance and the correlation coefficient are computed as follows:

.
……..$\displaystyle \text{Cov}(X,Y)=E[XY]-E[X] E[Y]=\frac{-8}{75}$

……..$\displaystyle \rho=\frac{\text{Cov}(X,Y)}{\sigma_X \sigma_Y}=\frac{-2}{3}=-0.667$

.

The following diagram shows the support for the joint distribution in this example – the triangular area below the green line.

.

Figure 1

Based on the diagram, the negative $\rho$ is not surprising. It is clear that the larger the $X$, the smaller the $Y$ is and that the smaller the $X$, the larger the $Y$ is.

Rho is a Measure of Linear Dependence

The correlation coefficient is a standardized measure and is a measure of linear relationship between the two random variables. The following theorem makes this clear.

Theorem 1
For any two random variables $X$ and $Y$, the following statements are true.

• $-1 \le \rho \le 1$
• $\lvert \rho \lvert=1$ if and only of $Y=a+b X$ for some constants $a$ and $b$, except possibly on a set with zero probability.

Proof of Theorem 1
Let $X_1=(X-\mu_X)/\sigma_X$ and $Y_1=(Y-\mu_Y)/\sigma_Y$ be the standardized variables. Consider the variance of $X_1 \pm Y_1$.

.
……..\displaystyle \begin{aligned} Var(X_1 \pm Y_1)&=Var(X_1) \pm 2 \ \text{Cov}(X_1,Y_1) +Var(Y_1) \\&=1 \pm 2 \ \rho + 1 \\&=2 \pm 2 \ \rho \\&=2 [1 \pm \rho] \ge 0 \end{aligned}
.

If $\rho>1$, then $1-\rho$ would be negative. If $\rho<-1$, then $1+\rho$ would be negative. It follows that $-1 \le \rho \le 1$. For the second claim, note that if $\rho=1$, then $Var(X_1-Y_1)=0$ and if $\rho=-1$, then $Var(X_1+Y_1)=0$. A random variable with zero variance is a constant variable except possibly on a set with zero probability. The fact that $X_1-Y_1$ or $X_1+Y_1$ is constant means that $Y$ is a linear function of $X$.

For the other direction, suppose that $Y=a+ bX$ for some constants $a$ and $b$. The covariance and correlation coefficient are calculated as follows.

.
……..\displaystyle \begin{aligned} \text{Cov}(X,a+b X)&=E[X (a+b X)]-E[X] E[a+ b X] \\&=a E[X]+b E[X^2]-a E[X] - b E[X]^2 \\&=b \ Var(X)=b \ \sigma_X^2 \end{aligned}
.

……..$\displaystyle \sigma_Y^2=Var(a+ b X)=b^2 \ Var(X)=b^2 \ \sigma_X^2$
.

……..$\displaystyle \sigma_Y=\lvert b \lvert \ \sigma_X$
.

……..\displaystyle \begin{aligned} \rho&=\frac{\text{Cov}(X,a+b X)}{\sigma_X \sigma_Y} =\frac{b \ \sigma_X^2}{\sigma_X \ \lvert b \lvert \ \sigma_X}=\pm 1 \end{aligned}
.

The calculation of $\rho=\pm 1$ works as long as the linear relation $Y=a+b X$ holds every where except possibly on a set of zero probability. This completes the proof of Theorem 1. $\square$

Based on Theorem 1, the joint distribution of $X$ and $Y$ lies on a straight line if the correlation coefficient $\rho$ is 1 or -1. If the correlation coefficient $\rho$ is close to 1 or -1, the distribution of $X$ and $Y$ clusters around a straight line. Thus the correlation coefficient $\rho$ is a measure of linear dependence of $Y$ on $X$. It indicates both the direction and the strength of the linear dependence.

Regression Curves

In regression analysis, one focus is on estimating the relationship between a dependent variable (or response variable) and one or more independent variables (explanatory variables). Covariance and correlation coefficient do not distinguish between dependent variable and independent variable. The covariance $\text{Cov}(X,Y)$ is the same as $\text{Cov}(Y,X)$. Likewise the correlation coefficient $\rho(X,Y)$ is the same as $\rho(Y,X)$. Suppose that the random variable $Y$ is regarded as the response variable and $X$ is regarded as the explanatory variable. A common problem in regression analysis is to estimate the conditional expectation of the dependent variable $Y$ given the independent variable $X$. Thus given a realization of $X$, we would like to estimate $E[Y \lvert X=x]$. The conditional expected value $E[Y \lvert X=x]$ is called the regression curve (or regression function) of $Y$ on $X$.

The present discussion is from the view point that the joint distribution of $X$ and $Y$ is known. Of course, the joint distribution is rarely known in advance. So the discussion here is more like background information for subsequent discussion. Note that the regression curve $E[Y \lvert X]$ is the mean of the conditional distribution $Y \lvert X$. Given the joint density $f(x,y)$, derive the conditional density function $f(y \lvert x)$. Then compute the mean associated with $f(y \lvert x)$. Two examples illustrate this process.

Example 2
Let $f(x,y)=\frac{1}{2} \ xy$ where $0. Determine the regression curve $E[Y \lvert X]$.

To find the regression curve $E[Y \lvert X]$, we need to determine the distribution of the possible values of $Y$ given $X=x$. First, we need to determine the marginal distribution of $X$. The following is the marginal density function $f_X(x)$.

.
……..\displaystyle \begin{aligned} f_X(x)&=\int_x^2 f(x,y) \ dy =\int_x^2 \frac{1}{2} \ xy \ dy \\&=\frac{1}{4} \ x \ (4-x^2) \ \ \ \ \ \ \ \ \ \ \ \ 0
.

The conditional density and its conditional mean are then derived as follows:

.
……..\displaystyle \begin{aligned} f(y \lvert x)&=\frac{f(x,y)}{f_X(x)} \\&=\frac{\frac{1}{2} \ xy}{\frac{1}{4} \ x \ (4-x^2)}=\frac{2y}{4-x^2} \ \ \ \ \ \ \ \ \ \ x
.
……..\displaystyle \begin{aligned} E[Y \lvert x]&=\int_x^2 y \ f(y \lvert x) \ dy\\& =\int_x^2 y \ \frac{2y}{4-x^2} \ dy=\int_x^2 \frac{2y^2}{4-x^2} \ dy\\&=\frac{\frac{2}{3} \ (8-x^3)}{4-x^2}\\&=\frac{2 \ (4+2x+x^2)}{3(2+x)} \ \ \ \ \ \ \ \ 0

Here's a graph of the regression function $E[Y \lvert x]$.

.

Figure 2

The orange curve is the regression function $E[Y \lvert X=x]$ where $0. The green line is the line $y=x$. The support of the joint distribution of $X$ and $Y$ is the area above the green line.

The regression curve gives the mean response $Y$ for every fixed value of the explanatory variable $X$. For example, for $X=0.5$, the average response for Y is $1.4$. The possible $Y$ values for $X=0.5$ are randomly distributed around the mean 1.4.

Example 3
Consider the joint distribution in Example 1. The conditional density for $Y \lvert X=x$ is

.
……..\displaystyle \begin{aligned} f(y \lvert x)&=\frac{f(x,y)}{f_X(x)} \\&=\frac{\frac{3}{2} \ xy}{\frac{3}{4} \ x \ (2-x)^2}=\frac{2y}{(2-x)^2} \ \ \ \ \ \ \ \ \ \ 0
.

The mean of the conditional distribution is:

.
……..\displaystyle \begin{aligned} E[Y \lvert x]&=\int_0^{2-x} y \ f(y \lvert x) \ dy\\& =\int_0^{2-x} y \ \frac{2y^2}{(2-x)^2} \ dy \\&=\frac{\frac{2}{3} \ (2-x)^3}{(2-x)^2}\\&=\frac{2}{3} \ (2-x)=\frac{4}{3}-\frac{2}{3} x \ \ \ \ \ \ \ \ 0
.

The regression curve in this example is actually a regression line. The following diagram shows the regression line.

.

Figure 3

The triangular area below the green line is the support of the joint distribution. The orange line is the regression line $E[Y \lvert X=x]$. The orange line gives the average value of the dependent variable for a given value $x$ of the independent variable. For example, when $x=0.5$, the mean $Y$ is 1. The $Y$ values that are observed at random for $x=0.5$ are distributed according to the density $f(y \lvert 0.5)$.

When Regression Curve is a Straight Line

When the regression function is a straight line, the slope of the line is determined by the correlation coefficient.

Theorem 2
Suppose that $X$ and $Y$ are random variables such that for some constants $a$ and $b$ $E[Y \lvert x]=a+b x$ for all values of $x$ taken on by $X$. In other words, the regression curve of $Y$ on $X$ is a straight line. Then $E[Y \lvert x]$ is of the following form.

.
……..$\displaystyle E[Y \lvert x]=\mu_Y+ \rho \ \frac{\sigma_Y}{\sigma_X} \ (x-\mu_X)$
.
……..or $\displaystyle E[Y \lvert x]=\biggl( \mu_Y-\rho \ \frac{\sigma_Y}{\sigma_X} \ \mu_X \biggr)+\biggl( \rho \ \frac{\sigma_Y}{\sigma_X}\biggr) \ x$
.

where $\displaystyle b=\rho \ \frac{\sigma_Y}{\sigma_X}$ and $\displaystyle a=\mu_Y-b \ \mu_X$.

Proof of Theorem 2
We give a proof in the continuous case. For the discrete case, replace integrals with summations. First, establish the following fact.

.
……..$\displaystyle \int_{-\infty}^\infty E[Y \lvert x] \ f_X(x) \ dx =\mu_Y$
.

The following derivation establishes this fact.

.
……..\displaystyle \begin{aligned} \int_{-\infty}^\infty E[Y \lvert x] \ f_X(x) \ dx&=\int_{-\infty}^\infty \int_{\infty}^\infty y \ f(y \lvert x) \ dy \ f_X(x) \ dx\\& =\int_{-\infty}^\infty \int_{\infty}^\infty y \ \frac{f(x,y)}{f_X(x)} \ dy \ f_X(x) \ dx \\&=\int_{-\infty}^\infty \int_{\infty}^\infty y \ f(x,y) \ dy \ dx \\&=\int_{-\infty}^\infty y \ \int_{\infty}^\infty f(x,y) \ dx \ dy \ \ * \\&=\int_{-\infty}^\infty y \ f_Y(y) \ dy=\mu_Y \end{aligned}
.

The step with * is the result of changing the order of integration. The fact that has just been established is not surprising. It is saying that weighing $E[Y \lvert x]$, the conditional mean of $Y$ for $x$, with the density $f_X(x)$ is the unconditional mean $E[Y]$. Now we perform the same derivation but replacing $E[Y \lvert x]$ with $a+b X$.

.
……..\displaystyle \begin{aligned} \int_{-\infty}^\infty E[Y \lvert x] \ f_X(x) \ dx=\int_{-\infty}^\infty (a+b x) \ f_X(x) \ dx=a+b \ \mu_X=\mu_Y \end{aligned}
.

Immediately, $a=\mu_Y-b \ \mu_X$. Now $E[Y \lvert x]=a+b \ x$ becomes …………………. $E[Y \lvert x]=\mu_Y+b \ (x-\mu_X)$ and $E[Y \lvert x]-\mu_Y=b \ (x-\mu_X)$. Multiply both sides by $(x-\mu_X) \ f_X(x)$.

.
……..$\displaystyle [E[Y \lvert x]-\mu_Y]\ (x-\mu_X) \ f_X(x)=b \ (x-\mu_X)^2 \ f_X(x)$
.

The following further evaluates the left-hand-side of the above equation.

.
……..\displaystyle \begin{aligned} LHS&=[\biggl[\int_{-\infty}^\infty y \ \frac{f(x,y)}{f_X(x)} \ dy \biggr]-\mu_Y ]\ (x-\mu_X) \ f_X(x)\\&=\int_{-\infty}^\infty y \ (x-\mu_X) \ f(x,y) \ dy-\mu_Y \ (x-\mu_X) \ f_X(x)\\&=\int_{-\infty}^\infty y \ (x-\mu_X) \ f(x,y) \ dy-\mu_Y \ (x-\mu_X) \ \int_{-\infty}^\infty f(x,y) \ dy\\&=\int_{-\infty}^\infty (x-\mu_X) \ (y-\mu_Y) \ f(x,y) \ dy \end{aligned}
.

Setting the last line of the above equals to $b \ (x-\mu_X)^2 \ f_X(x)$.

.
……..$\displaystyle \int_{-\infty}^\infty (x-\mu_X) \ (y-\mu_Y) \ f(x,y) \ dy=b \ (x-\mu_X)^2 \ f_X(x)$
.

Integrate both sides with respect to $x$ and the left-hand-side becomes $\text{Cov}(X,Y)$.

.
……..$\displaystyle \text{Cov}(X,Y)=b \ \sigma_X^2$
.

The following gives the desired $b$.

.
……..\displaystyle \begin{aligned} b&=\frac{\text{Cov}(X,Y)}{\sigma_X^2}=\frac{\rho \ \sigma_X \ \sigma_Y}{\sigma_X^2}=\rho \ \frac{\sigma_Y}{\sigma_X} \end{aligned}
.

This completes the proof of Theorem 2. $\square$

Theorem 2 highlights the important role played by the correlation coefficient in linear regression. When the regression function is linear, its slope and y-intercept are obtained by $\rho$, $\mu_X$, $\mu_Y$, $\sigma_X$ and $\sigma_Y$. The following verifies that this is indeed the case with Example 3.

.
……..$\displaystyle b=\rho \ \frac{\sigma_Y}{\sigma_X}=\frac{-2}{3} \ \frac{2/5}{2/5}=\frac{-2}{3}$
.
……..$\displaystyle a=\mu_Y-b \ \mu_X=\frac{4}{5}+\frac{2}{3} \ \frac{4}{5}=\frac{4}{3}$

Least Squares Line

To gain more insight about the correlation coefficient $\rho$, consider fitting a straight line $y=a+b x$ to the joint probability distribution of $X$ and $Y$. Usually line fitting is done on observed data so that the fitted line can represent the overall behavior of the response variable $Y$ as the explanatory $X$ changes. In the present discussion we do not have observed data. Instead the joint probability distribution of $X$ and $Y$ is a given. Thus there is no need to estimate the mean of $Y$ given $X=x$. The goal here is to demonstrate the important roles played by the correlation coefficient $\rho$ in the topic of linear regression. Thus the ideas discussed here are background information useful for a subsequent discussion of linear regression.

We wish to fit a line $y=a+b x$ through the support of a joint distribution so that the line approximates the relationship between $X$ and $Y$. For example, in Figure 1 and Figure 3, the support is the triangular area below the green line. In Figure 3, the regression curve $E[Y \lvert x]$ is a straight line (the orange line) and we can take that as a fitted line in the triangular area. How good is that fitted line? In Figure 2, the regression curve is not a straight line. So we would like to fit a line in the triangular area above the green line.

There are many lines that can be drawn in the triangular area in Figure 2. The form of the fitted line is $Y=a+b X$. When a point $(X,Y)$ in the support is not on the fitted line, the deviation from the line is $Y-a-b X$. It is a good goal that the total amount of deviation is as small as possible. The criterion we wish to use is the least squares criterion. This means that the total of the squares of the deviations is as small as possible. More specifically, we wish to find $a$ and $b$ such that the following expectation is minimized.

.
……..$\displaystyle W(a,b)=E[(Y-(a+b X))^2]=E[(Y-a-b X)^2]$
.

To find the $a$ and $b$ such that $W(a,b)$ is the least, we find $\partial W/ \partial a$ and $\partial W/ \partial b$, the partial derivatives of $W$. Then solve the two equations $\partial W/ \partial a=0$ and $\partial W/ \partial b=0$ simultaneously for $a$ and $b$. Interestingly, the answers look similar to the line indicated in Theorem 2. The results are stated in the following theorem.

Theorem 3
Suppose that the random variables $X$ and $Y$ have means $\mu_X$ and $\mu_Y$, variances $\sigma_X^2$ and $\sigma_Y^2$, and correlation coefficient $\rho$. The line $Y=a + b X$ such that ………… $W(a,b)=E[(Y-a-b X)^2]$ is at a minimum has the following y-intercept $a$ and slope $b$.

……..$\displaystyle a=\mu_Y-b \ \mu_X$
.
……..$\displaystyle b=\rho \ \frac{\sigma_Y}{\sigma_X}$
.

The line indicated in Theorem 3 is called the least squares regression line. This straight line is identical to the line indicated in Theorem 2. For the probability distribution in Example 1 and Example 3, the regression curve and the least squares regression line are the same. For the distribution in Example 2, the regression curve is not a straight line. The following gives the least squares regression line for Example 2.

.

……..\displaystyle \begin{aligned} &\mu_X=\frac{16}{15} \ \ \ \ \ \ \sigma_X^2=\frac{44}{225} \\&\mu_Y=\frac{8}{5} \ \ \ \ \ \ \ \ \sigma_Y^2=\frac{24}{225} \ \ \ \ \ \ \ \rho=\frac{4}{\sqrt{66}}\\&b=\rho \ \frac{\sigma_Y}{\sigma_X}=\frac{4}{11}\\&a=\mu_Y-b \ \mu_X=\frac{40}{33}\\&y=\frac{40}{33}+\frac{4}{11} \ x \end{aligned}

.

The following diagram shows both the regression curve and the least squares line for Example 2.

.

Figure 4

The blue straight line is the least squares line. The orange curve is the regression curve. Though they are different, there is a general agreement between the two in a certain range. In Example 2, the least squares line is a good approximation of the regression curve in a certain interval containing $\mu_X$.

Remarks

The correlation coefficient is a measure of linear relationship between two random variables. A regression function (regression curve) is $E[Y \lvert X=x]$, the expected value of the dependent variable $Y$ for a given value of the independent variable $X$. The regression curve $E[Y \lvert X=x]$ may or may not be a linear function. When it is, it has the form indicated in Theorem 2.

.
……..$\displaystyle E[Y \lvert x]=\biggl( \mu_Y-\rho \ \frac{\sigma_Y}{\sigma_X} \ \mu_X \biggr)+\biggl( \rho \ \frac{\sigma_Y}{\sigma_X}\biggr) \ x$
.

Interestingly, the least squares regression line for a given joint distribution also has the same form as the above line. The slope of the least squares regression line is a function of the correlation coefficient. More specifically, the slope is $\rho$ multiplied by the ratio $\sigma_Y / \sigma_X$. When the two random variables are positively correlated, the slope of the least squares regression line is positive. When the two random variables are negatively correlated, the slope of the least squares regression line is negative. The least squares regression line discussed here can be called the population least squares regression line since it is calculated from the distribution that describes the population. The joint distribution for the population is rarely known in advance. The least squares line is usually calculated from observed data. This will be a topic in subsequent posts.

Practice problems to reinforce the concepts discussed here are found in this companion blog.

.

Dan Ma math

Daniel Ma mathematics

Dan Ma stats

Daniel Ma statistics

Dan Ma statistical

Daniel Ma statistical

$\copyright$ 2018 – Dan Ma

### 4 responses

7 12 2018

[…] provides practice problems to reinforce the concept of correlation coefficient discussed in this post in a companion blog. The post in the companion blog shows how to evaluate the covariance and the […]

Like

15 12 2018

[…] concept of covariance and correlation coefficient is given detailed treatment in this post in a companion blog. Practice problems are available […]

Like

18 12 2018

[…] to Theorem 2 in this previous post, whenever the conditional mean is a linear function, it must be of the form exactly as described […]

Like

1 01 2019

[…] mean is a linear function, it must be of the exact same form as in (2) (this fact is Theorem 2 in this previous post). Given that the linear form of is part of the definition of bivariate normal, equation (2) is not […]

Like