Covariance and correlation are two of the most fundamental concepts in statistics, data science, machine learning, finance, genomics, and many other quantitative fields.
Both concepts attempt to answer an important question:
How do two variables change together?
For example:
Covariance and correlation help quantify these relationships mathematically.
Although they are closely related, they are not identical.
This tutorial explains:
Before discussing covariance, it is important to understand variance.
Variance measures:
how much a variable varies around its mean.
For a variable ( X ):
$ (X) = _{i=1}^{n} (X_i - {X})^2 $
Variance tells us:
Covariance extends this idea to two variables simultaneously.
Covariance measures:
how two variables vary together.
It tells us whether:
For two variables ( X ) and ( Y ):
$ (X,Y) = _{i=1}^{n} (X_i - {X})(Y_i - {Y}) $
where:
Consider the term:
$ (X_i - {X})(Y_i - {Y}) $
There are several possibilities.
If:
then:
[ (+)(+) = + ]
This contributes positively.
Similarly:
Thus:
positive covariance indicates variables move together.
If:
then:
$ (+)(-) = - $
This contributes negatively.
Negative covariance indicates opposite movement.
| Covariance | Interpretation |
|---|---|
| Positive | Variables increase together |
| Negative | One increases while the other decreases |
| Near zero | Weak or no linear relationship |
Covariance depends on scale.
For example:
This makes covariance difficult to interpret directly.
To solve this problem, we use correlation.
Correlation is a standardized version of covariance.
It measures:
both the strength and direction of a linear relationship.
Unlike covariance, correlation is scale-independent.
The Pearson correlation coefficient is:
$ r = {_X _Y} $
where:
By dividing covariance by standard deviations:
Correlation always lies between:
$ -1 r $
| Correlation | Interpretation |
|---|---|
| +1 | perfect positive relationship |
| +0.8 | strong positive relationship |
| +0.5 | moderate positive relationship |
| 0 | no linear relationship |
| -0.5 | moderate negative relationship |
| -1 | perfect negative relationship |
Positive correlation means:
Examples:
Negative correlation means:
Examples:
Zero correlation indicates:
no linear relationship.
However, this does NOT always mean the variables are unrelated.
They may still have:
One of the most important statistical principles is:
correlation does not imply causation.
If two variables are correlated, it does not necessarily mean:
There may be:
Suppose:
This does not mean:
ice cream causes drowning.
The hidden factor is:
In multivariate statistics, covariance is often represented as a matrix.
For variables ( X_1, X_2, X_3 ):
$ = \[\begin{bmatrix} \text{Var}(X_1) & \text{Cov}(X_1,X_2) & \text{Cov}(X_1,X_3) \\ \text{Cov}(X_2,X_1) & \text{Var}(X_2) & \text{Cov}(X_2,X_3) \\ \text{Cov}(X_3,X_1) & \text{Cov}(X_3,X_2) & \text{Var}(X_3) \end{bmatrix}\]$
Covariance matrices are fundamental in:
Similarly, correlation matrices contain pairwise correlations.
All diagonal values equal 1 because:
$ (X,X) = 1 $
| Feature | Covariance | Correlation |
|---|---|---|
| Measures joint variability | Yes | Yes |
| Indicates direction | Yes | Yes |
| Standardized | No | Yes |
| Scale dependent | Yes | No |
| Range fixed | No | Yes (-1 to 1) |
| Easy interpretation | Difficult | Easier |
These concepts are extremely important in genetics.
Examples include:
For example:
Principal Component Analysis (PCA) operates on the covariance matrix.
PCA identifies directions of maximum variance.
This is widely used in:
# Python Example: Covariance and Correlation
import numpy as np
import pandas as pd
np.random.seed(42)
# Simulated data
x = np.random.normal(10, 2, 100)
y = x * 0.8 + np.random.normal(0, 1, 100)
df = pd.DataFrame({
"X": x,
"Y": y
})
df.head()
| X | Y | |
|---|---|---|
| 0 | 10.993428 | 7.379372 |
| 1 | 9.723471 | 7.358132 |
| 2 | 11.295377 | 8.693587 |
| 3 | 13.046060 | 9.634571 |
| 4 | 9.531693 | 7.464069 |
# Computing Covariance
covariance = df.cov()
print(covariance)
# Computing Correlation
correlation = df.corr()
print(correlation)
X Y
X 3.299080 2.402955
Y 2.402955 2.642802
X Y
X 1.000000 0.813799
Y 0.813799 1.000000# Visualizing the Relationship
import matplotlib.pyplot as plt
plt.figure(figsize=(6,5))
plt.scatter(df["X"], df["Y"])
plt.xlabel("X")
plt.ylabel("Y")
plt.title("Relationship Between X and Y")
plt.show()
# Visualizing Covariance Matrix
plt.figure(figsize=(5,4))
plt.imshow(covariance)
plt.colorbar(label="Covariance")
plt.xticks([0,1], ["X","Y"])
plt.yticks([0,1], ["X","Y"])
plt.title("Covariance Matrix")
plt.show()
# Visualizing Correlation Matrix
plt.figure(figsize=(5,4))
plt.imshow(correlation)
plt.colorbar(label="Correlation")
plt.xticks([0,1], ["X","Y"])
plt.yticks([0,1], ["X","Y"])
plt.title("Correlation Matrix")
plt.show()
Covariance matrices appear everywhere in statistical genetics because genetics itself is fundamentally about shared variation and dependence.
Individuals in genetic datasets are not completely independent. They may:
A covariance matrix provides a mathematical way to describe these relationships.
In statistical genetics, many methods are essentially trying to answer questions like:
All of these involve covariance structures.
Variance measures variability within one variable:
[ (X) ]
Covariance measures joint variability between two variables:
[ (X,Y) ]
A covariance matrix extends this idea to many variables simultaneously.
For variables ( X_1, X_2, …, X_n ):
$ = \[\begin{bmatrix} \text{Var}(X_1) & \text{Cov}(X_1,X_2) & \cdots \\ \text{Cov}(X_2,X_1) & \text{Var}(X_2) & \cdots \\ \vdots & \vdots & \ddots \end{bmatrix}\]$
This matrix summarizes the dependence structure of the system.
One of the most important covariance matrices in genetics is the Genetic Relationship Matrix (GRM).
The GRM measures genome-wide genetic similarity between individuals.
$ G = ZZ^T $
where:
The GRM is essentially a covariance matrix between individuals based on genotype data.
Interpretation:
The GRM allows mixed models to account for:
This is critical in GWAS.
Modern GWAS often use linear mixed models:
$ y = X+ g + $
with:
$ g N(0, G_g^2) $
This means:
The covariance matrix defines how phenotype correlations arise from genotype similarity.
Methods like GCTA-GREML estimate SNP heritability using covariance structures.
Core idea:
If genetically similar individuals also have similar phenotypes, then genetics explains part of the trait variance.
This is fundamentally a covariance argument.
The model compares:
Linkage disequilibrium measures correlation between nearby SNPs.
LD matrices are covariance/correlation matrices between variants.
Example:
[ R = (SNP_i, SNP_j) ]
LD matrices are essential in:
Population stratification correction often uses PCA.
PCA is performed on covariance-like structures derived from genotype matrices.
The eigenvectors of these covariance matrices reveal ancestry axes.
Thus, covariance matrices help uncover:
Methods like LDSC estimate genetic correlation between traits.
Example:
These methods estimate covariance of SNP effect sizes across traits.
Positive covariance suggests shared biology.
Multivariate genetic models directly model covariance between traits.
For example:
$ _G = \[\begin{bmatrix} \text{Var}(Trait_1) & \text{Cov}(Trait_1,Trait_2) \\ \text{Cov}(Trait_2,Trait_1) & \text{Var}(Trait_2) \end{bmatrix}\]$
This allows researchers to estimate:
Bayesian methods heavily rely on covariance structures.
Examples include:
Covariance matrices define prior dependence between variables.
Modern genomics involves:
These datasets contain thousands of correlated variables.
Covariance matrices summarize their dependence structure efficiently.
Genes do not function independently.
They operate in:
Covariance matrices naturally describe coordinated biological behavior.
In many classical statistical problems, the main focus is estimating mean effects.
In statistical genetics, much of the focus shifts toward:
This is because most complex traits are highly polygenic.
Thousands of variants contribute tiny correlated effects.
Many statistical genetics methods are based on this idea:
Similar genomes should produce similar phenotypes.
This statement is fundamentally about covariance.
If:
then genetic effects exist.
This simple idea underlies:
Covariance matrices appear in:
| Method | Covariance Matrix Used |
|---|---|
| GCTA | GRM |
| PCA | Covariance matrix |
| LDSC | LD covariance |
| SBayesR | LD matrix |
| Mixed models | Random effect covariance |
| Genomic prediction | Genomic covariance |
| eQTL analysis | Expression covariance |
| TWAS | SNP-expression covariance |
Covariance matrices are central in statistical genetics because genetics is fundamentally about shared variation and dependence.
They allow researchers to model:
Modern statistical genetics is therefore heavily built around covariance structures and multivariate modeling.
Covariance measures how two variables vary together, while correlation measures the strength and direction of their standardized linear relationship.
Covariance is scale-dependent and difficult to compare across datasets, whereas correlation is standardized between -1 and 1 and easier to interpret.
These concepts are foundational in:
Understanding covariance and correlation is essential for interpreting relationships between variables and for building more advanced statistical models.