Depdence

Introduction to Dependence Metrics

Recommended Prerequesites

1. Probability
2. Probability II
3. Introduction to Multivariate Distributions
4. Distribution Distance

General Types

Autocorrelation

Autocorrelation measures the correlation of a time series with a lagged version of itself.

Given a time series $\left\{X_t\right\}$ where $t=1,2,\dots ,n$, the autocorrelation at lag k is defined as: $$\rho (k)=\frac{\text{Cov}(X_t,X_{t-k})}{\text{Var}(X_t)}$$ Autocorrelation is used in time series analysis to detect trends, seasonality, and repeating patterns. It plays a key role in models like AR (Auto-Regressive) models, where current values are modeled as linear functions of past values.

Pairwise Independence

Pairwise Independence is a weaker condition of independence between random variables. Two variables are pairwise independent if knowing the outcome of one provides no information about the outcome of the other, but pairwise independence does not necessarily imply full independence across all variables.

Random variables $X_1,X_2,\dots ,X_n$ are pairwise independent if for all $i\ne j$: $$P(X_i\cap X_j)=P(X_i)P(X_j)$$

Conditional Dependence

Conditional Independence refers to a situation where two random variables are independent, given the value of a third variable. Conditional independence is a key concept in Bayesian networks, graphical models, and statistical modeling, where it allows for simplified representations of complex dependencies.

Let $X,Y,Z$ be random variables. X and Y are said to be conditionally independent given Z if: $$P(X|Y,Z)=P(X|Z)$$

Metrics

Spearman Correlation

Spearman correlation assesses how well the relationship between two variables can be described using a monotonic function. $$\rho =1-\frac{6\sum _{i=1}^n(R_X(i)-R_Y(i){)}^2}{n(n^2-1)}$$

Kendall's Tau

Let $(X_1,Y_1),(X_2,Y_2),\dots ,(X_n,Y_n)$ be a set of paired observations. This could be thought of as having been at the same time point, for example. Compare the pairs at two indices. Label the pair as:

• Concordant if $(X_i-X_j)(Y_i-Y_j)>0$
• Discordant if $(X_i-X_j)(Y_i-Y_j)<0$

In plain English, this means that if X is greater at time i than at time j, then Y is concordant if Y is greater at time i than it was at time j. If the two variables have the opposite relationship, it is discordant.

Now, let us define the counts for each type of pair; i.e.: $$C=\sum _i^{n-1}\sum _{j=i+1}^n{1}_{(X_i-X_j)(Y_i-Y_j)>0}$$ $$D=\sum _i^{n-1}\sum _{j=i+1}^n{1}_{(X_i-X_j)(Y_i-Y_j)<0}$$ Then we define Kendall's Tau as $$\tau =\frac{C-D}{(\genfrac{}{}{0ex}{}{n}{2})}$$

Mutual Information

Unlike Kendall's Tau and Spearman's Rho, which assess specific types of dependence (such as monotonic or rank-based), MI captures all forms of dependence, including non-linear and complex relationships. $$I(X,Y)=\int \int p_{XY}(x,y)\log \left(\frac{p_{XY}(x,y)}{p_X(x)p_Y(y)}\right)dxdy$$ where $p_{XY}(x,y)$ is the joint probability density function of X and Y; $p_X(x)$ and $p_Y(y)$ are the marginal densities of X and Y, respectively.

Tail Dependence

Tail dependence refers to the behavior of extreme values in multivariate distributions. It captures the likelihood that extreme outcomes in one variable will be accompanied by extreme outcomes in another.

For two random variables $X$ and $Y$, the upper tail dependence coefficient ${\lambda }_U$ is defined as the limit of the conditional probability that Y exceeds a high threshold, given that X also exceeds that threshold: $${\lambda }_U=\underset{u\to 1^{-}}{lim}P(Y>F_Y^{-1}(u)|X>F_X^{-1}(u))$$ where $F_X$ and $F_Y$ are the marginal cumulative functions of X and Y respectively, and $F_X^{-1}(u)$ and $F_Y^{-1}(u)$ are the quantiles corresponding to the probability level u.

Similarly, the lower tail dependence coefficient ${\lambda }_L$ is defined as: $${\lambda }_L=\underset{u\to 0^{+}}{lim}P(Y\le F_Y^{-1}(u)|X\le F_X^{-1}(u))$$ $${\lambda }_U,{\lambda }_L\in [0,1]$$ A value of 0 indicates no tail dependence. A value of 1 implies perfect tail dependence.

Dependence Practice Problems

1. Given the following data for two variables $X$ and $Y$:

   X	Y
   10	12
   20	18
   30	30
   40	42
   50	45

   1. Rank the values of $X$ and $Y$
   2. Calculate the Spearman rank correlation coefficient $\rho$ for this data.
   3. Count the number of concordant and discordant pairs.
   4. Compute Kendall's tau $\tau$ for this data.
2. Given the following data for two variables $X$ and $Y$:

   X	Y
   1	9
   2	7.5
   3	7
   4	4.5
   5	6.5
   6	5.5
   8	5
   9	6
   10	3.5

   1. Rank the values of $X$ and $Y$
   2. Calculate the Spearman rank correlation coefficient $\rho$ for this data.
   3. Count the number of concordant and discordant pairs.
   4. Compute Kendall's tau $\tau$ for this data.
3. Given the following data for two variables $X$ and $Y$:

   X	Y
   0	0.5
   5	5
   10	-3
   15	-4
   20	4
   25	6
   30	0
   35	-2
   40	-1

   1. Rank the values of $X$ and $Y$
   2. Calculate the Spearman rank correlation coefficient $\rho$ for this data.
   3. Count the number of concordant and discordant pairs.
   4. Compute Kendall's tau $\tau$ for this data.