Introduction to the Likelihood Ratio Test

The Likelihood Ratio Test (LRT) is a fundamental statistical tool for comparing the fit of two competing models. It evaluates whether a more complex model provides a significantly better explanation of the observed data than a simpler, nested baseline model. By quantifying the improvement in likelihood, the LRT helps researchers make principled decisions about model complexity, reducing the risk of overfitting while still capturing meaningful patterns. This test is widely applied in fields such as biostatistics, econometrics, machine learning, and social sciences, where model selection is critical for inference and prediction.

The core idea of the LRT dates back to the work of British statistician Sir Ronald Fisher and was later formalized by Samuel S. Wilks in 1938. Wilks demonstrated that under certain conditions, the test statistic asymptotically follows a chi-square distribution, making it practical for hypothesis testing. Today, the LRT remains a cornerstone of likelihood-based inference, often used alongside information criteria like AIC and BIC.

When to Use the Likelihood Ratio Test

The Likelihood Ratio Test is most appropriate when comparing nested models — that is, when the simpler model is a special case of the more complex model. For example, a linear regression model without interaction terms is nested within a model that includes them, or a model with only main effects is nested within a model that includes quadratic terms. In such cases, the LRT directly tests whether the additional parameters improve the model fit beyond what would be expected by chance.

Alternative tests for model comparison include the Wald test and the Score (Lagrange Multiplier) test. Each has strengths and weaknesses:

  • Wald test: Requires only estimation of the full model and is computationally simpler for testing individual coefficients. However, it can be less reliable for small samples or when parameters are near the boundary of the parameter space.
  • Score test: Only requires estimation of the restricted model and is often used in model building. It is particularly useful when the full model is difficult to estimate.
  • Likelihood Ratio Test: Generally considered the most reliable for moderate to large samples, as it compares the full likelihoods of both models directly. It requires fitting both models but is invariant to parameterization and often more powerful.

Choose the LRT when you have the computational resources to fit both models and you need a robust, well-behaved test statistic. It is especially favored in generalized linear models (GLMs), mixed-effects models, and structural equation modeling.

Assumptions of the Likelihood Ratio Test

For the LRT to produce valid results, the following assumptions should be met:

  • Nested models: The null model must be a special case of the alternative model, obtained by setting some parameters to fixed values (usually zero).
  • Regularity conditions: The models must satisfy standard regularity conditions for maximum likelihood estimation (MLE): the true parameter is in the interior of the parameter space, the likelihood function is smooth, and the expected Fisher information matrix is positive definite. This excludes boundary parameters (e.g., variance components at zero) where the asymptotic distribution may break down.
  • Sample size: The LRT is an asymptotic test — it relies on large-sample properties. While it often performs well with moderate sample sizes (e.g., n > 30 or n > 50 depending on model complexity), small samples can lead to inflated Type I error rates. Correction methods (e.g., Bartlett correction) exist but are beyond the scope of this guide.
  • Independent and identically distributed (i.i.d.) observations: Most standard forms of the LRT assume independent observations. For clustered or time-series data, robust versions or alternative tests may be needed.

If these assumptions are violated, the test statistic may not follow the chi-square distribution, and the p-value may be unreliable. In such cases, bootstrapping or likelihood ratio tests with robust covariance estimates can be used.

Step-by-Step Procedure for Performing the Likelihood Ratio Test

The LRT procedure can be broken down into clear, reproducible steps:

Step 1: Specify the Null and Alternative Models

Define the simpler model (null, H0) and the more complex model (alternative, H1). Ensure they are nested. For example, in a logistic regression predicting diabetes (yes/no), the null model might include only age, while the alternative model adds BMI and family history.

Step 2: Fit Both Models and Obtain Log-Likelihoods

Use maximum likelihood estimation to fit each model to the data. Record the log-likelihood (the natural logarithm of the likelihood function evaluated at the MLE). Most statistical software provides the log-likelihood directly (e.g., logLik in R, llf in Stata, log_lik in Python's scikit-learn or statsmodels). The log-likelihood is always negative or zero, with higher (closer to zero) values indicating better fit.

Step 3: Compute the Test Statistic

The LRT statistic is:

D = 2 × (log-likelihoodalternative – log-likelihoodnull)

This value is always nonnegative because the alternative model cannot have a lower log-likelihood than the null (since it contains the null as a special case). A large D indicates that the alternative model explains significantly more variation in the data.

Step 4: Determine the Degrees of Freedom

Degrees of freedom (df) equal the difference in the number of freely estimated parameters between the two models. For example, if the null model has 3 parameters (intercept + 2 predictors) and the alternative has 5 parameters (intercept + 4 predictors), df = 2.

Note: When parameters are constrained (e.g., variance components fixed to zero), the df calculation may not be simply the count difference — boundary issues can cause the asymptotic distribution to be a mixture of chi-squares.

Step 5: Compute the p-value

Under the null hypothesis, the test statistic D follows a chi-square distribution with df equal to the difference in parameters. The p-value is the probability of observing a statistic as extreme as D under that distribution. Use a chi-square table or computational tools:

  • In R: pchisq(D, df, lower.tail = FALSE)
  • In Python: 1 - scipy.stats.chi2.cdf(D, df)
  • Many software packages automate this step (e.g., lrtest() in R's lmtest package or anova() in R for nested models).

Step 6: Make a Decision

Compare the p-value to your chosen significance level (commonly α = 0.05). If p < α, reject the null hypothesis in favor of the alternative model, concluding that the additional parameters provide a statistically significant improvement in fit. If p ≥ α, fail to reject the null, meaning the simpler model is adequate.

Remember: Statistical significance does not guarantee practical importance. Consider effect sizes and the substantive context.

Practical Example: Comparing Linear Regression Models

Let's illustrate the LRT with a concrete example. Suppose we have data on 200 individuals with variables: income (dependent, in thousands of dollars), age (years), education (years of schooling), and experience (years of work). We want to test whether adding education and experience improves prediction of income beyond age alone.

Models

  • Null model (M0): income = β₀ + β₁·age + ε
  • Alternative model (M1): income = β₀ + β₁·age + β₂·education + β₃·experience + ε

These models are nested: M1 includes all terms of M0 plus two extra parameters (β₂ and β₃). The degrees of freedom for the test = 2.

Fitting and Log-Likelihoods

After fitting both models using ordinary least squares (which under normality yields MLE), we obtain:

  • Log-likelihood for M0: -602.3
  • Log-likelihood for M1: -589.7

Notice M1 has a higher (less negative) log-likelihood, indicating better fit.

Test Statistic

D = 2 × (-589.7 – (-602.3)) = 2 × 12.6 = 25.2

p-value Calculation

With 2 degrees of freedom, we look up the chi-square distribution: pchisq(25.2, 2, lower.tail = FALSE) yields p ≈ 3.4 × 10⁻⁶. This is far below 0.05.

Conclusion

We reject the null model. Adding education and experience significantly improves the prediction of income. In practice, one would then examine individual coefficients, check residual diagnostics, and possibly consider further models (e.g., interactions).

Implementation in R

For R users, the LRT can be performed using the lrtest function from the lmtest package:


library(lmtest)
M0 <- lm(income ~ age, data = mydata)
M1 <- lm(income ~ age + education + experience, data = mydata)
lrtest(M0, M1)

This automatically computes D, df, and p-value. Alternatively, anova(M0, M1, test = "LRT") works for nested linear models and generalized linear models.

Interpreting Results and Common Pitfalls

Beyond the p-value, consider the following when using the LRT:

Effect Size and Practical Significance

A highly significant LRT can occur with trivial improvements if the sample size is large. Always examine the magnitude of the improvement — for example, the increase in R-squared or decrease in AIC. A significant test with a negligible effect may not warrant model complexity.

Multiple Comparisons

If you are testing many nested models sequentially (e.g., forward selection), the repeated use of LRT inflates Type I error. Use corrections such as Bonferroni or use information criteria (AIC, BIC) for model selection. The LRT is best for testing a specific, pre-planned hypothesis.

Boundary Issues

When testing whether a variance parameter is zero (e.g., in random-effects models), the null hypothesis lies on the boundary of the parameter space. In such cases, the test statistic follows a mixture of chi-square distributions (e.g., 50:50 mixture of χ²₀ and χ²₁ for a single variance). Specialized procedures are needed; standard LRT p-values are too conservative.

Sample Size and Asymptotics

For small sample sizes, the chi-square approximation may be poor. Consider using a parametric bootstrap: simulate data under the null distribution, compute the LRT statistic for each simulated dataset, and derive an empirical p-value. This approach is more reliable but computationally intensive.

Non-Nested Models

The LRT cannot be used to compare non-nested models (e.g., linear model vs. cubic model without nesting). For such comparisons, use AIC, BIC, or the Vuong test. The Cox test is another option for non-nested hypotheses.

Conclusion

The Likelihood Ratio Test is a powerful, asymptotic method for comparing nested models. By providing a statistically rigorous framework to decide whether additional complexity is justified, it enables researchers to build parsimonious yet accurate models. We have covered its theoretical foundation, assumptions, step-by-step application, and a practical example using regression. Remember to verify the assumptions — especially nesting, sample size, and regularity conditions — and to complement the LRT with effect size measures and diagnostic checks. When used appropriately, the LRT is an indispensable tool in the statistician's arsenal for data-driven decision-making.

For further reading, explore these external resources: