How to Use the Akaike Information Criterion for Model Selection

Understanding the Akaike Information Criterion

The Akaike Information Criterion (AIC) is a fundamental tool in statistical model selection, developed by Hirotugu Akaike in 1973. It provides an estimate of the relative information lost when a given model is used to represent the process that generated the data. By balancing model fit with complexity, AIC helps analysts avoid two common pitfalls: overfitting (models too complex for the available data) and underfitting (models too simple to capture the underlying pattern).

The core formula for AIC is:

AIC = 2k – 2ln(L)

where k is the number of estimated parameters in the model, and L is the maximum value of the likelihood function for the model given the data. The term 2k penalizes model complexity, while –2ln(L) measures how well the model fits the data. AIC does not exist in isolation; it is a relative measure used to compare candidate models on the same dataset. The model with the lowest AIC value is considered the best among the candidates, striking a practical balance between goodness-of-fit and parsimony.

AIC is grounded in information theory. It approximates the Kullback–Leibler divergence between the true data-generating process and the model. Because the true process is unknown, AIC provides a way to rank models based on expected predictive accuracy. This makes AIC particularly useful in fields such as ecology, econometrics, machine learning, and epidemiology, where multiple competing theories need to be tested against observed data.

Why AIC Matters for Model Selection

Modern statistical analysis often involves comparing several plausible models. Without a rigorous criterion, analysts might rely on raw likelihood or R-squared values, which tend to favor overly complex models. AIC introduces a penalty for each additional parameter, discouraging unnecessary complexity. This penalty is derived from a theoretical foundation: as the number of parameters increases, the variance of the model estimates grows, which can degrade out-of-sample predictions.

The key insight is that every additional parameter uses up some of the data's information. AIC quantifies the trade-off: a model that adds a parameter must improve the likelihood enough to offset the penalty of 2. For example, if adding a variable increases the log-likelihood by less than 1, the AIC will actually increase, indicating the extra complexity is not worthwhile. This property makes AIC a practical tool for variable selection, polynomial order determination, and model averaging.

AIC is especially valuable in exploratory research where no single model is universally accepted. By comparing AIC values across a set of candidate models, researchers can identify which models are most consistent with the data while avoiding the temptation to chase better in-sample fit endlessly. It also facilitates model averaging, where predictions from multiple models are weighted by their AIC-based probabilities, leading to more robust estimates and narrower confidence intervals.

Step-by-Step Guide to Using AIC

Step 1: Define a Set of Candidate Models

Start by formulating the research question and identifying plausible models. These models should be based on theory, prior research, or exploratory data analysis. All models must be fit to the same dataset. For linear regression, candidates might include different combinations of predictors, interaction terms, or polynomial transformations. For time series, models could vary by number of lags or structural components.

Step 2: Fit Each Model and Obtain the Likelihood

For each candidate model, estimate the parameters and compute the maximum log-likelihood value. Most statistical software (R, Python, SAS, SPSS) can output log-likelihood directly. Ensure that the likelihood function is comparable across models—this usually requires that the models are fit to exactly the same response variable and sample size. Models with different transformations (e.g., logged vs. unlogged outcome) are not directly comparable using AIC.

Step 3: Compute the AIC Value

Apply the formula AIC = 2k – 2ln(L) using the number of parameters k (including the intercept, variance term, and any estimated coefficients) and the log-likelihood. Many software packages compute AIC automatically. Be consistent: if one model estimates a variance parameter and another does not, both must account for it in k.

Step 4: Compare AIC Values Across Models

Sort models by AIC from lowest to highest. The model with minimum AIC is the best among the candidates. However, avoid discarding models too quickly. Compute ΔAIC for each model: ΔAIC = AIC model – AIC min. Models with ΔAIC less than 2 have substantial support; those with ΔAIC between 4 and 7 have considerably less support; and models with ΔAIC greater than 10 are essentially unsupported relative to the best model (Burnham & Anderson 2004).

Step 5: Consider Model Simplicity and Practical Context

If two models have nearly identical AIC values (ΔAIC < 1), the simpler model is often preferred, especially if the more complex model introduces variables that are difficult to interpret or measure. AIC is a guide, not a rule. Domain knowledge, data quality, and the intended use of the model should always inform the final choice.

Step 6: (Optional) Perform Model Averaging

Instead of selecting a single model, you can compute Akaike weights (w_i) for each model: w_i = exp(-0.5ΔAIC_i) / Σ exp(-0.5ΔAIC_j). These weights sum to 1 and represent the probability that model i is the best for the data (given the candidate set). Weighted predictions and parameter estimates reduce the risk of relying on a single potentially overfit model.

Interpreting AIC Values in Practice

AIC values are always positive and can be large, but only their relative differences matter. The absolute magnitude of AIC is meaningless without a comparative baseline. When reporting AIC results, include the ΔAIC values and Akaike weights. The following thresholds are widely used in the literature:

ΔAIC ≤ 2: The model has substantial support and should be considered equally plausible as the best model.
4 ≤ ΔAIC ≤ 7: The model has considerably less support. It may still be worth considering in a model averaging context.
ΔAIC > 10: The model has essentially no support and can be confidently excluded from further consideration.

These thresholds are rules of thumb. In practice, the size of the candidate set, sample size, and the specific scientific context can affect interpretation. For example, in large datasets, even small AIC differences may indicate meaningful differences in predictive ability. Conversely, with small sample sizes, the corrected AICc (discussed below) should be used.

When to Use the Corrected AIC (AICc)

The standard AIC formula works well when the sample size n is large relative to the number of parameters k. A common rule of thumb is that n/k should be at least 40. When the sample size is small, AIC tends to underpenalize complexity and may select models with too many parameters. To address this, Sugiura (1978) derived a corrected version, AICc:

AICc = AIC + (2k(k+1)) / (n – k – 1)

AICc converges to AIC as n grows, so it is safe to use AICc in all cases, especially when n/k < 40. It is recommended as the default choice for model selection in most applied statistics settings. Many software packages offer AICc as an option or allow manual calculation.

For small samples, the additional penalty can change model rankings substantially. For instance, with n = 20 and k = 5, the correction term adds about 3.4 to AIC, which can be the difference between selecting a simple model and an overparametrized one. Always report whether you used AIC or AICc, and if AICc, the sample size and number of parameters used.

AIC vs Other Model Selection Criteria

AIC is one of several information criteria. Understanding its relatives helps contextualize its strengths and weaknesses.

Bayesian Information Criterion (BIC)

BIC is another widely used criterion with formula BIC = ln(n)k – 2ln(L). The penalty for each parameter is ln(n) instead of 2. For typical sample sizes, ln(n) > 2, so BIC penalizes complexity more heavily than AIC. BIC is derived from a Bayesian perspective and favors simpler models, especially in large datasets. Which criterion is better depends on the goal: AIC aims to minimize expected prediction error (Kullback–Leibler distance), while BIC aims to identify the true model among a set of candidates—assuming such a model exists. In practice, many researchers report both.

Cross-Validation

k-fold cross-validation (CV) directly estimates out-of-sample prediction error by partitioning data. AIC is asymptotically equivalent to leave-one-out cross-validation for linear models. For nonlinear models or when n is very small, CV may be more reliable. However, CV is computationally more intensive and can be unstable. AIC provides a convenient closed-form approximation that works well in many parametric settings.

Adjusted R-squared

Adjusted R-squared penalizes R² by the number of predictors: R²_adj = 1 – (1–R²)(n–1)/(n–k–1). It is easy to compute but only applies to linear regression models and does not have a strong theoretical basis for general model comparison. AIC is more general, applicable to any model where a likelihood can be defined.

Common Pitfalls and Limitations

Despite its popularity, AIC has several limitations that analysts must consider.

Assumes all models are nested or comparable? Not necessarily. AIC can compare non-nested models as long as they are fit to the same dataset and use the same response variable. However, models with different underlying distribution families (e.g., Gaussian vs. Poisson) should not be compared unless the likelihoods are properly normalized.
Risk of selecting overly complex models with small samples. This is exactly the reason AICc was developed. Always prefer AICc when sample size is small relative to parameter count.
Does not test model assumptions. AIC does not check whether a model is a reasonable fit to the data—it only ranks candidates. Residual diagnostics, outlier detection, and goodness-of-fit tests should always accompany AIC analysis.
Sample size dependency. The absolute AIC value and even the relative rankings can change if the dataset changes. Use AIC only on the same sample across all models.
May favor models with high variance. In the presence of multicollinearity or extreme outliers, AIC can still select a model that fits the training data well but fails to generalize. Robust modeling and prior variable screening are essential.
Not suitable for comparing models with different error distributions. For example, comparing a linear regression (normal errors) with a logistic regression (binomial errors) is invalid because the likelihood functions scale differently.

Another subtle limitation: AIC assumes that the candidate set includes at least one good model. If all models are poor representations of reality, AIC will still select the best among them, but that model may be inadequate. Model selection should always be combined with external validation, cross-validation, and substantive knowledge.

Practical Example: Model Selection with AIC

Suppose a researcher studies how tree growth (Y, cm/year) relates to soil nitrogen (N, mg/kg), soil phosphorus (P, mg/kg), and annual rainfall (R, mm). Thirty plots are measured. Four candidate linear models are considered:

Model 1: Y = β₀ + β₁N + ε
Model 2: Y = β₀ + β₁N + β₂P + ε
Model 3: Y = β₀ + β₁N + β₂R + ε
Model 4: Y = β₀ + β₁N + β₂P + β₃R + ε

Each model is fitted using ordinary least squares. The log-likelihood values (ln L) are obtained: Model 1: -105.3, Model 2: -101.0, Model 3: -104.1, Model 4: -99.8. Number of parameters k (including intercept and error variance) for each: Model 1: 3, Model 2: 4, Model 3: 4, Model 4: 5.

Compute AIC:

Model 1: AIC = 2*3 – 2*(-105.3) = 6 + 210.6 = 216.6
Model 2: AIC = 2*4 – 2*(-101.0) = 8 + 202.0 = 210.0
Model 3: AIC = 2*4 – 2*(-104.1) = 8 + 208.2 = 216.2
Model 4: AIC = 2*5 – 2*(-99.8) = 10 + 199.6 = 209.6

Model 4 has the lowest AIC (209.6), followed closely by Model 2 (210.0). ΔAIC for Model 2 = 0.4, which is less than 2, indicating that both models have strong support. Model 3 and Model 1 have ΔAIC of 6.6 and 7.0, respectively, indicating much weaker support. The researcher might choose Model 2 because it is simpler (one less parameter) while almost as good as Model 4. Alternatively, they could perform model averaging: weights w₂ = exp(-0.5*0.4)/(exp(-0.5*0.4)+exp(0)+exp(-0.5*6.6)+exp(-0.5*7.0)) ≈ 0.55, w₄ ≈ 0.45. Predictions would combine both models proportionally.

Software Implementation of AIC

Most statistical packages compute AIC automatically. Here are common examples:

R: Use the lm() function for linear models; call AIC(model) or use extractAIC(). For generalized linear models, glm() and AIC() work similarly. The MuMIn package offers model averaging and AICc calculations.
Python (statsmodels): After fitting a model (e.g., OLS), use the aic attribute. For AICc, you can manually compute it or use the aic_c property available in some models.
SAS: The MODEL statement in PROC REG or PROC GLM produces AIC. In PROC MIXED, AIC is part of fit statistics output.
SPSS: In regression dialogs, select "Information criteria" under "Statistics" to display AIC and BIC.
Excel: Not natively supported, but you can compute AIC manually using the log-likelihood obtained from solver or add-ins.

For mixed models and time series, ensure you use the correct degrees of freedom. Penalties for random effects can be debated; consult expert guidelines or use conditional AIC (cAIC) for mixed models.