Why Effect Size Matters More Than You Think

For decades, researchers have placed heavy emphasis on p-values to decide whether a result is “significant.” But a statistically significant p-value only tells you that an effect is unlikely to have occurred by chance—it says nothing about the size or practical importance of that effect. This is where effect size enters the picture. Effect size quantifies the magnitude of a difference or relationship, giving researchers and practitioners a clear sense of how meaningful a finding truly is. Understanding effect size is not just a statistical nicety; it is essential for interpreting results in real-world contexts, conducting meta-analyses, and building a cumulative science.

Many fields now recognize that relying solely on null hypothesis significance testing (NHST) can lead to misguided conclusions. A large sample can make even a trivially small effect statistically significant, while a small sample might hide a large and important effect behind a non-significant p-value. Reporting effect sizes moves the conversation from “is there an effect?” to “how big is the effect?”—a far more relevant question for practitioners, policy makers, and anyone who wants to use research findings to make decisions.

What Is Effect Size?

Effect size is a standardized, unit‑free measure that describes the strength of a phenomenon. It answers the question “how large is the effect?” rather than “does an effect exist?” For example, a study might find that a new teaching method leads to higher test scores with p < 0.05. But if the effect size is very small, the improvement may be negligible in practice. Conversely, an effect with p = 0.06 could be large and practically important, but dismissed because it fell short of the arbitrary alpha threshold. Effect size helps you see the forest beyond the single p-value tree.

Unlike p-values, which are heavily influenced by sample size, effect sizes are largely independent of sample size. A large study can produce a tiny p-value for a trivial effect, while a small study may miss a large effect. Reporting effect size alongside confidence intervals provides a far richer picture of what the data actually say. The effect size itself is not affected by sample size, though its precision (confidence interval) narrows as sample size increases.

To compute an effect size, you typically standardize the raw difference or association by a measure of variability. For example, Cohen’s d divides the mean difference by the pooled standard deviation. This standardization allows comparison across studies that use different scales or measurements, which is why effect sizes are the foundation of meta-analysis. They turn disparate findings into a common metric.

Common Types of Effect Size

Choosing the right effect size depends on your study design and research question. Below are the most widely used measures, with guidance on when to apply each.

Cohen’s d (Standardized Mean Difference)

Cohen’s d is one of the most frequently reported effect sizes. It expresses the difference between two group means in units of the pooled standard deviation. A d of 0.5 means the two means differ by half a standard deviation. This measure is the go‑to for independent-groups t‑tests and can also be adapted for paired designs (using the standard deviation of the difference scores, often called Cohen’s dz). You can compute Cohen’s d from summary statistics (means and standard deviations) or use statistical software. For a quick online tool, many researchers turn to Psychometrica’s effect size calculator.

Cohen’s d is also used in meta‑analysis of mean differences. However, when sample sizes are very unequal, a pooled standard deviation weighted by sample size (Hedges’ g) may be preferred to reduce bias. Many meta‑analysis software packages automatically compute Hedges’ g, which is very similar to Cohen’s d but applies a small correction factor.

Pearson’s r (Correlation Coefficient)

Pearson’s r measures the strength and direction of a linear relationship between two continuous variables. Its value ranges from –1 (perfect negative) to +1 (perfect positive), with 0 indicating no linear relationship. Because r is already standardized, it functions directly as an effect size. Squaring r gives r², the proportion of variance shared between the two variables—a useful metric for understanding practical importance. For example, an r of 0.3 corresponds to 9% shared variance, which is considered a medium effect in the behavioral sciences. However, r can be inflated by range restriction or outliers, so always visualize your data and report confidence intervals.

Pearson’s r is appropriate not only for simple correlations but also for regression models (as standardized coefficients). When reporting r, include a confidence interval to communicate precision—especially important because r itself is slightly biased upward in small samples.

Eta‑Squared (η²) and Partial Eta‑Squared

In analysis of variance (ANOVA), eta‑squared indicates the proportion of total variance in the dependent variable accounted for by a factor. It is analogous to r² in regression. For a one‑way ANOVA, η² = SSbetween / SStotal. For factorial designs, researchers typically report partial eta‑squared (η²p), which removes variance from other factors and interactions before computing the proportion. Partial eta‑squared is therefore comparable across studies with different designs, making it the standard in many fields. However, partial eta‑squared can overestimate the true population effect size, especially in small samples. A less biased alternative is partial omega‑squared (ω²p), which many methodologies now recommend. Still, because partial eta‑squared is so common, it is acceptable to report it while also noting the potential for bias.

Odds Ratio (OR)

The odds ratio is the effect size of choice for binary outcomes, such as disease vs. no disease, success vs. failure, or survival vs. death. It is used in logistic regression, case‑control studies, and 2×2 contingency tables. An OR of 1 indicates no association; OR > 1 indicates increased odds of the outcome in the exposed group; OR < 1 indicates decreased odds. Because the odds ratio scale is non‑symmetric (range 0 to ∞), analysts often log‑transform it for analysis (log odds). The magnitude of an OR can be hard to interpret intuitively—for rare outcomes, it approximates the relative risk, but for common outcomes it does not. Nevertheless, it remains a standard effect size for biomedical and epidemiological research.

Cramer’s V

For chi‑square tests of association in contingency tables larger than 2×2, Cramer’s V adjusts phi (φ) to range between 0 and 1, providing a correlation‑like effect size. A small Cramer’s V might be 0.1, medium 0.3, and large 0.5, though these benchmarks depend on the degrees of freedom. Cramer’s V is useful when you want to quantify the strength of association between two categorical variables, regardless of the number of categories.

Cohen’s f²

Cohen’s f² is used in the context of multiple regression and ANOVA to quantify the local effect size for a set of predictors. It is defined as f² = R² / (1 − R²) for the entire model, or f² = (R²full − R²reduced) / (1 − R²full) for a subset of predictors. Cohen’s f² is directly related to the F‑test and is the effect size required for power analysis using G*Power. Benchmarks: 0.02 (small), 0.15 (medium), 0.35 (large).

Why Effect Size Is Critical in Research

From Statistical to Practical Significance

A treatment that “works” statistically may produce only a tiny improvement that is not worth the cost or effort. For instance, a weight‑loss drug might show a statistically significant reduction of 0.5 kg over placebo. With a large sample, that could be p < 0.001, yet the effect is clinically trivial. Reporting Cohen’s d alongside the p‑value forces the researcher to confront the magnitude. In educational research, an intervention that raises test scores by an effect size of 0.1 might be statistically significant with thousands of students, but the practical gain might not justify the expense of implementing the program. Effect size helps stakeholders—teachers, doctors, policymakers—decide whether an observed effect is worth acting upon.

Meta‑Analyses and Reproducibility

Effect sizes are the currency of meta‑analysis. Combining results across studies requires a standardized metric. Without effect sizes, researchers cannot aggregate findings, evaluate consistency, or test for moderators. The Reproducibility Project in psychology found that many original studies had small to medium effect sizes that replication studies could not confirm, partly because original studies were underpowered to detect such effects. Larger meta‑analyses using effect sizes have since clarified which findings are robust. By converting all results to a common effect size (e.g., Cohen’s d or r), meta‑analysts can weight studies by precision and produce an overall estimate with a confidence interval. This process is far more informative than counting how many individual studies “won” with p < 0.05.

In medicine, meta‑analyses of randomized controlled trials rely on effect sizes like odds ratios or standardized mean differences to determine the overall benefit of a treatment. The Cochrane Collaboration explicitly requires effect sizes and confidence intervals in its reviews. Without such standardization, the cumulative evidence would be a jumble of incompatible statistics.

Understanding Real‑World Impact

In education, an effect size of 0.2 (small) on a national test might still be meaningful if it applies to millions of students. In clinical medicine, an odds ratio of 1.2 (small) for a serious side effect could be alarming. Context matters. Effect size provides the raw number; you then interpret it within the domain. The APA Task Force on Statistical Inference has long recommended reporting effect sizes and confidence intervals. Similarly, the American Statistical Association’s statement on p‑values explicitly encourages supplementing p‑values with effect sizes and intervals.

Interpreting Effect Sizes: Beyond the Rule of Thumb

Thanks to Cohen, many researchers use the benchmarks: small (d = 0.2), medium (d = 0.5), large (d = 0.8). But these are only starting points. Cohen himself warned that the values were “arbitrary” and should be interpreted relative to the field. In social psychology, a d of 0.5 might be considered medium; in physics, an effect half a standard deviation would be enormous. In educational interventions, effect sizes above 0.2 are often reported as meaningful. It is far better to compare your effect size to those found in prior studies or meta‑analyses in your specific area than to rely on generic thresholds.

Moreover, effect sizes should be reported with confidence intervals. A wide interval around a medium effect suggests that the true effect could be small or large, reducing confidence in the exact magnitude. For instance, a Cohen’s d of 0.4 with a 95% CI from 0.1 to 0.7 tells a much more nuanced story than just “medium effect.” The interval width also informs future research: if it includes zero, the result is not statistically significant; if it excludes zero but is wide, you know the effect likely exists but its size remains uncertain.

Another important point is that effect sizes from different study designs are not always directly comparable. For example, Cohen’s d from a between‑subjects design differs from Cohen’s dz from a within‑subjects design because the latter uses the standard deviation of differences rather than the pooled standard deviation. Similarly, odds ratios cannot be directly compared to risk ratios when the outcome is common. Always be aware of the computational details when interpreting effect sizes from other studies.

Effect Size in Meta‑Analysis

In meta‑analysis, effect sizes are combined, and heterogeneity is assessed. A common metric is the standard difference in means (Cohen’s d or Hedges’ g) for continuous outcomes, and the log odds ratio for binary outcomes. The choice of effect size can affect the conclusions if effect sizes are not properly standardized. Methods like random‑effects models respect that true effect sizes vary across studies. Forest plots visually display each study’s effect size and confidence interval, and the overall pooled estimate. The Cochrane Handbook provides extensive guidance on selecting and interpreting effect sizes in systematic reviews.

Using Effect Size Effectively in Your Own Work

Reporting Standards

When you write up results, include the effect size measure, its value, and a confidence interval. For a t‑test, report Cohen’s d with 95% CI. For an ANOVA, report partial eta‑squared or omega‑squared. For regression, report standardized coefficients (β) or Cohen’s f². Avoid cherry‑picking the effect size that makes your result look largest; choose the one that matches your design and research question. Many journals now require effect sizes as part of their reporting guidelines (e.g., APA Style 7th edition).

It is also good practice to interpret the effect size in context. For example: “The intervention led to a medium‑sized improvement in reading scores (Cohen’s d = 0.54, 95% CI [0.28, 0.80]), indicating that the average student in the intervention group scored roughly half a standard deviation higher than the average control student.” Such a statement is far more informative than simply reporting p < 0.01.

Computing Effect Sizes in Software

Many statistical packages now include effect size options. In SPSS, you can request Cohen’s d for t‑tests, and partial eta‑squared for ANOVA. In R, functions like cohensD in the lsr package, eta_squared in effectsize, and rquery.cormat for correlations simplify the process. For those using Microsoft Excel, there are free templates available from Psychometrica that compute Cohen’s d, r, and odds ratios. For power analysis, G*Power is a widely used free program that allows you to input effect size (d, f², etc.) along with alpha and desired power to get the required sample size.

Power Analysis and Sample Planning

Effect size is central to power analysis. To determine the sample size needed for a study, you must specify the expected effect size. If you overestimate the effect size (e.g., assume a large effect when the true effect is medium), your study will be underpowered. It is wise to base your effect size estimate on prior literature or a small pilot study. Many online calculators and programs like G*Power facilitate this. When planning a study, always perform a power analysis: identify the smallest effect size of practical interest, set your desired power (typically 0.80) and alpha (0.05), and compute the required N. This step helps ensure that your study can detect meaningful effects, and that you don’t waste resources on an underpowered sample.

A common mistake is to use the effect size observed in a previous study as the expected effect size for a power analysis. Because observed effect sizes are often inflated due to publication bias (the “winner’s curse”), you should instead use a meta‑analytic estimate or a “smallest effect size of interest” (SESOI). The SESOI can be determined by clinical significance, cost‑benefit analysis, or normative benchmarks in your field.

Common Pitfalls

  • Ignoring effect size: Relying solely on p‑values leads to overconfidence in trivial effects.
  • Misusing Cohen’s d for repeated measures: Paired designs require an adjustment (e.g., Cohen’s dz using the standard deviation of the difference scores). Reporting a between‑groups d for a within‑subjects design will inflate the estimate because it ignores the correlation between measurements.
  • Reporting effect sizes without confidence intervals: A point estimate is less informative without its precision, and the confidence interval allows the reader to assess the range of plausible effect sizes.
  • Overinterpreting small effect sizes: Even if statistically significant, a very small effect may have no practical use. Always interpret the magnitude in the context of the field and the costs/benefits of applying the finding.
  • Using raw effect sizes without considering scale: For example, an unstandardized mean difference of 5 points on an IQ test (SD=15) is a Cohen’s d of 0.33; but the same raw difference on a test with SD=30 would be d=0.17. Always standardize properly.

Conclusion

Effect size is not an optional supplement to traditional null‑hypothesis testing—it is the primary currency for communicating the magnitude of findings. By moving beyond “p < 0.05” and embracing Cohen’s d, Pearson’s r, eta‑squared, odds ratios, and other measures, researchers give their audience the information needed to assess practical importance, compare results across studies, and design well‑powered future experiments. Reporting effect sizes with confidence intervals should become standard practice in every quantitative discipline. The next time you run a statistical test, remember: statistical significance tells you if something happened; effect size tells you how much it matters. Both are necessary for a complete picture.

As the field moves toward open and reproducible science, effect size reporting is a simple yet powerful way to improve transparency. By adopting these practices, researchers can help ensure that their findings are not just statistically significant, but genuinely meaningful.