The Kolmogorov-Smirnov (K‑S) test stands as one of the most widely used non‑parametric statistical tests for comparing two probability distributions. Unlike parametric tests such as the two‑sample t‑test, the K‑S test makes no assumption about the underlying distributional form, making it an indispensable tool when analyzing real‑world data that may not follow a normal or known distribution. This article provides a comprehensive, authoritative guide to using the K‑S test effectively – covering the theory, step‑by‑step implementation, practical applications, software usage, and important limitations.

What Is the Kolmogorov-Smirnov Test?

The K‑S test assesses whether two independent samples come from the same continuous distribution by comparing their empirical distribution functions (EDFs). The null hypothesis states that both samples are drawn from the same underlying distribution, while the alternative hypothesis states that they come from different distributions. The test statistic D measures the greatest vertical distance between the two EDFs. A small D value suggests the distributions are similar; a large D value indicates they differ significantly.

Two primary variants exist:

  • Two‑sample K‑S test – compares two empirical distributions (focus of this article).
  • One‑sample K‑S test – compares an empirical distribution against a fully specified theoretical distribution (e.g., normal, exponential).

The two‑sample version is often preferred for exploratory data analysis, quality control, model validation, and A/B testing when the underlying distributional assumptions of parametric tests are violated.

Assumptions of the K‑S Test

To obtain valid results, the following conditions should be met:

  • Continuous data – the test is designed for continuous distributions. Ties in the data (identical values) reduce its accuracy; if ties are present, the test becomes conservative, and an exact p‑value calculation is unavailable.
  • Independent samples – observations within and between samples must be independent. Paired or dependent data require a different approach.
  • Measured on at least an ordinal scale – the test uses the order of observations, so the data must be comparable in rank.
  • Sample size – the test works for any sample size, but very small samples (e.g., n < 5) have limited power. Large samples can detect even trivial differences, so consider effect size (the D statistic itself) alongside p‑values.

How the Two‑Sample K‑S Test Works

Let the two samples be X (size m) and Y (size n). Their empirical distribution functions are:

Fm(x) = (number of X values ≤ x) / m

Gn(x) = (number of Y values ≤ x) / n

The K‑S statistic D is defined as the supremum (maximum) of the absolute difference between these two functions over all x:

D = maxi |Fm(xi) – Gn(xi)|

In practice, D is computed at each unique value that appears in the combined data set. The p‑value is then derived from the sampling distribution of D under the null hypothesis. For small samples, exact p‑values are available (e.g., via the scipy.stats.ks_2samp function). For large samples, an asymptotic approximation is used.

The test can be performed as two‑sided (distributions differ in any way) or one‑sided (one distribution is stochastically greater than the other). Default is typically two‑sided.

Step‑by‑Step Procedure for the Two‑Sample K‑S Test

  1. State the hypotheses. H₀: The two samples come from the same continuous distribution. Hₐ: The two samples come from different distributions (two‑sided).
  2. Choose a significance level α (commonly 0.05).
  3. Combine and sort the data from both samples.
  4. Calculate the EDFs for each sample at every unique value.
  5. Compute D as the maximum absolute vertical distance between the two EDFs.
  6. Obtain the p‑value using statistical software or exact critical tables. Reject H₀ if p ≤ α.
  7. Interpret. If the test is significant, conclude that the two distributions are unlikely to be identical. If not significant, you cannot rule out that they came from the same distribution (but note: failure to reject does not prove equivalence).

Interpreting the Results

The D statistic itself measures the maximum deviation between the two cumulative distributions. For example, D = 0.30 means that at some point, the cumulative proportion differs by 30 percentage points between samples. This can be a useful effect size measure.

When reporting results, include both D and the p‑value. For a comprehensive analysis, consider plotting the two EDFs (or using a Q‑Q plot) to visually inspect where the differences occur. The K‑S test is sensitive to any kind of difference – location, scale, shape, or tail behavior – but it is most powerful at detecting shifts in the center of the distribution.

Be cautious with very large samples. With thousands of observations, the test can detect minuscule differences that are statistically significant but practically irrelevant. In such cases, focus on the magnitude of D.

Comparison with Other Tests

  • Two‑sample t‑test – assumes normality and equal variance; only tests difference in means. The K‑S test is distribution‑free and detects any distributional difference.
  • Mann‑Whitney U test – also non‑parametric, but tests whether one group tends to have larger values (stochastic ordering). The K‑S test is more sensitive to differences in shape and spread.
  • Anderson‑Darling test – often more powerful than K‑S, especially for detecting differences in the tails. It gives more weight to the tails of the distribution.
  • Chi‑squared test for independence – works on binned (categorical) data; K‑S is superior for continuous data.

Choose the K‑S test when you need a general, distribution‑free comparison of two continuous distributions and you are not focused exclusively on location or tail differences. For better tail sensitivity, consider the Anderson‑Darling test.

Practical Applications Across Domains

  • Data science & machine learning – comparing the distribution of predicted probabilities from a model to the actual binary outcomes; validating that training and test sets come from the same distribution.
  • A/B testing – checking that the control and treatment groups have similar distributions before the intervention, or comparing the distribution of a key metric (e.g., session duration) between the two groups.
  • Quality control – assessing if a batch of manufactured items has the same distribution of a critical dimension as a reference batch.
  • Clinical trials – evaluating whether the distribution of a biomarker is the same in two patient populations, especially when the data are not normally distributed.
  • Finance & econometrics – comparing the distribution of daily returns across two different periods or assets to detect changes in volatility or risk.
  • Environmental science – testing whether pollutant concentration distributions differ between two monitoring sites or seasons.

Using Statistical Software

The K‑S test is implemented in nearly all major statistical platforms. Here are concise examples for Python and R.

Python (with SciPy)

import numpy as np
from scipy.stats import ks_2samp

sample1 = np.random.normal(0, 1, 1000)
sample2 = np.random.normal(0.2, 1, 1000)

statistic, p_value = ks_2samp(sample1, sample2)
print(f"K-S statistic: {statistic:.4f}, p-value: {p_value:.4f}")

The ks_2samp function returns the two‑sided D statistic and p‑value. Additional parameters allow one‑sided tests and exact p‑value calculation for small samples. See the official SciPy documentation for full details.

R (with base stats)

sample1 <- rnorm(1000, mean = 0, sd = 1)
sample2 <- rnorm(1000, mean = 0.2, sd = 1)

result <- ks.test(sample1, sample2)
print(result)

The ks.test() function also supports one‑sample and two‑sample tests, and can compute exact p‑values for samples up to a certain size. For more control, refer to the R documentation for ks.test.

MATLAB

MATLAB provides the kstest2 function for two‑sample K‑S tests:

rng(0);
sample1 = randn(1000,1);
sample2 = randn(1000,1) + 0.2;
[h, p, ksstat] = kstest2(sample1, sample2);
fprintf('K-S stat: %.4f, p-value: %.4f\n', ksstat, p);

Limitations and Important Considerations

  • Sensitivity to ties – as the K‑S test assumes continuous data, ties lower its accuracy. The test becomes conservative (p‑value too large). Consider adding tiny random jitter if ties are sparse, or use an alternative test (e.g., Anderson‑Darling) that handles ties better.
  • Discrete data – the test is not recommended unless the discrete distribution has many levels; for ordinal or discrete data, consider the chi‑squared test or the discrete K‑S test variant.
  • Power – the K‑S test has relatively low power near the middle of the distribution compared to tests like the Cramér‑von Mises test. For detecting specific differences (e.g., variance changes), a more targeted test may be preferable.
  • Multiple comparisons – if you run several K‑S tests on the same data set, adjust the significance level using Bonferroni or FDR correction to avoid inflated Type I error.
  • One‑sample normality test – when using the one‑sample K‑S test to check for normality, you must specify the mean and variance of the theoretical normal distribution. If you estimate these from the data, the test is no longer valid (use Lilliefors test instead).

Advanced Variations

Beyond the basic two‑sided test, several extensions exist:

  • One‑sided K‑S test – tests whether one cumulative distribution function is always below the other (i.e., one sample is stochastically larger). Use the alternative parameter in software (e.g., 'less' or 'greater').
  • K‑S test for composite hypotheses – when parameters must be estimated, the standard asymptotic distribution no longer holds. Use a bootstrap or a specialized test (e.g., Lilliefors for normality, or rely on the two‑sample version which does not require parameter estimation).
  • Bootstrapped K‑S test – resample from the combined data to compute a more robust p‑value, particularly when sample sizes are moderate and handling ties.
  • Weighted K‑S statistic – modifications like the Anderson‑Darling statistic give more weight to the tails, increasing power for certain alternatives.

Example: Analysing Real‑World Sensor Data

Suppose a manufacturing plant measures the diameter of ball bearings produced by two different machines. The engineer suspects the machines may produce bearings with slightly different size distributions. After collecting 200 bearings from each machine, the K‑S test yields D = 0.145 with p = 0.009. At α = 0.05, the engineer rejects the null hypothesis. The EDF plot shows the deviation occurs at the lower end of the size range, suggesting Machine A produces slightly smaller bearings. The engineer can then investigate the cause.

Conclusion

The Kolmogorov-Smirnov test is a versatile, non‑parametric tool for comparing two distributions without imposing restrictive assumptions. By understanding its mechanics, assumptions, and limitations, you can apply it confidently in fields ranging from data science to clinical research. Always pair the test statistic with visual inspections of the empirical distributions, and be mindful of sample size effects and data characteristics such as ties. When used correctly, the K‑S test provides a rigorous foundation for distributional comparisons and actionable insights.

For further reading, the NIST Engineering Statistics Handbook offers a thorough explanation of K‑S tests and other goodness‑of‑fit procedures. For a deeper dive into test power and alternatives, consult advanced statistical textbooks or online resources specifically on distribution comparison methods.