Chapter 1: Why Small-Sample Research Matters

Small samples appear routinely across many substantive disciplines: in clinical piloting, ethnographic research, specialised occupational settings, and programme evaluations. In each of these contexts, the productive response to limited n is careful reasoning about power, uncertainty, and method choice rather than apology or retreat. It explains why large-sample approximations can mislead when data are modest, shows how sample size changes what can realistically be detected, and introduces the families of methods that remain useful when information is scarce. The broader aim is to set up the rest of the book: choosing analyses that fit the data you actually have, rather than the data you wish you had.

Learning Objectives

By the end of this chapter, you will be able to explain why small samples are common in applied research, identify where large-sample approximations become unreliable, construct and interpret power curves for different effect sizes, and select methods that fit the data and question at hand rather than the sample size you might have preferred.

Why Small Samples Are Often Unavoidable

Many textbooks assume that researchers can collect hundreds or thousands of observations. In practice, however, numerous research contexts yield small samples. Clinical studies of rare diseases, evaluations of pilot programmes, classroom-based educational interventions, community-based participatory research, and studies in Small Island Developing States (SIDS) often involve fewer than 100 participants. Resource constraints, logistical barriers, and ethical considerations (such as minimising burden on vulnerable populations) make small samples the norm rather than the exception.

For this book, a “small sample” usually means a dataset where routine large-sample approximations cannot be taken for granted: often n ≤ 50 per group for two-group comparisons, n ≤ 100 total for simple regression, or fewer than about 20 outcome events for binary or count models. These are working boundaries rather than universal cut-offs. A sample of 80 can still be small for a multivariable logistic model, while a sample of 24 paired observations may be informative for a tightly controlled within-person comparison.

In health sciences, rare-disease trials, feasibility studies, and single-site hospital evaluations may each involve only a few dozen participants. In education, a classroom-based intervention may be tested in one class of 15 to 25 students, while in business and the social sciences the accessible population may be small from the outset, as in niche-market A/B tests, studies of remote communities, or work with specialised occupational groups. Across these settings, the statistical problem runs deeper than a shortfall in recruitment: the population available for study may itself be limited.

Despite their ubiquity, small samples are often treated as deficient or temporary. In practice, this apologetic framing appears when authors describe modest n as a weakness by default, or when reviewers demand larger samples without asking whether a larger pool actually exists, whether recruitment would be ethical, or whether the research question is already well matched to a small but carefully analysed dataset. That response is methodologically unwarranted because sample size is not a free-floating quality marker: its adequacy depends on the population, the effect size, the design, and the inferential goal. In some settings, focused questions about a single outcome or a few key comparisons can often be addressed with modest samples, and even small studies can be informative when effects are large and variability is low. This mindset overlooks the fact that many important questions can only be addressed with small datasets. Rather than apologising, researchers should select methods that are appropriate for the sample size at hand.

When Large-Sample Approximations Fail

Classical parametric tests (t-tests, ANOVA, standard logistic regression) rely on asymptotic theory. They assume that sampling distributions approximate normality as sample size increases. With small samples, these approximations can be inaccurate. P-values may be misleading, confidence intervals may have poor coverage, and maximum likelihood estimates may be unstable or even undefined (for example, in logistic regression with separation).

Small samples also amplify the impact of outliers and violations of distributional assumptions. A single extreme value can dominate a mean or distort a regression slope. Skewed or heavy-tailed distributions, which cause few problems in large samples, become serious concerns when n is small. When n is limited, researchers should also select outcome measures carefully, because continuous or ordinal outcomes usually preserve more information per observation than coarse binary outcomes.

The sample mean remains meaningful, but the reference distribution used by a familiar test may no longer match the actual sampling behaviour well enough. The simulation below shows the point using a one-sample t-test at a nominal two-sided \(\alpha = 0.05\) when observations come from a strongly right-skewed distribution with true mean zero. The test is still centred on the correct null value, but the Type I error rate is higher than the nominal 5% level at small n.

Simulated Type I error for a one-sample t-test under strong right skew.
Sample size Nominal alpha Simulated Type I error
5 0.05 0.120
10 0.05 0.103
20 0.05 0.079
30 0.05 0.075

Interpretation: This simulation is deliberately simple, but it gives a concrete reason for caution. At n = 5 or 10, a nominal 5% t-test rejects too often under this skewed null distribution. The error rate moves closer to the target as n increases, but the improvement is gradual rather than automatic.

Visualising Power Trade-offs

Even modest reductions in sample size can have a dramatic impact on statistical power. Figure 1.1 uses base R’s power.t.test() function to illustrate how power declines as per-group sample size falls from 60 to 10, shown separately for small, medium, and large effects (Cohen’s d = 0.3, 0.5, and 0.8).

To calculate one point on the figure, you supply the per-group sample size n, the expected mean difference delta, the within-group standard deviation sd, the significance level sig.level, and the test design through type = "two.sample". For example, power.t.test(n = 35, delta = 0.5, sd = 1, sig.level = 0.05, type = "two.sample", alternative = "two.sided")$power returns a value a little above 0.50, meaning that a two-group study with 35 participants per group has only a little better than a fifty-fifty chance of detecting a true effect of d = 0.5 at the 5% level. Reading Figure 1.1 in this way helps translate abstract power calculations into concrete design choices.

NoteInterpretation

Figure 1.1 shows that with medium effects (about d = 0.5), power does not rise above 50% until per-group sample size reaches roughly the mid-30s, and it still remains below the conventional 80% threshold even at 60 participants per group. Detecting smaller effects (about d = 0.3) would require far more observations than are typically feasible in small-sample settings. This visual reinforces the need to report minimum detectable effects and to focus on estimation rather than binary significance testing when n is limited.

Appropriate Methods for Small Samples

When standard asymptotic approximations cannot be trusted, three broad classes of method remain available. Exact tests (such as Fisher’s exact test, exact binomial tests, and exact Poisson tests) compute p-values directly from the combinatorial distribution of the data and are especially useful for small discrete datasets. Resampling methods (bootstrap and permutation tests) use the observed data to approximate the sampling distribution, often yielding more accurate inferences than large-sample formulas when an exact calculation is unavailable or when the statistic of interest is more complex than a standard test handles well.

Nonparametric rank-based tests (Mann–Whitney U, Wilcoxon signed-rank, Kruskal–Wallis) make fewer distributional assumptions and are less sensitive to outliers, making them natural choices for ordinal or skewed outcomes. Penalised regression (Firth logistic regression, ridge, LASSO) can stabilise coefficient estimates when events are sparse (that is, when outcome events are rare relative to the number of predictors, as in binary logistic regression with few observed cases of the outcome). Bayesian methods incorporate prior information and quantify uncertainty through posterior distributions, which remain well-defined even when data are limited. In practice, these families complement rather than replace one another: exact tests are often most natural for small discrete tables, resampling methods are flexible for estimation and custom statistics, and penalised or Bayesian models become practical when the research question requires regression rather than a single comparison.

Example: Comparing Two Small Groups

Suppose we wish to compare customer satisfaction scores (on a 1–10 scale) between two service branches, each with only 12 observations. The scores are ordinal and may not be normally distributed.

The Mann–Whitney U test compares the distributions of the two groups without assuming normality. The p-value is the probability of observing a rank difference at least as large as the one obtained, assuming the two groups share the same underlying distribution. Because the test is based on ranks, it is robust to skewness and outliers.

NoteInterpretation

In this example, Branch A scores higher than Branch B, with a Hodges–Lehmann shift estimate of 2 points and a large standardised rank-based effect size (Rosenthal’s r = 0.67). The small p-value (p = 0.001) suggests that the observed rank difference would be unlikely if the two branches had the same underlying distribution. With only 12 observations per branch, the example shows how a rank-based method can still produce a clear, interpretable result when the outcome is ordinal and the effect is sizeable.

That gap between what standard rules of thumb demand and what is actually achievable in constrained settings is precisely the problem this book addresses.

Key Takeaways

Small samples are a common feature of many substantive fields rather than a deficiency to be apologised for, and large-sample approximations can fail when n is modest, leading to inaccurate p-values and confidence intervals. Exact tests, resampling methods, and rank-based procedures offer valid alternatives that do not require large samples to behave well. In practice, the choice of method should match the research question, outcome type, and available sample size. Small studies are strongest when the analysis is calibrated to the information actually observed and when uncertainty is reported transparently rather than hidden behind a binary significant/non-significant decision. That principle will guide the chapters that follow.

Self-Assessment Quiz

Test your understanding of the key concepts from Chapter 1.

Question 1. A study with n = 12 per group has 25% power to detect d = 0.5. What does this mean?

Explanation.

Statistical power is the probability of correctly rejecting a false null hypothesis when a specific effect size exists. With 25% power, there is a 75% chance of a Type II error (failing to detect a real effect of d = 0.5). This concept is directly illustrated in the power curve figure in this chapter, which shows how power declines as sample size decreases.

Question 2. Why might large-sample approximations fail with n = 15?

Explanation.

The Central Limit Theorem requires sufficient sample size for sampling distributions to approximate normality. With n = 15, especially if data are skewed or have outliers, parametric test assumptions may be violated. This is why the chapter emphasizes that “large-sample approximations can fail when n is small, leading to inaccurate p-values and confidence intervals.”

Question 3. Which research question is MOST appropriate for n = 20?

Explanation.

Focused comparisons are feasible with small samples, whereas broad multivariate prediction questions are not. Complex questions like A, C, and D require many more observations to estimate parameters stably and avoid overfitting.

Question 4. A pilot study with n = 8 finds a mean difference of 5 points (95% CI: [-2, 12], p = 0.14). The correct interpretation is:

Explanation.

The wide confidence interval reflects substantial uncertainty. The study cannot rule out small negative effects (-2) or large positive effects (12). Non-significance with small n indicates insufficient evidence, not absence of effect. This aligns with the chapter’s emphasis on focusing “on estimation rather than binary significance testing when n is limited.”

Question 5. Which outcome measure provides the MOST statistical information per observation?

Explanation.

Continuous measures preserve all variation in the data. Dichotomising or coarsening into categories discards information, reduces statistical power, and limits the ability to detect effects. This is why the chapter notes that, when n is limited, researchers should select outcome measures carefully.

Question 6. With n = 10 per group, which statement about power is TRUE?

Explanation.

Statistical power increases with larger effect sizes, larger sample sizes, and lower variance. Even with n = 10, a very large effect (d = 1.5) might have adequate power, while a small effect (d = 0.2) would not. This is demonstrated in the power curve plot showing different effect sizes (d = 0.3, 0.5, 0.8).

Question 7. A study finds p = 0.048 with n = 8 per group. Which concern is MOST valid?

Explanation.

P-values near cutoffs (0.05) are highly variable with small samples. A slight change in data or analysis could flip the result. Emphasis should be on effect size magnitude and confidence intervals, not borderline p-values. The chapter warns that “small samples amplify the impact of outliers and violations of distributional assumptions.”

Question 8. When is a small sample (n < 30) potentially sufficient?

Explanation.

Small samples can still be informative when effects are large and variability is low. The chapter argues that some important questions can only be addressed with small datasets, provided the method and interpretation are matched to the limited information available.

Question 9. Which is a legitimate reason for small sample size?

Explanation.

Rare populations, pilot studies, ethical constraints (minimising burden on vulnerable groups), and resource limitations in SIDS contexts are all legitimate reasons for small samples. The chapter explicitly mentions “clinical studies of rare diseases” as one context where “small samples are the norm rather than the exception.”

Question 10. A researcher states: “My study has n = 15, so I’ll just use nonparametric tests.” What is the problem with this reasoning?

Explanation.

Test selection should consider outcome type, distributional properties, and the research question. Small n is one consideration, but not the sole criterion. As the chapter concludes, “The choice of method should match the research question, outcome type, and available sample size.”