Chapter 9: Assessing Multiple Imputation Quality

Learning Objectives

By the end of this chapter, you will be able to explain why multiple-imputation diagnostics matter, inspect the main convergence and plausibility checks produced by mice, evaluate whether pooled estimates are stable across different values of m, and report imputation diagnostics in a way that makes downstream analyses defensible.

Why Imputation Diagnostics Matter

Multiple imputation (MI) requires deliberate specification at each stage. The quality of the imputed values depends on whether the imputation models are specified sensibly, whether the chained equations have converged, whether the imputed values remain plausible relative to the observed data, and whether the pooled estimates are stable across different choices of m. If those checks are ignored, the result can be biased parameter estimates, incorrect standard errors, implausible imputations, and misplaced confidence in apparently polished output.

Diagnostic 1: Convergence Checks

The mice algorithm uses iterative chained equations: it cycles through variables, updating imputations based on the current values of other variables. Convergence occurs when these iterations stabilise (no systematic trends).

Trace Plots

Trace plots show the mean and SD of imputed values across iterations for each variable. Good convergence looks like a fuzzy caterpillar: trace lines fluctuate randomly around a stable mean, show no systematic upward or downward trend across iterations, and chains from different imputations intermingle rather than remaining separated. Figure 9.1 shows the full set of trace plots for the three simulated variables.

Rendered Output

Figure 9.1: Trace plots for age, satisfaction, and income across imputations.

Cell Code

Interpretation: Figure 9.1 should resemble a fuzzy caterpillar rather than a drifting set of lines. If the traces still drift after roughly 20 iterations, increase maxit. If chains remain separated, inspect the imputation model specification rather than assuming convergence has occurred.

Checking Specific Variables

If you have many variables, focus on those with the most missingness. Figure 9.2 shows how that targeted check looks when attention is restricted to age.

Rendered Output

Figure 9.2: Trace plot for age only.

Cell Code

If you still see clear trends after the first 10 to 20 iterations, increase maxit to 50 or even 100. In many routine MCAR or MAR settings, maxit = 20–50 is usually enough, but the diagnostics should drive that decision rather than a hard default.

Diagnostic 2: Imputed vs. Observed Distributions

Imputed values should resemble the observed data distribution (but not be identical). Large discrepancies suggest model misspecification. With n < 30, imputed distributions may appear jagged or narrower because predictive mean matching has few donor values. Focus on whether imputed values fall within a plausible observed range rather than demanding smooth density curves. Figure 9.3 shows the density comparison for the simulated example.

Density Plots

Rendered Output

Figure 9.3: Observed and imputed density plots for the simulated example.

Cell Code

Interpretation: Figure 9.3 should show substantial overlap between observed and imputed distributions without making them identical. In very small examples the red curve can look more jagged or narrower because it is based on few imputed values. The red flags are collapse toward a single implausible value or a clear shift outside the observed range.

Strip Plots (Univariate)

Strip plots show individual imputed values (red) alongside observed values (blue). Figure 9.4 uses age as the example variable.

Rendered Output

Figure 9.4: Strip plot for age across imputations.

Cell Code

Interpretation: Figure 9.4 should show the imputed values filling gaps in the observed data without behaving like a separate distribution. A useful question here is whether any imputed points fall implausibly far outside the observed range, because that usually signals a poor imputation model rather than legitimate uncertainty.

Diagnostic 3: Sensitivity to m (Number of Imputations)

The number of imputations (m) affects the precision of pooled estimates. With more imputations, pooled estimates become more stable and standard errors more accurate. Table 9.1 compares the coefficient and standard-error estimates across three values of m.

Rule of Thumb for m

The number of imputations should increase as the fraction of missing information (FMI) increases. When FMI is below about 10%, m = 5 to 10 is often adequate. When FMI is around 10% to 30%, m = 20 to 50 is more defensible. When FMI exceeds 30%, values such as m = 50 to 100 may be needed. A useful heuristic from White, Royston, and Wood (2011) (White, Royston, and Wood 2011) is \(m \geq 100 \times \text{FMI}\), where FMI is averaged across the parameters that matter for the analysis. For example, average FMI = 0.15 suggests at least 15 imputations, and average FMI = 0.30 suggests at least 30. Round up to convenient reporting values such as m = 20 or m = 50.

Testing Sensitivity

Rendered Output

Table 9.1

Sensitivity of the age coefficient to the number of imputations

m	Age coefficient	Age SE
5	-0.022	0.013
20	-0.021	0.015
50	-0.019	0.014

Note. Only modest changes across m values are expected once Monte Carlo error is under control.

Cell Code

Interpretation: Table 9.1 should show broadly similar coefficient estimates across different values of m, with only modest differences due to Monte Carlo error, and the standard errors should stabilise as m increases. If the coefficients move substantially, for example by more than about 10%, that is a sign to increase m rather than treating the smaller-imputation result as settled.

When to Use Larger m

Larger values of m are especially sensible when missingness is high, when the sample itself is small and Monte Carlo error is therefore more noticeable, or when the analysis is sensitive enough that conservative inference matters. In practice, that means using at least m = 20 once missingness moves beyond about 20%, and considering m = 50 or more for particularly consequential analyses.

Diagnostic 4: Checking Imputation Model Assumptions

Inspect Imputation Methods

Table 9.2 records which imputation method is being used for each variable.

Rendered Output

Table 9.2

Imputation methods used for each variable

Variable	Method
age	pmm
satisfaction	pmm
income	pmm

Cell Code

Common methods: pmm uses predictive mean matching and is usually the safest default for continuous variables because it preserves plausible observed values. norm uses Bayesian linear regression and therefore leans harder on normality assumptions. logreg is intended for binary variables, while polyreg is used for unordered categorical variables. In most small-sample settings, pmm is the best starting point for continuous variables unless you have a strong reason to prefer a parametric normal model.

Check Predictor Matrix

Table 9.3 shows the predictor matrix that tells mice which variables are used to impute each target variable.

Rendered Output

Table 9.3

Predictor matrix for the illustrative imputation model

Imputed variable	age	satisfaction	income
age	0	1	1
satisfaction	1	0	1
income	1	1	0

Note. A value of 1 means the column variable is used to predict the row variable.

Cell Code

Interpretation: In the predictor matrix, rows identify the variables being imputed and columns identify the variables used to predict them. A 1 means the column variable is included as a predictor for the row variable, while a 0 means it is excluded.

Modify if needed:

# Example: Exclude a variable from predicting another
pred <- imp$predictorMatrix
pred["age", "satisfaction"] <- 0  # Don't use satisfaction to predict age

# Re-run imputation with modified predictor matrix
imp_modified <- mice(mi_data, m = 5, predictorMatrix = pred, print = FALSE)

Diagnostic 5: Fraction of Missing Information (FMI)

The FMI quantifies how much uncertainty is introduced by imputation. It is automatically reported by pool(), and Table 9.4 shows the relevant columns for the pooled regression model.

Rendered Output

Table 9.4

Pooled regression estimates with lambda and FMI

Term	Estimate	Std. error	Lambda	FMI
(Intercept)	0.101	0.643	0.296	0.342
age	-0.021	0.015	0.214	0.258
income	0.075	0.015	0.232	0.276

Cell Code

Columns to examine: The fmi column reports the fraction of missing information for each coefficient, while lambda shows the proportion of total variance attributable to missingness.

Interpretation: Values of fmi below about 0.10 indicate low missing-information burden and are often compatible with m = 5 to 10. Values around 0.10 to 0.30 suggest a moderate burden and support using m = 20 to 50. Once fmi exceeds about 0.30, missingness is making a large contribution to uncertainty. Consider larger m, add defensible auxiliary variables, or state that MI may not fully recover information in a very small sample.

Example: Full Diagnostic Workflow

The earlier sections show the individual diagnostics. This closing example condenses them into a reporting workflow so the end result is not a second copy of the same plots, but a compact summary of what should be written up after those checks have been completed.

Rendered Output

Table 9.5

Workflow summary for assessing multiple-imputation quality

Step	Action
Describe missingness	satisfaction 20.0 % missing; age 10.0 % missing; income 6.0 % missing
Choose imputation settings	Use predictive mean matching with m = 20, maxit = 30, and random seed = 2025.
Check convergence	Inspect trace plots for drift or separated chains.
Check plausibility	Compare observed and imputed distributions with density and strip plots.
Pool model estimates	Pool the regression of satisfaction on age and income.
Summarise FMI	FMI ranges from 0.208 to 0.302.
Write the report	Example write-up: We used predictive mean matching with m = 20 imputations, maxit = 30, and random seed = 2025. Diagnostic plots indicated adequate convergence and plausible imputations, and FMI values ranged from 0.208 to 0.302.

Cell Code

Red Flags and Troubleshooting

Problem	Symptom	Solution
Non-convergence	Trace plots show trends	Increase maxit (try 50–100)
Imputed values at one value	Density plot shows spike	Use method = "pmm" instead of norm
Imputed values out of range	Strip plot shows outliers	Check variable type (e.g., use logreg for binary)
Unstable estimates across m	Coefficients vary > 10%	Increase m (try 50–100)
High FMI (> 0.50)	Large uncertainty	Consider whether MI is appropriate; may need auxiliary variables or accept wider CIs
Separation warnings (logistic regression)	Model fails to converge	Use penalized imputation methods or increase sample size

Reporting MI Diagnostics

When reporting MI results, include:

1. Missingness pattern: “Three variables had missing data (age: 20%, income: 18%, satisfaction: 10%)”
2. Imputation model: “We used predictive mean matching with m = 20 imputations, maxit = 30, and random seed = 2025”
3. Convergence: “Trace plots showed convergence after 20 iterations (see Supplementary Figure S1)”
4. Plausibility: “Imputed values were visually consistent with observed distributions (density plots in Supplementary Figure S2)”
5. Sensitivity: “Results were stable across m = 5, 20, and 50 imputations (coefficient differences < 5%)”
6. FMI: “Fraction of missing information ranged from 0.12 to 0.25, indicating moderate impact of missingness”

Key Takeaways

Multiple-imputation results are only as defensible as the diagnostics behind them. Convergence checks, distributional comparisons, stability across different values of m, and inspection of FMI all help determine whether the imputation model is behaving plausibly rather than merely producing polished output. In small samples especially, report the seed, m, maxit, method, package versions, and remaining uncertainty clearly.

---

Self-Assessment Quiz

Question 1. What is the main purpose of a trace plot in multiple-imputation diagnostics?

To check whether the imputation chains have stabilised across iterationsTo count the number of missing values in each variableTo show the final pooled regression coefficientsTo choose between complete-case analysis and LOCF

Explanation.

Trace plots are a convergence diagnostic. They show whether the imputed values are fluctuating around a stable level or still drifting across iterations.

Question 2. What pattern in a trace plot would most strongly suggest that maxit should be increased?

A systematic upward or downward drift across iterationsRandom fluctuation around a stable meanSeveral chains overlapping each otherSlightly different starting values for each chain

Explanation.

If the trace continues to trend upward or downward, the chained-equations algorithm has not yet stabilised. That is the clearest signal to increase maxit.

Question 3. What is the main diagnostic question answered by a density plot of observed and imputed values?

Whether the imputed values are plausible relative to the observed distributionWhether the predictor matrix contains enough zerosWhether the sample size exceeds 100Whether Little’s MCAR test is significant

Explanation.

Density plots compare the observed and imputed distributions. The goal is to check plausibility, not to force the two distributions to be identical.

Question 4. In a strip plot, what would count as a red flag?

Imputed values falling well outside the observed rangeObserved and imputed points using different colorsThe x-axis being labelled by imputation numberHaving more than one imputed dataset

Explanation.

Strip plots should show imputed values filling plausible gaps in the observed data. Values far outside the observed range usually indicate model misspecification.

Question 5. Why does the chapter compare pooled results across m = 5, 20, and 50?

To check whether the estimates are stable as Monte Carlo error is reducedTo prove that larger m always changes the coefficient directionTo replace the need for FMITo test whether the outcome is normally distributed

Explanation.

Changing m is a sensitivity check for Monte Carlo error. If the pooled estimates move substantially, then the smaller value of m was not yet stable enough.

Question 6. What is the rule of thumb from White, Royston, and Wood (2011) for choosing m?

Use m ≥ 100 × FMIAlways use m = 5Set m equal to the sample sizeUse m = 2 whenever missingness is below 10%

Explanation.

The chapter recommends the shorthand m >= 100 x FMI, which scales the number of imputations to the amount of missing information in the model. For example, an average FMI of 0.30 suggests at least 30 imputations.

Question 7. What does the predictor matrix tell you in a mice analysis?

Which variables are used to predict each imputed variableWhich cases were dropped before imputationHow many imputations are neededWhether the pooled p-values are significant

Explanation.

Rows identify the variable being imputed and columns identify the available predictors. A 1 means the predictor is used for that row variable, and a 0 means it is excluded.

Question 8. Why is predictive mean matching (pmm) often preferred to norm for continuous variables in small samples?

Because pmm preserves plausible observed values and is less dependent on strict normalityBecause pmm requires no predictorsBecause norm only works for binary dataBecause pmm guarantees narrower confidence intervals

Explanation.

The chapter treats pmm as the safer default because it draws on observed donor values. That makes it more robust when normal-theory assumptions are not especially convincing.

Question 9. What does a high FMI value mean for a pooled coefficient?

That missingness is contributing substantially to uncertainty in that estimateThat the predictor matrix is incorrectThat the data are definitely MNARThat convergence has failed

Explanation.

FMI quantifies how much uncertainty is coming from the missing data rather than only from observed-data sampling variation. High FMI values justify larger m and more cautious interpretation.

Question 10. What should you do if the MI results differ materially from the complete-case results?

Report both and discuss the sensitivity of the conclusion to the missing-data assumptionsAutomatically prefer the imputed resultDelete the complete-case analysis from the paperIncrease the significance threshold

Explanation.

Material disagreement between MI and complete-case results is itself important information. The chapter recommends reporting that sensitivity rather than hiding it.

Question 11. What information should appear in a transparent write-up of an MI analysis?

The imputation method, the number of imputations, the convergence/plausibility checks, and the FMI summaryOnly the final p-values from the pooled modelOnly the percentage of missing valuesOnly the complete-case results

Explanation.

The final reporting section stresses that readers need to know the method, m, the main diagnostics, and the FMI burden. Reporting only the pooled coefficients would hide the quality checks that make the analysis credible.

Ayre, Colin, and Andrew J. Scally. 2014. ‘Critical Values for Lawshe’s Content Validity Ratio: Revisiting the Original Methods of Calculation’. Measurement and Evaluation in Counseling and Development 47 (1): 79–86. https://doi.org/10.1177/0748175613513808.

Briggs, Steven R., and Jonathan M. Cheek. 1986. ‘The Role of Factor Analysis in the Development and Evaluation of Personality Scales’. Journal of Personality 54 (1): 106–48.

Buuren, Stef van. 2018. Flexible Imputation of Missing Data. 2nd ed. Boca Raton, FL: Chapman; Hall/CRC.

Cameron, A. Colin, and Pravin K. Trivedi. 2013. Regression Analysis of Count Data. 2nd ed. Cambridge: Cambridge University Press. https://doi.org/10.1017/CBO9781139013567.

Chen, Henian, Patricia Cohen, and Sophie Chen. 2010. ‘How Big Is a Big Odds Ratio? Interpreting the Magnitudes of Odds Ratios in Epidemiological Studies’. Communications in Statistics - Simulation and Computation 39 (4): 860–64. https://doi.org/10.1080/03610911003650383.

Cicchetti, Domenic V. 1994. ‘Guidelines, Criteria, and Rules of Thumb for Evaluating Normed and Standardized Assessment Instruments in Psychology’. Psychological Assessment 6 (4): 284–90. https://doi.org/10.1037/1040-3590.6.4.284.

Clark, Lee Anna, and David Watson. 1995. ‘Constructing Validity: Basic Issues in Objective Scale Development’. Psychological Assessment 7 (3): 309–19.

Cohen, Jacob. 1988. Statistical Power Analysis for the Behavioral Sciences. 2nd ed. Hillsdale, NJ: Lawrence Erlbaum Associates.

Conover, W. J. 1999. Practical Nonparametric Statistics. 3rd ed. New York: John Wiley & Sons.

Costello, Anna B., and Jason Osborne. 2005. ‘Best Practices in Exploratory Factor Analysis: Four Recommendations for Getting the Most from Your Analysis’. Practical Assessment, Research, and Evaluation 10 (1): 1–9. https://doi.org/10.7275/jyj1-4868.

Davison, A. C., and D. V. Hinkley. 1997. Bootstrap Methods and Their Application. Cambridge Series in Statistical and Probabilistic Mathematics. Cambridge: Cambridge University Press.

DeVellis, Robert F., and Carolyn T. Thorpe. 2021. Scale Development: Theory and Applications. 5th ed. Thousand Oaks, CA: SAGE Publications.

Dienes, Zoltan. 2014. ‘Using Bayes to Get the Most Out of Non-Significant Results’. Frontiers in Psychology 5: 781. https://doi.org/10.3389/fpsyg.2014.00781.

Efron, Bradley, and Robert J. Tibshirani. 1993. An Introduction to the Bootstrap. New York, NY: Chapman; Hall/CRC.

Elm, Erik von, Douglas G. Altman, Matthias Egger, Stuart J. Pocock, Peter C. Gøtzsche, and Jan P. Vandenbroucke. 2007. ‘The Strengthening the Reporting of Observational Studies in Epidemiology (STROBE) Statement: Guidelines for Reporting Observational Studies’. PLoS Medicine 4 (10): e296. https://doi.org/10.1371/journal.pmed.0040296.

Fay, Michael P. 2010. ‘Confidence Intervals That Match Fisher’s Exact or Blaker’s Exact Tests’. Biostatistics 11 (2): 373–74. https://doi.org/10.1093/biostatistics/kxp050.

Firth, David. 1993. ‘Bias Reduction of Maximum Likelihood Estimates’. Biometrika 80 (1): 27–38. https://doi.org/10.1093/biomet/80.1.27.

Gelman, Andrew, Daniel Simpson, and Michael Betancourt. 2017. ‘The Prior Can Often Only Be Understood in the Context of the Likelihood’. Entropy 19 (10): 555. https://doi.org/10.3390/e19100555.

Good, Phillip I. 2005. Permutation, Parametric, and Bootstrap Tests of Hypotheses. 3rd ed. New York: Springer.

Graham, John W. 2009. ‘Missing Data Analysis: Making It Work in the Real World’. Annual Review of Psychology 60: 549–76. https://doi.org/10.1146/annurev.psych.58.110405.085530.

Hair, Joseph F., G. Tomas M. Hult, Christian M. Ringle, and Marko Sarstedt. 2017. A Primer on Partial Least Squares Structural Equation Modeling (PLS-SEM). 2nd ed. Thousand Oaks, CA: SAGE Publications.

Harrell, Frank E. 2015. Regression Modeling Strategies: With Applications to Linear Models, Logistic and Ordinal Regression, and Survival Analysis. 2nd ed. Springer Series in Statistics. New York: Springer. https://doi.org/10.1007/978-3-319-19425-7.

Heinze, Georg, and Michael Schemper. 2002. ‘A Solution to the Problem of Separation in Logistic Regression’. Statistics in Medicine 21 (16): 2409–19. https://doi.org/10.1002/sim.1047.

Hodges, Jr., J. L., and E. L. Lehmann. 1963. ‘Estimates of Location Based on Rank Tests’. The Annals of Mathematical Statistics 34 (2): 598–611. https://doi.org/10.1214/aoms/1177704172.

Hoenig, John M., and Dennis M. Heisey. 2001. ‘The Abuse of Power: The Pervasive Fallacy of Power Calculations for Data Analysis’. The American Statistician 55 (1): 19–24.

Hosmer, David W., Stanley Lemeshow, and Rodney X. Sturdivant. 2013. Applied Logistic Regression. 3rd ed. Wiley Series in Probability and Statistics. Hoboken, NJ: John Wiley & Sons. https://doi.org/10.1002/9781118548387.

Hu, Li-tze, and Peter M. Bentler. 1999. ‘Cutoff Criteria for Fit Indexes in Covariance Structure Analysis: Conventional Criteria Versus New Alternatives’. Structural Equation Modeling 6 (1): 1–55. https://doi.org/10.1080/10705519909540118.

Huberty, Carl J., and Stephen Olejnik. 2006. Applied MANOVA and Discriminant Analysis. 2nd ed. Hoboken, NJ: Wiley. https://doi.org/10.1002/047178947X.

Hwang, Ching-Lai, and Kwangsun Yoon. 1981. Multiple Attribute Decision Making: Methods and Applications. Berlin: Springer. https://doi.org/10.1007/978-3-642-48318-9.

Hyndman, Rob J., and George Athanasopoulos. 2021. Forecasting: Principles and Practice. 3rd ed. Melbourne, Australia: OTexts. https://otexts.com/fpp3/.

Koo, Terry K., and Mae Y. Li. 2016. ‘A Guideline of Selecting and Reporting Intraclass Correlation Coefficients for Reliability Research’. Journal of Chiropractic Medicine 15 (2): 155–63. https://doi.org/10.1016/j.jcm.2016.02.012.

Kraft, Matthew A. 2020. ‘Interpreting Effect Sizes of Education Interventions’. Educational Researcher 49 (4): 241–53. https://doi.org/10.3102/0013189X20912798.

Lakens, Daniël. 2013. ‘Calculating and Reporting Effect Sizes to Facilitate Cumulative Science: A Practical Primer for t-Tests and ANOVAs’. Frontiers in Psychology 4: 863. https://doi.org/10.3389/fpsyg.2013.00863.

Lakens, Daniël, Anne M. Scheel, and Peder M. Isager. 2018. ‘Equivalence Testing for Psychological Research: A Tutorial’. Advances in Methods and Practices in Psychological Science 1 (2): 259–69. https://doi.org/10.1177/2515245918770963.

Lawshe, Charles H. 1975. ‘A Quantitative Approach to Content Validity’. Personnel Psychology 28 (4): 563–75. https://doi.org/10.1111/j.1744-6570.1975.tb01393.x.

Lipsey, Mark W., and David B. Wilson. 1993. ‘The Efficacy of Psychological, Educational, and Behavioural Treatment: Confirmation from Meta-Analysis’. American Psychologist 48 (12): 1181–1209. https://doi.org/10.1037/0003-066X.48.12.1181.

Little, Roderick J. A. 1988. ‘A Test of Missing Completely at Random for Multivariate Data with Missing Values’. Journal of the American Statistical Association 83 (404): 1198–1202. https://doi.org/10.1080/01621459.1988.10478722.

MacKinnon, David P., Jennifer L. Krull, and Chondra M. Lockwood. 2000. ‘Equivalence of the Mediation, Confounding and Suppression Effect’. Prevention Science 1 (4): 173–81.

Mair, Patrick, and Rand R. Wilcox. 2020. ‘Robust Statistical Methods in r Using the WRS2 Package’. Behavior Research Methods 52 (2): 464–88. https://doi.org/10.3758/s13428-019-01246-w.

Mann, Henry B., and Donald R. Whitney. 1947. ‘On a Test of Whether One of Two Random Variables Is Stochastically Larger Than the Other’. The Annals of Mathematical Statistics 18 (1): 50–60. https://doi.org/10.1214/aoms/1177730491.

McDonald, Roderick P. 1999. Test Theory: A Unified Treatment. Mahwah, NJ: Lawrence Erlbaum Associates.

Morey, Richard D., and Jeffrey N. Rouder. 2011. ‘Bayes Factor Approaches for Testing Interval Null Hypotheses’. Psychological Methods 16 (4): 406–19. https://doi.org/10.1037/a0024377.

Nielsen, Jakob. 1993. Usability Engineering. San Francisco, CA: Morgan Kaufmann.

Olsson, Ulf. 1979. ‘Maximum Likelihood Estimation of the Polychoric Correlation Coefficient’. Psychometrika 44 (4): 443–60.

Opricovic, Serafim, and Gwo-Hshiung Tzeng. 2004. ‘Compromise Solution by MCDM Methods: A Comparative Analysis of VIKOR and TOPSIS’. European Journal of Operational Research 156 (2): 445–55. https://doi.org/10.1016/S0377-2217(03)00020-1.

Page, Matthew J., Joanne E. McKenzie, Patrick M. Bossuyt, Isabelle Boutron, Tammy C. Hoffmann, Cynthia D. Mulrow, Larissa Shamseer, et al. 2021. ‘The PRISMA 2020 Statement: An Updated Guideline for Reporting Systematic Reviews’. BMJ 372: n71. https://doi.org/10.1136/bmj.n71.

R Core Team. 2026. R: A Language and Environment for Statistical Computing. Vienna, Austria: R Foundation for Statistical Computing. https://www.R-project.org/.

Rubin, Donald B. 1987. Multiple Imputation for Nonresponse in Surveys. New York, NY: Wiley.

Saaty, Thomas L. 1980. The Analytic Hierarchy Process. New York, NY: McGraw-Hill.

Schulz, Kenneth F., Douglas G. Altman, and David Moher. 2010. ‘CONSORT 2010 Statement: Updated Guidelines for Reporting Parallel Group Randomised Trials’. BMJ 340: c332. https://doi.org/10.1136/bmj.c332.

Shan, Guogen. 2018. Exact Statistical Inference for Categorical Data. London: Academic Press.

Sterne, Jonathan A. C., Ian R. White, John B. Carlin, Michael Spratt, Patrick Royston, Michael G. Kenward, Angela M. Wood, and James R. Carpenter. 2009. ‘Multiple Imputation for Missing Data in Epidemiological and Clinical Research: Potential and Pitfalls’. BMJ 338: b2393. https://doi.org/10.1136/bmj.b2393.

Teare, M. Dawn, Munyaradzi Dimairo, Neil Shephard, Amanda Hayman, Amy Whitehead, and Stephen J. Walters. 2014. ‘Sample Size Requirements to Estimate Key Design Parameters from External Pilot Randomised Controlled Trials: A Simulation Study’. Trials 15: 264. https://doi.org/10.1186/1745-6215-15-264.

Tomczak, Maciej, and Ewa Tomczak. 2014. ‘The Need to Report Effect Size Estimates Revisited: An Overview of Some Recommended Measures of Effect Size’. Trends in Sport Sciences 21 (1): 19–25.

Trizano-Hermosilla, Italo, and Jose M. Alvarado. 2016. ‘Best Alternatives to Cronbach’s Alpha Reliability in Realistic Conditions: Congeneric and Asymmetrical Measurements’. Frontiers in Psychology 7: 769. https://doi.org/10.3389/fpsyg.2016.00769.

Van de Schoot, Rens, and Milica Miočević. 2020. Small Sample Size Solutions: A Guide for Applied Researchers and Practitioners. London: Routledge.

Vehtari, Aki, Andrew Gelman, and Jonah Gabry. 2017. ‘Practical Bayesian Model Evaluation Using Leave-One-Out Cross-Validation and WAIC’. Statistics and Computing 27 (5): 1413–32. https://doi.org/10.1007/s11222-016-9696-4.

Wagenmakers, Eric-Jan, Jonathon Love, Maarten Marsman, Tahira Jamil, Alexander Ly, Josine Verhagen, Ravi Selker, et al. 2018. ‘Bayesian Inference for Psychology. Part II: Example Applications with JASP’. Psychonomic Bulletin & Review 25 (1): 58–76. https://doi.org/10.3758/s13423-017-1323-7.

White, Ian R., Patrick Royston, and Angela M. Wood. 2011. ‘Multiple Imputation Using Chained Equations: Issues and Guidance for Practice’. Statistics in Medicine 30 (4): 377–99. https://doi.org/10.1002/sim.4067.