Chapter 4: Measurement Quality and Scale Development

Learning Objectives

By the end of this chapter, you will be able to explain the distinctions between content, construct, and criterion validity, pilot and refine short scales with limited samples, compute and interpret basic item-level diagnostics in R, and report measurement evidence transparently when full psychometric validation is not yet feasible.

The Challenge of Measurement in Small Studies

Many small-sample studies rely on brief, custom-developed measurement instruments. Standard scale development protocols (large pilot studies, factor analysis, item response theory) require hundreds of observations. With small samples, researchers must balance the need for reliable, valid measurement with practical constraints.

Short scales (3–5 items) can be internally consistent and valid if items are carefully chosen. Pilot testing with qualitative feedback (cognitive interviews, think-aloud protocols) can identify ambiguous wording, response biases, and cultural appropriateness. Quantitative pilot data (even with n of about 20 to 30) can reveal extreme floor or ceiling effects, items with no variance, and obvious inconsistencies.

Content and Face Validity

Content validity refers to whether items comprehensively and appropriately represent the construct being measured. Face validity refers to whether items appear relevant and appropriate to respondents. Face validity is not a psychometric property in its own right and therefore carries limited evidentiary weight; it should complement, not substitute for, stronger content or construct validity evidence. Both are assessed through expert review and respondent feedback during scale development, before quantitative pilot testing, typically with input from both domain experts and representatives of the target population.

Content Validity Ratio (CVR)

Lawshe’s Content Validity Ratio provides a simple index of expert agreement on whether an item is “essential” to a construct. With \(N\) experts and \(n_e\) rating an item as essential, the CVR is:

\[ \operatorname{CVR} = \frac{n_e - N/2}{N/2} \]

CVR ranges from −1 to +1. Positive values indicate majority agreement that the item is essential, but retention thresholds depend on panel size. Under the conventional one-tailed \(\alpha = 0.05\) rule, a panel of eight experts requires a CVR of 0.75 or higher, which means at least seven experts must judge the item essential (Lawshe 1975; Ayre and Scally 2014). These critical values correspond to exact binomial probabilities and were re-examined by Ayre and Scally (Ayre and Scally 2014). Use CVR alongside qualitative feedback to decide which items to retain.

Rendered Output

[1] 0.5

Cell Code

Interpretation: A CVR of 0.50 reflects that 6 of 8 experts (75%) rated the item essential. However, for an 8-member panel the usual retention threshold is 0.75, which requires agreement from at least 7 experts (Lawshe 1975; Ayre and Scally 2014). Because 0.50 is below 0.75, this item would not satisfy Lawshe’s criterion for statistically significant content validity. Researchers should consult verified critical-value tables when using CVR for item-retention decisions.

Content Validity Index (CVI)

The Content Validity Index is a more descriptive companion to CVR and is common in health, education, and applied measurement work. For an item-level CVI (I-CVI), experts rate each item for relevance, often on a 1–4 scale, and the analyst computes the proportion of experts rating the item as either 3 or 4. A scale-level average CVI (S-CVI/Ave) is then the mean of the item-level CVIs across the candidate item set. CVI does not replace qualitative review, but it gives a transparent summary of expert agreement before a small quantitative pilot begins.

For example, suppose five experts rated six candidate items on a four-point relevance scale. Ratings of 3 or 4 are counted as content-valid endorsements.

Item	Experts rating 3 or 4	I-CVI	Pilot decision
Item 1	5 of 5	1.00	Retain
Item 2	4 of 5	0.80	Retain, check wording
Item 3	3 of 5	0.60	Revise before pilot
Item 4	5 of 5	1.00	Retain
Item 5	2 of 5	0.40	Drop or rewrite
Item 6	4 of 5	0.80	Retain, check redundancy

The S-CVI/Ave for this set is 0.77, calculated as the average of the six I-CVI values. The table is a discussion tool, not a mechanical keep/drop decision rule. Items 3 and 5 need substantive review because several experts did not judge them clearly relevant. Items 2 and 6 may be acceptable, but the qualitative comments should be checked for ambiguity or overlap.

Construct and Criterion Validity in Small Samples

Content validity is only one part of measurement quality. Researchers also need to consider whether scores behave as theory predicts, which is the core of construct validity, and whether they relate to a meaningful external benchmark, which is the core of criterion validity. In small studies, these checks should usually be modest and pre-specified. Rather than claiming a full psychometric validation from n = 20–40, look for tentative supporting evidence by comparing groups that theory predicts should differ on the construct, examining correlations with one or two closely related measures while reporting effect sizes and confidence intervals, or checking whether scores predict a practically relevant external outcome. Report effect sizes and uncertainty, avoid post-hoc fishing for significant associations, and avoid strong claims from unstable estimates.

Reliability, Validity, and Measurement Error

Reliability concerns consistency; validity concerns whether the instrument measures the intended construct. A scale can be reliable without being valid. In an early pilot, item statistics and internal consistency estimates are best treated as screening tools that flag obvious weaknesses, not as definitive proof that the scale is ready for high-stakes use. Chapter 5 extends this discussion for short scales.

If items are scored or coded by human raters, agreement among raters becomes part of measurement quality. With two raters, report a chance-corrected agreement statistic such as Cohen’s kappa alongside the raw agreement percentage; with more than two raters, use a multi-rater extension such as Fleiss’ kappa or an intraclass correlation coefficient when scores are numeric. In small samples these estimates can be unstable, so describe the coding protocol, rater training, and disagreement-resolution process rather than reporting a coefficient alone.

Steps for Scale Development with Small Samples

1. Define the construct clearly. Start by specifying what you are measuring and which dimensions or facets belong inside that construct.
2. Generate candidate items. Write more items than you expect to keep so that weak or redundant items can be removed later without leaving gaps in content coverage.
3. Expert review. Ask domain experts to rate each candidate item for relevance, clarity, and representativeness, and use their feedback to flag wording or content problems early.
4. Cognitive interviews. Ask a small number of respondents (often n = 5–10) to complete the scale while thinking aloud so you can hear how they interpret the wording and response options.
5. Quantitative pilot. Administer the revised scale to a small sample (often n = 20–40) and compute descriptive item statistics to identify obvious weaknesses.
6. Item analysis. Review patterns such as low variance, weak corrected item-total correlations, and floor or ceiling effects to decide which items need revision.
7. Refine and re-test. Remove or revise problematic items and, if resources permit, test the revised version again because scale development is iterative rather than strictly linear (DeVellis and Thorpe 2021).

Example: Item Analysis for a Pilot Scale

We pilot a 5-item job satisfaction scale with n = 25 employees. Each item uses a 1–7 Likert response.

Rendered Output

Table 4.1

Item descriptive statistics for the pilot scale

Item	Mean	SD	Min	Max
clear_communication	4.32	1.97	1	7
staff_courtesy	4.20	1.71	1	7
wait_time_ok	6.08	0.76	5	7
problem_resolved	3.60	1.80	1	7
overall_satisfied	4.12	1.72	1	7

Note. Item 3 has the narrowest spread and the highest mean, which is consistent with a possible ceiling effect.

Table 4.2

Inter-item correlation matrix for the pilot scale

Item	clear_communication	staff_courtesy	wait_time_ok	problem_resolved	overall_satisfied
clear_communication	1.00	0.34	0.18	-0.20	0.46
staff_courtesy	0.34	1.00	0.12	0.03	0.55
wait_time_ok	0.18	0.12	1.00	0.27	0.41
problem_resolved	-0.20	0.03	0.27	1.00	0.06
overall_satisfied	0.46	0.55	0.41	0.06	1.00

Note. Positive values indicate items moving in the same direction; negative values flag possible misfit, reverse coding, or a different latent facet.

Table 4.3

Corrected item-total correlations for the pilot scale

Item	Corrected item-total correlation	Mean	SD
clear_communication	0.29	4.32	1.97
staff_courtesy	0.45	4.20	1.71
wait_time_ok	0.38	6.08	0.76
problem_resolved	-0.01	3.60	1.80
overall_satisfied	0.62	4.12	1.72

Note. Items 3 and 4 fall below the usual screening threshold of about 0.30. Cronbach’s alpha for the initial 5-item scale was 0.540.

Cell Code

Reproducibility note: When adapting this code for your own pilot, record your set.seed() value and package versions so the same simulated item patterns can be regenerated.

Cronbach’s alpha for the initial 5-item scale was 0.540. Because this pilot uses only 25 respondents, the estimate should be treated cautiously: alpha can be unstable with small samples, so in applied work it is sensible either to report confidence intervals or to note the sampling uncertainty around the point estimate.

Interpretation: Look for a broadly coherent pattern rather than demanding that every statistic fall within a rigid threshold. In this pilot, Items 1, 2, and 5 move together reasonably well. Item 3 has a restricted high-end range and a negative corrected item-total correlation, suggesting that it is not functioning like the rest of the scale. Item 4 contributes very little to the total score, which makes it a candidate for revision or removal as well. The corrected item-total correlation (r.drop) is especially useful here: values below approximately 0.30 often indicate weak discrimination and deserve closer review, but the threshold is context-dependent. Very short scales may legitimately show lower average inter-item correlations, and substantively important facets should not be dropped solely on statistical grounds.

Identifying Problematic Items

Problematic items usually reveal themselves through a combination of weak descriptive and correlational signals. If an item has very low variance, most respondents are giving essentially the same answer, which often means the wording is too obvious, too extreme, or too narrow. If the corrected item-total correlation is weak, the item may be discriminating poorly or may be tapping a different construct. Floor or ceiling effects limit an item’s ability to distinguish among respondents because most answers cluster at one end of the scale. Negative correlations are especially important warning signs because they often indicate incorrect reverse coding or an item working against the rest of the scale. Treat these patterns as heuristics rather than automatic deletion rules: a statistically weak item may still capture an important facet of the construct and be worth revising rather than dropping.

Refining the Scale

Based on item analysis, revise or remove problematic items. For example, if Item 3 shows a ceiling effect and Item 4 has weak item-total correlation, consider removing them. Compute alpha for the revised scale.

Rendered Output

Table 4.4

Internal consistency before and after item refinement

Scale	Cronbach’s alpha	Average inter-item correlation
Initial 5-item scale	0.540	0.220
Revised 3-item scale	0.702	0.447

Note. Removing wait_time_ok and problem_resolved improves both alpha and the average inter-item correlation in this simulated pilot example.

Cell Code

Interpretation: In this example, dropping Items 3 and 4 raises alpha from the initial 5-item version to the revised 3-item version. That does not mean researchers should always delete items with low statistics. Item removal should also reflect the construct definition, expert review, and respondent feedback. If Item 3 captures an important facet of job satisfaction, revising its wording may be preferable to dropping it entirely. For very short scales, it is also useful to inspect the average inter-item correlation rather than relying on alpha alone (DeVellis and Thorpe 2021).

Qualitative Feedback and Cognitive Interviews

With very small samples (n < 20), quantitative item analysis is unreliable. Qualitative methods (cognitive interviews, focus groups) are more informative. Ask respondents:

• What does each item mean to you?
• Were any items confusing, ambiguous, or difficult to answer?
• Are the response options appropriate?
• Are any items culturally inappropriate or offensive?

This feedback can prevent major problems before larger-scale data collection.

Key Takeaways

Measurement quality matters especially in small-sample research because unreliable measures reduce power and can distort substantive conclusions. In practice, strong small-sample scale development depends less on claiming full psychometric validation and more on combining expert review, respondent feedback, and cautious item-level diagnostics to identify obvious weaknesses. Iterative refinement, rather than one definitive pilot, is what gradually improves the scale when resources are limited.

Self-Assessment Quiz

Question 1. What is the primary advantage of using qualitative methods (cognitive interviews) over quantitative methods when pilot testing scales with very small samples (n < 20)?

Cognitive interviews provide more statistical powerQualitative feedback can identify ambiguous wording and cultural issues without requiring statistical reliabilityQuantitative item analysis is too expensiveCognitive interviews automatically calculate Cronbach’s alpha

Explanation.

“With very small samples (n < 20), quantitative item analysis is unreliable. Qualitative methods (cognitive interviews, focus groups) are more informative.” Cognitive interviews reveal ambiguous wording and cultural issues without requiring the large samples needed for statistical reliability indices.

Question 2. In Lawshe’s Content Validity Ratio (CVR), if 8 experts are consulted and 6 judge an item as “essential,” what is the CVR?

0.250.500.751.00

Explanation.

CVR = (6 - 8/2) / (8/2) = (6 - 4) / 4 = 2/4 = 0.50. That means 75% of experts judged the item essential. However, with N = 8 the usual Lawshe threshold is 0.75, so a CVR of 0.50 would not satisfy the standard retention criterion.

Question 3. What does a “ceiling effect” in item analysis indicate?

Most respondents give the lowest possible responseMost respondents give the highest possible responseThe item has perfect reliabilityThe item correlates negatively with the total score

Explanation.

“If most responses cluster at the low or high end, the item cannot differentiate among respondents.” A ceiling effect occurs when most respondents give the highest possible response, limiting discrimination.

Question 4. Which corrected item-total correlation threshold typically indicates that an item discriminates poorly and should be considered for removal?

Above 0.7Between 0.3 and 0.7Below 0.3Exactly 0.5

Explanation.

“The corrected item-total correlation (r.drop) indicates how well each item correlates with the total score excluding itself. Values below about 0.3 often suggest weak items that deserve closer review.” Such items may discriminate poorly, but final decisions should still consider scale length, construct coverage, and respondent feedback.

Question 5. What does content validity assess?

Whether items comprehensively and appropriately represent the construct being measuredWhether the scale has high Cronbach’s alphaWhether factor analysis confirms a one-dimensional structureWhether the scale predicts future behavior

Explanation.

“Content validity refers to whether items comprehensively and appropriately represent the construct being measured.” Content validity is assessed through expert review, not statistical tests.

Question 6. In a small pilot study, what would count as preliminary evidence for construct validity?

A single high Cronbach’s alphaScores relate to other variables or known groups in the direction theory predictsRespondents say the scale looks professionalA factor analysis with n = 25 produces one factor

Explanation.

Construct validity concerns whether scores behave in ways theory predicts. In small studies, this usually means modest, pre-specified checks such as known-groups comparisons or correlations with closely related measures, reported with effect sizes and uncertainty rather than post-hoc fishing for significant associations.

Question 7. In pilot testing with small samples (n of about 20 to 30), what is the primary limitation of conducting factor analysis?

Factor analysis requires specialised softwareFactor analysis usually needs much larger samples for stable loadings; n of about 20 to 30 is typically too smallFactor analysis only works with 7-point Likert scalesFactor analysis cannot handle missing data

Explanation.

Factor analysis usually needs substantially larger samples than a small pilot can provide. Common rules of thumb suggest about 5 to 10 participants per item or an absolute N of at least 100, although the exact requirement depends on communalities, item quality, and model complexity. A pilot sample of 20 to 30 is therefore usually too small for stable factor loadings, though it can still support preliminary item screening.

Question 8. If an item correlates negatively with the total score and with other items, what is the most likely explanation?

The item has high content validityThe item may be reverse-coded incorrectly or measuring an opposite constructThe sample size is too largeThe item has a ceiling effect

Explanation.

“If an item correlates negatively with the total or with other items, it may be reverse-coded incorrectly or measuring an opposite construct.” Negative correlations suggest coding errors or conceptual misalignment with the scale.

Question 9. What does criterion validity ask?

Whether scale scores align with an external benchmark or relevant outcomeWhether experts agree an item is essentialWhether all items load on one factorWhether a scale has no missing values

Explanation.

Criterion validity asks whether scores relate to an external benchmark, either at the same time (concurrent validity) or later (predictive validity). In small studies, this evidence should be reported cautiously with effect sizes and uncertainty.

Question 10. In the example of the 5-item job satisfaction scale, Item 3 had a restricted range (responses only from 4–7 on a 1–7 scale). What does this suggest?

Item 3 has perfect reliabilityItem 3 may have a ceiling effect, limiting its ability to differentiate among respondentsItem 3 should be kept because high scores are desirableThe sample size should be increased to 1,000

Explanation.

“Item 3 has a restricted range (4–7), which may indicate a ceiling effect.” Restricted ranges (especially at the high end) indicate ceiling effects that limit the item’s ability to differentiate among respondents.

Ayre, Colin, and Andrew J. Scally. 2014. ‘Critical Values for Lawshe’s Content Validity Ratio: Revisiting the Original Methods of Calculation’. Measurement and Evaluation in Counseling and Development 47 (1): 79–86. https://doi.org/10.1177/0748175613513808.

Briggs, Steven R., and Jonathan M. Cheek. 1986. ‘The Role of Factor Analysis in the Development and Evaluation of Personality Scales’. Journal of Personality 54 (1): 106–48.

Buuren, Stef van. 2018. Flexible Imputation of Missing Data. 2nd ed. Boca Raton, FL: Chapman; Hall/CRC.

Cameron, A. Colin, and Pravin K. Trivedi. 2013. Regression Analysis of Count Data. 2nd ed. Cambridge: Cambridge University Press. https://doi.org/10.1017/CBO9781139013567.

Chen, Henian, Patricia Cohen, and Sophie Chen. 2010. ‘How Big Is a Big Odds Ratio? Interpreting the Magnitudes of Odds Ratios in Epidemiological Studies’. Communications in Statistics - Simulation and Computation 39 (4): 860–64. https://doi.org/10.1080/03610911003650383.

Cicchetti, Domenic V. 1994. ‘Guidelines, Criteria, and Rules of Thumb for Evaluating Normed and Standardized Assessment Instruments in Psychology’. Psychological Assessment 6 (4): 284–90. https://doi.org/10.1037/1040-3590.6.4.284.

Clark, Lee Anna, and David Watson. 1995. ‘Constructing Validity: Basic Issues in Objective Scale Development’. Psychological Assessment 7 (3): 309–19.

Cohen, Jacob. 1988. Statistical Power Analysis for the Behavioral Sciences. 2nd ed. Hillsdale, NJ: Lawrence Erlbaum Associates.

Conover, W. J. 1999. Practical Nonparametric Statistics. 3rd ed. New York: John Wiley & Sons.

Costello, Anna B., and Jason Osborne. 2005. ‘Best Practices in Exploratory Factor Analysis: Four Recommendations for Getting the Most from Your Analysis’. Practical Assessment, Research, and Evaluation 10 (1): 1–9. https://doi.org/10.7275/jyj1-4868.

Davison, A. C., and D. V. Hinkley. 1997. Bootstrap Methods and Their Application. Cambridge Series in Statistical and Probabilistic Mathematics. Cambridge: Cambridge University Press.

DeVellis, Robert F., and Carolyn T. Thorpe. 2021. Scale Development: Theory and Applications. 5th ed. Thousand Oaks, CA: SAGE Publications.

Dienes, Zoltan. 2014. ‘Using Bayes to Get the Most Out of Non-Significant Results’. Frontiers in Psychology 5: 781. https://doi.org/10.3389/fpsyg.2014.00781.

Efron, Bradley, and Robert J. Tibshirani. 1993. An Introduction to the Bootstrap. New York, NY: Chapman; Hall/CRC.

Elm, Erik von, Douglas G. Altman, Matthias Egger, Stuart J. Pocock, Peter C. Gøtzsche, and Jan P. Vandenbroucke. 2007. ‘The Strengthening the Reporting of Observational Studies in Epidemiology (STROBE) Statement: Guidelines for Reporting Observational Studies’. PLoS Medicine 4 (10): e296. https://doi.org/10.1371/journal.pmed.0040296.

Fay, Michael P. 2010. ‘Confidence Intervals That Match Fisher’s Exact or Blaker’s Exact Tests’. Biostatistics 11 (2): 373–74. https://doi.org/10.1093/biostatistics/kxp050.

Firth, David. 1993. ‘Bias Reduction of Maximum Likelihood Estimates’. Biometrika 80 (1): 27–38. https://doi.org/10.1093/biomet/80.1.27.

Gelman, Andrew, Daniel Simpson, and Michael Betancourt. 2017. ‘The Prior Can Often Only Be Understood in the Context of the Likelihood’. Entropy 19 (10): 555. https://doi.org/10.3390/e19100555.

Good, Phillip I. 2005. Permutation, Parametric, and Bootstrap Tests of Hypotheses. 3rd ed. New York: Springer.

Graham, John W. 2009. ‘Missing Data Analysis: Making It Work in the Real World’. Annual Review of Psychology 60: 549–76. https://doi.org/10.1146/annurev.psych.58.110405.085530.

Hair, Joseph F., G. Tomas M. Hult, Christian M. Ringle, and Marko Sarstedt. 2017. A Primer on Partial Least Squares Structural Equation Modeling (PLS-SEM). 2nd ed. Thousand Oaks, CA: SAGE Publications.

Harrell, Frank E. 2015. Regression Modeling Strategies: With Applications to Linear Models, Logistic and Ordinal Regression, and Survival Analysis. 2nd ed. Springer Series in Statistics. New York: Springer. https://doi.org/10.1007/978-3-319-19425-7.

Heinze, Georg, and Michael Schemper. 2002. ‘A Solution to the Problem of Separation in Logistic Regression’. Statistics in Medicine 21 (16): 2409–19. https://doi.org/10.1002/sim.1047.

Hodges, Jr., J. L., and E. L. Lehmann. 1963. ‘Estimates of Location Based on Rank Tests’. The Annals of Mathematical Statistics 34 (2): 598–611. https://doi.org/10.1214/aoms/1177704172.

Hoenig, John M., and Dennis M. Heisey. 2001. ‘The Abuse of Power: The Pervasive Fallacy of Power Calculations for Data Analysis’. The American Statistician 55 (1): 19–24.

Hosmer, David W., Stanley Lemeshow, and Rodney X. Sturdivant. 2013. Applied Logistic Regression. 3rd ed. Wiley Series in Probability and Statistics. Hoboken, NJ: John Wiley & Sons. https://doi.org/10.1002/9781118548387.

Hu, Li-tze, and Peter M. Bentler. 1999. ‘Cutoff Criteria for Fit Indexes in Covariance Structure Analysis: Conventional Criteria Versus New Alternatives’. Structural Equation Modeling 6 (1): 1–55. https://doi.org/10.1080/10705519909540118.

Huberty, Carl J., and Stephen Olejnik. 2006. Applied MANOVA and Discriminant Analysis. 2nd ed. Hoboken, NJ: Wiley. https://doi.org/10.1002/047178947X.

Hwang, Ching-Lai, and Kwangsun Yoon. 1981. Multiple Attribute Decision Making: Methods and Applications. Berlin: Springer. https://doi.org/10.1007/978-3-642-48318-9.

Hyndman, Rob J., and George Athanasopoulos. 2021. Forecasting: Principles and Practice. 3rd ed. Melbourne, Australia: OTexts. https://otexts.com/fpp3/.

Koo, Terry K., and Mae Y. Li. 2016. ‘A Guideline of Selecting and Reporting Intraclass Correlation Coefficients for Reliability Research’. Journal of Chiropractic Medicine 15 (2): 155–63. https://doi.org/10.1016/j.jcm.2016.02.012.

Kraft, Matthew A. 2020. ‘Interpreting Effect Sizes of Education Interventions’. Educational Researcher 49 (4): 241–53. https://doi.org/10.3102/0013189X20912798.

Lakens, Daniël. 2013. ‘Calculating and Reporting Effect Sizes to Facilitate Cumulative Science: A Practical Primer for t-Tests and ANOVAs’. Frontiers in Psychology 4: 863. https://doi.org/10.3389/fpsyg.2013.00863.

Lakens, Daniël, Anne M. Scheel, and Peder M. Isager. 2018. ‘Equivalence Testing for Psychological Research: A Tutorial’. Advances in Methods and Practices in Psychological Science 1 (2): 259–69. https://doi.org/10.1177/2515245918770963.

Lawshe, Charles H. 1975. ‘A Quantitative Approach to Content Validity’. Personnel Psychology 28 (4): 563–75. https://doi.org/10.1111/j.1744-6570.1975.tb01393.x.

Lipsey, Mark W., and David B. Wilson. 1993. ‘The Efficacy of Psychological, Educational, and Behavioural Treatment: Confirmation from Meta-Analysis’. American Psychologist 48 (12): 1181–1209. https://doi.org/10.1037/0003-066X.48.12.1181.

Little, Roderick J. A. 1988. ‘A Test of Missing Completely at Random for Multivariate Data with Missing Values’. Journal of the American Statistical Association 83 (404): 1198–1202. https://doi.org/10.1080/01621459.1988.10478722.

MacKinnon, David P., Jennifer L. Krull, and Chondra M. Lockwood. 2000. ‘Equivalence of the Mediation, Confounding and Suppression Effect’. Prevention Science 1 (4): 173–81.

Mair, Patrick, and Rand R. Wilcox. 2020. ‘Robust Statistical Methods in r Using the WRS2 Package’. Behavior Research Methods 52 (2): 464–88. https://doi.org/10.3758/s13428-019-01246-w.

Mann, Henry B., and Donald R. Whitney. 1947. ‘On a Test of Whether One of Two Random Variables Is Stochastically Larger Than the Other’. The Annals of Mathematical Statistics 18 (1): 50–60. https://doi.org/10.1214/aoms/1177730491.

McDonald, Roderick P. 1999. Test Theory: A Unified Treatment. Mahwah, NJ: Lawrence Erlbaum Associates.

Morey, Richard D., and Jeffrey N. Rouder. 2011. ‘Bayes Factor Approaches for Testing Interval Null Hypotheses’. Psychological Methods 16 (4): 406–19. https://doi.org/10.1037/a0024377.

Nielsen, Jakob. 1993. Usability Engineering. San Francisco, CA: Morgan Kaufmann.

Olsson, Ulf. 1979. ‘Maximum Likelihood Estimation of the Polychoric Correlation Coefficient’. Psychometrika 44 (4): 443–60.

Opricovic, Serafim, and Gwo-Hshiung Tzeng. 2004. ‘Compromise Solution by MCDM Methods: A Comparative Analysis of VIKOR and TOPSIS’. European Journal of Operational Research 156 (2): 445–55. https://doi.org/10.1016/S0377-2217(03)00020-1.

Page, Matthew J., Joanne E. McKenzie, Patrick M. Bossuyt, Isabelle Boutron, Tammy C. Hoffmann, Cynthia D. Mulrow, Larissa Shamseer, et al. 2021. ‘The PRISMA 2020 Statement: An Updated Guideline for Reporting Systematic Reviews’. BMJ 372: n71. https://doi.org/10.1136/bmj.n71.

R Core Team. 2026. R: A Language and Environment for Statistical Computing. Vienna, Austria: R Foundation for Statistical Computing. https://www.R-project.org/.

Rubin, Donald B. 1987. Multiple Imputation for Nonresponse in Surveys. New York, NY: Wiley.

Saaty, Thomas L. 1980. The Analytic Hierarchy Process. New York, NY: McGraw-Hill.

Schulz, Kenneth F., Douglas G. Altman, and David Moher. 2010. ‘CONSORT 2010 Statement: Updated Guidelines for Reporting Parallel Group Randomised Trials’. BMJ 340: c332. https://doi.org/10.1136/bmj.c332.

Shan, Guogen. 2018. Exact Statistical Inference for Categorical Data. London: Academic Press.

Sterne, Jonathan A. C., Ian R. White, John B. Carlin, Michael Spratt, Patrick Royston, Michael G. Kenward, Angela M. Wood, and James R. Carpenter. 2009. ‘Multiple Imputation for Missing Data in Epidemiological and Clinical Research: Potential and Pitfalls’. BMJ 338: b2393. https://doi.org/10.1136/bmj.b2393.

Teare, M. Dawn, Munyaradzi Dimairo, Neil Shephard, Amanda Hayman, Amy Whitehead, and Stephen J. Walters. 2014. ‘Sample Size Requirements to Estimate Key Design Parameters from External Pilot Randomised Controlled Trials: A Simulation Study’. Trials 15: 264. https://doi.org/10.1186/1745-6215-15-264.

Tomczak, Maciej, and Ewa Tomczak. 2014. ‘The Need to Report Effect Size Estimates Revisited: An Overview of Some Recommended Measures of Effect Size’. Trends in Sport Sciences 21 (1): 19–25.

Trizano-Hermosilla, Italo, and Jose M. Alvarado. 2016. ‘Best Alternatives to Cronbach’s Alpha Reliability in Realistic Conditions: Congeneric and Asymmetrical Measurements’. Frontiers in Psychology 7: 769. https://doi.org/10.3389/fpsyg.2016.00769.

Van de Schoot, Rens, and Milica Miočević. 2020. Small Sample Size Solutions: A Guide for Applied Researchers and Practitioners. London: Routledge.

Vehtari, Aki, Andrew Gelman, and Jonah Gabry. 2017. ‘Practical Bayesian Model Evaluation Using Leave-One-Out Cross-Validation and WAIC’. Statistics and Computing 27 (5): 1413–32. https://doi.org/10.1007/s11222-016-9696-4.

Wagenmakers, Eric-Jan, Jonathon Love, Maarten Marsman, Tahira Jamil, Alexander Ly, Josine Verhagen, Ravi Selker, et al. 2018. ‘Bayesian Inference for Psychology. Part II: Example Applications with JASP’. Psychonomic Bulletin & Review 25 (1): 58–76. https://doi.org/10.3758/s13423-017-1323-7.

White, Ian R., Patrick Royston, and Angela M. Wood. 2011. ‘Multiple Imputation Using Chained Equations: Issues and Guidance for Practice’. Statistics in Medicine 30 (4): 377–99. https://doi.org/10.1002/sim.4067.