Chapter 6: Developing Short Scales for Small Samples

Learning Objectives

By the end of this chapter, you will be able to explain why short-scale development with small samples requires staged validation rather than one definitive study, match psychometric tools to the sample sizes they can realistically support, use item-level diagnostics to refine candidate items, and report scale-development evidence transparently without overstating what a pilot can show.

The Scale Development Lifecycle

Scale development is inherently a multi-stage process. With small samples, researchers must be strategic about which psychometric analyses to conduct at each stage and which to reserve for later validation.

The small helper functions used below (for example, chapter6_simulate_scale() and chapter6_problem_item_diagnostics()) are defined in R/chapter_helpers.R and sourced in the setup chunk. Readers who want to run individual chunks can first run source("R/chapter_helpers.R") from the project root.

The Iterative Process

Stage 1: Item Generation (n = 5–10 cognitive interviews)

A first round of about 5 to 10 cognitive interviews usually surfaces the clearest wording and comprehension problems in an initial item pool (Nielsen 1993).

Goal: Generate a pool of 2–3× your target number of items and ensure they are comprehensible.

Methods:

• Literature review: Identify existing scales and adapt items
• Expert consultation: Subject matter experts suggest relevant content
• Cognitive interviews: Think-aloud protocols with 5–10 participants from the target population

Example:

Rendered Output

Table 6.1

Candidate items flagged during cognitive interviewing.

Item ID	Candidate Item	Interview Note
2	I bounce back quickly from setbacks	Ambiguous: what counts as 'quickly'?
7	I rebound after difficult situations	Too similar to item 2
13	I seek support when needed	Double-barreled: 'seek' and 'support'

Note. Items flagged during think-aloud work should be revised before any pilot administration.

Cell Code

Key Point: At this stage, do NOT collect quantitative data. Focus on qualitative feedback about item clarity, relevance, and comprehensiveness.

Stage 2: Pilot Testing (n = 20–30)

Goal: Identify problematic items before committing to a larger study.

Methods:

• Administer all items to a small pilot sample
• Compute item-total correlations (r.cor)
• Check for ceiling/floor effects (> 80% at extreme response)
• Examine item means and SDs (avoid items with no variance)

What you CAN do with n = 20–30:

At this stage, item-total correlations are screening tools rather than definitive item-deletion criteria. With n = 20–30, values near the usual 0.30 heuristic have substantial sampling variability, so a result like 0.28 versus 0.32 should be treated as a provisional flag for review rather than a mechanical keep-or-drop rule. The Briggs and Cheek (Briggs and Cheek 1986) mean-interitem-correlation guidance is useful here, but it is still a heuristic: for n = 25, a sample correlation of r = 0.30 has an approximate Fisher-z 95% CI from about -0.11 to 0.62. That interval is too wide to support automatic item deletion based on a single pilot correlation.

Rendered Output

Table 6.2

Pilot-stage item diagnostics for the 12-item candidate scale.

Item	Item-total r	Mean	SD	Extreme %	Flag
WRS5	NA	5.00	0.00	100	Ceiling/floor effect
WRS8	0.10	1.64	0.49	36	Low variance
WRS11	0.32	3.84	0.37	0	Low variance

Note. The pilot flags one ceiling item, one weak item-total correlation, and one low-variance item for revision or removal.

Cell Code

Interpretation:

• WRS5: Ceiling effect (100% of responses at the maximum). Remove or reword.
• WRS8: Weak item-total correlation of 0.10. Consider dropping or rewriting.
• WRS11: Low variance (SD = 0.37) and weak discrimination. Reword for clarity or replace it.

What you CANNOT do with n = 20–30:

WarningDo NOT Overinterpret Alpha with n < 30

Cronbach’s alpha estimates are highly unstable with n < 30. The 95% confidence interval will usually be very wide. For example, when the observed alpha is around 0.65, an approximate interval of about [0.40, 0.85] would not be unusual. The exact width depends on both the observed alpha value and the sample size, so use the psych::alpha() output to report the interval from your own data.

Instead: Focus on item-level diagnostics (means, SDs, ceiling or floor patterns, and item-total correlations) to refine your scale. If software computes alpha while extracting those diagnostics, do not treat the pilot alpha as a stable reliability result. Defer formal reliability reporting to Stage 3.

Stage 3: Refinement (n = 50–100)

Goal: Estimate reliability and assess dimensionality.

Methods:

• Cronbach’s alpha with confidence intervals
• McDonald’s omega (if you suspect multidimensionality)
• Split-half reliability as a robustness check
• Exploratory Factor Analysis (EFA) if n ≥ 100 and you suspect subscales

Example:

Rendered Output

Table 6.3

Refinement-stage reliability summary for the 8-item scale.

Metric	Value
Cronbach's alpha	0.833
Bootstrap 95% CI lower	0.747
Bootstrap 95% CI upper	0.885
Average inter-item correlation	0.383
Mean split-half reliability	0.832
Minimum split-half reliability	0.779
Maximum split-half reliability	0.901

Note. The confidence interval is a percentile bootstrap interval from row resampling. Split-half values are Spearman-Brown-adjusted reliability coefficients from psych::splitHalf().

Cell Code

Interpretation:

• Here the scale shows good research reliability (alpha = 0.835) with an approximate 95% CI of about [0.772, 0.898].
• The mean split-half reliability is 0.836, with adjusted split-half values ranging from 0.789 to 0.895, which supports the same conclusion from a second perspective.
• With n = 60, uncertainty is still present, but the interval is now narrow enough to support cautious reporting.

Exploratory Factor Analysis (EFA) with n = 50–100:

Rendered Output

Figure 6.1: Parallel analysis for the candidate short scale.

Cell Code

Figure 6.1 shows the parallel-analysis result for the simulated 8-item scale. In this example, the factor solution is the relevant guide: one factor is retained, and the fitted one-factor model explains about 43.9% of the total variance. Some software also prints a separate “components” recommendation. For early scale development, the factor recommendation is the substantively relevant result (Costello and Osborne 2005).

Caution: Even when parallel analysis suggests one dominant factor in a simulated example like this, n = 100 still only supports preliminary guidance. Treat EFA results as provisional until a larger validation sample can confirm the structure. For early psychological scale development, a first factor explaining roughly 40% to 60% of the variance is often acceptable as an initial signal rather than a final validation result (Costello and Osborne 2005).

Stage 4: Validation (n = 150+)

Goal: Confirm scale structure and establish validity.

In this chapter, SEM refers to structural equation modelling. In Chapter 5, SEM denoted the standard error of measurement. The context distinguishes the two uses.

Methods:

• Confirmatory Factor Analysis (CFA): Test hypothesized factor structure
• Test-retest reliability: Administer scale twice (2–4 weeks apart)
• Convergent validity: Correlate with theoretically related measures
• Discriminant validity: Show low correlation with unrelated constructs
• Known-groups validity: Scale discriminates between relevant groups

Example:

# CFA requires lavaan package and n ≥ 150
library(lavaan)

# Define 1-factor model
model <- '
  resilience =~ WRS1 + WRS2 + WRS3 + WRS4 + WRS5 + WRS6 + WRS7 + WRS8
'

# Fit model
cfa_result <- cfa(model, data = validation_data)
summary(cfa_result, fit.measures = TRUE, standardized = TRUE)

# Fit indices to assess model adequacy (conventional heuristics):
# - CFI >= 0.90 (acceptable), >= 0.95 (good)
# - RMSEA <= 0.08 (acceptable), <= 0.06 (good)
# - SRMR <= 0.08 (acceptable)
# Evaluate fit holistically rather than treating cutoffs as automatic rules.

Use these fit indices as heuristics rather than absolute pass-fail rules. CFI compares the hypothesised model with a baseline model in which the items are treated as unrelated. Larger values indicate better relative fit. RMSEA estimates lack of fit per model degree of freedom. Smaller values indicate closer approximate fit. SRMR summarises the average standardised residual discrepancy between the observed and model-implied correlations. Conventional thresholds such as CFI >= 0.95, RMSEA <= 0.06, and SRMR <= 0.08 are useful vocabulary, but they were developed mostly for larger samples and should not be applied mechanically in small validation studies (Hu and Bentler 1999). If fit is poor, report the indices, inspect the pattern of residuals and loadings, and explain the limitation. Do not repeatedly refit the model until the cutoffs are met.

Some short scales also contain both a broad general construct and narrower item clusters. In that setting, a bifactor model can separate the general factor from group factors and can support omega-hierarchical as an estimate of reliability for the general factor. This is a larger-sample validation tool, not a pilot-stage shortcut.

Test-retest reliability:

# Compute scale scores at Time 1 and Time 2
validation_data <- validation_data %>%
  dplyr::mutate(
    resilience_t1 = rowMeans(dplyr::select(., WRS1_t1:WRS8_t1), na.rm = TRUE),
    resilience_t2 = rowMeans(dplyr::select(., WRS1_t2:WRS8_t2), na.rm = TRUE)
  )

# Intraclass correlation coefficient (ICC)
# Specify the model explicitly because ICC values depend on the chosen form.
library(irr)
icc_result <- icc(
  cbind(validation_data$resilience_t1, validation_data$resilience_t2),
  model = "twoway",
  type = "agreement",
  unit = "single"
)
print(icc_result)

# ICC > 0.75 is often treated as good test-retest reliability,
# but always report the ICC model and confidence interval.

When reporting ICC, specify the model and interpretation rule directly in the text. A two-way agreement ICC above about 0.75 is often treated as good evidence of test-retest reliability, but the exact value depends on the ICC form chosen (Cicchetti 1994; Koo and Li 2016).

Special Considerations for n < 50

What You CANNOT Do

WarningAnalyses That Require Larger Samples

With n < 50, the following analyses are not feasible or will produce unreliable results:

1. Exploratory Factor Analysis (EFA): Rule of thumb is n ≥ 100 or 5–10 participants per item
2. Confirmatory Factor Analysis (CFA): Requires n ≥ 150–200 for stable parameter estimates
3. Measurement Invariance Testing: Requires n ≥ 200 per group
4. Structural Equation Modelling (SEM): Complex models need n ≥ 200–400
5. PLS-SEM (Partial Least Squares SEM): Despite “small-sample” marketing claims, stable path estimates still usually need at least n ≈ 100–150
6. Item Response Theory (IRT): Most models require n ≥ 250–500
7. Reliable Cronbach’s alpha: With n < 30, alpha estimates have 95% CIs spanning 0.3–0.4 units

Keep these methods for larger studies. Forced application to very small samples produces misleading results.

---

NoteWhy Structural Equation Modelling (SEM) Requires Large Samples

Question: “Can I use SEM, CFA, or PLS-SEM with my small sample (n < 100)?”

Short Answer: No. SEM-based methods require substantially larger samples than this book’s target range (n = 10–100).

Minimum Sample Size Requirements

Method	Minimum n	Realistic n	Why?
Confirmatory Factor Analysis (CFA)	150	200-300	Stable factor loadings, fit indices
Structural Equation Modelling (SEM)	200	300-500	Complex path models, multiple latent variables
PLS-SEM	100	150-300	Despite “small-sample” marketing claims, stable path estimates usually need at least 100-150 cases
Multi-Group SEM	200/group	300/group	Measurement invariance testing

Rule of Thumb: 10-20 observations per estimated parameter is a common starting heuristic (e.g., 5 indicators + 3 paths = 8 parameters → need 80-160 observations), but actual requirements vary with model complexity, indicator quality, and estimation method.

That is why the table above should be read as planning guidance rather than a universal rulebook. Even methods sometimes marketed as “small-sample friendly,” such as PLS-SEM, still need enough observations for stable path estimates and standard errors (Hair et al. 2017).

What Happens If You Ignore These Requirements?

With n < 100, SEM/CFA/PLS-SEM tends to produce unstable parameter estimates. Factor loadings can fluctuate sharply after very small data changes, path coefficients often carry very large standard errors, and conclusions may change when only a handful of observations are added or removed.

Small samples also increase the risk of non-convergent or improper solutions. Maximum-likelihood estimation may fail to converge, Heywood cases such as negative variances or loadings above 1.0 become more likely, and analysts can end up imposing arbitrary constraints simply to force the model to run.

Even when the software returns output, the fit statistics are difficult to trust. With small n, χ², CFI, TLI, and RMSEA can look reassuring for the wrong reasons, modification indices often suggest spurious changes, and a model may appear to fit the sample well only because it has overfit noise that will not replicate in new data.

The final danger is false confidence. SEM software will still print parameter estimates, p-values, confidence intervals, and polished path diagrams, but that appearance of technical completeness does not make the results trustworthy. Reviewers and readers will rightly question claims built on latent-variable models that the sample size cannot support.

What Should You Do Instead? (For n < 100)

Use the methods in THIS book:

SEM Goal	Small-Sample Alternative	Chapter	Minimum n
Assess scale reliability	Cronbach’s α, McDonald’s ω, split-half	Ch 6	30-50
Validate items	Item-total correlations, alpha-if-deleted	Ch 6	30-50
Reduce dimensionality	Sum/mean composite scores	Ch 6	20+
Test relationships (X → Y)	Regression with composite scores	Ch 5	30-50
Multiple predictors	Penalized regression (ridge/lasso/elastic net)	Ch 13	50-100
Mediation (X → M → Y)	Simple mediation with bootstrap CIs	Part E Project 5	80-100
Latent correlations	Polychoric correlations (exploratory)	Ch 6	50+
Measurement precision	Standard Error of Measurement (SEM statistic)	Ch 6	30+

Key Principle: Composite scores are your friend. - Sum or average your scale items to create observed composite variables - Use these composites in regression, t-tests, ANOVA - Acknowledge measurement error in limitations section - Plan larger validation study (n ≥ 200) for future CFA/SEM

Example: Replacing SEM with Composite-Score Analysis

Proposed SEM Model (n = 60):

Job_Satisfaction (5 items) → Turnover_Intent (3 items)
     ↑
Performance (4 items)

Small-Sample Alternative:

Advantages: This composite-score approach works with n = 60, remains interpretable because coefficients refer to changes in averaged scale scores, and is usually more robust than forcing a latent-variable model onto limited data. It is also more honest, because it acknowledges that observed composites are being analysed directly rather than pretending the sample is large enough for stable latent-variable estimation.

Limitations to Acknowledge: Composite scores still contain measurement error, they cannot test complex factor structures, and they do not separate within-item from between-item variance. Missing-data rules also need to be stated explicitly so readers know when partial composites were allowed and when cases were excluded.

When to Pursue SEM: Collect n ≥ 200 in a follow-up study. Then: 1. Use EFA to explore factor structure (if theory is unclear) 2. Use CFA to confirm measurement model 3. Test structural paths with latent variables 4. Assess model fit rigorously

Software Will Let You Do Bad Things

Warning: SmartPLS, AMOS, Mplus, and other SEM software will happily run with n = 50. They will produce: - Parameter estimates - p-values - Fit indices - Pretty path diagrams

This does NOT mean the results are trustworthy. Software cannot judge whether your sample size is adequate—you must.

Recommended Reading (For Future Large-Sample Studies)

When you collect n ≥ 200, consult these resources:

1. Kline, R. B. (2016). Principles and Practice of Structural Equation Modeling (4th ed.). Guilford Press.
   • Gold standard SEM textbook
   • Sample size guidelines (pp. 15-18, 264-270)
2. Brown, T. A. (2015). Confirmatory Factor Analysis for Applied Research (2nd ed.). Guilford Press.
   • CFA-specific guidance
   • Measurement invariance testing
3. Hair, J. F., Hult, G. T. M., Ringle, C. M., & Sarstedt, M. (2022). A Primer on Partial Least Squares Structural Equation Modeling (PLS-SEM) (3rd ed.). Sage.
   • PLS-SEM methods (but note: still needs n ≥ 100-150 realistically)
4. Hoyle, R. H. (Ed.). (2023). Handbook of Structural Equation Modeling (2nd ed.). Guilford Press.
   • Advanced topics, sample size planning

Bottom Line

For n = 10–100, the methods in this book are the defensible default because they are appropriate for small samples, robust to common assumption problems, honest about uncertainty, and interpretable for substantive readers and reviewers. Save SEM for a later study with n >= 200. Until then, composite scores combined with transparent regression-style analyses will usually serve the research question much better.

---

What You CAN Do

With n = 20–50, focus on these feasible and informative analyses:

1. Content validity. Use expert review, cognitive interviews, and explicit links to the theoretical framework to decide whether the item pool is clear, relevant, and broad enough before you rely on any statistics.

2. Item-level diagnostics. Examine item means, standard deviations, skewness, corrected item-total correlations, floor or ceiling effects, and the inter-item correlation matrix to flag items that are obviously misbehaving.

3. Preliminary reliability (n ≥ 30). Report Cronbach’s alpha with its 95% confidence interval, add split-half reliability as a robustness check, and inspect the mean inter-item correlation to see whether the scale is too loose or too redundant.

4. Known-groups validity. Compare scale scores between groups that theory predicts should differ, and use nonparametric methods such as the Mann–Whitney U test if the scale scores are skewed or ordinal.

5. Preparation for a larger validation study. Document the item-generation process, report pilot decisions transparently including which items were dropped and why, and use the pilot to specify the hypotheses that a later CFA or validation study will test.

Example: Documenting a Small-Sample Scale Development

Rendered Output

Table 6.4

Illustrative staged reporting summary for a short-scale development project.

Development Stage	Sample Size	Analyses Conducted	Key Findings
Item Generation	n = 8 (cognitive interviews)	Think-aloud protocols; expert review (CVI = 0.88)	Generated 15 items; 2 flagged as ambiguous
Pilot Testing	n = 25	Item-total correlations; ceiling/floor checks; 3 items dropped	Dropped WRS5 (ceiling), WRS8 (r.cor = 0.10), WRS11 (low variance)
Refinement	n = 60	Alpha = 0.84 [95% CI: 0.77, 0.90]; split-half = 0.79-0.90	8-item scale shows acceptable reliability for research use
Validation	Planned: n = 200	CFA, test-retest ICC, convergent validity (planned)	Pending larger validation sample

Note. Report the actual pilot sample sizes, item deletions, and planned validation targets transparently.

Cell Code

Reporting Guidelines for Small-Sample Scale Development

When publishing or reporting scale development work with n < 50:

1. Acknowledge limitations explicitly:

   • “With n = 25, we could not reliably estimate Cronbach’s alpha. Instead, we focused on item-total correlations to identify weak items.”
   • “Exploratory factor analysis was not feasible (n = 60); we plan CFA with a larger sample (target n = 200).”

2. Report what you did (not what you wish you could do): Avoid presenting alpha as a definitive reliability estimate when n < 30. If a journal requires it, report the confidence interval and describe the result as preliminary screening information. Do not force EFA or CFA with inadequate samples. Instead, explain the theoretical rationale for the proposed item grouping and reserve the structural test for a later study.

3. Frame as a preliminary/pilot study: State plainly that the work is a pilot or refinement study and explain what that means for interpretation. For example: “This pilot study (n = 35) was designed to refine item wording and identify problematic items before a larger validation study,” or “Results are preliminary and should be interpreted with caution pending validation with n ≥ 150.”

4. Provide detailed item-level information:

   • Publish item means, SDs, item-total correlations
   • Report which items were dropped and why
   • Share cognitive interview feedback (qualitative)

5. Plan and fund the validation study:

   • Use pilot data to justify sample size for validation (power analysis for CFA)
   • Secure resources for n ≥ 150–200 before claiming a “validated” scale

Key Takeaways

Scale development with small samples is best understood as a staged process rather than a single psychometric event. Very small pilots are most useful for qualitative refinement and item-level diagnostics, somewhat larger samples can support cautious preliminary reliability work, and only much larger studies can justify structural validation through EFA, CFA, and broader validity testing. Across all of those stages, transparency about what was and was not feasible is what makes the evidence credible.

---

Self-Assessment Quiz

Question 1. Why is scale development with small samples best treated as a multi-stage process?

Because all psychometric evidence can be collected from one pilot sampleBecause different stages support different goals, from item clarity to later validation, and each requires different sample sizesBecause short scales never need validationBecause only qualitative methods are allowed with small samples

Explanation.

The chapter presents scale development as an iterative process. Early stages focus on item generation and pilot diagnostics, while later stages support reliability estimation, factor analysis, and validation. Small samples can support some of these steps, but not all of them at once.

Question 2. At the item-generation stage, what is the main priority?

Estimating Cronbach’s alphaRunning confirmatory factor analysisChecking whether items are clear, relevant, and comprehensible through literature review, expert consultation, and cognitive interviewsComputing test-retest reliability

Explanation.

Stage 1 emphasises item generation and comprehension. The chapter explicitly states that this stage should focus on literature review, expert consultation, and cognitive interviews rather than quantitative psychometric estimation.

Question 3. With a pilot sample of about 20 to 30 participants, which analysis is most appropriate?

A full CFA with fit indicesItem-total correlations, ceiling or floor checks, and item means and SDsA definitive test-retest reliability studyA published claim that the scale is fully validated

Explanation.

The chapter recommends item-level diagnostics at the pilot stage: item-total correlations, ceiling and floor checks, and basic descriptive statistics. These analyses help identify weak items before larger validation work.

Question 4. Why does the chapter warn against relying on Cronbach’s alpha with n < 30?

Alpha cannot be computed in RAlpha estimates are highly unstable and have very wide confidence intervals with such small samplesAlpha is only for binary itemsAlpha is irrelevant for short scales

Explanation.

The warning callout explains that alpha estimates are highly unstable with n < 30, making the confidence interval so wide that the estimate is not very informative for decision-making.

Question 5. What should you do when an item shows a ceiling effect in the pilot data?

Automatically keep it because everyone agreesRemove or reword it because respondents are not differentiating on that itemUse CFA immediatelyAverage it with another item

Explanation.

The chapter’s pilot example flags ceiling effects as a sign that an item may not discriminate well. The recommended response is to remove or reword the item.

Question 6. At roughly what stage does the chapter consider exploratory factor analysis potentially feasible?

Immediately after cognitive interviewsAt the refinement stage, ideally when n approaches or exceeds 100, and even then results are tentative with modest samplesOnly after test-retest reliabilityNever in small-sample research

Explanation.

The chapter states that EFA is more defensible at the refinement stage and ideally with n around 100 or more. With smaller samples, the results are described as exploratory and unstable.

Question 7. What is the main goal of the validation stage?

To decide whether items are comprehensibleTo confirm scale structure and establish validity evidence such as CFA, test-retest reliability, and convergent validityTo compute item means onlyTo avoid collecting any more data

Explanation.

Stage 4 is the validation phase. It is where the chapter places confirmatory factor analysis, test-retest reliability, convergent validity, discriminant validity, and known-groups validity.

Question 8. Which reporting practice best fits a pilot scale-development study with n = 35?

Claim the scale is fully validatedReport preliminary item-level diagnostics, acknowledge the limitations, and state that larger-sample validation is still neededSuppress the sample size because it looks weakPresent CFA fit indices from an underpowered model as definitive evidence

Explanation.

The reporting guidelines emphasise honesty about what the study could and could not show. With a pilot sample, the chapter recommends framing findings as preliminary and stating the need for a larger validation study.

Question 9. Why is transparency especially important in small-sample scale development?

Because reviewers ignore sample sizesBecause readers need to know which psychometric claims are supported now and which must wait for later validationBecause short scales do not need theoryBecause only qualitative evidence matters

Explanation.

The chapter repeatedly stresses transparency: report what you did, what you did not do, and why. Small samples can support useful refinement decisions, but they do not justify overclaiming full validation.

Question 10. Which conclusion is most consistent with the chapter’s overall message?

A small pilot sample can fully validate a new instrumentScale development with small samples can be rigorous if it proceeds in stages and reserves stronger psychometric claims for later, larger studiesFactor analysis is always mandatory before any scale can be usedItem wording matters less than alpha

Explanation.

Small samples require staged, realistic claims. Early work can be rigorous when researchers match their analyses to the evidence their sample can support, even if full validation must wait for a larger dataset.

Ayre, Colin, and Andrew J. Scally. 2014. ‘Critical Values for Lawshe’s Content Validity Ratio: Revisiting the Original Methods of Calculation’. Measurement and Evaluation in Counseling and Development 47 (1): 79–86. https://doi.org/10.1177/0748175613513808.

Briggs, Steven R., and Jonathan M. Cheek. 1986. ‘The Role of Factor Analysis in the Development and Evaluation of Personality Scales’. Journal of Personality 54 (1): 106–48.

Buuren, Stef van. 2018. Flexible Imputation of Missing Data. 2nd ed. Boca Raton, FL: Chapman; Hall/CRC.

Cameron, A. Colin, and Pravin K. Trivedi. 2013. Regression Analysis of Count Data. 2nd ed. Cambridge: Cambridge University Press. https://doi.org/10.1017/CBO9781139013567.

Chen, Henian, Patricia Cohen, and Sophie Chen. 2010. ‘How Big Is a Big Odds Ratio? Interpreting the Magnitudes of Odds Ratios in Epidemiological Studies’. Communications in Statistics - Simulation and Computation 39 (4): 860–64. https://doi.org/10.1080/03610911003650383.

Cicchetti, Domenic V. 1994. ‘Guidelines, Criteria, and Rules of Thumb for Evaluating Normed and Standardized Assessment Instruments in Psychology’. Psychological Assessment 6 (4): 284–90. https://doi.org/10.1037/1040-3590.6.4.284.

Clark, Lee Anna, and David Watson. 1995. ‘Constructing Validity: Basic Issues in Objective Scale Development’. Psychological Assessment 7 (3): 309–19.

Cohen, Jacob. 1988. Statistical Power Analysis for the Behavioral Sciences. 2nd ed. Hillsdale, NJ: Lawrence Erlbaum Associates.

Conover, W. J. 1999. Practical Nonparametric Statistics. 3rd ed. New York: John Wiley & Sons.

Costello, Anna B., and Jason Osborne. 2005. ‘Best Practices in Exploratory Factor Analysis: Four Recommendations for Getting the Most from Your Analysis’. Practical Assessment, Research, and Evaluation 10 (1): 1–9. https://doi.org/10.7275/jyj1-4868.

Davison, A. C., and D. V. Hinkley. 1997. Bootstrap Methods and Their Application. Cambridge Series in Statistical and Probabilistic Mathematics. Cambridge: Cambridge University Press.

DeVellis, Robert F., and Carolyn T. Thorpe. 2021. Scale Development: Theory and Applications. 5th ed. Thousand Oaks, CA: SAGE Publications.

Dienes, Zoltan. 2014. ‘Using Bayes to Get the Most Out of Non-Significant Results’. Frontiers in Psychology 5: 781. https://doi.org/10.3389/fpsyg.2014.00781.

Efron, Bradley, and Robert J. Tibshirani. 1993. An Introduction to the Bootstrap. New York, NY: Chapman; Hall/CRC.

Elm, Erik von, Douglas G. Altman, Matthias Egger, Stuart J. Pocock, Peter C. Gøtzsche, and Jan P. Vandenbroucke. 2007. ‘The Strengthening the Reporting of Observational Studies in Epidemiology (STROBE) Statement: Guidelines for Reporting Observational Studies’. PLoS Medicine 4 (10): e296. https://doi.org/10.1371/journal.pmed.0040296.

Fay, Michael P. 2010. ‘Confidence Intervals That Match Fisher’s Exact or Blaker’s Exact Tests’. Biostatistics 11 (2): 373–74. https://doi.org/10.1093/biostatistics/kxp050.

Firth, David. 1993. ‘Bias Reduction of Maximum Likelihood Estimates’. Biometrika 80 (1): 27–38. https://doi.org/10.1093/biomet/80.1.27.

Gelman, Andrew, Daniel Simpson, and Michael Betancourt. 2017. ‘The Prior Can Often Only Be Understood in the Context of the Likelihood’. Entropy 19 (10): 555. https://doi.org/10.3390/e19100555.

Good, Phillip I. 2005. Permutation, Parametric, and Bootstrap Tests of Hypotheses. 3rd ed. New York: Springer.

Graham, John W. 2009. ‘Missing Data Analysis: Making It Work in the Real World’. Annual Review of Psychology 60: 549–76. https://doi.org/10.1146/annurev.psych.58.110405.085530.

Hair, Joseph F., G. Tomas M. Hult, Christian M. Ringle, and Marko Sarstedt. 2017. A Primer on Partial Least Squares Structural Equation Modeling (PLS-SEM). 2nd ed. Thousand Oaks, CA: SAGE Publications.

Harrell, Frank E. 2015. Regression Modeling Strategies: With Applications to Linear Models, Logistic and Ordinal Regression, and Survival Analysis. 2nd ed. Springer Series in Statistics. New York: Springer. https://doi.org/10.1007/978-3-319-19425-7.

Heinze, Georg, and Michael Schemper. 2002. ‘A Solution to the Problem of Separation in Logistic Regression’. Statistics in Medicine 21 (16): 2409–19. https://doi.org/10.1002/sim.1047.

Hodges, Jr., J. L., and E. L. Lehmann. 1963. ‘Estimates of Location Based on Rank Tests’. The Annals of Mathematical Statistics 34 (2): 598–611. https://doi.org/10.1214/aoms/1177704172.

Hoenig, John M., and Dennis M. Heisey. 2001. ‘The Abuse of Power: The Pervasive Fallacy of Power Calculations for Data Analysis’. The American Statistician 55 (1): 19–24.

Hosmer, David W., Stanley Lemeshow, and Rodney X. Sturdivant. 2013. Applied Logistic Regression. 3rd ed. Wiley Series in Probability and Statistics. Hoboken, NJ: John Wiley & Sons. https://doi.org/10.1002/9781118548387.

Hu, Li-tze, and Peter M. Bentler. 1999. ‘Cutoff Criteria for Fit Indexes in Covariance Structure Analysis: Conventional Criteria Versus New Alternatives’. Structural Equation Modeling 6 (1): 1–55. https://doi.org/10.1080/10705519909540118.

Huberty, Carl J., and Stephen Olejnik. 2006. Applied MANOVA and Discriminant Analysis. 2nd ed. Hoboken, NJ: Wiley. https://doi.org/10.1002/047178947X.

Hwang, Ching-Lai, and Kwangsun Yoon. 1981. Multiple Attribute Decision Making: Methods and Applications. Berlin: Springer. https://doi.org/10.1007/978-3-642-48318-9.

Hyndman, Rob J., and George Athanasopoulos. 2021. Forecasting: Principles and Practice. 3rd ed. Melbourne, Australia: OTexts. https://otexts.com/fpp3/.

Koo, Terry K., and Mae Y. Li. 2016. ‘A Guideline of Selecting and Reporting Intraclass Correlation Coefficients for Reliability Research’. Journal of Chiropractic Medicine 15 (2): 155–63. https://doi.org/10.1016/j.jcm.2016.02.012.

Kraft, Matthew A. 2020. ‘Interpreting Effect Sizes of Education Interventions’. Educational Researcher 49 (4): 241–53. https://doi.org/10.3102/0013189X20912798.

Lakens, Daniël. 2013. ‘Calculating and Reporting Effect Sizes to Facilitate Cumulative Science: A Practical Primer for t-Tests and ANOVAs’. Frontiers in Psychology 4: 863. https://doi.org/10.3389/fpsyg.2013.00863.

Lakens, Daniël, Anne M. Scheel, and Peder M. Isager. 2018. ‘Equivalence Testing for Psychological Research: A Tutorial’. Advances in Methods and Practices in Psychological Science 1 (2): 259–69. https://doi.org/10.1177/2515245918770963.

Lawshe, Charles H. 1975. ‘A Quantitative Approach to Content Validity’. Personnel Psychology 28 (4): 563–75. https://doi.org/10.1111/j.1744-6570.1975.tb01393.x.

Lipsey, Mark W., and David B. Wilson. 1993. ‘The Efficacy of Psychological, Educational, and Behavioural Treatment: Confirmation from Meta-Analysis’. American Psychologist 48 (12): 1181–1209. https://doi.org/10.1037/0003-066X.48.12.1181.

Little, Roderick J. A. 1988. ‘A Test of Missing Completely at Random for Multivariate Data with Missing Values’. Journal of the American Statistical Association 83 (404): 1198–1202. https://doi.org/10.1080/01621459.1988.10478722.

MacKinnon, David P., Jennifer L. Krull, and Chondra M. Lockwood. 2000. ‘Equivalence of the Mediation, Confounding and Suppression Effect’. Prevention Science 1 (4): 173–81.

Mair, Patrick, and Rand R. Wilcox. 2020. ‘Robust Statistical Methods in r Using the WRS2 Package’. Behavior Research Methods 52 (2): 464–88. https://doi.org/10.3758/s13428-019-01246-w.

Mann, Henry B., and Donald R. Whitney. 1947. ‘On a Test of Whether One of Two Random Variables Is Stochastically Larger Than the Other’. The Annals of Mathematical Statistics 18 (1): 50–60. https://doi.org/10.1214/aoms/1177730491.

McDonald, Roderick P. 1999. Test Theory: A Unified Treatment. Mahwah, NJ: Lawrence Erlbaum Associates.

Morey, Richard D., and Jeffrey N. Rouder. 2011. ‘Bayes Factor Approaches for Testing Interval Null Hypotheses’. Psychological Methods 16 (4): 406–19. https://doi.org/10.1037/a0024377.

Nielsen, Jakob. 1993. Usability Engineering. San Francisco, CA: Morgan Kaufmann.

Olsson, Ulf. 1979. ‘Maximum Likelihood Estimation of the Polychoric Correlation Coefficient’. Psychometrika 44 (4): 443–60.

Opricovic, Serafim, and Gwo-Hshiung Tzeng. 2004. ‘Compromise Solution by MCDM Methods: A Comparative Analysis of VIKOR and TOPSIS’. European Journal of Operational Research 156 (2): 445–55. https://doi.org/10.1016/S0377-2217(03)00020-1.

Page, Matthew J., Joanne E. McKenzie, Patrick M. Bossuyt, Isabelle Boutron, Tammy C. Hoffmann, Cynthia D. Mulrow, Larissa Shamseer, et al. 2021. ‘The PRISMA 2020 Statement: An Updated Guideline for Reporting Systematic Reviews’. BMJ 372: n71. https://doi.org/10.1136/bmj.n71.

R Core Team. 2026. R: A Language and Environment for Statistical Computing. Vienna, Austria: R Foundation for Statistical Computing. https://www.R-project.org/.

Rubin, Donald B. 1987. Multiple Imputation for Nonresponse in Surveys. New York, NY: Wiley.

Saaty, Thomas L. 1980. The Analytic Hierarchy Process. New York, NY: McGraw-Hill.

Schulz, Kenneth F., Douglas G. Altman, and David Moher. 2010. ‘CONSORT 2010 Statement: Updated Guidelines for Reporting Parallel Group Randomised Trials’. BMJ 340: c332. https://doi.org/10.1136/bmj.c332.

Shan, Guogen. 2018. Exact Statistical Inference for Categorical Data. London: Academic Press.

Sterne, Jonathan A. C., Ian R. White, John B. Carlin, Michael Spratt, Patrick Royston, Michael G. Kenward, Angela M. Wood, and James R. Carpenter. 2009. ‘Multiple Imputation for Missing Data in Epidemiological and Clinical Research: Potential and Pitfalls’. BMJ 338: b2393. https://doi.org/10.1136/bmj.b2393.

Teare, M. Dawn, Munyaradzi Dimairo, Neil Shephard, Amanda Hayman, Amy Whitehead, and Stephen J. Walters. 2014. ‘Sample Size Requirements to Estimate Key Design Parameters from External Pilot Randomised Controlled Trials: A Simulation Study’. Trials 15: 264. https://doi.org/10.1186/1745-6215-15-264.

Tomczak, Maciej, and Ewa Tomczak. 2014. ‘The Need to Report Effect Size Estimates Revisited: An Overview of Some Recommended Measures of Effect Size’. Trends in Sport Sciences 21 (1): 19–25.

Trizano-Hermosilla, Italo, and Jose M. Alvarado. 2016. ‘Best Alternatives to Cronbach’s Alpha Reliability in Realistic Conditions: Congeneric and Asymmetrical Measurements’. Frontiers in Psychology 7: 769. https://doi.org/10.3389/fpsyg.2016.00769.

Van de Schoot, Rens, and Milica Miočević. 2020. Small Sample Size Solutions: A Guide for Applied Researchers and Practitioners. London: Routledge.

Vehtari, Aki, Andrew Gelman, and Jonah Gabry. 2017. ‘Practical Bayesian Model Evaluation Using Leave-One-Out Cross-Validation and WAIC’. Statistics and Computing 27 (5): 1413–32. https://doi.org/10.1007/s11222-016-9696-4.

Wagenmakers, Eric-Jan, Jonathon Love, Maarten Marsman, Tahira Jamil, Alexander Ly, Josine Verhagen, Ravi Selker, et al. 2018. ‘Bayesian Inference for Psychology. Part II: Example Applications with JASP’. Psychonomic Bulletin & Review 25 (1): 58–76. https://doi.org/10.3758/s13423-017-1323-7.

White, Ian R., Patrick Royston, and Angela M. Wood. 2011. ‘Multiple Imputation Using Chained Equations: Issues and Guidance for Practice’. Statistics in Medicine 30 (4): 377–99. https://doi.org/10.1002/sim.4067.