Project 3. Evaluating a Process Improvement Intervention (Paired Design)

Background

A small operations team tracked incident counts for 20 units before and after a process change. Because each unit is measured twice, the analysis should use the paired structure. Lower post-intervention counts are favourable, so the practical estimand is the within-unit reduction.

Descriptive Summary

Table P3.1

Before-after incident summaries

Time Mean SD Median
Before 4.30 1.56 4.5
After 2.75 1.65 3.0
Reduction (before - after) 1.55 2.24 2.0

Note. Reduction is defined as before minus after; positive values indicate fewer incidents after the change.

Figure P3.1: Within-unit incident counts before and after the process change.

Most units show lower incident counts after the process change, although the individual trajectories are not identical. The connected-line display is more informative than two independent boxplots because it shows the paired changes directly.

Primary and Sensitivity Analyses

Table P3.2

Paired analysis of the process change

Analysis Estimate 95% CI p-value
Paired t-test Mean after-before = -1.55 -2.60 to -0.50 0.006
Wilcoxon signed-rank Pseudomedian after-before = -2.00 -2.50 to -1.00 0.005
Paired standardised mean change dz = -0.69 Not computed --

Note. The signed-rank test is a robustness check for skewed paired differences. Negative after-before estimates indicate improvement because lower counts are better; dz is computed as after minus before, so the negative sign denotes a standardised reduction.

The paired t-test and signed-rank test agree that incident counts were lower after the process change. The mean reduction is about 1.55 incidents per unit. With n = 20, the evidence is strong for this dataset, but the estimate should still be reported with its confidence interval rather than only as a p-value.

Diagnostic Checks

Table P3.3

IQR-screened unusual paired changes

Unit Department Before After Reduction Flag
1 QA 1 7 -6 Review

Note. Outlier screening identifies units for audit; it is not an automatic exclusion rule.

Table P3.4

Descriptive reductions by department

Department n Mean reduction Median
Logistics 4 2.75 3.0
Production 8 1.88 2.0
QA 8 0.62 1.5

Note. Department summaries are exploratory because subgroup sizes are small.

The outlier screen and department summaries support interpretation rather than hypothesis testing. If a unit is unusual, the correct next step is to check records and implementation notes. If departments differ descriptively, that should guide future sampling or process review rather than a formal subgroup claim.

Reporting Summary

Incident counts decreased after the process change, with a mean after-before difference of -1.55, 95% CI [-2.60, -0.50], p = 0.006. The signed-rank sensitivity analysis led to the same substantive conclusion. The finding should be framed as strong pilot evidence of improvement, with the usual caution that department-level patterns remain exploratory.

Extension Task

Repeat the paired analysis after excluding the largest absolute reduction as a sensitivity check. Report whether the mean difference, confidence interval and signed-rank conclusion change enough to affect the practical interpretation.