A/B testing HR interventions: experimentation for people programs

60-Second Summary

Most HR programs are launched company-wide with no control group. Then we 'measure success' against pre-launch trends and call any improvement victory. That isn't measurement — it's marketing.
True A/B testing in HR is possible more often than people think — but requires randomisation at the right unit (often manager or team, not employee) and statistical literacy most HR teams haven't built.
When randomisation isn't possible (most policy changes), use quasi-experimental methods: difference-in-differences (compare changes in treated vs untreated groups), regression discontinuity (when eligibility has a sharp cutoff), or synthetic controls.
Sample-size math matters: detecting a 5pp engagement lift in a 200-person team requires ~315 per arm for 80% power at α=0.05. Most HR experiments are underpowered and report 'no significant effect' when there's a real one.
The political reality: telling managers 'your team isn't getting the new program because you're the control' creates equity issues. Design experiments leaders can defend — phased rollouts, opt-in vs assigned variants, or sequential designs.

Product teams have spent 15 years building experimentation muscle. HR has not. This guide brings the techniques over — true A/B tests where possible, quasi-experimental designs where not — with the political reality of testing programs on humans built into the design from the start.

Why HR needs experiments

Without a control group, you cannot distinguish 'the program worked' from 'engagement went up because we hired better managers last quarter'. Almost every HR claim of program effectiveness is unfalsifiable — which is why most programs survive long past their value.

Companies that run real HR experiments (Google's Project Oxygen, Microsoft's Viva research, Bersin clients) report that 30–50% of programs they test show no effect or negative effect — programs they would otherwise have scaled. The ROI of experimentation is mostly in what you stop doing.

True A/B tests in HR

When randomised experiments are feasible

1
Manager training
Randomise managers (not employees) into trained vs control cohorts. Measure team-level outcomes 3–6 months later.
2
Recruiting interventions
Randomise candidates to interview format A vs B. Compare offer rates, accept rates, 90-day performance.
3
Onboarding variants
Randomise new hires into different onboarding tracks. Measure time-to-productivity, 12-month retention.
4
Engagement nudges
Randomise teams to receive vs not receive a structured intervention (e.g. mandatory weekly 1:1s).
5
Comp communication
Randomise managers in how they communicate the comp cycle. Measure perceived fairness and retention.

Unit of randomisation matters

If the intervention is delivered by managers, randomise managers (not employees) — otherwise contamination ruins the experiment. If managers train differently within a team, employees in the control group are affected by treated peers. Cluster-randomised designs are usually the right answer.

Quasi-experimental designs

Most HR changes can't be randomised — you can't deny half your employees parental leave to measure its effect. Quasi-experimental methods extract causal estimates without randomisation.

Method	When to use	Key assumption
Difference-in-differences (DiD)	A policy is rolled out to one region/BU but not another. Compare change-over-time in both.	Parallel trends pre-treatment — both groups would have moved similarly without the policy
Regression discontinuity (RDD)	Eligibility has a sharp cutoff (e.g. only managers of 5+ get a coaching budget). Compare just-above vs just-below.	Continuity: people just-above and just-below are otherwise similar
Synthetic control	One BU adopts something; construct a 'synthetic twin' from a weighted combination of other BUs as the counterfactual.	Pre-period match between BU and synthetic is good
Propensity-score matching	Compare program participants to non-participants matched on observables.	Selection on observables — no unmeasured confounders (often violated)
Interrupted time series	A program is launched company-wide. Compare pre/post trend slopes.	No other major change at the same time

Watch the assumptions

Each method has a load-bearing assumption. When the assumption fails, the estimate is biased — sometimes by more than the effect being measured. Always report what the assumption is and how you've stress-tested it.

Sample-size math

The power calculation question: how many people in each arm do I need to detect an effect of size X with 80% confidence at α=0.05?

Outcome	Baseline	Minimum detectable effect	Per-arm sample size
Engagement score (1–5 scale, SD ~1)	3.5	+0.2 (small)	~393
Engagement score (1–5 scale, SD ~1)	3.5	+0.3 (medium)	~175
12-month retention (binary)	85%	+5pp	~609
12-month retention (binary)	85%	+10pp	~149
Quality-of-hire (1–5)	3.8	+0.25	~252

The honest implication

Most HR experiments below 300 per arm are underpowered to detect anything below a medium effect. Either pool data across cycles, target larger detectable effects, or accept that 'no significant effect' from a small experiment means 'we couldn't tell either way' — not 'no effect'.

The political reality

Frame as phased rollout, not 'control group': 'We're piloting in 3 teams first to learn what works' is acceptable; 'we're denying you this program' is not.
Randomise at units where the comparison is invisible to participants: managers don't know who else is in the cohort; new hires don't compare onboarding tracks.
Get exec sponsor and legal review before any program that affects pay, promotion, or termination — those have legal exposure even if the experiment is well-intentioned.
Have a stopping rule: if interim analysis shows large negative effect, end the experiment and roll out the control to the treated group.
Publish results internally — including the failures. Teams that hide negative results lose credibility within two cycles.

Worked examples

Example 1: New manager training program

Design: Randomise 60 first-time managers into 30 treated (training) vs 30 control (no training, will receive it 6 months later).
Outcome: Team engagement Q12 score at 6 months; team attrition; team performance distribution.
Sample-size check: 30 per arm is underpowered for engagement effects under +0.3 — pre-register that as the minimum detectable effect.
Result interpretation: report effect size with 95% CI. 'Training was associated with +0.18 (CI: −0.08 to +0.44) on engagement' is the honest read.

Example 2: Parental leave extension (cannot randomise)

Design: company rolls out enhanced leave Jan 1. Use difference-in-differences vs a peer-company benchmark cohort (Pave, Mercer) for control.
Assumption to test: pre-2026 retention trends in both populations parallel. Check with 24+ months of pre-data.
Outcome: 12-month retention of new parents pre vs post, vs benchmark.
Limitation: pandemic-era data may violate parallel trends; segment carefully.

FAQ

Frequently asked questions

Is it ethical to A/B test on employees?

Yes when (a) both arms receive an acceptable experience, (b) the experiment is approved by leadership and (where relevant) legal, (c) results aren't used to disadvantage individuals, and (d) you'd be comfortable publishing the design. The standard is lower than medical IRB but higher than 'we'll just see what happens'.

Should employees be told they're in an experiment?

For interventions that materially affect their work (training, policy), yes — frame as a pilot. For invisible variants (e.g. two onboarding email sequences), notice in the privacy notice usually suffices, but check local law (Germany requires more disclosure than the US).

Can we use Bayesian methods?

Yes — Bayesian designs are well-suited to HR because sample sizes are small and prior knowledge from benchmarks is real. Reporting credible intervals is often more honest than significance tests. But your audience needs to be educated for the results to land.