Sunny Mehta recently suggested that the observed range of goaltender save percentages over the course of a single season could essentially be duplicated by an average goalie and random factors. It dawned on me that, if this were true, that observed goaltender performance metrics ought to similarly be indistinguishable from metrics derived from an average goaltender plus random factors. To explore this, I ran some simulations using a pseudo-random number generator. For these simulations I used an ES save percentage of 0.917, PK save percentage of 0.868, and PP save percentage of 0.914
The range of GVA seen in a single season can be duplicated by an average goaltender and random factors. GVA is a fairly imprecise estimate, with a large standard error and a large confidence interval.
The average NHL team in 2009 faced 2478 shots, 1971 at even strength, 437 on the penalty kill, and 70 on the power play. If a hypothetical starter played 2/3 of his teams minutes, he faced 1314 shots at ES, 291 on the PK, and 47 on the PP. Running 10,000 simulated seasons gives:
GVA
Min. :-45.4900
1st Qu.: -7.4900
Median : 0.5100
Mean : 0.0952
3rd Qu.: 8.5100
Max. : 47.5100
Std. Dev. 11.57677
Thus a 95% CI for GVA is [22.69, -22.69]
Average Goaltender Ryan Miller workload
Miller led the league in GVT. He faced 1690 ES shots, 321 PK shots, and 87 PP shots. 10,000 simulated seasons with an average goaltender facing that workload:
GVA
Min. :-46.8800
1st Qu.: -8.8800
Median : 0.1200
Mean : 0.1159
3rd Qu.: 9.1200
Max. : 48.1200
Std. Dev. 13.10482
Thus a 95% CI for GVA at this workload is [25.69, -25.69]. Miller's GVA was 40.12, which is outside the CI, but within the range generated.
Average Goaltender Vesa Toskala workload
Toskala, on the other hand, had the worst GVT in the league. He faced 540 ES shots, 151 PK shots, and 23 PP shots. 10,000 simulated seasons with an average goaltender facing that workload:
GVA
Min. :-37.3700
1st Qu.: -5.3700
Median : 0.6300
Mean : 0.0421
3rd Qu.: 5.6300
Max. : 30.6300
Std. Dev. 8.026439
Thus a 95% CI for GVA at this workload is [15.73, -15.73]. Toskala's GVA was -21.37, which is outside the CI, but again within the range generated.
Conclusion
The range of GVA seen in a single season can be duplicated by an average goaltender and random factors. Extreme values of GVA do lie outside the 95% CI for average GVA.
As an estimate of goaltender performance, GVA is subject to estimation error. This simulation data suggests that observed GVA is a fairly imprecise estimate, with a large standard error and a large confidence interval.
Acknowledgment
I want to thank Sunny Mehta for his very helpful correspondence.
Addendum
Ryan Miller at Ryan Miller workload
GVA
Min. : 0.1200
1st Qu.: 32.1200
Median : 40.1200
Mean : 39.6262
3rd Qu.: 47.3700
Max. : 83.1200
Std. Dev. 11.79438
95% CI [63.24, 17.00]
Vesa Toskala at Vesa Toskala workload
GVA
Min. :-61.3700
1st Qu.: -27.3700
Median : -20.3700
Mean : -20.0783
3rd Qu.: -14.3700
Max. : 13.6300
Std. Dev. 8.950644
95% CI [-2.54, -37.62]