The notion of "shot quality" is an appealing one. So appealing, in fact, that it is the perhaps the only statistical concept in hockey that is palatable to people who otherwise reject statistical analysis in its entirety.
It's easy to understand why "shot quality" caught on - there are good scoring chances and bad ones, and there are players who consistently work around the net and there are ones who never get themselves in good position. Clearly, at the individual level, it exists. It also clearly exists at the individual game level. But is it a true and repeatable talent at the team level over the course of a season or more?
First, let's bound the impact that shot quality can possibly have. I took team shooting data going back to 1988 and simulated each season to determine the role of skill versus that of luck in each year:
Not surprisingly, as the NHL has become more competitive, the role of chance has become more pronounced. Keep in mind that what we call "skill" here is not entirely due to "shot quality" - spending more or less time on the power-play would qualify as skill, as does goaltender talent and individual shooting talent, along with deterministic changes in shooting percentage that depend on the score or the time left in the game. As others have shown, that doesn't leave a lot of real estate for "shot quality."
We can quantify the role of "shot quality" at the team-level directly by looking at where teams shot from, using data from the last five seasons. In order to reduce confounding factors, such as higher shooting percentage depending on man-advantage, the score or the time left in the game, or simply scorer bias, we only look at even-strength shooting percentage in road games with the score tied.
If "shot quality" is a true talent possessed by teams, then we should expect to see two things: 1) high correlation between "shot quality" in even and odd shots (ie shot #1, #3, etc... vs shot #2, #4, etc...); 2) high year-to-year correlation in "shot quality" because of continuity of personnel or coaching systems. Here's what we get:
The amount of "shot quality" at the team level regresses roughly 50% in both cases, and the observed "shot quality" in one year or one half of the year is a very poor predictor of performance in the other half of the sample. The standard deviation of observed "shot quality" is on the order of 10 goals per season. But only 25% of this is skill, so the league-wide spread due to "shot quality" talent is on the order of six goals, or one win. (If we include data from the 1st and 2nd periods of close games, we end up with the same overall result.)
So it's not nothing. But it's less than the talent difference between Roberto Luongo and Michael Leighton (2.5-3 wins), for example. While "shot quality" may sometimes explain the role of luck in past performance, it has very little predictive value at the team level.