I'll start off by admitting that this post may not be totally convincing to some people because there's no real data in it. I'm not showing the numbers for any team, or goalie, or anything, but rather trying to explain why CNS has discovered that all the big hockey stats guys -- Messrs. Desjardins, Ferrari, Zona, etc. -- say the same thing: that shot quality has no discernible effect on shooting percentage (or save percentage, which is exactly the opposite of shooting percentage).
It's not exactly an intuitive result. After all, shouldn't a better shot go in more often? And if you have more better shots, shouldn't more of them go in? This isn't made any easier by the fact that, over at Arctic Ice Hockey (formerly known as Behind The Net), they have a post that shows that shooting percentage really does go up as you get closer to the net!
In the end, what the statisticians conclude seems to really rile people up. "Luck is a bigger factor than shot quality." What the heck does that mean? If you claim to be a stats guy, why are you bringing luck into the equation?
Let's take a look at Gabe's shooting percentage vs. shot distance chart:
People look at this chart and say, okay, if I take a shot 5-on-5, if I'm 10 feet away from the net it goes in 17% of the time and if it's from 40 feet it goes in 2% of the time. That may be true on average, but somehow, over the course of a season and across all the teams in the NHL, where you shoot from doesn't impact a player's shooting percentage (or a goalie's save percentage). Why?! It took me forever to figure this out.
Shot distance matters...
I think the best way to understand this is to look at some "toy" examples of the stats. By that I mean simplify things until they make sense (and let me know if they don't!). Let's start with one where there are only three kinds of shots: near the net, medium, and far:
Here I'm showing exactly what happens: either a shot goes in or it doesn't. A circle is bigger if there's a lot of shots there. Most of the far shots miss, but the near shots are more even. Let's go one step further:
Now what I've done is group shots into three categories. The blue dots are "good" shots that go in more often. Maybe the goalie is screened, or the shooter has a clean breakaway. The red dots are "bad" shots -- the goalie sees it all the way, or an errant pass is deflected on-goal and counted as a shot. The purple dots fall in between. What I'm not doing is calling this "shot quality." Shots are like snowflakes -- no two shots are the same. Whether a shot goes in or not can be affected by unquantifiable factors as well as randomness -- "luck," if you will.
Again, shot distance does matter; no matter what is happening, shots taken closer to the net will go in more often. Of course, in reality, players shoot from more than just 3 spots on the ice, so here's a slightly more realistic graph (left):
I don't make any assumptions about what makes a shot red, purple, or blue; I added the colors myself, just based on whether they were higher or lower in the graph. I'll get to this in a moment. On the right, I've fitted a line to the points on the left. This is essentially Gabe's distance plot: the average shooting percentages at each distance interval from the net.
... But it gets washed out by "luck"
Now, I don't have the data to show that shorter shot distances don't lead to significantly higher shooting percentages in the long run. This article, and many of the links in it, shows that luck plays a bigger role, even for measures of "shot quality" beyond distance (angle, rebounds, exact location, etc.). In short, if you assumed that everybody shot at the same percentage, and flipped a coin that came up heads at that percentage, you would easily see the same kinds of numbers that teams/players/goalies got in real life. Good shooting, then, is indistinguishable from randomness. This is what the stats guys mean by luck.
I know this might not make a whole lot of sense, because it's not intuitive. If the numbers say that shot quality doesn't matter, where does the advantage in shots close to the net disappear? Somehow, a team that is able to shoot higher quality shots doesn't score more. Here's how:
On the left, we have a team that takes more of its shots up close. I call this the "good team." The team on the right is the opposite -- a team of Bickells, if you will. We'll call them the "bad team." When somebody takes a shot, you don't know how likely it is to go in, because it could be a red or a purple or a blue shot. The point is, to the extent that we're able to measure "shot quality," it's random.
If you want to call it luck, then the good team is a little bit unlucky. A lot of its near shots tend to be lower percentage shots. And bad team -- well, it got lucky with those two big juicy blue shots. And, what do you know? The two teams' shooting percentages are about the same.
Of course, it doesn't always happen this way. Sometimes a good team will also be lucky. Or a bad team can be unlucky. But what is causes this "luck" is either random or unquantifiable. And in general, it's less likely for a team to be lucky and good than to just have average luck. It wipes out a lot of the advantage of always taking shots close to the net.
You can try this yourself -- number all of the points in my graph (ignore the color), then put the near shots in one hat and the far shots in another. To simulate a good team, draw 6 near shots and 3 far shots. Do the reverse for a bad team. More often than not, the resulting shooting percentages will all be quite close to average.
People are asking the wrong questions
In the world of statistics, the burden of proof is always on the side claiming that a relationship exists. You can't say that "shot quality leads to higher shooting percentage" if it doesn't in the long run.
The questions, then, aren't "does shot quality exist?" or "what does luck have to do with it?" Obviously, some shots will be more likely to go in. If you could quantify clean breakaways, they no doubt would go in more often. Right now, we are hamstrung by the quantifiable stats we can think of. We call them measures of "shot quality" and everything else "luck."
What we really should ask, then, is: are there better ways to define shot quality? From a statistical point of view, it has to be something you can quantify and measure. And nobody has figured out a number that impacts shooting/save percentages meaningfully any more than randomness (luck) does.
Personally, I'd go one step further and ask if "luck" is really the same as randomness. Why don't teams that shoot better score more? In my good team graph above, a lot of its near shots end up being red dots. I have to wonder, if a team has a reputation for crashing the crease (say, if they have a Holmstrom or a Hornqvist), whether opposing defenses tend to do a better job clearing the area in front of the net. Or even, if a team is good, if the opposition tends to play up to the competition. If so, then you actually have a factor working against shot quality and making it much, much harder to be both lucky and good.
I would love to see if anyone could think of a more creative way to measure shot quality. It would satisfy the deep curiosity I developed when I first starting messing around with hockey stats. But a part of me wants it to remain a mystery. I think zacked put it best when he commented in CNS's post:
Really what we want to capture is the degree of "oh shit!" feeling you get when an opponent sets up a shot, or how much of a boner you pop when Sharpie winds up, and see what effect that has.
Isn't that why we watch this game?