Stats: Why we do what we do

You are not a coach, scout, or general manager. Watch the game. 

The numbers are imperfect and do not cover all aspects of the game. Statistics are not everything. The game isn’t played on the spreadsheet. Statistics only tell you what happened. Do you think you are smarter then the coach, general manager, or scouts?

Why does the coach choose player A over player B? Why is player utilization often not evidence driven? Why are players with better results sometimes not used more than some players with worse? Why does hockey management tend to gravitate towards certain type of players?

Why does this conflict exist between the eye-test and statistics?

Let’s take a look.

Whether you’re using the eye-test or statistical analysis, the end goal is the same. It is about rational individuals using evidence to help optimize future wins. The difference is in how one goes about gathering evidence.

The end goal is always wins. While “wins is the only stat that matters” is often used as a Straw Man attack on underlying metrics, it has some truth. Essentially all analysis relates to win optimization.

This desire for wins causes some to idolize those that have been successful before. This is why players are sometimes judged by their number of championships (rings: maybe the fanciest stat of them all). This is why some will use team records with and without the player.

People assume those with the desired results are driving those results.

Hockey is a team game though. The contemporary economics of NHL hockey results in even the best teams employing players that are worse than some on even the worst of teams.

Hockey is also a game of bounces. Past wins are not always indicative of future wins due to the massive influence that variance has on results. 

Outscoring is a byproduct of the same philosophy. The team that outscores its opponents wins. The goal is to have as many players as possible capable of out scoring their competition.

This outlook drives analysis using goal-based methods like traditional plus-minus. (plus-minus has other inherent flaws as a statistic but that is for another day).

Like with wins, outscoring is the desired result, but using it effectively. requires large samples to overcome the huge impact of variance.

The eye-test takes analysis a step further than using desired results.

Traditional scouting looks at the attributes of winning teams and compares them to players. These attributes are what drive on-ice results. They are the reasons why some players and teams beget desired results more often than others.

When comparing two players, it is not typical for one to be better in almost all attributes. There is only one Sidney Crosby. Typically there are tradeoffs. One could be the better skater while the other is smarter. One may be better positionally in the defensive zone while the other breaks the puck out better.

A scout unconsciously takes qualitative information and quantifies it. They may not realize it, but they are unknowingly ranking attributes relative to each other. They are adding up the qualitative attributes, assuming value in each, combining these values, and then comparing.

This leads us to the new wave in statistics, or the underlying metrics.

The method combines the philosophies of the two previous analysis methods. It looks at what predicts desired results while accounting for a player’s or team’s control over these results.

While Corsi percentage is not the only underlying metric used, it is the most common. This is for a reason. It predicts goal differentials for players and teams in the short run better than past goal differentials. It serves as a proxy for effectively used possession time, and is a byproduct of players with the opportunity to create scoring chances.

Corsi is a desired result that certain players can sustainably achieve more often than others. It is desired in a different way than Goals though. Goals are the ultimate target, while underlying results are desired to achieve the ultimate target. Corsi can be thought of as representing one of the major inputs a player controls in outscoring an opponent.

Analytics is simply using quantitative results to make more informed and likely better decisions.

Both the eye-test and statistics have their flaws. The strengths of one often makes up for the shortcomings of the other.

Statistics are powerful. They can accurately describe how a player or team performed in certain areas. 

They cannot breakdown why a player drives particular results if that area is not directly measurable.

A player may be under performing over a short period of time, but statistics may struggle in telling us why. Is a player performing poorly due to the natural variance inherent hockey? Is it due to the player doing something different? Is the player holding on to the puck too long? Is he not going to the ‘dirty’ or ‘inside’ areas of the ice?

Statistics may also struggle in finding the answer. Does the player lack the ability to win board battles and could use a linemate who could assist them in that area? Could the player benefit from attacking the blue line from a different angle? Is the player’s confidence causing a rut to sustain itself longer?

The eye-test can tell you the strengths and weaknesses to a player but struggles in quantifying a player’s impact in a way that can be accurately compared without bias. No matter what, all people have inherent biases brought on by how one subjectively values certain attributes.

Eyes can tell you that Dustin Byfuglien struggles in the defensive zone. They can tell you that Byfuglien moves the puck well and sustains pressure in the offensive zone. No one though can objectively determine the impact value of each attribute and compare by how much one is superior than the other from simply looking.

Attributes are useless if they do not impact results in one form or the other.

All indications are that coaches, general managers, and scouts are quite effective in terms of the eye-test. The disconnect usually comes in being able to measure the opportunity costs where there is a conflict between different attributes. 

A coach likely knows their fourth line grinder is an inferior player for 5-on-5 and penalty kill situations. The issue is knowing which situation is the optimal deployment. The eye-test cannot compare impact opportunity costs, only attributes relative to each other.

Jonathan Willis once put it best:

One should not take one method as being right all the time, as both are flawed.

When making decisions and investigating further though, it should be acted out with full understanding that each method has particular strengths and weaknesses.

Contrary to arguments on twitter and in internet comment sections and on message boards, the separation between the eye-test and statistics is heavily overstated.

Skaters who perform better statistically tend to be used more than those who do not. Goaltenders with higher save percentages tend to play more games than those who do not. Draft eligible prospects with higher points per game tend to get drafted earlier than those who do not.

Hockey is not quite the same as baseball when it comes to the “statistical revolution.” There does not exist a glut of first-line talents on NHL fourth lines being grotesquely undervalued.

Conflict does exist at times though, which is why the proper and most efficient way of approaching things is to combine the two.