Draft Theory: Re-Defining the Roles of Scouts and Stats
By Rhys Jessop
8 years agoNumbers and scouts would seem to be at odds over Pavel Zacha. But are they really?
I don’t know precisely why this is, but I think I have a pretty good idea. Whenever we talk about looking at prospects through the lens of statistics and quantitative analysis, the discussion almost inevitably turns to “stats vs. scouts.” Often times, people like myself are as guilty of going here as anybody. “Can you believe those idiots ranked Lawson Crouse ahead of Mitch Marner,” I’ll begin. “Don’t they know that the numbers say this is insane?”
The thing is, going down the stats-against-scouts road is missing the point entirely. This isn’t a pissing contest to see who’s right more often. It’s a continuous journey towards consistently identifying the best talent that has the highest chance of contributing at a significant level in the NHL. Numbers and watchers-of-the-games shouldn’t be at odds with one another, since our end goal is the same – to identify the junior-aged players that project to be the best future NHLers.
As such, although the Twitter snark is fun, the actual debate when we’re trying to advance our knowledge shouldn’t be centred on who’s right and who’s wrong. Instead, we should be trying to leverage the strengths of every angle we can look at talent identification from to build a drafting and scouting approach that is as accurate, precise, and predictive as we can possibly make it. We want to use numbers to get the most out of our scouts, and we want to use scouts to get the most out of our numbers.
So how should we go about constructing a talent identification system that is grounded in quantitative analysis and also takes advantage of a rigorous qualitative approach? Let’s explore after the jump.
Perhaps the most stats-friendly amateur scout in the public realm is ESPN’s Corey Pronman, formerly of Hockey Prospectus. It’s part of the reason he’s cited so frequently in these circles – his personal scouting philosophy aligns with a lot of the teachings of stats-based analysis, such as an emphasis on skill and play-driving ability before the consideration of inches, pounds, and leadershiptensity and gritruculence, so we’re predisposed to find his views more reasonable and perceive them as more accurate.
Pronman has taken a swing at this topic before, and I don’t completely agree with his analysis. He wrote his piece on how stats and scouts can be happy bedfellows back in October 2012, so I don’t know if his thoughts on the matter are still what he wrote back then, so let’s take this as a jumping off point rather than what Pronman thinks right at this very second. Information changes. Attitude changes. People change in turn.
You can read Pronman’s article here, but his basic stats-and-scouts outline in his own words is as follows:
- Analyze the type of player and the type of skills you would want a team to emphasize.
- Assess the qualitative information of a particular player.
- Evaluate a player’s performance through quantitative measures.
Personally, I think this approach is backwards. In order to best use the tools at your disposal, you have to have an intimate understanding of not only their strengths, but their limitations as well. The suggested approach not only fails to consider the human limits of gathering and synthesizing large amounts of data, but it also ignores the fact that the best numbers at our disposal are really poor at assessing most things on a micro level.
Where the Eye Test Fails
In his most recent post about contentious prospect Lawson Crouse, Pronman points to longtime analytically-inclined baseball writer Keith Law for his thoughts on the role of scouting in finding future players:
ESPN colleague Keith Law once wrote that the point of scouting is to develop a mental database over time to identify trends in players, as well as particular skills and physical tools that have predictive value. The idea is that scouting hundreds if not thousands of players during a long time period creates value for the scout in identifying these particular cases. Although scouting data might not be binary like certain stats, it can still be used in a predictive manner. For example: A player who skates at X level, with Y hands and Z hockey sense tends to slot into a certain role in the NHL.
This sentiment is, both on the surface and in theory, pretty reasonable. We know that the NHL is clearly the best hockey league in the world, and we know that it takes a certain skill base to get there. The bigger and faster and smarter a guy is, the better a chance they have at succeeding against the biggest and fastest and smartest guys on the planet.
This isn’t like EA Sports NHL 15 where you can pull up a player’s attributes page and see all the little inputs that go in to his overall talent though. We refer to stuff like skating ability, physical game, and hockey IQ as “soft” because we can’t currently measure it, but we still know it matters. Because we know it matters, we still have to gather this information in some capacity and base our judgments off of it. Hence, scouts are a critical piece of the puzzle in talent identification.
The problem is that as a general rule, humans are horrible at handling large amounts of sophisticated information, processing it, and distilling it into a form that we can accurately use to make good decisions. Saying that scouts “build a mental database” sounds fine in theory, but in practice, any such database is going to be riddled with countless cognitive biases, contain inaccurate information, and have massive holes.
Just think about what such a database would require. A scout would have to remember years and years worth of information on every player they’ve ever seen, and be able to recall that information accurately and with enough clarity to compare the intricacies and nuances of the player right in front of them to hundreds and thousands of past players in bygone seasons. I like to think I have a pretty good memory, but I can hardly remember both of my parents’ birthdays – as much as we don’t like to admit it, human memory just isn’t good enough to do what Law and Pronman suggest.
The Holes in our Numbers
While numbers and a quantitative approach are immune to many of the pitfalls inherent in trusting analysis to the eye test, it’s absolutely vital to understand the limits of your numbers too. When analyzing draft-eligible prospects, we’re pretty limited in terms of the stuff we can use. Only the NHL tracks events in detail, and to do a more holistic analysis of player inputs, we need even greater detail than what’s given. SportVU and player tracking are so highly anticipated for a reason.
The problem is that once you get into lower and lower level leagues, the data just gets more and more sparse. Josh Weissbock has been able to scrape more stuff than we had previously over at CHLStats.com, but that still doesn’t change the fact that in most cases, our best available data on junior-age players consists of one (short) year of scoring data.
It has long been established that total points and points-per-game are horrible ways to evaluate players at the NHL level since scoring is heavily influenced by so many factors. Ice time is a major one, and while we can estimate it, we can’t estimate it well enough to determine how much it influenced a player’s total scoring. On-ice shooting percentage is another factor, and we have no way of knowing this for an individual player whatsoever. And this is to say nothing about defensive ability, which is hard enough to pin down with shot location and frequency models. We can’t even begin to quantify defense in 17-year olds.
With this in mind, we have to consider a pretty broad range of point totals to be roughly similar when comparing prospects. An an example this year, we can’t say that Lawson Crouse’s 0.91 points per game is any different than Travis Konecny’s 1.13 or Blake Speers’ 1.18. There are too many factors at play to be able to make that call. All we know is that they are two players with roughly similar statistical profiles that will receive roughly similar quantitative evaluations.
Playing to Each Other’s Strengths
So we’ve established that humans, while having a good eye for micro-level detail, are prone to making mental shortcuts when processing large amounts of data, and mis-identifying macro-level trends in turn – this is how you wind up with busts like Ryan Parent being labelled as the surest bet to become an NHL regular in his draft year. On the other hand, as evidenced by previous work done by people like myself and Josh Weissbock and Moneypuck among others, we know that the numbers we have are good enough to spot macro level trends, but fall short of being able to identify the nuances in player talent that separate two guys.
As such, your first pass at ranking players for a draft should not fall on your scouts, but rather your quantitative analysis. You should build a model that considers all relevant statistical factors and makes use of the best data available in order to identify the macro-level trends at play. There will always be guys that buck the trends and develop beyond what was projected of them, but the goal is to maximize your chances at getting the best NHL asset. If the math tells you that the odds of one player succeeding are significantly higher than another, then you’ll find more talent in the long run by drafting the guy who’s more likely to be successful year after year.
Often though, the choice is hardly black-and-white. At any given time in the draft, you’re likely to be in a position where your numbers aren’t highlighting just one guy – remember, the data we have simply is not good enough for this – but rather a handful of players who, statistically, are more or less indistinguishable from one another. As professional skills coach Darryl Belfry likes to point out though, no two players are identical. Each player you’re going to be looking at is going to have a slightly different skillset, and you are going to have to make the call on which you think is better, lest you incur a hugely negative opportunity cost.
This is where excellent scouting is paramount. You’ve already distilled the prospect pool to a level in which all the players you’re considering are extremely similar in terms of mathematically projected future value, so the margins in which player is a better bet than the others are surely razor-thin. While you’re not asking your scouts to make a list out of 211 guys anymore, you’re still asking for huge amounts of information, and if you’re going to make the right call, that information has to be accurate. If you project two players to each make the NHL 50% of the time based on your quantitative models, your scouts have to be right on which guy has the more projectable skillset far more often than 50% of the time.
Putting it All Together
A quantitative approach to drafting is not building algorithms or picking from spreadsheets, but determining what, if any, macro level trends occur in drafting, identifying areas where excess value can be derived, and gathering the requisite soft information to make the best bet at a given draft position. Our current best available numbers simply are not good enough to make specific calls, but should play foundational role in targeting junior-aged talent.
Scouts still have to make the final call on which player a team picks though, so having individuals who are at the top of their field in proficiency is crucial to building the best hockey team going forwards. A quantitative approach may ask scouts to project players less and focus on identifying skills and abilities more, but that doesn’t make scouts any less important.
To bring it all back around to Corey Pronman’s idea, here’s my revised approach to incorporating rigorous quantitative analysis into traditional scouting methods:
- Construct a rigorous statistical framework to identify macro-level trends in player success. This will distill the prospect pool you’re considering at any one given time and help eliminate betting on low-chance players (like Nathan Smith, Taylor Ellington, Patrick White, etc.). Your scouts will operate within this framework to find the best player available.
- Collect and carefully consider qualitative information on each player to appraise micro-level differences in ability. Look at puck skills, physical ability, skating, defense, on-ice intelligence, etc. to determine which of the players you’re considering spending a draft pick on possesses the most projectable talent. You’re still trying to identify the best players available here since the point of the draft is to maximize the future value of each of your picks to be able to accumulate the most NHL assets down the road. An excess of assets allows you to act to fill needs as they arise rather than trying to crystal ball what your organization will look like and what needs will arise 3-5 years down the road and beyond.
- Consider the current needs of your organization, and what you want to emphasize in your organization going forward. By this point, you’ve distilled the pool of talent you’re looking at to probably a handful of similarly skilled players. If you have a notable lack of prospects at one position, then perhaps one of the group of the best available players you’ve identified can help alleviate that future need. Maybe you want to emphasize speed and offensive skill, so you’ll lean towards the player who you believe to have the best puck skills and quickest feet. Maybe you place a strong emphasis on being an upstanding citizen and a model representative of your organization. This is the point at which you can consider intangibles too – only after you’ve exhausted every hard and soft tangible indicator of future ability.
Given the differing strengths and weaknesses of traditional scouting and our current numbers, a top-down approach such as this is necessary for putting each of the tools at your disposal in the best position to do their intended job. Numbers are good at some things, and scouts are good at others, and when used properly they can cover each other’s weaknesses well as well as emphasize each other’s strengths.
Quantitative analysis shouldn’t be at odds with traditional approaches to scouting, especially since the end goal of both approaches is identical: to find the best players. And working in concert with one another, our approaches to finding the best players should only get better.
Recent articles from Rhys Jessop
