Draft Theory: Re-Defining the Roles of Scouts and Stats

Numbers and scouts would seem to be at odds over Pavel Zacha. But are they really?

I don’t know precisely why this is, but I think I have a pretty good idea. Whenever we talk about looking at prospects through the lens of statistics and quantitative analysis, the discussion almost inevitably turns to “stats vs. scouts.” Often times, people like myself are as guilty of going here as anybody. “Can you believe those idiots ranked Lawson Crouse ahead of Mitch Marner,” I’ll begin. “Don’t they know that the numbers say this is insane?”

The thing is, going down the stats-against-scouts road is missing the point entirely. This isn’t a pissing contest to see who’s right more often. It’s a continuous journey towards consistently identifying the best talent that has the highest chance of contributing at a significant level in the NHL. Numbers and watchers-of-the-games shouldn’t be at odds with one another, since our end goal is the same – to identify the junior-aged players that project to be the best future NHLers.

As such, although the Twitter snark is fun, the actual debate when we’re trying to advance our knowledge shouldn’t be centred on who’s right and who’s wrong. Instead, we should be trying to leverage the strengths of every angle we can look at talent identification from to build a drafting and scouting approach that is as accurate, precise, and predictive as we can possibly make it. We want to use numbers to get the most out of our scouts, and we want to use scouts to get the most out of our numbers.

So how should we go about constructing a talent identification system that is grounded in quantitative analysis and also takes advantage of a rigorous qualitative approach? Let’s explore after the jump.

Perhaps the most stats-friendly amateur scout in the public realm is ESPN’s Corey Pronman, formerly of Hockey Prospectus. It’s part of the reason he’s cited so frequently in these circles – his personal scouting philosophy aligns with a lot of the teachings of stats-based analysis, such as an emphasis on skill and play-driving ability before the consideration of inches, pounds, and leadershiptensity and gritruculence, so we’re predisposed to find his views more reasonable and perceive them as more accurate.

Pronman has taken a swing at this topic before, and I don’t completely agree with his analysis. He wrote his piece on how stats and scouts can be happy bedfellows back in October 2012, so I don’t know if his thoughts on the matter are still what he wrote back then, so let’s take this as a jumping off point rather than what Pronman thinks right at this very second. Information changes. Attitude changes. People change in turn.

You can read Pronman’s article here, but his basic stats-and-scouts outline in his own words is as follows:

  1. Analyze the type of player and the type of skills you would want a team to emphasize.
  2. Assess the qualitative information of a particular player.
  3. Evaluate a player’s performance through quantitative measures.
Personally, I think this approach is backwards. In order to best use the tools at your disposal, you have to have an intimate understanding of not only their strengths, but their limitations as well. The suggested approach not only fails to consider the human limits of gathering and synthesizing large amounts of data, but it also ignores the fact that the best numbers at our disposal are really poor at assessing most things on a micro level.

Where the Eye Test Fails

In his most recent post about contentious prospect Lawson Crouse, Pronman points to longtime analytically-inclined baseball writer Keith Law for his thoughts on the role of scouting in finding future players:

ESPN colleague Keith Law once wrote that the point of scouting is to develop a mental database over time to identify trends in players, as well as particular skills and physical tools that have predictive value. The idea is that scouting hundreds if not thousands of players during a long time period creates value for the scout in identifying these particular cases. Although scouting data might not be binary like certain stats, it can still be used in a predictive manner. For example: A player who skates at X level, with Y hands and Z hockey sense tends to slot into a certain role in the NHL. 

This sentiment is, both on the surface and in theory, pretty reasonable. We know that the NHL is clearly the best hockey league in the world, and we know that it takes a certain skill base to get there. The bigger and faster and smarter a guy is, the better a chance they have at succeeding against the biggest and fastest and smartest guys on the planet. 

This isn’t like EA Sports NHL 15 where you can pull up a player’s attributes page and see all the little inputs that go in to his overall talent though. We refer to stuff like skating ability, physical game, and hockey IQ as “soft” because we can’t currently measure it, but we still know it matters. Because we know it matters, we still have to gather this information in some capacity and base our judgments off of it. Hence, scouts are a critical piece of the puzzle in talent identification.

The problem is that as a general rule, humans are horrible at handling large amounts of sophisticated information, processing it, and distilling it into a form that we can accurately use to make good decisions. Saying that scouts “build a mental database” sounds fine in theory, but in practice, any such database is going to be riddled with countless cognitive biases, contain inaccurate information, and have massive holes. 

Just think about what such a database would require. A scout would have to remember years and years worth of information on every player they’ve ever seen, and be able to recall that information accurately and with enough clarity to compare the intricacies and nuances of the player right in front of them to hundreds and thousands of past players in bygone seasons. I like to think I have a pretty good memory, but I can hardly remember both of my parents’ birthdays – as much as we don’t like to admit it, human memory just isn’t good enough to do what Law and Pronman suggest.

The Holes in our Numbers

While numbers and a quantitative approach are immune to many of the pitfalls inherent in trusting analysis to the eye test, it’s absolutely vital to understand the limits of your numbers too. When analyzing draft-eligible prospects, we’re pretty limited in terms of the stuff we can use. Only the NHL tracks events in detail, and to do a more holistic analysis of player inputs, we need even greater detail than what’s given. SportVU and player tracking are so highly anticipated for a reason.

The problem is that once you get into lower and lower level leagues, the data just gets more and more sparse. Josh Weissbock has been able to scrape more stuff than we had previously over at CHLStats.com, but that still doesn’t change the fact that in most cases, our best available data on junior-age players consists of one (short) year of scoring data.

It has long been established that total points and points-per-game are horrible ways to evaluate players at the NHL level since scoring is heavily influenced by so many factors. Ice time is a major one, and while we can estimate it, we can’t estimate it well enough to determine how much it influenced a player’s total scoring. On-ice shooting percentage is another factor, and we have no way of knowing this for an individual player whatsoever. And this is to say nothing about defensive ability, which is hard enough to pin down with shot location and frequency models. We can’t even begin to quantify defense in 17-year olds.

With this in mind, we have to consider a pretty broad range of point totals to be roughly similar when comparing prospects. An an example this year, we can’t say that Lawson Crouse’s 0.91 points per game is any different than Travis Konecny’s 1.13 or Blake Speers’ 1.18. There are too many factors at play to be able to make that call. All we know is that they are two players with roughly similar statistical profiles that will receive roughly similar quantitative evaluations.

Playing to Each Other’s Strengths

So we’ve established that humans, while having a good eye for micro-level detail, are prone to making mental shortcuts when processing large amounts of data, and mis-identifying macro-level trends in turn – this is how you wind up with busts like Ryan Parent being labelled as the surest bet to become an NHL regular in his draft year. On the other hand, as evidenced by previous work done by people like myself and Josh Weissbock and Moneypuck among others, we know that the numbers we have are good enough to spot macro level trends, but fall short of being able to identify the nuances in player talent that separate two guys.

As such, your first pass at ranking players for a draft should not fall on your scouts, but rather your quantitative analysis. You should build a model that considers all relevant statistical factors and makes use of the best data available in order to identify the macro-level trends at play. There will always be guys that buck the trends and develop beyond what was projected of them, but the goal is to maximize your chances at getting the best NHL asset. If the math tells you that the odds of one player succeeding are significantly higher than another, then you’ll find more talent in the long run by drafting the guy who’s more likely to be successful year after year.

Often though, the choice is hardly black-and-white. At any given time in the draft, you’re likely to be in a position where your numbers aren’t highlighting just one guy – remember, the data we have simply is not good enough for this – but rather a handful of players who, statistically, are more or less indistinguishable from one another. As professional skills coach Darryl Belfry likes to point out though, no two players are identical. Each player you’re going to be looking at is going to have a slightly different skillset, and you are going to have to make the call on which you think is better, lest you incur a hugely negative opportunity cost

This is where excellent scouting is paramount. You’ve already distilled the prospect pool to a level in which all the players you’re considering are extremely similar in terms of mathematically projected future value, so the margins in which player is a better bet than the others are surely razor-thin. While you’re not asking your scouts to make a list out of 211 guys anymore, you’re still asking for huge amounts of information, and if you’re going to make the right call, that information has to be accurate. If you project two players to each make the NHL 50% of the time based on your quantitative models, your scouts have to be right on which guy has the more projectable skillset far more often than 50% of the time.

Putting it All Together

A quantitative approach to drafting is not building algorithms or picking from spreadsheets, but determining what, if any, macro level trends occur in drafting, identifying areas where excess value can be derived, and gathering the requisite soft information to make the best bet at a given draft position. Our current best available numbers simply are not good enough to make specific calls, but should play foundational role in targeting junior-aged talent.

Scouts still have to make the final call on which player a team picks though, so having individuals who are at the top of their field in proficiency is crucial to building the best hockey team going forwards. A quantitative approach may ask scouts to project players less and focus on identifying skills and abilities more, but that doesn’t make scouts any less important.

To bring it all back around to Corey Pronman’s idea, here’s my revised approach to incorporating rigorous quantitative analysis into traditional scouting methods:

  1. Construct a rigorous statistical framework to identify macro-level trends in player success. This will distill the prospect pool you’re considering at any one given time and help eliminate betting on low-chance players (like Nathan Smith, Taylor Ellington, Patrick White, etc.). Your scouts will operate within this framework to find the best player available.
  2. Collect and carefully consider qualitative information on each player to appraise micro-level differences in ability. Look at puck skills, physical ability, skating, defense, on-ice intelligence, etc. to determine which of the players you’re considering spending a draft pick on possesses the most projectable talent. You’re still trying to identify the best players available here since the point of the draft is to maximize the future value of each of your picks to be able to accumulate the most NHL assets down the road. An excess of assets allows you to act to fill needs as they arise rather than trying to crystal ball what your organization will look like and what needs will arise 3-5 years down the road and beyond.
  3. Consider the current needs of your organization, and what you want to emphasize in your organization going forward. By this point, you’ve distilled the pool of talent you’re looking at to probably a handful of similarly skilled players. If you have a notable lack of prospects at one position, then perhaps one of the group of the best available players you’ve identified can help alleviate that future need. Maybe you want to emphasize speed and offensive skill, so you’ll lean towards the player who you believe to have the best puck skills and quickest feet. Maybe you place a strong emphasis on being an upstanding citizen and a model representative of your organization. This is the point at which you can consider intangibles too – only after you’ve exhausted every hard and soft tangible indicator of future ability.

Given the differing strengths and weaknesses of traditional scouting and our current numbers, a top-down approach such as this is necessary for putting each of the tools at your disposal in the best position to do their intended job. Numbers are good at some things, and scouts are good at others, and when used properly they can cover each other’s weaknesses well as well as emphasize each other’s strengths.

Quantitative analysis shouldn’t be at odds with traditional approaches to scouting, especially since the end goal of both approaches is identical: to find the best players. And working in concert with one another, our approaches to finding the best players should only get better.

  • Graphic Comments

    Think there is one more way to marry the two approaches and that is to convert subjective assessments into numerical ratings and then run analytics on those ratings just like we do with counted statistics. Because I can tell you right now that those qualitative assessments of “soft” attributes are being made. So in essence they *are* being measured. But the question is, how are they analyzed.

    So when I think about Cory’s mental database, I have no problem if that is being used to assess player attributes in a numerical rating system, i.e. scored. That is actually what the human brain is good at with some level of training and repetition. What I don’t want that brain doing is trying to analyze the data because that’s where biases set in.

    So ideally, you would combine the ratings from the purely stats based analysis with the ratings from the scouting assessments into a model that weights each attribute by how much value your organization places on it. And again, I think this is where an analysis of historical trends and cohort success rates across both hard and soft metrics could help identify those that are more important in terms of the probability of success.

  • wojohowitz

    The weakness in the process is the inability to calculate the `compete level` on the individual and this question should be clarified during the interview by asking the right questions. What do players like Stefan, Lawton and Daigle have in common? They are all first overall draft picks that did not have the necessary compete level.

    Then look at the players like Burrows who was undrafted or Brendan Gallagher – a fifth rounder. There is no questioning their compete level.

    I feel like I`m playing `MoneyBall` by questioning the competence of the professional scouting staffs. Ask the right questions.

    • BuffaloBillsOfHockey

      “Compete level”…? I hope you aren’t serious.

      If you are, that means you actually believe that when you boil all the hockey players on the planet earth down to the best 690 roster players that some of them are lazy or lacksidasical in some way (or at least compared to others) while some of their peers “have what it takes”. You can’t get to the world stage, the absolute top level of ANY sport without being an extremely serious competitor. These are guys who’ve never washed out of anything in their lives (not yet, anyway, as some eventually wash out of the NHL).

      Furthermore, the seriousness, commitment and effort that these men put in is basically indistinguishable from one another. I’m tired of the phrase “they just wanted it more” for similar reasons. You think that these highly competitive, elite athletes beat the near impossible odds – never giving up – made the NHL, gave their best every game for 82 regular seasoned game to secure a playoff spot, fought tooth and nail – some of them battling through injuries that would make ordinary people like you and I cry tears of blood, much less be able to continue playing hockey at this or any level – for a chance at Lord Stanley’s Cup, only to get to Game 7 of the Conference Final and say, “screw it, this is too much work”, or “meh, do I really need another trophy to my name”? I highly doubt it. Why don’t we all just recognize what really happened here most likely, the two most frequent outcomes that happen in this situation? Either they A) Ran up against a superior team or – especially in the case of Game 7 or 1-goal overtime outcomes – B) they got unlucky.

      Finally, interviewing 18 year old hockey players for a “job” in the NHL is the number one practice in this industry that needs to stop: You wouldn’t interview highly trained dogs to see which one you want to enter into a competition, would you? No. Chances are you can watch video tapes of them performing tricks and otherwise behaving. You may have access to some nice basic numbers stats like how many shows they’ve won, or their best times to complete an obstacle course. You have all the things you actually need to make a decision in both these cases. People lie in job interviews because they know they’re getting close to their goal of securing a job. All this talk about “character”…what for? You don’t care about what charities your mailman or waiter support, do you? No. All you care about is that they do their fracking job and bring you your mail or food in a timely fashion. Why is it any different for these kids? Hell, if Connor McDavid worshipped Satan and ate an all-baby diet, he should definitely still go number one in the draft because he’s clearly the best player available and he’s being paid to play hockey, not be a role model for the rest of us. Heck, why ask an 18 year old kid anything relating to morals, ethics or ANYTHING else for that matter? THEY’RE 18. They have no life experience. What does the line of questioning sound like? “So how bad do you want to play in the NHL?” “Real bad.” “Alright, well we also talked to you teammate Bob about selecting him at the draft and he said he wanted it ‘real, real bad.’ Do you think you have that kind of compete level in you?” “Bob said that? OK, well then I want it real,real, REAL bad!” You won’t find out “compete level” or anything else of value by interviewing these players. If anything, it puts the interviewer, presuambly the GM, Head Scout or other decision maker at risk of developing a personal bias or some other stupid attachment because he likes him or he has a good sense of humor or “he reminds me of me at his age/my own son”. Serisouly, this happens. Picking a player in a draft because you like his sense of humor or “character” in an interview is unbelievably stupid and irresponsible. Seriously, just stop interviewing draft picks.

        • BuffaloBillsOfHockey

          OK, I’m dumb, then.

          Please explain to the big idiot how you can use verbal questioning to gauge someone’s competitiveness into a quantitative amount that can be put against their contemporaries. Oh, wait: you can’t. OK, if I gave you a group of ten highly competitive top level young athletes, what questions would you use to repeatably (year after year) isolate the top one. Oh, you can’t do that either. For that reason, using interviews to gauge competitiveness is stupid. Just like using questions without problems in them to gauge a person’s IQ score is stupid. It’s a problem solving test, they’d need to be given problem solving questions. You understand what I’m saying? You’re using a screwdriver on a nail: it doesn’t work because you’re doing it wrong.

          • pheenster

            You very clearly don’t have any understanding whatsoever about people or how to manage them. Because you see, hockey players are just that: people. Some are smart, some are stupid, some work hard, some don’t, some are a$$holes, some aren’t; I could go on. And all of these qualities are pretty much impossible to encapsulate into a statistic so that they can quantitatively measured. That’s why interviews are important. Teams already have a pretty good idea about a prospects skill level thanks to endless analysis by themselves and others but in most cases this is the only chance the player has to show a team what kind of a person they are. Are they a fit for the room, possibly difficult to coach, confident, less so, loud, mouthy, quiet, funny even? A draft pick is worth so much these days that teams want and need to know these things. If they don’t, they’re not a very well managed team. And for the player, the same is true for them as it is for anyone else in a job interview: you never get a second chance to make a first impression.

            So you, good sir, are the one that’s “doing it wrong”.

      • pheenster

        I agree with you regarding their “compete level.” Everyone in the NHL has a ridiculously high compete level. Although, I will say that if you’ve ever played sports at a reasonably high level, then you know there is a vast range of personalities and while everyone is very competitive some people do train more, eat better and work just a little harder than the rest of us… and, on top of all that, some of these people are also genetically blessed for their sport of choice, which helps immensely. I mean, no two people are exactly the same, even regarding compete level, right?

        But I totally disagree about the interview. The interview is your best chance to assess the mental makeup of someone. To find out about their personal life, find out their goals in life beyond simply ‘making the NHL.’ A player can look great on paper but have a less than ideal mind or personality for professional hockey. I guarantee you some of the guys who put up great numbers this year will never make it to the show and I think it’s more likely because of who they are/the life they lead and not because of how they play/a lack of skill. Obviously, these guys are only 18 and people change a lot once they’re adults but the interview lets you see them as a person and not just a player. I think that’s pretty important, especially when this potential draft pick could be with your franchise for the next decade or more

  • X

    The Canucks mind as well hire accountants to scout for players because their regular scouts sure as hell don’t know what they’re doing.

    In fact, a ouija board could make better draft selections than the Canucks ever will or have.

  • Andy

    Solid article – maybe a good intro piece for draft traditionalists?

    I hope in the future you’ll have an article outlining the specific macro/micro-level trends so we can see what the stats suggest are important factors to prospect success.

  • BuffaloBillsOfHockey

    With so many small D in this draft Morrissey, who has poise already to be a plus player, looks like a minor draft error. After the 7 guy wouldn’t shutup I’ve got the draft preview. I like Raslovic’s stats, and Ahl to complement the smaller Jets picks. The 2003 Avalanche deadline strategy looks right, they just stuck with Abby. I had a dream that was only a bit not mine own. It was the underground tunnels with carvings of businesses past. I recognized Home Depot. The msg was: it isn’t only the Scottish Enlightenment, the winter, it was commerce too. Getting rewarded for your effort in the exchange district days helped Winnipeg win World Wars. Expanding existing underground tunnels is the 1st step for civil defense from space phenomena.
    Sam will be a fan favourite. Oversexualizing maybe was good in the days of 1200AD chivalry, but it seems like a parasitic character trait while women don’t behave as equals. The flamers aren’t trying to make the world better, and there is a higher incidence of mental illness, so I don’t know why you’d want to stand up for all of them. Take away the above two populations, sure.
    S.Colbert is now looking good for attacking Lalas’s defence of European mores.

    I too had made winning my whole life, and when you make winning your whole life, you have to keep on winning, no matter what. Understand?
    They told me, the left hook kingside pawn storm counterattack is the real danger, don’t let him move me back. And I started throwing my queenside pawns up, and I’m thinking: Fritz has ****. And then g4, and I fell right off my chair. And I couldn’t hear too good. And I thought I moved in time. I flagged.
    Tom and Bill, a Super Bowl is a wonderful thing. But if you’re not enough without it, you’ll never be enough with it. You’ll never know if you would’ve won while both QBs were equals.

  • BuffaloBillsOfHockey

    …The Home Depot was I think their election lobby for public housing, and part of a reference to why Wpg boomed. They mentioned Halifax is bitter because of a canal construction, which was also what my old landlord there said. But Wpg never got bitter…the tunnels in the HSC, U of M, and Portage and Main, would need some civil defence stockpiles, especially suits or radiation-proof vehicles to collect supplies from elsewhere.

  • wojohowitz

    @ BuffaloBillsOfHockey

    I`m not talking about roster players although I could still estimate 10% of 690 players have the problem of being non-competitive. Here`s three examples; For the Canucks two players who were noticeably invisible during the playoffs were Matthias and Vrbata. Probably because the competitiveness ramps up a couple of notches during the playoffs and these two were not willing to pay the price. The third player who comes to mind easily is Alex Semin who is a very talented player but only plays hard when he needs a contract.

    As for examples of roster players who are competitive besides the aforementioned Burrows and Gallagher have you noticed Duncan Keith during these playoffs. He`s a front runner for the Conn Smythe because he has elevated his game which is competitiveness personified.

    What I was referring to in my post was the success rate of the draft where we could assume that as few as 20% of these young players could make the NHL and the question becomes how can we identify these 40 players of the 200 drafted. How did the Canadiens identify Gallagher – a 5`9″ fifth round pick – as a guy with a naturally competitive spirit.

    What would be a question to ask a young player in an attempt to gauge his competitiveness? Here`s one example; I see you ran track (played football, baseball, whatever) in high school so why did you quit? The quest to gauge competitiveness is not necessarily in the players answer but in his non-verbal response.

    • BuffaloBillsOfHockey

      Are you sure you’re not mistaking jump in quality of competition (the bottom 14 teams are no longer playing in the playoffs, after all) for a lack of competitiveness in a series of individuals. Take the Calgary Flames. Please. They convincingly but not overwhelmingly beat the Canucks, another below average team in the first round in 6 games only to be utterly annihilated in the second round by the Ducks in 5. One of these performances would shake out the same way time and time again if you kept replaying the series (Ducks vs. Flames) and the other would not (Canucks vs. Flames). This is because one is a much more even matchup, while the other is basically a sacrificial offering. This has nothing to do with competitive spirit of the Flames team or the individuals on it. They tried their best and lost. The truth is, the Ducks are a superior team composed of better players, period.

      As for Duncan Keith, he’s a great player. When someone lucks out or scores specially significant goals (as Rhys pointed out in this very piece) human beings tend to develop a bias where they only remember the good times and best-of plays and treat him like a hero of mythical status when in fact it’s far more likely that he played really well, was really lucky, or most likely, a combination of both of the above.

      On the subject of Gahllager, how do you know that it was the Canadiens intent to find a guy with “competitive spirit”? Maybe it was luck. Maybe they were looking at another attribute of his game.

      Finally, I can tell you that the vast majority of response see to your hypothetical question will be to the effect of “I quit track to focus on hockey, which is my true passion that I’m 100% committed to.” Or “I was worried that I was putting too much strain on my feet and legs as I started putting in more and more skating practice”. This is of course doing two things. The first is giving the interviewer the answer that they want to hear. The second is that it shows them you can think on your feet under pressure (possibly the only truly useful aspect of an interview process). If you can’t think on your feet, we don’t want you. If you crumble under the pressure and blurt out “I quit track so I could eat Oreos and play Mario Kart all summer!” then we DEFINITELY don’t want you. However, save for culling all but the dumbest candidates, I maintain that the interview process for athletes is redundant and missing the point.