analytics are the big gay

Hockey advanced stats primer, Part 2: How can (and should) we measure play-driving ability?
By Charlie O'Connor Dec 5, 2019 61

Spend even just a bit of time in a hockey conversation with someone who believes in the utility of advanced statistics, and the term “play-driving” is almost certain to come up.

For those in the analytics community, it’s a simple way to articulate a key concept: Teams that control play by outshooting and outchancing opponents tend to be the most successful at outscoring opponents. “Driving play,” therefore, directly leads to wins — at the team level, and in turn, the player level as well.

But how does one measure this ability? What are the best ways to determine which players are helping their teams most to win the underlying shot and chance battles that lead to goals and victories?

Back in 2017, I wrote what I dubbed as an “advanced stat primer” for readers at The Athletic Philadelphia and the broader site who were interested in understanding the meanings behind metrics I regularly reference in articles. I broke down the definitions of Corsi and expected goals, why rate stats like “per 60” were superior to raw totals, the point of “relative” metrics and more. The goal: Put together what amounted to an “Advanced Metrics 101” course, giving beginners an easy entry point to the core concepts that most in the analytics community take for granted.

Two years have passed, and it feels like the right time for a “201” class that delves a little deeper into the hockey stats world.

The subject of the day is play-driving ability — specifically, metrics in the public sphere that are commonly used in trying to judge which players drive play best. For those who dabble in hockey stats, “Which player looks best by the stats?” is a recurring topic of conversation, but those debates often fail to clarify exactly what the stats actually mean or how they work. Today, the goal is to provide that clarification.

Before we get to the lesson, a request: If you haven’t done the required reading of Part 1 of the primer, please do so. Our 201 review assumes basic knowledge of Corsi For Percentage, how xG differs from Corsi and why stats are often put into per 60 form. If you read the primer in 2017 but feel your base-level knowledge is rusty, it’s worth giving Part 1 a quick review. We don’t want anyone to fall behind here.

We won’t be diving deeply into the mathematical formulas behind the stats covered today. Our goals are more modest: Explain the most widely cited play-driving metrics in the public sphere, what information they provide, what limitations they might have, where they can be found and when it’s best to use each one.

Students, please take your seats.
Microstats (or micro stats)

Where to find it: Corey Sznajder’s Patreon

In the strictest sense, raw “microstats” aren’t much different than secondary metrics tracked by the NHL, such as hits, blocks or faceoffs won. They’re simply events that occur during a game that are recorded by trackers. The key difference between the two? Microstats (also called manually tracked stats) are gathered by a third party, since the NHL doesn’t record them, and these stats tend to focus on the play-driving and offense creation aspects of hockey.

The vast majority of microstat data available in the public sphere are produced by Corey Sznajder of The Energy Line, though others have completed tracking projects. Companies in the industry also track these stats and make their data accessible to teams (for a fee, of course). But if you see a comprehensive article referencing microstats, it likely utilizes (or was inspired by) Sznajder’s data.

But what are microstats? Most of them center on two on-ice events: offensive zone entries and defensive zone exits. In recent years, trackers like Sznajder have expanded their work to include passes all over the ice (largely inspired by Ryan Stimson’s Passing Project), but entries/exits — and the stats derived from them — still are the most widely cited.

These stats allow us to answer intriguing questions. Who is the best defenseman on a team in generating breakouts? Which forwards dump the puck into the offensive zone the most? Which defensemen turn the puck over in the defensive zone the most or least? Before, we had to guess at these answers. Now, as a result of manually tracking, we have cold, hard evidence.

What does this have to do with play-driving, though? It goes to the heart of what it means to “drive play”: moving the puck from defense to offense, pushing play forward. “Transition” actions like zone entries and exits are, by nature, at the root of play-driving.

And so is the nature of those entries and exits. Research has shown that controlled entries (carry-ins) are about twice as valuable as uncontrolled ones (dump-ins) in creating shots. The analytics community also found that controlled defensive zone exits help to create offensive zone entries for the team with the puck far more often than uncontrolled defensive zone exits.

In other words, controlled is good and uncontrolled is not nearly as good. It’s why stats like controlled entry percentage and controlled exit percentage are regularly cited to champion an individual’s “play-driving” ability.

But a note of caution: Microstats (at least in their current form) are only a part of the play-driving puzzle.

Think of it in terms of competitive sprinting. Imagine a runner who is elite at breaking out of the starting block; no other sprinter gets a quicker jump on the competition. But does that mean our wannabe Olympian is the fastest person in the world? Of course not. Maybe the athlete noticeably tires halfway through races; maybe the runner’s strides are surprisingly short, which allows opponents to blast past as the race progresses. Just being good at one aspect of sprinting — even a very important one — doesn’t necessarily make a sprinter elite.

It’s the same with hockey. Being able to regularly engineer controlled defensive zone exits is a valuable skill, especially for defensemen. Even novice fans intuitively understand the importance of moving the puck up ice to launch an attack. But the process of play-driving doesn’t stop there. There are still puck battles to be won, passes in the offensive zone to be made, shots to be taken. Excelling at one aspect of play-driving is great — but by itself, it doesn’t make someone a play-driver. There are better, more comprehensive metrics that shed light on who truly deserves that tag.

That doesn’t mean microstats aren’t important — after all, getting a good start in the 100-meter dash is a key part of winning the race. Microstats are vital for deeper hockey research projects that answer pointed questions about tactics or playing styles. They also can be helpful in constructing theories to explain why a player performs well (or poorly) by metrics like Corsi and xG.

But be skeptical if you read that a player is “good” solely because of strong microstats. The core reason metrics like Corsi and xG have value is because, as outlined in Part 1, they predict future goal-based outcomes better than past goal-based results. Think of it this way: Microstats describe actions that are just pieces of the play-driving whole, rather than the entire equation.

Perhaps in the future — maybe when the NHL fully implements its universal tracking package, which is in the works — a comprehensive public metric that accounts for exits, entries, battles won, passes made, turnovers created and God knows what else might exist that proves statistically superior to the Corsi and xG-centric stats of today. But for now, it’s best to view microstats as a supplement to shot and chance differential metrics (and the more complex metrics derived from them), not as a replacement.
WOWY

Where to find it: NaturalStatTrick.com player pages

The rare hockey stat name with a cool backstory, WOWY is a reference to the classic U2 song, “With or Without You.” The concept behind WOWYs is simple: They measure how well someone drives play, via Corsi or xG differential with a specific teammate, and how well without that teammate.

The goal? Isolate play-driving to figure out who is really doing the most work to help the team control play.

The simplicity of WOWYs help to explain their popularity in online hockey circles. At its core, a WOWY is a straightforward comparison between two players that tells an obvious story. And usually, that story involves showing how one player is “dragging down” another’s results.

A classic example is Ryan McDonagh’s 5-on-5 results from 2013-14 through 2015-16 — with and without regular defensive partner Dan Girardi.

When McDonagh skated with Girardi, he produced mediocre-to-passable shot and chance differentials — just below 50 percent in both. Without Girardi, however, he easily cleared the break-even mark. On the other hand, Girardi got crushed when he wasn’t skating alongside his talented partner.

Again, the metric tells an easy-to-understand story: McDonagh is a good defenseman, and his underlying numbers were a lot better when he wasn’t with Girardi. It’s easy to see why WOWYs presented in this form appeal to fans who are trying to isolate play-driving ability.
Ryan McDonagh (left) and Dan Girardi (right) flank Ottawa’s Tommy Wingels in 2017. (Brad Penner / USA Today)

So why have WOWYs become the most hated advanced stat by thought leaders in the analytics community?

I highly recommended the ESPN.com article linked above, which includes some of the smartest stats people detailing the flaws and limitations of WOWYs. In short, the primary concern is that WOWYs are often presented as an end-all, be-all argument for or against a player, when in reality, they leave out a lot of key information.

For starters, WOWYs focus on only two players. In hockey, of course, each team has five skaters on the ice (during 5-on-5 play). WOWYs inherently ignore the impact of the other three, who are clearly playing a part in driving results.

WOWYs can also miss important information about the nature of minutes spent without another player. What if, hypothetically, Girardi skated every one of his 997 minutes without McDonagh stapled to your 50-year-old next-door neighbor named Joe? Suddenly, Girardi’s poor results away from McDonagh are understandable, even impressive. After all, Joe can barely make it around the block to walk his dog, Sparky, without huffing and puffing; for Girardi to carry play to a 43-44 percent rate while saddled with him ain’t too shabby.

Of course, this is an extreme, cartoonish example. (Also, Girardi truly was not a very good defenseman during that stretch, so don’t blame Joe and Sparky.) But it illuminates the risk of boiling the equation down to only two variables when there are far more at play.

Finally, there are simply better, more comprehensive stats available in the public sphere that utilize the concept behind WOWY. Individual WOWYs are mere components of those superior stats, in the same way that microstats describe only pieces of the play-driving ability measured by metrics like Corsi or xG.

That’s not to say WOWYs don’t have value, just like exits per 60 minutes or controlled entry percentage do. The “with” side of WOWYs can be a quick check to see if a new defensive pairing is working, for example. And there’s always storytelling value in breaking down a more complex metric into its components to better explain why a player grades out well in it.

But when trying to isolate an individual’s play-driving ability, why use only a piece of the equation when there are accessible ways to look at the whole thing?
RelTM

Where to find it: Evolving-Hockey.com

This brings us to the RelTM metric.

A disclaimer: There’s a big difference between relative metrics (explained in Part 1) and RelTM, despite the similarities in name. Relative (or Rel) metrics simply calculate the percentage point difference between how a team drives play when a specific player is on the ice, and how the club performs when he sits on the bench. It’s just an on/off differential.

RelTM, on the other hand, is more like WOWY on steroids.

Remember the problem with individual WOWYs, that they focus on only two players and ignore everyone else? RelTM gets around that issue by measuring how basically every single player on a team performs with the player being evaluated, and then combines all of the information into one comprehensive metric.

RelTM answers this question: Do a player’s teammates tend to deliver better results by Corsi and/or xG when playing with him (a positive RelTM) or do they usually produce worse results with him (a negative RelTM)?

Now, not all WOWYs in the RelTM equation are weighted equally — the less time a player spends with a teammate, the less weight it’s given. For example, let’s say over the course of a full season, Claude Giroux has a 30 percent Corsi while playing alongside Tyler Pitlick, and a 55 percent Corsi away from him. That doesn’t speak well for Pitlick at first glance, right?

But assume Giroux and Pitlick spent only 30 minutes together — the poor “with” performance was likely just a couple of shifts here and there over an 82-game slate. Therefore, Pitlick doesn’t get penalized much by RelTM even though it might originally appear that he’s dramatically dragging down Giroux. If Pitlick spent 500 minutes with Giroux and “dragged him down” that much? That’s a different story, and it would be given much more weight in the equation.

Grading out as “plus-3” by Corsi plus/minus per 60 RelTM (or Corsi ±/60 RelTM) means that a player (on average) improves the Corsi differential of his teammates by three shot attempts every 60 minutes of play; a negative-3 Corsi ±/60 RelTM indicates the player deflates his teammates differentials by an average of three shots. The expected goals version of the metric (xG ±/60 RelTM) functions in the same way.

So what qualifies as a “good” performance by these metrics? Without getting too mathematical, here’s a rough breakdown of acceptable full-season RelTM ranges for each position, broken down into tiers. (Note: This is a general guide for ease of understanding and should not be taken as gospel.)

All metrics are even-strength only. The ranges were determined by taking average ranges of the last three seasons for players with at least 200 minutes at even strength, and are rounded for clarity.

Don’t forget that a key underlying goal of advanced metrics is to identify players who help their teams to outshoot and outchance the opposition, because those underlying advantages ultimately lead to goal advantages (which lead to wins). That’s what positive RelTM players do — acquire a bunch of them, and your team is probably going to win a lot.

Like any metric, however, RelTM has limitations. It doesn’t account for the impact of zone starts, quality of competition or schedule effects on play-driving results. Score effects can be added through after-the-fact adjustment, but there are other stats that account for those more thoroughly by including them in the overarching mathematical model. (RelTM is basically just an equation, not a full-fledged model.) And in small samples, RelTM can spit out some pretty extreme results, so be sure to avoid jumping to concrete conclusions about a player due to a sky-high or basement-level-low RelTM early in a season.

These weaknesses don’t render the metric useless, even though some important variables aren’t fully present. On the whole, however, RelTM is a perfectly acceptable way to get a read on an individual’s play-driving ability at even strength.
Rel vs. RelTM

So, we have both Rel and RelTM. Which is the better metric when trying to determine if (and to what degree) an individual actually drives play?

The diplomatic answer is that both have their uses, so long as it’s clear what is being measured and what question is being answered.

The short answer is that RelTM is usually better.

Why? Let’s run through a completely fictional, non-hockey story to help explain RelTM’s superiority.

Meet Lucy. She’s a worker on a teddy bear assembly line who works the night shift putting the legs on each adorable bear. At the factory, a teddy bear is constructed by the full assembly line in 15 minutes on average. But during the day shift — when Lucy doesn’t work — the average bear is finished in just 12 minutes.

By Rel metrics, Lucy would grade out poorly. After all, the bears get built faster when she’s not there, right? The logical conclusion of Rel stats is that Lucy isn’t doing a good job and is likely dragging down the assembly line’s performance.

But what if the night shift is often used for breaking in new employees on the assembly line? (Lucy is an exception; she simply prefers night work even though she’s a 10-year veteran of the company.) What if Lucy is actually great at putting legs on teddy bears in a quick and correct manner, and the reason bears are built three minutes slower on her watch has nothing to do with her?

In fact, during a week when she was sick and had to miss work, it took 20 minutes on average for bears to be finished during the night shift. Given this information, wouldn’t it be more accurate to say that she’s keeping a poorly performing group from being a total disaster than to argue that she’s dragging down the efficiency of the assembly line as a whole?

Well, Rel would simply disregard her as a negative-3 Teddy Bear Rel worker, unworthy of commendation. Teddy Bear RelTM, on the other hand, would recognize that Lucy’s co-workers perform far better when Lucy is on the assembly line as compared to when she’s not, and credit her for the solid work accordingly.

To be clear, Rel can be useful, especially when RelTM isn’t easily accessible. It’s certainly better than raw Corsi or xG differential in determining which players on a team are truly best at driving play. And there are rare instances when RelTM unfairly punishes or credits players for team-wide dramatic improvements or drop-offs in play-driving — Rel can be better suited to avoid that trap.

But on the whole, RelTM is the better metric to use when trying to measure individual play-driving ability. Now, go out and buy a teddy bear.
Isolated shot rate (via Threat)

Where to find it: HockeyViz.com

Does the above mean that RelTM is the end-all, be-all of play-driving measurement for players? Maybe a few years ago, but not anymore.

The smartest minds in the public sphere haven’t accepted the inherent weaknesses of RelTM, which at its core, is a relatively simple metric in mathematical terms. More complex models are necessary to account for the missing variables in the RelTM formula.

One such popular model is Micah Blake McCurdy’s isolated shot rate, quantified as “Threat,” which is available for public viewing on HockeyViz.com. It’s often cited by fans on social media when judging players or teams; if you’re active on hockey Twitter, you likely have seen the inimitable visualizations from McCurdy’s model.

Why use isolated shot rate? A weakness of RelTM is that it really only measures the impact one player has on the shot and chance differentials of teammates. Isolated shot rate is far more ambitious in its aims — on the single-player level, it looks to completely isolate an individual’s impact on his team’s ability to create and prevent offense.

It starts out simple enough. McCurdy’s “Threat” model is the starting point, and among quality-adjusted shot models, Threat is about as straightforward as it gets. It begins with raw shot differential, and then weights every shot based on location, using the league-average shooting percentage from that particular spot. That’s it. No accounting for rebounds, rushes and backhand shots. It’s just location-adjusted shot differential. If 3 percent of all NHL shots at 5-on-5 taken from the right faceoff dot over a given year go into the net, then the model will give every shot from the right faceoff dot a 3 percent chance of success. Easy to understand, right?

The complexity comes later. Isolated shot rate takes the results from the basic Threat model, and then accounts for a ridiculous amount of outside factors that also affect a player’s weighted shot differential: teammate impact, the effect of competition, shift starts, score effects, schedule-related fatigue and coaching impact on results. It even factors in an individual’s play-driving results from past seasons. (Remember this distinguishing factor of isolated shot rate — it’ll be important in the next section.)

After gathering all of these variables, a mathematical regression (specifically, a ridge regression) is used, and results in the model’s estimation of a player’s impact on his team’s 5-on-5 offense and defense.

To better understand how it all works, let’s first look at offensive Threat with the example of Claude Giroux.

Per the model, Giroux’s isolated impact on offense creation is +20 percentage points. But what does that mean? According to isolated shot rate, when Giroux is on the ice, he personally drives up the likelihood of a Flyers goal — assuming league-average shooting percentages and a league-average goalie for the other team — by 20 percentage points, as compared to league-average results. That’s a massive impact.

It’s just one half of the full play-driving equation, though — we can’t forget about defense. And on that side, Giroux grades out less impressively — though still well enough to avoid being deemed a liability.



In 2019-20, Giroux’s isolated impact on offense suppression is negative-7 percentage points — actually a good thing, because with defense, a player wants to be in the negative (it means the team is less likely to allow a goal). So assuming league-average goaltending, Giroux’s teams are 7 percentage points less likely to allow goals as compared to the NHL average.

McCurdy chose to isolate offensive and defensive impact in his presentation of the model, but it’s not difficult to combine the two to come up with a rough aggregate differential. For Giroux, his offensive impact (plus-20) is his true strength, but he hasn’t been a problem defensively (negative-7) in 2019-20, either.

So in aggregate, Giroux grades out at plus-27 percentage points in overall isolated shot rate impact. The Flyers perform 27 percentage points better than league-average in driving play with Giroux on the ice. Not too bad.
Claude Giroux takes a shot as Cam Atkinson defends on Oct. 26. (Eric Hartline / USA Today)

The model also provides visualizations alongside the numbers. It’s a key aspect of the presentation — it’s no coincidence that McCurdy’s website is titled HockeyVIZ after all.

The maps show where shots tend to be generated (and allowed) with a player on the ice at even strength — red means more shots than league-average in a particular region; blue indicates fewer shots. This gives insight into the “nuts and bolts” of isolated shot rate in a way that numbers alone fail to do. With this model, a person can actually see where shots tend to originate from with a player on the ice, which helps to quickly identify how the team is succeeding (or failing) when a player skates.

What are isolated shot rate’s limitations? For starters, it intentionally shies away from placing a single-number grade on a player. While one can roughly combine the offensive and defensive Threat numbers, there’s a reason why the metric is not presented as such by its creator: It’s not meant to “rank” players in a set order, or to measure “total value added” as some metrics (like Wins Above Replacement) try to do. It’s simply a measurement of even-strength play-driving ability that primarily uses visualizations to tell the story.

The simplicity of the initial Threat model is another limitation. There’s a reason public standalone xG models account for additional information beyond shot volume and location — such as rebounds, rush chances and shot type. It’s because these layers provide more detail about the quality of the underlying shots. Isolated shot rate assumes every shot taken from a specific location is equal, and that’s not necessarily the case, as a shot from the right faceoff circle on a rush has a better chance of fooling a goaltender than a shot from the same spot generated on the cycle.

Isolated shot rate doesn’t account for shooting talent, either. Some players have the ability to outperform shot- and chance-based expectations, simply because their shooting is just that good or because they have a knack for making their teammates’ shots more dangerous than “expected” due to especially creative passing. The model, by design, doesn’t pick up on either skill.

But if you’re looking solely to evaluate an individual’s play-driving ability at even strength, isolated shot rate will do nicely.
RAPM

Where to find it: Evolving-Hockey.com

And now, we come to Regularized-Adjusted Plus Minus, or RAPM.

Even more than isolated shot rate — which stands alone due to its visual element — RAPM is the spiritual successor to RelTM, though RAPM didn’t directly evolve out of RelTM from a mathematical standpoint. Like RelTM, RAPM results can be presented in Corsi and xG form. And “good” RAPM results usually look very similar to strong RelTM grades.

It’s easy to transition between the two — Sean Couturier, for example, had a plus-8.9 Corsi differential RelTM and a +8.83 Corsi differential RAPM in 2018-19. We’re talking about similar-looking results here, for the most part.

In fact, when compared directly, individual results by Corsi RelTM and Corsi RAPM were correlated by about 84 percent last season; the correlation between the two metrics by expected goals was even higher — around 87 percent. Full-population RAPM ranges tend to be less extreme than those of RelTM, but it’s rare to find players who grade out fantastically by one and terribly by the other.

Still, there’s far more “under the hood,” so to speak, with RAPM.

Originally, RAPM was a basketball metric, constructed with the goal of teasing out individual player impact on an NBA team’s point differential. Hockey is different, of course; teams don’t combine for 100-plus goals in a single game like they do in basketball. So when Brian MacDonald — and later Josh and Luke Younggren (the Evolving Wild twins) — adapted RAPM for hockey, they understandably focused on shots as their foundational metric (which occur more often than goals, and therefore amass large, reliable samples much more quickly).

RAPM starts out with raw shot creation (and suppression) results for a player. It then takes almost every outside influence on Corsi or xG differential that a skeptic could possibly fathom — teammates, competition, zone starts, home/road impact, schedule effects — and uses a mathematical model and ridge regression to attempt to account for each one of them.

Sound familiar? It should, if you paid attention to the previous section, ahem. RAPM and isolated shot rate have many similarities, in both their aims (isolate offensive and defensive impact from all other factors) and how they go about achieving those aims from a math standpoint. Not all of the add-ons are the same — for example, isolated shot rate attempts to account for the impact of a coach, RAPM does not — but the overarching goal of bringing them all into the statistical party is the same: to successfully isolate play-driving ability.

Yet there are key differences, even beyond the intricacies of the math involved. For starters, RAPM doesn’t share isolated shot rate’s “hesitation” regarding single-number results — the end play-driving stats of the model (Corsi RAPM/60 and xG RAPM/60) take into account offensive and defensive impacts and combine them into one number, which can be used to determine the best play-drivers at a given time.

RAPM also splits its results into raw shot differential (Corsi) and quality-adjusted differential (xG), and lets people pick and choose between the two. Isolated shot rate, as covered earlier, solely uses its underlying Threat model, which is sort of a middle ground between Corsi and xG, since it takes into account shot location but not factors such as rebounds or rushes.

And perhaps the largest functional difference between isolated shot rate and RAPM is their differing approaches to including prior information (before the season being measured). If you’re looking at a player’s single-season RAPM results, that’s all you’re looking at — data from just that season. Every year, RAPM begins with the assumption that each player has an equal talent level — essentially, league-average talent. Isolated shot rate’s formula, on the other hand, accounts for a player’s past results.

For example, isolated shot rate would look at Couturier in 2019-20 and say, “We’re giving your numbers a bit of a boost because you’ve been an awesome play-driver for years.” On the other hand, RAPM would tell Couturier to go prove he’s great all over again. (Spoiler alert: He’s doing just that.)
Sean Couturier posted a plus-8.9 Corsi differential RelTM and a +8.83 Corsi differential RAPM in 2018-19. (Eric Hartline / USA Today)

There are benefits to both approaches. Isolated shot rate isn’t going to be as “fooled” as RAPM by an early season play-driving hot streak by a historically poor player, since the metric includes past results in its formula. But RAPM is inherently going to be a more on-the-nose description of what a player’s results look like now (the season in question), since it’s almost exclusively looking at just a single year.

This brings us a to key aspect of statistical models that is incredibly important in understanding their purpose and usefulness. In building a model, statisticians make choices about what they want it to present. A single model isn’t going to be able to answer every question about every subject simultaneously. The fact that RAPM doesn’t account for past results might qualify as a limitation — especially if you’re trying to zero in on a player’s true talent level as a play-driver — but it shouldn’t necessarily be viewed as a flaw of the model. RAPM is telling exactly the story it was programmed to tell. It’s on the user of the model to understand the precise nature of that story, and present the results accordingly.

So what is the “right” way to use RAPM?

It should be viewed as an upgrade to RelTM as a measurement of individual play-driving ability over a single season, for starters. Like RelTM, it can be easily presented in single-number form (the results tend to look similar), but it accounts for far more outside factors than the comparatively rudimentary RelTM is capable of including.

Also, RAPM should be used as a play-driving metric, and not one that measures total value. Like isolated shot rate, it doesn’t account for shooting talent, and it’s not a Wins Above Replacement metric that tries to rank players according to how much value they add to their teams. RAPM is a fantastic tool if you’re trying to learn which player on a team has pushed play in the right direction most at even strength. Looking for a list of the best players in hockey? When attempting to answer that question, RAPM results are just a component, not the whole story, of course.

As a single-number metric that measures individual impact on Corsi and xG differential, however, RAPM is probably the best around.